Commit Graph

412 Commits

Author SHA1 Message Date
Andrew Turner
8ff003011d Rework how we store the VFP registers in the pcb. This will be used when
creating a floating-point context within the kernel without having to move
the stored values in memory.

Sponsored by:	The FreeBSD Foundation
2016-10-14 15:53:48 +00:00
Oleksandr Tymoshenko
563193fd0c Add initial Raspberry Pi 3 support
RPI3 kernel config builds kernel compatible with latest upstream device
tree and firmware: https://github.com/raspberrypi/firmware/tree/master/boot
As of today it's 597c662a613df1144a6bc43e5f4505d83bd748ca

Default console is PL01x, so pi3-disable-bt dt overlay should be configured
in config.txt and stock U-Boot should be patched to use proper serial port.

Yet unsupported: SMP, VCHIQ, RNG driver. RNG requires some work due to
upstream device tree incompatibility.

Multiple people contributed to this work over time: db@, loos@, manu@
2016-10-14 03:37:35 +00:00
Andrew Turner
1890f18cbe Move printing the AArch64 ID registers to a new SYSINIT, the previous
location only prints them when booting on SMP with multiple cores.

MFC after:	1 week
Sponsored by:	DARPA, AFRL
2016-10-13 09:06:29 +00:00
Warner Losh
b2a7ac4802 Fix building on i386 and arm. But 'public domain' headers on the files
with no creative content. Include "lost" changes from git:
o Use /dev/efi instead of /dev/efidev
o Remove redundant NULL checks.

Submitted by: kib@, dim@, zbb@, emaste@
2016-10-13 06:56:23 +00:00
Oleksandr Tymoshenko
c38fb7809f INTRNG: Propagate IRQ activation error to API consumer
Keep resource state consistent with INTRNG state - if intr_activate_irq
fails - deactivate resource and propagate error to calling function

Reviewed by:	mmel
2016-10-12 17:10:59 +00:00
Jonathan T. Looney
bd79708dbf In the TCP stack, the hhook(9) framework provides hooks for kernel modules
to add actions that run when a TCP frame is sent or received on a TCP
session in the ESTABLISHED state. In the base tree, this functionality is
only used for the h_ertt module, which is used by the cc_cdg, cc_chd, cc_hd,
and cc_vegas congestion control modules.

Presently, we incur overhead to check for hooks each time a TCP frame is
sent or received on an ESTABLISHED TCP session.

This change adds a new compile-time option (TCP_HHOOK) to determine whether
to include the hhook(9) framework for TCP. To retain backwards
compatibility, I added the TCP_HHOOK option to every configuration file that
already defined "options INET". (Therefore, this patch introduces no
functional change. In order to see a functional difference, you need to
compile a custom kernel without the TCP_HHOOK option.) This change will
allow users to easily exclude this functionality from their kernel, should
they wish to do so.

Note that any users who use a custom kernel configuration and use one of the
congestion control modules listed above will need to add the TCP_HHOOK
option to their kernel configuration.

Reviewed by:	rrs, lstewart, hiren (previous version), sjg (makefiles only)
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D8185
2016-10-12 02:16:42 +00:00
Jung-uk Kim
493deb390b Merge ACPICA 20160930. 2016-10-04 20:27:15 +00:00
Konstantin Belousov
559a7b209a Add a way for the architecture to specify the calling ABI for methods
in the EFI Runtime Services Table.  On amd64, the calling conventions
are MS.

Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
2016-09-21 10:35:44 +00:00
Wojciech Macek
a134b98878 Add support for SPI-mapped MSI interrupts in GICv3.
PIC_SETUP_INTR implementation in GICv3 did not allow
for setting up interrupts without included FDT
description. GICv2m-like MSI interrupts, which map
MSI messages to SPI interrupt lines, may not have
a description in FDT. Add support for such interrupts
by setting the trigger and polarity to the appropriate
values for MSI (edge, high) and get the hardware
IRQ number from the corresponding ISRC.

Obtained from:         Semihalf
Submitted by:          Michal Stanek <mst@semihalf.com>
Sponsored by:          Annapurna Labs
Reviewed by:           wma
Differential Revision: https://reviews.freebsd.org/D7662
2016-09-21 05:22:49 +00:00
Wojciech Macek
3502e7b37c Register GICv3 xref.
This allows other drivers to retrieve interrupt parent node.

Obtained from:         Semihalf
Submitted by:          Michal Stanek <mst@semihalf.com>
Sponsored by:          Annapurna Labs
Reviewed by:           wma, zbb
Differential Revision: https://reviews.freebsd.org/D7568
2016-09-21 05:15:50 +00:00
Wojciech Macek
988d9b3505 Add driver for PCIe root complex on Annapurna Alpine platform.
The driver subclasses pci-host-generic and additionally
performs configuration of vendor-specific PCIe registers.

Obtained from:         Semihalf
Submitted by:          Michal Stanek <mst@semihalf.com>
Sponsored by:          Annapurna Labs
Reviewed by:           wma
Differential Revision: https://reviews.freebsd.org/D7571
2016-09-20 11:11:06 +00:00
Konstantin Belousov
944e0bab86 Consolidate four efi_next_descriptor() definitions.
Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
2016-09-18 17:38:02 +00:00
Andrew Turner
6de3c41534 Add a missing opt_ddb.h include from places that include debug_moditor.h
to fix the build.

Obtained from:	ABT Systems Ltd
X-MFC with:	r305771
Sponsored by:	The FreeBSD Foundation
2016-09-13 16:20:31 +00:00
Andrew Turner
5ed824174b Fix the arm64 kernel build when DDB is disabled, debug_monitor.c depends on
DDB, and is unused when it's disabled.

Obtained from:	ABT Systems Ltd
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2016-09-13 15:45:22 +00:00
Andrew Turner
8b52054063 Add a warning about a known erratum we have observed on ThunderX pass 1.1.
As this is evaluation hardware with only a few users, and there is a lack
of information add a warning when booting on this hardware.

Reported by:	cognet
Obtained from:	ABT Systems Ltd
MFC after:	Instant
Sponsored by:	The FreeBSD Foundation
2016-09-13 14:10:49 +00:00
Alan Cox
8cb0c1029d Various changes to pmap_ts_referenced()
Move PMAP_TS_REFERENCED_MAX out of the various pmap implementations and
into vm/pmap.h, and describe what its purpose is.  Eliminate the archaic
"XXX" comment about its value.  I don't believe that its exact value, e.g.,
5 versus 6, matters.

Update the arm64 and riscv pmap implementations of pmap_ts_referenced()
to opportunistically update the page's dirty field.

On amd64, use the PDE value already cached in a local variable rather than
dereferencing a pointer again and again.

Reviewed by:	kib, markj
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D7836
2016-09-10 16:49:25 +00:00
Andrew Turner
13db69623b Trap msr/mrs instructions. These are privileged arm64 instructions and
shouldn't normally be used.

Obtained from:	ABT Systems Ltd
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2016-09-08 12:53:01 +00:00
Andrew Turner
e0c6c1d1fd Don't panic when we don't handle a userland exception, not all we may see
are currently handled.

Obtained from:	ABT Systems Ltd
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2016-09-08 12:39:03 +00:00
Andrew Turner
77c02eccb8 When synchronising the instruction and data caches we only need to clean
the data cache to the point of unification. This is the point where the
two caches are unified to a single unified cache so cleaning past here
is just extra unneeded work.

This was noticed when investigating r305545.

Reported by:	bz
Obtained from:	ABT Systems Ltd
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2016-09-07 16:46:54 +00:00
Andrew Turner
3b34364450 Only call cpu_icache_sync_range when inserting an executable page. If the
page is non-executable the contents of the i-cache are unimportant so this
call is just adding unneeded overhead when inserting pages.

While doing research using gem5 with an O3 pipeline and 1k/32k/1M iTLB/L1
iCache/L2 Bjoern Zeeb (bz@) observed a fairly high rate of calls into
arm64_icache_sync_range() from pmap_enter() along with a high number of
instruction fetches and iTLB/iCache hits.

Limiting the calls to arm64_icache_sync_range() to only executable pages,
we observe the iTLB and iCache Hit going down by about 43%. These numbers
are quite misleading when looked at alone as at the same time instructions
retired were reduced by 19.2% and instruction fetches were reduced by 38.8%.
Overall this reduced the runtime of the test program by 22.4%.

On Juno hardware, in steady-state, running the same test, using the cycle
count to determine runtime, we do see a reduction of up to 28.9% in runtime.

While these numbers certainly depend on the program executed, we expect an
overall performance improvement.

Reported by:	bz
Obtained from:	ABT Systems Ltd
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2016-09-07 16:22:05 +00:00
Wojciech Macek
9d6cd3d858 Introduce support for Annapurna Alpine CCU and NB devices
This commit adds drivers for Alpine Cache Coherency Unit
and North Bridge Service whose task is to configure
the system fabric and enable cache coherency.

Obtained from:         Semihalf
Submitted by:          Michal Stanek <mst@semihalf.com>
Sponsored by:          Annapurna Labs
Reviewed by:           wma
Differential Revision: https://reviews.freebsd.org/D7565
2016-09-07 05:34:41 +00:00
Jared McNeill
0fbb017195 Add generic device-tree cpufreq driver. 2016-09-06 21:18:14 +00:00
Andrew Turner
705cb30cae Enable superpages on arm64 by default. These seem to be stable, having
survived multiple world and kernel builds, and of poudriere building full
package sets.

I have observed a 3% reduction in buildworld times with superpages enabled,
however further testing is needed to see if this is observed in other
workloads.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-09-04 17:50:23 +00:00
Mark Johnston
dbbaf04f1e Remove support for idle page zeroing.
Idle page zeroing has been disabled by default on all architectures since
r170816 and has some bugs that make it seemingly unusable. Specifically,
the idle-priority pagezero thread exacerbates contention for the free page
lock, and yields the CPU without releasing it in non-preemptive kernels. The
pagezero thread also does not behave correctly when superpage reservations
are enabled: its target is a function of v_free_count, which includes
reserved-but-free pages, but it is only able to zero pages belonging to the
physical memory allocator.

Reviewed by:	alc, imp, kib
Differential Revision:	https://reviews.freebsd.org/D7714
2016-09-03 20:38:13 +00:00
Jared McNeill
d69d5ab04f Add support for Allwinner A64 thermal sensors. 2016-09-03 15:26:00 +00:00
Andrew Turner
af6936890a Add a pc_clock pcpu field and use it to implement cpu_est_clockrate. This
will allow drivers that manage the clock frequency to communicate this with
the reset of the kernel.

Reported by:	jmcneill
MFC after:	1 week
Sponsored by:	ABT Systems Ltd
2016-09-02 10:13:51 +00:00
Andrew Turner
2c97ce35a7 Fix arm64 superpages bugs in pmap_enter:
* Pass the correct virtual address when demoting a superpage
 * Use the correct l3 table after demoting a superpage
 * Remove an invalid KASSERT hit demoting then promoting a superpage [1]

With this it is believed that  superpages on arm64 is stable.

Reported by:	[1] cognet
Obtained from:	ABT Systems Ltd
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2016-09-01 10:26:06 +00:00
Andrew Turner
d953ec3222 Also handle instruction traps. We might hit these when the page we are
executing is being promoted to a superpage.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-31 13:58:32 +00:00
Andrew Turner
7090a45142 Because we need to use a break-before-make sequence when promoting pages
there is a short period where functions that walk the kernel page table
without locking them may see an invalid entry. One solution would be to add
locking to these functions, however some may be called from locations where
we are unable to sleep.

Until a better solution can be found stop promoting pages in the kernel
pmap so these functions work as expected.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-30 16:45:15 +00:00
Andrew Turner
bc3aca4bf9 Print both the kernel read and write translation in DDB when asking for
a virtual to physical translation. These may be different, e.g. when a
page is mapped as read-only.

MFC after:	1 month
Sponsored by:	ABT Systems Ltd
2016-08-27 10:30:20 +00:00
Andrew Turner
d92a1b6ab3 Fix an assert, it should check if, when moving from 1 l1 to 512 l2 blocks,
the l2 entry is a block type and not an l3 page.

While here fix the string to correct the level name and add a missing ')'.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-25 14:42:29 +00:00
Jared McNeill
1567e084b3 Add support for Allwinner A64.
Reviewed by:	andrew, manu
Relnotes:	yes
2016-08-25 13:59:19 +00:00
Andrew Turner
d5911afb79 Map coherent memory in a non-coherent dma tag as uncached. This is similar
to what the 32-bit arm code does, with the exception that it always assumes
the tag is non-coherent.

Tested by:	jmcneill
Obtained from:	ABT Systems Ltd
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2016-08-25 10:53:03 +00:00
Andrew Turner
aa9872243a Allow superpages on arm64 to be enabled. It is disabled for now to allow
for more testing across more hardware.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-24 13:00:50 +00:00
Andrew Turner
262432bbb1 Add support to promote and demote managed superpages. This is the last part
needed before enabling superpages on arm64. This code is based on the amd64
pmap with changes as needed to handle the differences between the two
architectures.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-24 12:57:40 +00:00
Andrew Turner
3969d2f921 Teach the parts of the arm64 pmap that need to iterate over pages to also
iterate over superpages. We don't yet create these, but soon will.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-24 12:32:18 +00:00
Ed Schouten
48734c99d3 Convert pointers obtained from the threadattr_t structure with TO_PTR().
In all of these source files, the userspace pointer size corresponds
with the kernelspace pointer size, meaning that casting directly works.
As I'm planning on making 32-bit execution on 64-bit systems work as
well, use TO_PTR() here as well, so that the changes between source
files remain minimal.
2016-08-24 10:13:18 +00:00
Andrew Turner
e0a66c6379 Also adjust the virtual address passed to vm_page_pa_tryrelock.
Reported by:	alc
Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-23 16:37:34 +00:00
Andrew Turner
56d9fe8567 Map memory as read-only in pmap_enter_quick_locked as is done in other
pmap implementations.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-23 16:20:56 +00:00
Andrew Turner
f17394c340 If we find we have a superpage in pmap_enter_quick_locked return without
trying to add a new level 3 page.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-23 16:12:25 +00:00
Andrew Turner
dd9faf6dc4 Include the offset the virtual address is within an L1 or L2 block when
finding the vm_page_t in pmap_extract_and_hold. Previously it would return
the vm_page_t of the first page in a block. This would cause issues when,
for example, fsck reads from a device into the middle of a superpage. In
this case the read call would write to the start of the block, and not to
the buffer passed in.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-23 15:48:27 +00:00
Andrew Turner
724c3e47f6 Fix the arm64 non-SMP build, active_irq is a uint64_t so cast it through
a uintmax_t.

Obtained from:	ABT Systems Ltd
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2016-08-22 19:05:11 +00:00
Andrew Turner
1d479540a5 Ensure map is valid, even before userland exists and the fault address
register points to an address in the userland range.

Obtained from:	ABT Systems Ltd
MFC after:	1 week
Sponsored by:	the FreeBSD Foundation
2016-08-22 18:19:46 +00:00
Andrew Turner
8e7fcbc322 Fix pmap_update_entry, pmap_invalidate_range takes the end address, not
the size.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-22 18:12:44 +00:00
Andrew Turner
c7d40f1843 Use switch statements in pmap_remove_pages. While only one level of
pagetable is supported more will be added soon to support removing
superpages.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-22 14:53:39 +00:00
Andrew Turner
a3c1bc2eae Use pmap_update_entry in pmap_enter when updating an entry with a new
physical address. This is required when either mapping is writeable.

While here remove an unneeded call to pmap_pde, we already have the pde
from earlier in the function.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-22 12:56:40 +00:00
Andrew Turner
aed6b9701b Add sysctls to report on superpages statistics. While here add extra
logging to these paths.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-22 12:17:40 +00:00
Andrew Turner
109dddc015 Add a size argument to pmap_update_entry.
Make use of this in pmap_promote_l2.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-22 10:50:30 +00:00
Andrew Turner
1d32c3222e Add KASSERTS in pmap_alloc_l3 to ensure we are not encountering superpages
when we don't yet expect them;

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-22 10:21:09 +00:00
Michal Meloun
895c8b1c39 INTRNG: Rework handling with resources. Partially revert r301453.
- Read interrupt properties at bus enumeration time and store
   it into global mapping table.
 - At bus_activate_resource() time, given mapping entry is resolved and
   connected to real interrupt source. A copy of mapping entry is attached
   to given resource.
 - At bus_setup_intr() time, mapping entry stored in resource is used
   for delivery of requested interrupt configuration.
 - For MSI/MSIX interrupts, mapping entry is created within
   pci_alloc_msi()/pci_alloc_msix() call.
 - For legacy PCI interrupts, mapping entry must be created within
   pcib_route_interrupt() by pcib driver itself.

Reviewed by: nwhitehorn, andrew
Differential Revision: https://reviews.freebsd.org/D7493
2016-08-19 10:52:39 +00:00