Commit Graph

466 Commits

Author SHA1 Message Date
Andrew Turner
aed6b9701b Add sysctls to report on superpages statistics. While here add extra
logging to these paths.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-22 12:17:40 +00:00
Andrew Turner
109dddc015 Add a size argument to pmap_update_entry.
Make use of this in pmap_promote_l2.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-22 10:50:30 +00:00
Andrew Turner
1d32c3222e Add KASSERTS in pmap_alloc_l3 to ensure we are not encountering superpages
when we don't yet expect them;

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-22 10:21:09 +00:00
Michal Meloun
895c8b1c39 INTRNG: Rework handling with resources. Partially revert r301453.
- Read interrupt properties at bus enumeration time and store
   it into global mapping table.
 - At bus_activate_resource() time, given mapping entry is resolved and
   connected to real interrupt source. A copy of mapping entry is attached
   to given resource.
 - At bus_setup_intr() time, mapping entry stored in resource is used
   for delivery of requested interrupt configuration.
 - For MSI/MSIX interrupts, mapping entry is created within
   pci_alloc_msi()/pci_alloc_msix() call.
 - For legacy PCI interrupts, mapping entry must be created within
   pcib_route_interrupt() by pcib driver itself.

Reviewed by: nwhitehorn, andrew
Differential Revision: https://reviews.freebsd.org/D7493
2016-08-19 10:52:39 +00:00
Konstantin Belousov
1680854946 Implement userspace gettimeofday(2) with HPET timecounter.
Right now, userspace (fast) gettimeofday(2) on x86 only works for
RDTSC.  For older machines, like Core2, where RDTSC is not C2/C3
invariant, and which fall to HPET hardware, this means that the call
has both the penalty of the syscall and of the uncached hw behind the
QPI or PCIe connection to the sought bridge.  Nothing can me done
against the access latency, but the syscall overhead can be removed.
System already provides mappable /dev/hpetX devices, which gives
straight access to the HPET registers page.

Add yet another algorithm to the x86 'vdso' timehands. Libc is updated
to handle both RDTSC and HPET.  For HPET, the index of the hpet device
to mmap is passed from kernel to userspace, index might be changed and
libc invalidates its mapping as needed.

Remove cpu_fill_vdso_timehands() KPI, instead require that
timecounters which can be used from userspace, to provide
tc_fill_vdso_timehands{,32}() methods.  Merge i386 and amd64
libc/<arch>/sys/__vdso_gettc.c into one source file in the new
libc/x86/sys location.  __vdso_gettc() internal interface is changed
to move timecounter algorithm detection into the MD code.

Measurements show that RDTSC even with the syscall overhead is faster
than userspace HPET access.  But still, userspace HPET is three-four
times faster than syscall HPET on several Core2 and SandyBridge
machines.

Tested by:	Howard Su <howard0su@gmail.com>
Sponsored by:	The FreeBSD Foundation
MFC after:	1 month
Differential revision:	https://reviews.freebsd.org/D7473
2016-08-17 09:52:09 +00:00
Andrew Turner
2bafd72fdb Add the ARMv8.1 identification registers to the list we print when booting.
MFC after:	1 week
Sponsored by:	ABT Systems Ltd
2016-08-15 09:23:08 +00:00
Andrew Turner
510a3f1b79 Implement promotions and demotions in the arm64 pmap code. For now we don't
promote memory as I am not sure all the demotion cases are handled, however
it is useful to implement pmap_page_set_memattr. This is used, for example,
when mapping uncached memory for bus_dma(9).

pmap_page_set_memattr needs to demote the DMAP region as on ARM we need to
ensure all mappings to the same physical address have the same attributes.

Reviewed by:	kib
Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D6987
2016-08-12 10:29:34 +00:00
Ed Schouten
13b4b4df98 Provide the CloudABI vDSO to its executables.
CloudABI executables already provide support for passing in vDSOs. This
functionality is used by the emulator for OS X to inject system call
handlers. On FreeBSD, we could use it to optimize calls to
gettimeofday(), etc.

Though I don't have any plans to optimize any system calls right now,
let's go ahead and already pass in a vDSO. This will allow us to
simplify the executables, as the traditional "syscall" shims can be
removed entirely. It also means that we gain more flexibility with
regards to adding and removing system calls.

Reviewed by:	kib
Differential Revision:	https://reviews.freebsd.org/D7438
2016-08-10 21:02:41 +00:00
Ed Schouten
66bd4c2eeb Make cpu_set_user_tls() work when called on the running thread.
On all the other architectures, this function can also be called on the
currently running thread. In this case, we shouldn't fix up the address
in the PCB, but also patch up the register itself. Otherwise it will not
become active and will simply become overwritten by the next switch.

Reviewed by:	imp
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D7437
2016-08-10 15:45:25 +00:00
Andrew Turner
4cf6e978b4 Uncomment the vm.kvm_size and vm.kvm_free sysctls. These work as expected so
there is no reason to leave them commented out.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-10 10:36:11 +00:00
Andrew Turner
4088f71f9b Implement pmap_align_superpage on arm64 based on the amd64 implementation.
This will be needed when superpage support is added.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-10 10:13:34 +00:00
Andrew Turner
b0316526b0 Remove the pvh_global_lock lock from the arm64 pmap. It is unneeded on arm64
as invalidation will have completed before the pmap_invalidate_* functions
have complete.

Discussed with:	alc, kib
Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-04 13:49:36 +00:00
Andrew Turner
27c9d42531 Remove trailing whitespace from the arm64 pmap
Obtained from:	ABT Systems Ltd
MFC after:	3 weeks
Sponsored by:	The FreeBSD Foundation
2016-08-02 15:26:46 +00:00
Andrew Turner
698c14e189 Add a kernel variable to let the user to select their preferred order
between ACPI and FDT. This will be needed on machines with both, e.g. the
SoftIron Overdrive 3000. The kernel will accept one or more comma separated
values of either 'acpi' or 'fdt'. Any other values are skipped.

To set it the user can either set it on the loader command line, or
in loader.conf e.g. in loader.conf:
kern.cfg.order=acpi,fdt

This will try using ACPI then FDT. If none of the selected options work the
kernel tries to use one to get the serial console, then panics.

Reviewed by:	emaste (earlier version)
Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D7274
2016-08-01 12:17:44 +00:00
Andrew Turner
49a92cd4a5 Add the fields for the PAR_EL1 register. This is used when performing an
address lookup with the AT instructions.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-01 10:36:58 +00:00
Andrew Turner
63512a1276 Add the Data Fault Status Code values to the ESR_ELx registers for when the
fault code is a Data Abort.

Obtained from:	AT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-07-31 18:58:20 +00:00
Andrew Turner
ed24579135 Extract the common parts of pmap_kenter_device to a new function. This will
be used when superpage support is added.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-07-31 17:53:09 +00:00
Andrew Turner
7592321ad1 Fix the comment above pmap_invalidate_page. tlbi will invalidate the tlb
on all CPUs.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-07-31 14:59:44 +00:00
Andrew Turner
c5b3b20907 Relax the barriers around a TLB invalidation to only wait on
inner-shareable memory accesses. There is no need for full system barriers.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-07-31 12:59:10 +00:00
Andrew Turner
eda295b9e5 Add a generic EHCI USB driver based on the Allwinner A10 driver. It is ACPI
only for now, but wouldn't be too difficult to add support for FDT.

Reviewed by:	hselasky
Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D7352
2016-07-29 08:50:36 +00:00
Andrew Turner
1af4207389 Remove an unused variable.
Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-07-25 16:18:20 +00:00
Andrew Turner
0f0b7aa289 Fix a typo in a string in a KASSERT to sanity check the CPU IDs.
Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-07-25 15:59:31 +00:00
Andrew Turner
8ec2b5df73 Rework how we number CPUs on arm64 to try and keep clusters together.
Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-07-25 15:57:13 +00:00
Andrew Turner
abf5bdb441 Enable the generic OHCI driver on arm64
Obtained from:	ABT Systems Ltd
MFC after:	1 month
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
2016-07-25 14:49:15 +00:00
Alexander Motin
fb112f72a8 Add more UEFI/e820 memory types from latest specifications.
This is only cosmetics.

MFC after:	2 weeks
2016-07-24 09:15:11 +00:00
Andrew Turner
e0f50f02e4 Implement bus_print_child to print the resources used by the ITS driver.
Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-07-15 13:25:47 +00:00
Andrew Turner
5d94bbc241 Finish removing the non-INTRNG support from sys/arm64.
Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-07-14 17:31:29 +00:00
Andrew Turner
ec1172ed92 Remove the non-INTRNG support from the ThunderX PCIe drivers.
Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-07-14 17:23:49 +00:00
Andrew Turner
c55795369b Move gic_v3_irqsrc into the GICv3 driver source as it's only needed there.
Remove unused macros from the GICv3 header.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-07-14 17:16:51 +00:00
Andrew Turner
a509d56d7f Move structures only used by the GICv3 ITS driver from a shared header to
the ITS driver file. There is no need for other drivers to need to know
about these structures.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-07-14 17:10:54 +00:00
Andrew Turner
b30fd6a080 Remove the non-INTRNG support from the GICv3 interrupt controller driver.
This is no longer needed.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-07-14 17:05:25 +00:00
Andrew Turner
07b8a57120 Add memmmap on arm64 so we can mmap /dev/mem and /dev/kmem.
Obtained from:	ABT Systems Ltd
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2016-07-13 23:03:34 +00:00
Andrey A. Chernov
239e5577a2 Undo r302601, WCHAR_MAX may not be a valid wchar value. 2016-07-12 04:20:44 +00:00
Andrey A. Chernov
713a61670c I don't know why unsigned int is choosed for wchar_t here, but WCHAR_MAX
should be <= WINT_MAX. It is bigger, __UINT_MAX > INT32_MAX
2016-07-12 00:37:48 +00:00
Ed Schouten
d96aeddf2f Don't forget to set sa->narg for CloudABI system calls.
It turns out that this value is not used within the system call code
under normal conditions, except when using tracing tools like ktrace.
If we forget to set this value, it is set to random garbage. This may
cause ktrace to hang indefinitely, making it impossible to kill.

Reported by: Michael Plass
PR: 210800
MFC before: 11.0-RELEASE
2016-07-08 20:09:21 +00:00
Andrew Turner
b760df65c8 Remove the old pre-INTRNG arm64 interrupt framework. GENERIC was switched
to INTRNG in r301565 with the old code no longer being built by default with
no reports of issues on any supported hardware.

Approved by:	re (gjb)
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-07-06 16:20:10 +00:00
Wojciech Macek
e46f2622b4 ARM64: fix DMAP calculation
Use arithmetic operators instead of logical. This fixes
    DMAP ranges calculation for ThunderX Dual Socket.

    Obtained from:         Semihalf
    Sponsored by:          Cavium
    Reviewed by:           zbb
    Differential Revision: https://reviews.freebsd.org/D7023
    Approved by:           re (gjb)
2016-06-30 04:58:19 +00:00
Andrew Turner
add8a9c2d2 Fix a race when the hardware has raised an exception with an unknown or
uncategorised reason. We need to read the fault address register before
enabling interrupts as the interrupt handler may cause this register to
change.

Approved by:	re (marius, kib)
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-06-22 12:05:08 +00:00
Konstantin Belousov
5c2cf81845 Update comments for the MD functions managing contexts for new
threads, to make it less confusing and using modern kernel terms.

Rename the functions to reflect current use of the functions, instead
of the historic KSE conventions:
  cpu_set_fork_handler -> cpu_fork_kthread_handler (for kthreads)
  cpu_set_upcall -> cpu_copy_thread (for forks)
  cpu_set_upcall_kse -> cpu_set_upcall (for new threads creation)

Reviewed by:	jhb (previous version)
Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
Approved by:	re (hrs)
Differential revision:	https://reviews.freebsd.org/D6731
2016-06-16 12:05:44 +00:00
Andrew Turner
04ff877e4c Switch arm64 to use intrng by default. The old interrupt handling code can
still be used, however this is expected to be removed soon.

Obtained from:	ABT Systems Ltd
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
2016-06-07 20:14:08 +00:00
Svatopluk Kraus
ad5244ece1 INTRNG - change the way how an interrupt mapping data are provided
to the framework in OFW (FDT) case.

This is a follow-up to r301451.

Differential Revision:	https://reviews.freebsd.org/D6634
2016-06-05 16:20:12 +00:00
Andrew Turner
550d01a211 Add the GICv3 ITS intrng driver. As the interface to the interrupt
framework has significantly changed the driver has moved to a new file.
While it shares some code with the existing driver this has been modified
to work better with the intrng framework.

This has been tested on the ThunderX servers in the netperf cluster and has
been used to boot them for other testing, including DTrace and hwpmc.

With this we can use intrng on all supported arm64 platforms I was able to
test on. It is expected we will move to intrng soon, and disable the old
arm64 interrupt framework.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D6437
2016-06-03 10:28:06 +00:00
Zbigniew Bodek
32fb8300d4 Return real error value instead of hard-coded ENXIO (fix after r300149)
It is possible to return real error value in case of gic_v3_attach()
failure instead of hard-coded ENXIO.

Obtained from:	Semihalf
Sponsored by:	Cavium
2016-06-01 08:20:10 +00:00
Andrew Turner
920c8a57c0 dpcpu_init should have also passed in the calculated cpuid, not the
devicetree ID.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-31 19:17:32 +00:00
Andrew Turner
e0db04b3de Allow the kernel to boot on a CPU where the devicetree has numbered it with
a non-zero ID. To do this we increment the cpuid of any CPUs with a smaller
devicetree ID by one to stop them conflicting with the boot CPU.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-31 18:45:52 +00:00
Andrew Turner
cea2a7fe6d Enable setting BF_COHERENT on DMA tags. This allows the kernel to start
using the cache handling functions.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-31 16:28:56 +00:00
Andrew Turner
2760c2cad4 Mark the ThunderX and generic PCI drivers as cache-coherent when we know
this to be the case. This will mean we don't try and handle the cache in
bus_dmamap_sync when it is not needed.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D6605
2016-05-31 09:15:21 +00:00
Andrew Turner
02759f14f2 Extract the correct bits from the GICD_TYPER register. The interrupt count
is encoded in the bottom 5 bits.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-20 15:43:51 +00:00
Andrew Turner
da257dda10 Add more useful GICv3 register definitions. While here fix
GITS_CBASER_CACHE_MASK to use the correct shift macro.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-20 13:11:07 +00:00
Andrew Turner
98670cad72 Filter out BUS_DMASYNC_POSTWRITE sync operations, there is nothing for us
to do on these.

Reported by:	wma
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-20 12:38:48 +00:00
Andrew Turner
fa13831f86 Enable NEW_PCIB on arm64.
Obtained from:	ABT Systems Ltd
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
2016-05-20 08:43:18 +00:00
Andrew Turner
6d2f7ef617 Handle PCI_RES_BUS on the generic and ThunderX PCIe drivers. This has been
tested on the Pass 1.1 and 2.0 ThunderX machines in the Netperf cluster.

Reviewed by:	jhb
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D6453
2016-05-20 08:29:00 +00:00
Andrew Turner
b7c02deed2 Define PCI_RES_BUS for NEW_PCIB
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-19 14:00:18 +00:00
Andrew Turner
9346e9130d Return the struct intr_pic pointer from intr_pic_register. This will be
needed in later changes where we may not be able to lock the pic list lock
to perform a lookup, e.g. from within interrupt context.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-18 15:05:44 +00:00
Zbigniew Bodek
b02a807b46 Add support for MSI/MSIX deallocation on GICv3-ITS
Allow to deallocate previously allocated ITS device along with
its interrupts. Interrupt numbers are being freed when the last
LPI number is no longer busy.

Reviewed by:	wma
Obtained from:	Semihalf
Sponsored by:	Cavium
Differential Revision:	https://reviews.freebsd.org/D6351
2016-05-18 10:09:07 +00:00
Bjoern A. Zeeb
1f420ed469 The GIC (v2 at least) has a bit in the TYPER register to indicate whether the GIC
supports the Security Extensions or not. This bit is not the same as the CPU one.
Currently we are not checking for either before trying to write to the special
registers.  This can lead to problems on hardware or simulators that do not
provide the security extensions.  Add the missing checks. Their interactions with
the CPU flag is not entirely clear to me but using a macro will make it easier
to quickly adjust the condition once the CPU bits are sorted as well.

Reviewed by:	br
Sponsored by:	DARPA/AFRL
Differential Revision:	https://reviews.freebsd.org/D6397
2016-05-17 13:12:26 +00:00
Andrew Turner
e34357f56d Add an arm64 kernel config to help testing intrng. It is expected this
config will only last a few weeks until we switch to this interrupt
framework.

Obtained from:	 ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-17 12:48:30 +00:00
Andrew Turner
9a38d21983 Clean up the GICv3 intrng code:
* In gic_v3_attach free the correct data on failure.
 * Implement gic_v3_teardown_intr.
 * Update the panic string when enabling/disabling an invalid interrupt.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-17 12:46:50 +00:00
Andrew Turner
88f7980a81 Add intrng support to the GICv3 driver. It lacks ITS support so won't handle
MSI or MSI-X interrupts, however this is enought to boot FreeBSD under the
ARM Foundation Model with a GICv3 interrupt controller.

Approved by:	ABT Systems Ltd
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
2016-05-16 14:07:43 +00:00
Andrew Turner
2b48ec8e6d Move the call to intr_pic_init_secondary to the same place as in the
non-intrng case.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-16 12:02:06 +00:00
Andrew Turner
72b3f638b7 Add support for intrng to arm64. As the GICv3 drivers will need to be
updated, and until further testing can be done, this is disabled for now.

It is expected arm64 will switch to this interface, and the old interface
will be removed before 11.0 is released.

Obtained from:	ABT Systems Ltd
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
2016-05-16 10:48:51 +00:00
Andrew Turner
4e1763d34c Teach the ThunderX PCI PEM driver about intrng. This will be used later
when arm64 is supported by intrng.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-16 10:03:57 +00:00
Andrew Turner
1e43b18c4b Add a pcib interface for use by interrupt controllers that need to
translate the pci rid to a controller ID. The translation could be based
on the 'msi-map' OFW property, a similar ACPI option, or hard-coded for
hardware lacking the above options.

Reviewed by:	wma
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-16 09:31:44 +00:00
Andrew Turner
5db4448f06 Add support to the arm64 busdma to handle the cache. For now this is
disabled, however when we enable it it will default to assume memory is
not cache-coherent, unless either the tag was created or the parent was
marked as cache-coherent.

Obtained from:	ABT Systems Ltd
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
2016-05-13 16:03:50 +00:00
Andrew Turner
f82b6f5255 Rename the internal BUC_DMA_* flags to BF_* so they won't conflict with
the flags in sys/bus_dma.h.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-12 15:24:36 +00:00
Andrew Turner
2b45fde379 Restrict the memory barriers in bus_dmamap_sync to just the operations
where it's needed.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-12 12:13:28 +00:00
Andrew Turner
9615213bef Call busdma_swi from swi_vm as is done from other architectures.
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-11 18:48:47 +00:00
Andrew Turner
b7a7ee3af1 On arm64 always create a bus_dmamap_t object. This will be use to hold the
list of memory that the kernel will need to sync when operating with a
non-cache coherent DMA engine.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-11 16:53:41 +00:00
Andrew Turner
7ca01e8f89 Add data barriers to the arm64 bus_dmamap_sync function. We need these
to ensure ordering between the CPU and device. As the CPU and DMA target
may be in different shareability domains they need to be full system
barriers.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-11 14:59:54 +00:00
Zbigniew Bodek
46542fb652 Fix I/O coherence issues on ThunderX when SMP is disabled
To maintain coherence between cache and DMA memory appropriate
shareability flags need to be set in the PTE regardless of SMP
option.

Reviewed by:	wma
Obtained from:	Semihalf
Sponsored by:	Cavium
Differential Revision: https://reviews.freebsd.org/D6231
2016-05-11 13:23:56 +00:00
Andrew Turner
fb853e4270 Push the logic to talk with the MSI/MSI-X interrupt controller to the FDT
attachment. This is where it will live when we import intrng as it will
need to look at either the msi-parent or msi-map FDT properties.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-10 15:45:59 +00:00
John Baldwin
82cb5c3b5b Native PCI-express HotPlug support.
PCI-express HotPlug support is implemented via bits in the slot
registers of the PCI-express capability of the downstream port along
with an interrupt that triggers when bits in the slot status register
change.

This is implemented for FreeBSD by adding HotPlug support to the
PCI-PCI bridge driver which attaches to the virtual PCI-PCI bridges
representing downstream ports on HotPlug slots. The PCI-PCI bridge
driver registers an interrupt handler to receive HotPlug events. It
also uses the slot registers to determine the current HotPlug state
and drive an internal HotPlug state machine. For simplicty of
implementation, the PCI-PCI bridge device detaches and deletes the
child PCI device when a card is removed from a slot and creates and
attaches a PCI child device when a card is inserted into the slot.

The PCI-PCI bridge driver provides a bus_child_present which claims
that child devices are present on HotPlug-capable slots only when a
card is inserted. Rather than requiring a timeout in the RC for
config accesses to not-present children, the pcib_read/write_config
methods fail all requests when a card is not present (or not yet
ready).

These changes include support for various optional HotPlug
capabilities such as a power controller, mechanical latch,
electro-mechanical interlock, indicators, and an attention button.
It also includes support for devices which require waiting for
command completion events before initiating a subsequent HotPlug
command. However, it has only been tested on ExpressCard systems
which support surprise removal and have none of these optional
capabilities.

PCI-express HotPlug support is conditional on the PCI_HP option
which is enabled by default on arm64, x86, and powerpc.

Reviewed by:	adrian, imp, vangyzen (older versions)
Relnotes:	yes
Differential Revision:	https://reviews.freebsd.org/D6136
2016-05-05 22:26:23 +00:00
Zbigniew Bodek
42638f9c5a Fix GICv3 build after r299090
Obtained from:	Semihalf
Sponsored by:	Cavium
2016-05-05 17:51:14 +00:00
Andrew Turner
ef6b515b69 Disable ACPI on arm64 ad it has only had minimal testing and is causing
boot issues when booting with FDT. It is planned to re-enable this at a
later date.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-04-26 14:21:39 +00:00
Ruslan Bukin
30b72b6871 Move arm's devmap to some generic place, so it can be used
by other architectures.

Reviewed by:	imp
Differential Revision:	https://reviews.freebsd.org/D6091
Sponsored by:	DARPA, AFRL
Sponsored by:	HEIF5
2016-04-26 11:53:37 +00:00
Andrew Turner
ad76caf642 Use the yield instruction in the arm64 cpu_spinwait. This instruction is
a hint to the hardware the software is not performing a task.

Sponsored by:	ABT Systems Ltd
2016-04-25 17:32:08 +00:00
Pedro F. Giffuni
d9c9c81c08 sys: use our roundup2/rounddown2() macros when param.h is available.
rounddown2 tends to produce longer lines than the original code
and when the code has a high indentation level it was not really
advantageous to do the replacement.

This tries to strike a balance between readability using the macros
and flexibility of having the expressions, so not everything is
converted.
2016-04-21 19:57:40 +00:00
Andrew Turner
be95c24986 Group the ThunderX PCIe PEM newbus methods to help find them.
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-04-20 14:12:40 +00:00
Andrew Turner
cf9c1ecfd2 Pull out the MSI/MSI-X handling calls to simplify future intrng
integration.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-04-20 13:23:06 +00:00
Pedro F. Giffuni
41b610a8ee arm: for pointers replace 0 with NULL.
These are mostly cosmetical, no functional change.

Found with devel/coccinelle.
2016-04-15 14:30:40 +00:00
Andrew Turner
a48bd01151 Fix the types for the start, end, and count arguments to
arm_gic_fdt_alloc_resource. These were the old u_long where they should be
rman_res_t. Both of these are the same size on arm64 so this is just for
correctness, and would not have led to incorrect behaviour.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-04-14 14:44:23 +00:00
Andrew Turner
f8a39033c2 Set the upper limit of the DMAP region to the limit of RAM as was found in
the physmap. This will reduce the likelihood of an issue where we have
device memory mapped in the DMAP. This can only happen if it is within the
same 1G block of normal memory.

Reviewed by:	kib
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D5938
2016-04-14 10:43:28 +00:00
Ed Maste
6b580fd720 arm64: Avoid null dereference in its_init_cpu
its_init_cpu() is called from gic_v3_init_secondary(), and its_sc will
be NULL if its did not attach.

Sponsored by:	The FreeBSD Foundation
2016-04-13 21:00:00 +00:00
Andrew Turner
0b4890b034 Document the memory ranges within the kernel region to help with debugging
to track down which region an address is from.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-04-13 11:43:03 +00:00
Andrew Turner
9853d10411 Increase the arm64 kernel address space to 512GB, and the DMAP region to
2TB. The latter can be increased in 512GB chunks by adjusting the lower
address, however more work will be needed to increase the former.

There is still some work needed to only create a DMAP region for the RAM
address space as on ARM architectures all mappings should have the same
memory attributes, and these will be different for device and normal memory.

Reviewed by:	kib
Obtained from:	ABT Systems Ltd
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D5859
2016-04-13 09:44:32 +00:00
Pedro F. Giffuni
74b8d63dcc Cleanup unnecessary semicolons from the kernel.
Found with devel/coccinelle.
2016-04-10 23:07:00 +00:00
Zbigniew Bodek
017b6ebc0a Fix interrupts delivery on ThunderX for VF IDs beyond 8
SR-IOV devices usually use Alternative Routing ID (ARI).
In that case slot/device is always assumed to be 0 and
function/identifier is extended to 8 bits.

Fix interrupts delivery to VF IDs beyond 8 by using a correct
DevID if ARI is enabled.

Reviewed by:   jhb, wma
Obtained from: Semihalf
Sponsored by:  Cavium
Differential Revision: https://reviews.freebsd.org/D5855
2016-04-07 10:36:50 +00:00
Andrew Turner
5d58666c93 Use PHYS_IN_DMAP to check if a physical address is within the DMAP region.
Approved by:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-04-06 14:16:37 +00:00
Andrew Turner
70e7278593 Cleanup the early pagetable creation code in preperation for increasing
the size of the arm64 DMAP region.

Approved by:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-04-06 14:12:00 +00:00
Andrew Turner
c7d4b461b3 Allow vmparam.h to be included from assembly files on arm64.
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-04-06 14:08:10 +00:00
Ed Schouten
ab83575070 Make CloudABI's way of doing TLS more friendly to userspace emulators.
We're currently seeing how hard it would be to run CloudABI binaries on
operating systems cannot be modified easily (Windows, Mac OS X). The
idea is that we want to just run them without any sandboxing. Now
that CloudABI executables are PIE, this is already a bit easier, but TLS
is still problematic:

- CloudABI executables want to write to the %fs, which typically
  requires extra system calls by the emulator every time it needs to
  switch between CloudABI's and its own TLS.

- If CloudABI executables overwrite the %fs base unconditionally, it
  also becomes harder for the emulator to store a backup of the old
  value of %fs. To solve this, let's no longer overwrite %fs, but just
  %fs:0.

As CloudABI's C library does not use a TCB, this space can now be used
by an emulator to keep track of its internal state. The executable can
now safely overwrite %fs:0, as long as it makes sure that the TCB is
copied over to the new TLS area.

Ensure that there is an initial TLS area set up when the process starts,
only containing a bogus TCB. We don't really care about its contents on
FreeBSD.

Reviewed by:	kib
Differential Revision:	https://reviews.freebsd.org/D5836
2016-04-06 11:11:31 +00:00
Wojciech Macek
1c7c13aa0e Implement dtrace_getupcstack in ARM64
Allow using DTRACE for performance analysis of userspace
applications - the function call stack can be captured.
This is almost an exact copy of AMD64 solution.

Obtained from:         Semihalf
Sponsored by:          Cavium
Reviewed by:           emaste, gnn, jhibbits
Differential Revision: https://reviews.freebsd.org/D5779
2016-04-06 05:13:36 +00:00
Andrew Turner
53b832b091 Add a table to map from the FreeBSD CPUID space to the GIC CPUID space. On
many SoCs these two are the same, however there is no requirement for this
to be the case, e.g. on the ARM Juno we boot on what the GIC thinks of as
CPU 2, but FreeBSD numbers it CPU 0.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-04-04 17:04:33 +00:00
Andrew Turner
bc5a80161c Reduce the diff for when we switch to intrng. The IPI interrupts will be
split out to multiple handlers.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-04-04 15:13:17 +00:00
Wojciech Macek
4d1dd74a50 arm64: pagezero improvement
This change has been provided to improve pagezero call performance.

Submitted by:          Dominik Ermel <der@semihalf.com>
Obtained from:         Semihalf
Sponsored by:          Cavium
Reviewed by:           kib
Differential Revision: https://reviews.freebsd.org/D5741
2016-04-04 07:16:43 +00:00
Wojciech Macek
73ffb5e8a4 Add bzero.S to ARM64 machdep
Add fille missing from https://svnweb.freebsd.org/changeset/base/297536
2016-04-04 07:11:33 +00:00
Wojciech Macek
db27818234 arm64: bzero optimization
This optimization attempts to utylize as wide as possible register store instructions to zero large buffers.
The implementation, if possible, will use 'dc zva' to zero buffer by cache lines.

Speedup: 60x faster memory zeroing

Submitted by:          Dominik Ermel <der@semihalf.com>
Obtained from:         Semihalf
Sponsored by:          Cavium
Reviewed by:           kib
Differential Revision: https://reviews.freebsd.org/D5726
2016-04-04 07:06:20 +00:00
Ed Schouten
4a8b3b18cc Make Position Independent Executables work for CloudABI.
- Set BI_CAN_EXEC_DYN, so we can execute ET_DYN ELF files in addition to
  regular ET_EXECs.
- Provide an AT_BASE entry in the auxiliary vector, so the executable
  knows at which address it got loaded and can apply relocations.
2016-03-31 18:52:00 +00:00
Andrew Turner
f2f21faf62 Add support for 4 level pagetables. The userland address space has been
increased to 256TiB. The kernel address space can also be increased to be
the same size, but this will be performed in a later change.

To help work with an extra level of page tables two new functions have
been added, one to file the lowest level table entry, and one to find the
block/page level. Both of these find the entry for a given pmap and virtual
address.

This has been tested with a combination of buildworld, stress2 tests, and
by using sort to consume a large amount of memory by sorting /dev/zero. No
new issues are known to be present from this change.

Reviewed by:	kib
Obtained from:	ABT Systems Ltd
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D5720
2016-03-31 11:07:24 +00:00
Andrew Turner
6c5b1ed4b6 Read the CPU ID for the current CPU from the GIC. The GIC may have a
different ID space than the kernel. Because of this we need to read the
ID from the hardware. The hardware will provide this value to the CPU by
reading any of the first 8 Interrupt Processor Targets Registers.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D5706
2016-03-29 13:51:26 +00:00
Wojciech Macek
f379b4636a arm64: Fixing user space boudary checking in copyinout.S
Big buffer size could cause integer overflow and as a result
attempt to copy beyond VM_USERMAX_ADDRESS.

Fixing copyinstr boundary checking where compared value has been
overwritten by accident when setting fault handler.

Submitted by:          Dominik Ermel <der@semihalf.com>
Obtained from:         Semihalf
Sponsored by:          Cavium
Reviewed by:           kib
Differential Revision: https://reviews.freebsd.org/D5719
2016-03-24 13:28:33 +00:00
Wojciech Macek
f3e730a1e5 ARM64 copyinout improvements
The first of set of patches.
Use wider load/stores when aligned buffer is being copied.

In a simple test:
  dd if=/dev/zero of=/dev/null bs=1M count=1024
the performance jumped from 410MB/s up to 3.6GB/s.

TODO:
 - better handling of unaligned buffers (WiP)
 - implement similar mechanism to bzero

Submitted by:          Dominik Ermel <der@semihalf.com>
Obtained from:         Semihalf
Sponsored by:          Cavium
Reviewed by:           kib, andrew, emaste
Differential Revision: https://reviews.freebsd.org/D5664
2016-03-23 13:29:52 +00:00
Andrew Turner
f6c7371c81 Use the saved program state register to detect when an exception frame is
from userpsace. Previously we could have triggered a panic by trying to
jump to a kernel address from userland as the trap handling code thought we
received an ast in kernel mode.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-03-22 08:36:25 +00:00
Andrew Turner
b011fce09f Move the opt_ files to be included first so their definitions can be used
from within all further included files.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-03-18 16:32:22 +00:00
Andrew Turner
a9056bbb93 Rename COUNT_IPI to INTR_IPI_COUNT to reduce the diff with intrng.
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-03-18 16:29:58 +00:00
Andrew Turner
4d00d27b3a Reduce the diff with intrng by renaming similar functions. This is a noop,
but will help move to use the common interrupt handling code later.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-03-18 16:18:29 +00:00
Andrew Turner
7285efe8cd Remove the invalid L0_BLOCK definition. ARMv8 doesn't support block
translation in the level 0 descriptor.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-03-18 10:01:25 +00:00
Wojciech Macek
f54153bb08 pmap arm64: fixing pmap_invalidate_range
It seems that if range within one page is given this page will not be
invalidated at all. Clean it up.

Submitted by:          Dominik Ermel <der@semihalf.com>
Obtained from:         Semihalf
Sponsored by:          Cavium
Reviewed by:           wma, zbb
Approved by:           cognet (mentor)
Differential Revision: https://reviews.freebsd.org/D5569
2016-03-14 07:26:38 +00:00
John Baldwin
6fc8053f1a Fix reporting of the CloudABI ABI in kdump.
- Advertise the word size for CloudABI ABIs via the SV_LP64 flag.  All of
  the other ABIs include either SV_ILP32 or SV_LP64.
- Fix kdump to not assume a 32-bit ABI if the ABI flags field is non-zero
  but SV_LP64 isn't set.  Instead, only assume a 32-bit ABI if SV_ILP32 is
  set and fallback to the unknown value of "00" if neither SV_LP64 nor
  SV_ILP32 is set.

Reviewed by:	kib, ed
Differential Revision:	https://reviews.freebsd.org/D5560
2016-03-09 18:38:30 +00:00
Bjoern A. Zeeb
3d9ac4ecfc Force re-routing PCI interrupts (this is for legacy INTx not MSI).
Need this for gem5, but was not needed on real hadrware (yet) as it
was always MSI.

Reviewed by:		andrew, jhb
Discovered by:		andrew
Sponsored by:		DARPA/AFRL
Differential Revision:	https://reviews.freebsd.org/D5494
2016-03-02 15:20:42 +00:00
Wojciech Macek
52ea10af69 Improve ThunderX PEM driver to work on pass2 revision
Things changed:
    * do not allocate 4GB of SLI space, because it's the waste of
      system resources. Allocate only small portions when needed.
    * provide own implementation of activate_resource which performs
      address translation between PCI bus and host PA address space.
      This is temporary solution, should be replaced by bus_map_resource
      once implemented.

Obtained from:         Semihalf
Sponsored by:          Cavium
Approved by:           cognet (mentor)
Reviewed by:           jhb
Differential revision: https://reviews.freebsd.org/D5294
2016-03-02 08:39:59 +00:00
Wojciech Macek
a5d461a398 Get memory ranges from FDT if no EFI API is available on ARM64
Obtained from:         Semihalf
Submitted by:          Michal Stanek <mst@semihalf.com>
Sponsored by:          Annapurna Labs
Approved by:           cognet (mentor)
Reviewed by:           andrew, wma
Differential revision: https://reviews.freebsd.org/D5408
2016-03-01 12:50:24 +00:00
Wojciech Macek
b2552c46b6 Enable SRE_EL2 on ARM64
Enable system register access for EL2. Alpine-V2 is
the first device requiring this to be enabled.
It is also in-sync with Linux initialization code,
and compatible with Alpine-V2 uboot requirements.

Obtained from:         Semihalf
Submitted by:          Michal Stanek <mst@semihalf.com>
Sponsored by:          Annapurna Labs
Approved by:           cognet (mentor)
Reviewed by:           wma
Differential revision: https://reviews.freebsd.org/D5394
2016-03-01 08:15:00 +00:00
Wojciech Macek
30d9287468 Add uart 8250 device to GENERIC arm64 configuration
Obtained from:         Semihalf
Submitted by:          Michal Stanek <mst@semihalf.com>
Sponsored by:          Annapurna Labs
Approved by:           cognet (mentor)
Reviewed by:           zbb, wma
Differential revision: https://reviews.freebsd.org/D5406
2016-03-01 07:06:36 +00:00
Justin Hibbits
e665eafb25 Correct the memory rman ranges to be to BUS_SPACE_MAXADDR
Summary:
As part of the migration of rman_res_t to be typed to uintmax_t, memory ranges
must be clamped appropriately for the bus, to prevent completely bogus addresses
from being used.

This is extracted from D4544.

Reviewed By: cem
Sponsored by:	Alex Perez/Inertial Computing
Differential Revision: https://reviews.freebsd.org/D5134
2016-03-01 02:59:06 +00:00
Wojciech Macek
2445e7c84e Restore ThunderX Pass1.1 PCI changes removed by r295962
If Enhanced Allocation is not used, we can't allocate any random
    range. All internal devices have hardcoded place where they can
    be located within PCI address space. Fortunately, we can read
    this value from BAR.

Obtained from:         Semihalf
Sponsored by:          Cavium
Approved by:           cognet (mentor)
Reviewed by:           zbb
Differential revision: https://reviews.freebsd.org/D5455
2016-02-26 12:16:11 +00:00
Wojciech Macek
fb05500b24 Make pci_host_generic and thunderx_pci common
* provided OFW interface for pci_host_generic (for handling devices which are present in DTS under the PCI node)
  * removed support for internal PCI from arm64/cavium
  * cleaned up and made most of the code common

Obtained from:         Semihalf
Sponsored by:          Cavium
Approved by:           cognet (mentor)
Reviewed by:           zbb
Differential revision: https://reviews.freebsd.org/D5261
2016-02-24 06:05:30 +00:00
Wojciech Macek
0422bce8b2 Add Intel 10Gb support to ARM64 GENERIC kernel config
Obtained from:         Semihalf
Sponsored by:          Cavium
Approved by:           cognet (mentor)
Reviewed by:           zbb
Differential revision: https://reviews.freebsd.org/D5347
2016-02-22 13:34:43 +00:00
Svatopluk Kraus
35a0bc1260 As <machine/vmparam.h> is included from <vm/vm_param.h>, there is no
need to include it explicitly when <vm/vm_param.h> is already included.

Suggested by:	alc
Reviewed by:	alc
Differential Revision:	https://reviews.freebsd.org/D5379
2016-02-22 09:08:04 +00:00
Svatopluk Kraus
d6849317c5 As <machine/param.h> is included from <sys/param.h>, there is no need
to include it explicitly when <sys/param.h> is already included.

Reviewed by:	alc, kib
Differential Revision:	https://reviews.freebsd.org/D5378
2016-02-22 09:04:36 +00:00
Svatopluk Kraus
a1e1814d76 As <machine/pmap.h> is included from <vm/pmap.h>, there is no need to
include it explicitly when <vm/pmap.h> is already included.

Reviewed by:	alc, kib
Differential Revision:	https://reviews.freebsd.org/D5373
2016-02-22 09:02:20 +00:00
Justin Hibbits
7915adb560 Introduce a RMAN_IS_DEFAULT_RANGE() macro, and use it.
This simplifies checking for default resource range for bus_alloc_resource(),
and improves readability.

This is part of, and related to, the migration of rman_res_t from u_long to
uintmax_t.

Discussed with:	jhb
Suggested by:	marcel
2016-02-20 01:32:58 +00:00
Zbigniew Bodek
b998c9656b Introduce bus_get_bus_tag() method
Provide bus_get_bus_tag() for sparc64, powerpc, arm, arm64 and mips
nexus and its children in order to return a platform specific default tag.

This is required to ensure generic correctness of the bus_space tag.
It is especially needed for arches where child bus tag does not match
the parent bus tag. This solves the problem with ppc architecture
where the PCI bus tag differs from parent bus tag which is big-endian.

This commit is a part of the following patch:
https://reviews.freebsd.org/D4879

Submitted by:  Marcin Mazurek <mma@semihalf.com>
Obtained from: Semihalf
Sponsored by:  Annapurna Labs
Reviewed by:   jhibbits, mmel
Differential Revision: https://reviews.freebsd.org/D4879
2016-02-18 13:00:04 +00:00
Wojciech Macek
98bc9384c5 Fix ThunderX external PEM bus offset
Obtained from:         Semihalf
Sponsored by:          Cavium
Approved by:           cognet (mentor)
Reviewed by:           zbb
Differential revision: https://reviews.freebsd.org/D5293
2016-02-18 11:26:08 +00:00
Svatopluk Kraus
1413a3ab64 Remove pd_prot and pd_cache members from struct arm_devmap_entry.
The struct is used for definition of static device mappings which
should always have same protection and attributes.
2016-02-17 12:36:24 +00:00
Andrew Turner
45fd186285 Allow callers of OF_decode_addr to get the size of the found mapping. This
will allow for code that uses the old fdt_get_range and fdt_regsize
functions to find a range, map it, access, then unmap to replace this, up
to and including the map, with a call to OF_decode_addr.

As this function should only be used in the early boot code the unmap is
mostly do document we no longer need the mapping as it's a no-op, at least
on arm.

Reviewed by:	jhibbits
Sponsored by:	ABT Systems Ltd
Differential Revision:	https://reviews.freebsd.org/D5258
2016-02-16 15:18:12 +00:00
Zbigniew Bodek
9ccaab6db5 Support PEM that is not a PCI endpoint on ThunderX
Some chip revisions don't have their external PCIe buses
behind the internal bridge. Add support for FDT-configurable
PEMs but keep ability for PCIe enumeration.

Reviewed by:   andrew, wma
Obtained from: Semihalf
Sponsored by:  Cavium
Differential Revision: https://reviews.freebsd.org/D5285
2016-02-16 11:43:57 +00:00
Andrew Turner
770fd1c976 Only update curthread and curpcb after we have finished using the old
values.

If switching from a thread that used floating-point registers to a thread
that is still running, but holding the blocked_lock lock we would switch
the curthread to the new (running) thread, then call critical_enter. This
will non-atomically increment td_critnest, and later call critical_exit to
non-atomically decrement this value.

This can happen at the same time as the new thread is still running on the
old core, also calling these functions. In this case there will be a race
between these non-atomic operations. This can be an issue as we could loose
one of these operations leading to the value to not return to zero.

If, later on, we then hit a data abort we check if the td_critnest is zero.
If this check fails we will panic the kernel.

This has been observed when running pcmstat on a Cavium ThunderX. The pcm
thread will use the blocked_lock lock and there is a high chance userspace
will use the floating-point registers. When, later on, pmcstat triggers a
data abort we will hit this panic.

The fix is to update these values after storing the floating-point state.
This means we use the correct curthread while storing the state so it will
not be an issue that the changes to td_critnest are non-atomic.

Sponsored by:	ABT Systems Ltd
2016-02-12 12:38:04 +00:00
Zbigniew Bodek
6cd36342c0 Support interrupts binding in GICv3 and ITS
- Add MOVI command and routine for the LPI migration
- Allow to search for the ITS device descriptor using
  not only devID but also LPI number.
- Bind SPIs in the Distributor
- Don't bind its_dev to collection. Keep track of the collection
  IDs for each LPI.

Reviewed by:   wma
Obtained from: Semihalf
Sponsored by:  Cavium
Differential Revision: https://reviews.freebsd.org/D5231
2016-02-11 12:04:58 +00:00
Zbigniew Bodek
907a0579aa Implement finer locking in ITS
- Change locks' names to be more suitable
- Don't use blocking mutex. Lock only basic operations such
  as lists or bitmaps modifications.

Reviewed by:   wma
Obtained from: Semihalf
Sponsored by:  Cavium
Differential Revision: https://reviews.freebsd.org/D5230
2016-02-11 12:03:11 +00:00
Zbigniew Bodek
47a1ff355e Initially bind all interrupts to the boot CPU when using GICv3
This should be done by routing all interrupts to CPU0,
different assignment will be induced by either interrupts
shuffling or bus_bind_intr().

Reviewed by:   wma
Obtained from: Semihalf
Sponsored by:  Cavium
Differential Revision: https://reviews.freebsd.org/D5229
2016-02-11 12:01:33 +00:00
Zbigniew Bodek
55bdcadded Call pmc_hook() correctly in the ARM64 interrupt handler
pmc_hook() was called only in case of the stray interrupt but should
rather be called on each interrupt. Move in to the arm_cpu_intr()
handler, out of the critical section too.

Reviewed by:   br
Obtained from: Semihalf
Sponsored by:  Cavium
Differential Revision: https://reviews.freebsd.org/D5161
2016-02-11 11:59:32 +00:00
Zbigniew Bodek
be7aab76ec Introduce bus_bind_intr method for ARM64
It can be used to bind specific interrupt to a particular CPU.
Requires PIC support for interrupts binding.

Reviewed by:   wma
Obtained from: Semihalf
Sponsored by:  Cavium
Differential Revision: https://reviews.freebsd.org/D5122
2016-02-11 11:58:27 +00:00
Zbigniew Bodek
513411c9f5 Fix bugs in interrupts allocation on ARM64
Separate interrupt descriptors lookup from allocation. It was possible
to perform config on non-existing interrupt simply by allocating spurious
descriptor.
Must lock the interrupt descriptors table lookup to avoid mismatches.
This ought to prevent trouble while setting up new interrupt
and dispatching existing one.
Use spin mutex rather than sleep mutex. This is mainly due to lock in
arm_dispatch_intr.
This should be eventually changed to a lock-less solution without
walking through a linked list on each interrupt.

Reviewed by:   andrew, wma
Obtained from: Semihalf
Sponsored by:  Cavium
Differential Revision: https://reviews.freebsd.org/D5121
2016-02-11 11:57:13 +00:00
Zbigniew Bodek
8133eda921 Minor clean-ups for ARM64 GICv3 and GIC drivers
GICv3:
- move ICC_SGI1R_EL1 definitions to armreg.h and use proper system
  register's names
GIC:
- remove unused functions

Reviewed by:   andrew
Obtained from: Semihalf
Sponsored by:  Cavium
Differential Revision: https://reviews.freebsd.org/D5119
2016-02-11 11:55:37 +00:00
Wojciech Macek
c7fc655f3f ARM64 disassembler: support for LDR instructions
Implemented disassembly for a whole bunch of
    various ldr instructions.

Obtained from:         Semihalf
Sponsored by:          Cavium
Approved by:           cognet (mentor)
Reviewed by:           zbb
Differential revision: https://reviews.freebsd.org/D5217
2016-02-11 06:50:11 +00:00
Andrew Turner
d664515e68 Revert an arm64 change that sneaked in with r295464. 2016-02-10 10:28:33 +00:00
Andrew Turner
907fe11655 Update of the Allwinner drivers to:
* Use the Linux compat string
 * Use EARLY_DRIVER_MODULE to attach at the right time
 * Add a generic A10 kernel config file
 * A20 now use generic_timer
 * Add two new dts files for Olimex boards
 * Update our custom DTS file for A10 and A20 to use the same compatible
   property names as the vendor ones.

Submitted by:	Emmanuel Vadot <manu@bidouilliste.com>
Differential Revision:	https://reviews.freebsd.org/D4792
2016-02-10 09:19:29 +00:00
Gleb Smirnoff
b28cc462ad Include sys/_task.h into uma_int.h, so that taskqueue.h isn't a
requirement for uma_int.h.

Suggested by:	jhb
2016-02-09 20:22:35 +00:00
Andrew Turner
f75ef22a66 Use designated initialisers for the db disassembler interface to help with
finding the struct when searching for one of its members.
2016-02-09 10:18:22 +00:00
Wojciech Macek
c00a03a11d Ignore invalid page descriptors in ARM64 pmap_mincore
Prevent the function from null-pointer-dereference when unexisting
    mapping is being processed.

Obtained from:         Semihalf
Sponsored by:          Cavium
Approved by:           cognet (mentor)
Reviewed by:           zbb, cognet
Differential revision: https://reviews.freebsd.org/D5228
2016-02-09 06:26:27 +00:00
Andrew Turner
55beb2a538 Implement kdb_cpu_sync_icache on arm64.
Sponsored by:	ABT Systems Ltd
2016-02-05 15:38:28 +00:00
Andrew Turner
f13ec4b40e Enable checking of the stack alignment. The stack should be aligned to a
16-byte value. With this the hardware will check if a memory access uses
an incorrectly aligned stack pointer as the base address.

Sponsored by:	ABT Systems Ltd
2016-02-04 17:22:15 +00:00
Ruslan Bukin
46c9b07f22 Fix build. 2016-02-04 11:52:53 +00:00
Gleb Smirnoff
6c95c7903c Fix build. 2016-02-04 03:55:41 +00:00
Andrew Turner
729ac0ee8c Handle a misaligned stack pointer exception from userspace. The exception
still needs to be enabled, but this will help with testing.

Sponsored by:	ABT Systems Ltd
2016-02-03 17:00:19 +00:00
Andrew Turner
3df911720b Increase the size of PHYS_AVAIL_SIZE to allow firmware to provide a large
number of physical memory locations we can access. This is the case on
some HiKey boards that may have UEFI reserved memory dispersed through the
physical space.

Sponsored by:	ABT Systems Ltd
2016-02-02 17:59:43 +00:00
Andrew Turner
332b343376 Ensure we don't overflow the phys_avail array. Some firmware may provide
more memory locations than we have space to record.

Sponsored by:	ABT Systems Ltd
2016-02-02 17:57:15 +00:00
Andrew Turner
50d9282660 Increase the space we use after the kernel to 8MiB. On 2GiB HiKey board we
would try to access data past this point stopping the boot.

Sponsored by:	ABT Systems Ltd
2016-02-02 16:35:37 +00:00
Andrew Turner
2c0ed87ef8 Only look for the ranges property when we have children. This fixes booting
on systems with a gicv2, but no PCIe so no gicv2m.

Sponsored by:	ABT Systems Ltd
2016-02-02 16:32:44 +00:00