Commit Graph

327 Commits

Author SHA1 Message Date
Andrew Turner
04ff877e4c Switch arm64 to use intrng by default. The old interrupt handling code can
still be used, however this is expected to be removed soon.

Obtained from:	ABT Systems Ltd
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
2016-06-07 20:14:08 +00:00
Svatopluk Kraus
ad5244ece1 INTRNG - change the way how an interrupt mapping data are provided
to the framework in OFW (FDT) case.

This is a follow-up to r301451.

Differential Revision:	https://reviews.freebsd.org/D6634
2016-06-05 16:20:12 +00:00
Andrew Turner
550d01a211 Add the GICv3 ITS intrng driver. As the interface to the interrupt
framework has significantly changed the driver has moved to a new file.
While it shares some code with the existing driver this has been modified
to work better with the intrng framework.

This has been tested on the ThunderX servers in the netperf cluster and has
been used to boot them for other testing, including DTrace and hwpmc.

With this we can use intrng on all supported arm64 platforms I was able to
test on. It is expected we will move to intrng soon, and disable the old
arm64 interrupt framework.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D6437
2016-06-03 10:28:06 +00:00
Zbigniew Bodek
32fb8300d4 Return real error value instead of hard-coded ENXIO (fix after r300149)
It is possible to return real error value in case of gic_v3_attach()
failure instead of hard-coded ENXIO.

Obtained from:	Semihalf
Sponsored by:	Cavium
2016-06-01 08:20:10 +00:00
Andrew Turner
920c8a57c0 dpcpu_init should have also passed in the calculated cpuid, not the
devicetree ID.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-31 19:17:32 +00:00
Andrew Turner
e0db04b3de Allow the kernel to boot on a CPU where the devicetree has numbered it with
a non-zero ID. To do this we increment the cpuid of any CPUs with a smaller
devicetree ID by one to stop them conflicting with the boot CPU.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-31 18:45:52 +00:00
Andrew Turner
cea2a7fe6d Enable setting BF_COHERENT on DMA tags. This allows the kernel to start
using the cache handling functions.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-31 16:28:56 +00:00
Andrew Turner
2760c2cad4 Mark the ThunderX and generic PCI drivers as cache-coherent when we know
this to be the case. This will mean we don't try and handle the cache in
bus_dmamap_sync when it is not needed.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D6605
2016-05-31 09:15:21 +00:00
Andrew Turner
02759f14f2 Extract the correct bits from the GICD_TYPER register. The interrupt count
is encoded in the bottom 5 bits.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-20 15:43:51 +00:00
Andrew Turner
da257dda10 Add more useful GICv3 register definitions. While here fix
GITS_CBASER_CACHE_MASK to use the correct shift macro.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-20 13:11:07 +00:00
Andrew Turner
98670cad72 Filter out BUS_DMASYNC_POSTWRITE sync operations, there is nothing for us
to do on these.

Reported by:	wma
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-20 12:38:48 +00:00
Andrew Turner
fa13831f86 Enable NEW_PCIB on arm64.
Obtained from:	ABT Systems Ltd
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
2016-05-20 08:43:18 +00:00
Andrew Turner
6d2f7ef617 Handle PCI_RES_BUS on the generic and ThunderX PCIe drivers. This has been
tested on the Pass 1.1 and 2.0 ThunderX machines in the Netperf cluster.

Reviewed by:	jhb
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D6453
2016-05-20 08:29:00 +00:00
Andrew Turner
b7c02deed2 Define PCI_RES_BUS for NEW_PCIB
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-19 14:00:18 +00:00
Andrew Turner
9346e9130d Return the struct intr_pic pointer from intr_pic_register. This will be
needed in later changes where we may not be able to lock the pic list lock
to perform a lookup, e.g. from within interrupt context.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-18 15:05:44 +00:00
Zbigniew Bodek
b02a807b46 Add support for MSI/MSIX deallocation on GICv3-ITS
Allow to deallocate previously allocated ITS device along with
its interrupts. Interrupt numbers are being freed when the last
LPI number is no longer busy.

Reviewed by:	wma
Obtained from:	Semihalf
Sponsored by:	Cavium
Differential Revision:	https://reviews.freebsd.org/D6351
2016-05-18 10:09:07 +00:00
Bjoern A. Zeeb
1f420ed469 The GIC (v2 at least) has a bit in the TYPER register to indicate whether the GIC
supports the Security Extensions or not. This bit is not the same as the CPU one.
Currently we are not checking for either before trying to write to the special
registers.  This can lead to problems on hardware or simulators that do not
provide the security extensions.  Add the missing checks. Their interactions with
the CPU flag is not entirely clear to me but using a macro will make it easier
to quickly adjust the condition once the CPU bits are sorted as well.

Reviewed by:	br
Sponsored by:	DARPA/AFRL
Differential Revision:	https://reviews.freebsd.org/D6397
2016-05-17 13:12:26 +00:00
Andrew Turner
e34357f56d Add an arm64 kernel config to help testing intrng. It is expected this
config will only last a few weeks until we switch to this interrupt
framework.

Obtained from:	 ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-17 12:48:30 +00:00
Andrew Turner
9a38d21983 Clean up the GICv3 intrng code:
* In gic_v3_attach free the correct data on failure.
 * Implement gic_v3_teardown_intr.
 * Update the panic string when enabling/disabling an invalid interrupt.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-17 12:46:50 +00:00
Andrew Turner
88f7980a81 Add intrng support to the GICv3 driver. It lacks ITS support so won't handle
MSI or MSI-X interrupts, however this is enought to boot FreeBSD under the
ARM Foundation Model with a GICv3 interrupt controller.

Approved by:	ABT Systems Ltd
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
2016-05-16 14:07:43 +00:00
Andrew Turner
2b48ec8e6d Move the call to intr_pic_init_secondary to the same place as in the
non-intrng case.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-16 12:02:06 +00:00
Andrew Turner
72b3f638b7 Add support for intrng to arm64. As the GICv3 drivers will need to be
updated, and until further testing can be done, this is disabled for now.

It is expected arm64 will switch to this interface, and the old interface
will be removed before 11.0 is released.

Obtained from:	ABT Systems Ltd
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
2016-05-16 10:48:51 +00:00
Andrew Turner
4e1763d34c Teach the ThunderX PCI PEM driver about intrng. This will be used later
when arm64 is supported by intrng.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-16 10:03:57 +00:00
Andrew Turner
1e43b18c4b Add a pcib interface for use by interrupt controllers that need to
translate the pci rid to a controller ID. The translation could be based
on the 'msi-map' OFW property, a similar ACPI option, or hard-coded for
hardware lacking the above options.

Reviewed by:	wma
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-16 09:31:44 +00:00
Andrew Turner
5db4448f06 Add support to the arm64 busdma to handle the cache. For now this is
disabled, however when we enable it it will default to assume memory is
not cache-coherent, unless either the tag was created or the parent was
marked as cache-coherent.

Obtained from:	ABT Systems Ltd
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
2016-05-13 16:03:50 +00:00
Andrew Turner
f82b6f5255 Rename the internal BUC_DMA_* flags to BF_* so they won't conflict with
the flags in sys/bus_dma.h.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-12 15:24:36 +00:00
Andrew Turner
2b45fde379 Restrict the memory barriers in bus_dmamap_sync to just the operations
where it's needed.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-12 12:13:28 +00:00
Andrew Turner
9615213bef Call busdma_swi from swi_vm as is done from other architectures.
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-11 18:48:47 +00:00
Andrew Turner
b7a7ee3af1 On arm64 always create a bus_dmamap_t object. This will be use to hold the
list of memory that the kernel will need to sync when operating with a
non-cache coherent DMA engine.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-11 16:53:41 +00:00
Andrew Turner
7ca01e8f89 Add data barriers to the arm64 bus_dmamap_sync function. We need these
to ensure ordering between the CPU and device. As the CPU and DMA target
may be in different shareability domains they need to be full system
barriers.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-11 14:59:54 +00:00
Zbigniew Bodek
46542fb652 Fix I/O coherence issues on ThunderX when SMP is disabled
To maintain coherence between cache and DMA memory appropriate
shareability flags need to be set in the PTE regardless of SMP
option.

Reviewed by:	wma
Obtained from:	Semihalf
Sponsored by:	Cavium
Differential Revision: https://reviews.freebsd.org/D6231
2016-05-11 13:23:56 +00:00
Andrew Turner
fb853e4270 Push the logic to talk with the MSI/MSI-X interrupt controller to the FDT
attachment. This is where it will live when we import intrng as it will
need to look at either the msi-parent or msi-map FDT properties.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-10 15:45:59 +00:00
John Baldwin
82cb5c3b5b Native PCI-express HotPlug support.
PCI-express HotPlug support is implemented via bits in the slot
registers of the PCI-express capability of the downstream port along
with an interrupt that triggers when bits in the slot status register
change.

This is implemented for FreeBSD by adding HotPlug support to the
PCI-PCI bridge driver which attaches to the virtual PCI-PCI bridges
representing downstream ports on HotPlug slots. The PCI-PCI bridge
driver registers an interrupt handler to receive HotPlug events. It
also uses the slot registers to determine the current HotPlug state
and drive an internal HotPlug state machine. For simplicty of
implementation, the PCI-PCI bridge device detaches and deletes the
child PCI device when a card is removed from a slot and creates and
attaches a PCI child device when a card is inserted into the slot.

The PCI-PCI bridge driver provides a bus_child_present which claims
that child devices are present on HotPlug-capable slots only when a
card is inserted. Rather than requiring a timeout in the RC for
config accesses to not-present children, the pcib_read/write_config
methods fail all requests when a card is not present (or not yet
ready).

These changes include support for various optional HotPlug
capabilities such as a power controller, mechanical latch,
electro-mechanical interlock, indicators, and an attention button.
It also includes support for devices which require waiting for
command completion events before initiating a subsequent HotPlug
command. However, it has only been tested on ExpressCard systems
which support surprise removal and have none of these optional
capabilities.

PCI-express HotPlug support is conditional on the PCI_HP option
which is enabled by default on arm64, x86, and powerpc.

Reviewed by:	adrian, imp, vangyzen (older versions)
Relnotes:	yes
Differential Revision:	https://reviews.freebsd.org/D6136
2016-05-05 22:26:23 +00:00
Zbigniew Bodek
42638f9c5a Fix GICv3 build after r299090
Obtained from:	Semihalf
Sponsored by:	Cavium
2016-05-05 17:51:14 +00:00
Andrew Turner
ef6b515b69 Disable ACPI on arm64 ad it has only had minimal testing and is causing
boot issues when booting with FDT. It is planned to re-enable this at a
later date.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-04-26 14:21:39 +00:00
Ruslan Bukin
30b72b6871 Move arm's devmap to some generic place, so it can be used
by other architectures.

Reviewed by:	imp
Differential Revision:	https://reviews.freebsd.org/D6091
Sponsored by:	DARPA, AFRL
Sponsored by:	HEIF5
2016-04-26 11:53:37 +00:00
Andrew Turner
ad76caf642 Use the yield instruction in the arm64 cpu_spinwait. This instruction is
a hint to the hardware the software is not performing a task.

Sponsored by:	ABT Systems Ltd
2016-04-25 17:32:08 +00:00
Pedro F. Giffuni
d9c9c81c08 sys: use our roundup2/rounddown2() macros when param.h is available.
rounddown2 tends to produce longer lines than the original code
and when the code has a high indentation level it was not really
advantageous to do the replacement.

This tries to strike a balance between readability using the macros
and flexibility of having the expressions, so not everything is
converted.
2016-04-21 19:57:40 +00:00
Andrew Turner
be95c24986 Group the ThunderX PCIe PEM newbus methods to help find them.
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-04-20 14:12:40 +00:00
Andrew Turner
cf9c1ecfd2 Pull out the MSI/MSI-X handling calls to simplify future intrng
integration.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-04-20 13:23:06 +00:00
Pedro F. Giffuni
41b610a8ee arm: for pointers replace 0 with NULL.
These are mostly cosmetical, no functional change.

Found with devel/coccinelle.
2016-04-15 14:30:40 +00:00
Andrew Turner
a48bd01151 Fix the types for the start, end, and count arguments to
arm_gic_fdt_alloc_resource. These were the old u_long where they should be
rman_res_t. Both of these are the same size on arm64 so this is just for
correctness, and would not have led to incorrect behaviour.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-04-14 14:44:23 +00:00
Andrew Turner
f8a39033c2 Set the upper limit of the DMAP region to the limit of RAM as was found in
the physmap. This will reduce the likelihood of an issue where we have
device memory mapped in the DMAP. This can only happen if it is within the
same 1G block of normal memory.

Reviewed by:	kib
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D5938
2016-04-14 10:43:28 +00:00
Ed Maste
6b580fd720 arm64: Avoid null dereference in its_init_cpu
its_init_cpu() is called from gic_v3_init_secondary(), and its_sc will
be NULL if its did not attach.

Sponsored by:	The FreeBSD Foundation
2016-04-13 21:00:00 +00:00
Andrew Turner
0b4890b034 Document the memory ranges within the kernel region to help with debugging
to track down which region an address is from.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-04-13 11:43:03 +00:00
Andrew Turner
9853d10411 Increase the arm64 kernel address space to 512GB, and the DMAP region to
2TB. The latter can be increased in 512GB chunks by adjusting the lower
address, however more work will be needed to increase the former.

There is still some work needed to only create a DMAP region for the RAM
address space as on ARM architectures all mappings should have the same
memory attributes, and these will be different for device and normal memory.

Reviewed by:	kib
Obtained from:	ABT Systems Ltd
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D5859
2016-04-13 09:44:32 +00:00
Pedro F. Giffuni
74b8d63dcc Cleanup unnecessary semicolons from the kernel.
Found with devel/coccinelle.
2016-04-10 23:07:00 +00:00
Zbigniew Bodek
017b6ebc0a Fix interrupts delivery on ThunderX for VF IDs beyond 8
SR-IOV devices usually use Alternative Routing ID (ARI).
In that case slot/device is always assumed to be 0 and
function/identifier is extended to 8 bits.

Fix interrupts delivery to VF IDs beyond 8 by using a correct
DevID if ARI is enabled.

Reviewed by:   jhb, wma
Obtained from: Semihalf
Sponsored by:  Cavium
Differential Revision: https://reviews.freebsd.org/D5855
2016-04-07 10:36:50 +00:00
Andrew Turner
5d58666c93 Use PHYS_IN_DMAP to check if a physical address is within the DMAP region.
Approved by:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-04-06 14:16:37 +00:00
Andrew Turner
70e7278593 Cleanup the early pagetable creation code in preperation for increasing
the size of the arm64 DMAP region.

Approved by:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-04-06 14:12:00 +00:00