83 Commits

Author SHA1 Message Date
andrew
58f81b7be8 Add support for intrng to arm64. As the GICv3 drivers will need to be
updated, and until further testing can be done, this is disabled for now.

It is expected arm64 will switch to this interface, and the old interface
will be removed before 11.0 is released.

Obtained from:	ABT Systems Ltd
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
2016-05-16 10:48:51 +00:00
andrew
297d2b0fcc Add support to the arm64 busdma to handle the cache. For now this is
disabled, however when we enable it it will default to assume memory is
not cache-coherent, unless either the tag was created or the parent was
marked as cache-coherent.

Obtained from:	ABT Systems Ltd
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
2016-05-13 16:03:50 +00:00
andrew
8c3616af7a Call busdma_swi from swi_vm as is done from other architectures.
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-11 18:48:47 +00:00
zbb
bc6d6fd49a Fix I/O coherence issues on ThunderX when SMP is disabled
To maintain coherence between cache and DMA memory appropriate
shareability flags need to be set in the PTE regardless of SMP
option.

Reviewed by:	wma
Obtained from:	Semihalf
Sponsored by:	Cavium
Differential Revision: https://reviews.freebsd.org/D6231
2016-05-11 13:23:56 +00:00
br
778cc5a811 Move arm's devmap to some generic place, so it can be used
by other architectures.

Reviewed by:	imp
Differential Revision:	https://reviews.freebsd.org/D6091
Sponsored by:	DARPA, AFRL
Sponsored by:	HEIF5
2016-04-26 11:53:37 +00:00
andrew
b40ffdceea Use the yield instruction in the arm64 cpu_spinwait. This instruction is
a hint to the hardware the software is not performing a task.

Sponsored by:	ABT Systems Ltd
2016-04-25 17:32:08 +00:00
andrew
6d3c93b78f Set the upper limit of the DMAP region to the limit of RAM as was found in
the physmap. This will reduce the likelihood of an issue where we have
device memory mapped in the DMAP. This can only happen if it is within the
same 1G block of normal memory.

Reviewed by:	kib
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D5938
2016-04-14 10:43:28 +00:00
andrew
fcf07e4e7b Document the memory ranges within the kernel region to help with debugging
to track down which region an address is from.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-04-13 11:43:03 +00:00
andrew
66ef37d31b Increase the arm64 kernel address space to 512GB, and the DMAP region to
2TB. The latter can be increased in 512GB chunks by adjusting the lower
address, however more work will be needed to increase the former.

There is still some work needed to only create a DMAP region for the RAM
address space as on ARM architectures all mappings should have the same
memory attributes, and these will be different for device and normal memory.

Reviewed by:	kib
Obtained from:	ABT Systems Ltd
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D5859
2016-04-13 09:44:32 +00:00
andrew
d532565c6d Allow vmparam.h to be included from assembly files on arm64.
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-04-06 14:08:10 +00:00
wma
f5a4347e1c Implement dtrace_getupcstack in ARM64
Allow using DTRACE for performance analysis of userspace
applications - the function call stack can be captured.
This is almost an exact copy of AMD64 solution.

Obtained from:         Semihalf
Sponsored by:          Cavium
Reviewed by:           emaste, gnn, jhibbits
Differential Revision: https://reviews.freebsd.org/D5779
2016-04-06 05:13:36 +00:00
wma
153a55cf7f arm64: pagezero improvement
This change has been provided to improve pagezero call performance.

Submitted by:          Dominik Ermel <der@semihalf.com>
Obtained from:         Semihalf
Sponsored by:          Cavium
Reviewed by:           kib
Differential Revision: https://reviews.freebsd.org/D5741
2016-04-04 07:16:43 +00:00
wma
4ba80483c0 arm64: bzero optimization
This optimization attempts to utylize as wide as possible register store instructions to zero large buffers.
The implementation, if possible, will use 'dc zva' to zero buffer by cache lines.

Speedup: 60x faster memory zeroing

Submitted by:          Dominik Ermel <der@semihalf.com>
Obtained from:         Semihalf
Sponsored by:          Cavium
Reviewed by:           kib
Differential Revision: https://reviews.freebsd.org/D5726
2016-04-04 07:06:20 +00:00
andrew
b45ea0fe80 Add support for 4 level pagetables. The userland address space has been
increased to 256TiB. The kernel address space can also be increased to be
the same size, but this will be performed in a later change.

To help work with an extra level of page tables two new functions have
been added, one to file the lowest level table entry, and one to find the
block/page level. Both of these find the entry for a given pmap and virtual
address.

This has been tested with a combination of buildworld, stress2 tests, and
by using sort to consume a large amount of memory by sorting /dev/zero. No
new issues are known to be present from this change.

Reviewed by:	kib
Obtained from:	ABT Systems Ltd
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D5720
2016-03-31 11:07:24 +00:00
andrew
acaab8fb10 Use the saved program state register to detect when an exception frame is
from userpsace. Previously we could have triggered a panic by trying to
jump to a kernel address from userland as the trap handling code thought we
received an ast in kernel mode.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-03-22 08:36:25 +00:00
andrew
d5c75c2b10 Rename COUNT_IPI to INTR_IPI_COUNT to reduce the diff with intrng.
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-03-18 16:29:58 +00:00
andrew
162f59a4a0 Reduce the diff with intrng by renaming similar functions. This is a noop,
but will help move to use the common interrupt handling code later.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-03-18 16:18:29 +00:00
andrew
df8c050ab1 Remove the invalid L0_BLOCK definition. ARMv8 doesn't support block
translation in the level 0 descriptor.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-03-18 10:01:25 +00:00
bz
983c664612 Force re-routing PCI interrupts (this is for legacy INTx not MSI).
Need this for gem5, but was not needed on real hadrware (yet) as it
was always MSI.

Reviewed by:		andrew, jhb
Discovered by:		andrew
Sponsored by:		DARPA/AFRL
Differential Revision:	https://reviews.freebsd.org/D5494
2016-03-02 15:20:42 +00:00
wma
ad7622cfab Enable SRE_EL2 on ARM64
Enable system register access for EL2. Alpine-V2 is
the first device requiring this to be enabled.
It is also in-sync with Linux initialization code,
and compatible with Alpine-V2 uboot requirements.

Obtained from:         Semihalf
Submitted by:          Michal Stanek <mst@semihalf.com>
Sponsored by:          Annapurna Labs
Approved by:           cognet (mentor)
Reviewed by:           wma
Differential revision: https://reviews.freebsd.org/D5394
2016-03-01 08:15:00 +00:00
skra
19710a8b47 Remove pd_prot and pd_cache members from struct arm_devmap_entry.
The struct is used for definition of static device mappings which
should always have same protection and attributes.
2016-02-17 12:36:24 +00:00
zbb
95137856d1 Introduce bus_bind_intr method for ARM64
It can be used to bind specific interrupt to a particular CPU.
Requires PIC support for interrupts binding.

Reviewed by:   wma
Obtained from: Semihalf
Sponsored by:  Cavium
Differential Revision: https://reviews.freebsd.org/D5122
2016-02-11 11:58:27 +00:00
zbb
95c8add1f2 Minor clean-ups for ARM64 GICv3 and GIC drivers
GICv3:
- move ICC_SGI1R_EL1 definitions to armreg.h and use proper system
  register's names
GIC:
- remove unused functions

Reviewed by:   andrew
Obtained from: Semihalf
Sponsored by:  Cavium
Differential Revision: https://reviews.freebsd.org/D5119
2016-02-11 11:55:37 +00:00
andrew
a718ec0580 Implement kdb_cpu_sync_icache on arm64.
Sponsored by:	ABT Systems Ltd
2016-02-05 15:38:28 +00:00
andrew
5b58d7a3f8 Increase the size of PHYS_AVAIL_SIZE to allow firmware to provide a large
number of physical memory locations we can access. This is the case on
some HiKey boards that may have UEFI reserved memory dispersed through the
physical space.

Sponsored by:	ABT Systems Ltd
2016-02-02 17:59:43 +00:00
andrew
f0c954c605 Implement single stepping on arm64. We need to set the single step bits in
the processor and debug state registers. A flag has been added to the pcb
to tell us when to enable single stepping for a given thread.

Reviewed by:	kib
Sponsored by:	ABT Systems Ltd
Differential Revision:	https://reviews.freebsd.org/D4730
2016-02-02 10:28:56 +00:00
wma
047ea0e83b Framework for ARM64 instruction disassembler
Provide an easy to use framework for ARM64 DDB disassembler.
    This commit does not contain full list of instruction opcodes.

Obtained from:         Semihalf
Sponsored by:          Cavium
Approved by:           cognet (mentor)
Reviewed by:           zbb, andrew, cognet
Differential revision: https://reviews.freebsd.org/D5114
2016-01-29 13:06:30 +00:00
nwhitehorn
b6345059e4 Make using the #address-cells property on the interrupt parent in device
tree parsing opt-out rather than opt-in. All FDT-based systems as well as
PowerPC systems with real Open Firmware use the CHRP-derived binding that
includes it, which makes SPARC the odd man out here. Making it opt-out
avoids astonishment on new platform bring up.
2016-01-02 19:28:35 +00:00
andrew
5e0178e881 Decode and print the ID_AA64* registers on boot. These registers hold
information on what the core supports. In most cases these will be
identical across most CPUs in the SoC, however there may be the case where,
with a big.LITTLE setup they may differ. In this case we print the
decoded data on all CPUs.

Reviewed by:	kib
Sponsored by:	ABT Systems Ltd
Differential Revision:	https://reviews.freebsd.org/D4725
2015-12-30 17:36:34 +00:00
andrew
585881df34 Increase the size and alignment of the setjmp buffer. This will allow for
possible future CPU extentions with larger registers.

jmp_buf's size and alignment are baked into the ABI of third party libraries
and thus are very hard to change later so it is best to waste a small amount
of space now.

Reviewed by:	brooks
Differential Revision:	https://reviews.freebsd.org/D3956
2015-12-23 15:22:44 +00:00
andrew
567d111884 Support the variant of the interrupt-map property where the parent bus has
the #address-cells property set. For this we need to read more data before
the parent interrupt description.

this is only enabled on arm64 for now as it's not quite compliant with the
ePAPR spec. We should use a default of 2 where the #address-cells property
is missing, however this will need further testing across architectures.

Obtained from:	ABT Systems Ltd
Sponsored by:	SoftIron Inc
Differential Revision:	https://reviews.freebsd.org/D4518
2015-12-17 17:00:04 +00:00
andrew
35ebdef5c3 Reduce the numbers of levels of indirection in arm64 pcib drivers by making
the MSI & MSI-X handler functions be directly callable from the driver
methods.

Sponsored by:	ABT Systems Ltd
2015-12-10 13:19:30 +00:00
kib
80e8626b43 Add support for usermode (vdso-like) gettimeofday(2) and
clock_gettime(2) on ARMv7 and ARMv8 systems which have architectural
generic timer hardware. It is similar how the RDTSC timer is used in
userspace on x86.

Fix a permission problem where generic timer access from EL0 (or
userspace on v7) was not properly initialized on APs.

For ARMv7, mark the stack non-executable. The shared page is added for
all arms (including ARMv8 64bit), and the signal trampoline code is
moved to the page.

Reviewed by:	andrew
Discussed with:	emaste, mmel
Sponsored by:	The FreeBSD Foundation
Differential revision:	https://reviews.freebsd.org/D4209
2015-12-07 12:20:26 +00:00
andrew
faf45ee16f Rework the atomic code to reduce the repetition. This merges some of the
atomic functions where they are almost identical, or have acquire/release
semantics.

While here clean these function up. The cbnz instruction doesn't change
the condition flags so drop cc, however they should have memory added to the
clobber list.

Reviewed by:	kib
Sponsored by:	ABT Systems Ltd
Differential Revision:	https://reviews.freebsd.org/D4318
2015-12-01 12:27:36 +00:00
andrew
27d63e061c Add support for moving the DMAP range. This is needed as some AMD SoCs
place physical memory at an address outside the old DMAP range. This is an
issue as we rely on being able to move from PA -> VA using this range.

Obtained from:	Patrick Wildt <patrick@bitrig.org> (earlier version)
Sponsored by:	ABT Systems Ltd
Differential Revision:	https://reviews.freebsd.org/D3885
2015-11-24 11:01:43 +00:00
andrew
401dfe5c9d Mark functions as such. This means we call them directly rather than have
the dynamic linker copy them, but not relocate them at the new location.
This allows us to run sqlite3 without it crashing.

Sponsored by:	ABT Systems Ltd
2015-10-27 22:24:57 +00:00
andrew
afaafc8bde Use 4 levels of page tables when enabling the MMU. This will allow us to
boot on an SoC that places physical memory at an address past where three
levels of page tables can access in an identity mapping.

Submitted by:   Wojciech Macek <wma@semihalf.com>,
                Patrick Wildt <patrick@bitrig.org>
Differential Revision:	https://reviews.freebsd.org/D3885 (partial)
Differential Revision:	https://reviews.freebsd.org/D3744
2015-10-19 13:20:23 +00:00
kib
19c7e603e0 Clear exclusive monitors when handling data aborts, the monitors are
in unknown state per spec.

Reviewed by:	andrew (previous version)
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D3668
2015-09-18 16:52:18 +00:00
zbb
24d0082cdc Block secondary ITS instances from attaching on ARM64
Currently FreeBSD supports only single PIC controller. Some systems
that have more than one (like ThunderX dual-socket) fails to boot.
Disable other PICes until proper handling is implemented in the
generic interrupt code.

Reviewed by:   imp
Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3682
2015-09-16 23:59:45 +00:00
andrew
04ea917e53 Remove pmap_kenter from pmap.h, the function doesn't exist. 2015-08-20 12:07:44 +00:00
andrew
83f856f5ff Add pmap_get_tables to get the page tables for a given virtual address. This
will be used for minidump support.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2015-08-20 12:05:42 +00:00
andrew
0bfc117108 Add the arm64 minidump header. This was missed from r286953. 2015-08-20 11:26:26 +00:00
zbb
773f34d7a5 Increase MAXCPU in ARM64
Increase MAXCPU number to the maximum known value the existing
hardware can support.

Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3405
2015-08-19 11:59:41 +00:00
zbb
9bb007f409 Add SMP support to GICv3 and ITS drivers
Introduce supprot for SMP to GICv3 and ITS drivers.

Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3299
2015-08-19 10:36:36 +00:00
andrew
bb4db62b94 Add the CNTHCTL_EL2 register bits missed in r286674 2015-08-12 17:09:57 +00:00
emaste
002d9943c1 Rationalize BSD license on sys/*/include/in_cksum.h
Remove the advertising clause from the Regents of the University of
California's license, per the letter dated July 22, 1999.

Update clause numbering.
2015-08-05 19:05:12 +00:00
emaste
50ae188f8f Rationalize BSD license on sys/*/include/float.h
Remove the advertising clause from the Regents of the University of
California's license, per the letter dated July 22, 1999.

Update clause numbering.
2015-08-05 17:05:35 +00:00
andrew
d388cc06c0 Pass the pcb to store the vfp state in to vfp_save_state. This fixes a bug
in savectx where it will be used to store the current state however will
pass in a pcb when vfp_save_state expected a thread pointer.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2015-08-03 11:05:02 +00:00
andrew
5d008d66a8 Add support for uma_small_alloc and uma_small_free, and make use of these.
This is copied from the amd64 version with minor changes. These should be
merged into a single file as from a quick look there are other copies of
the same file in other parts of the tree.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2015-07-31 14:17:26 +00:00
andrew
2403c30cf5 Add more atomic_swap_* functions.
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2015-07-31 13:34:43 +00:00