Commit Graph

107 Commits

Author SHA1 Message Date
andrew
26ea906520 Increase CACHE_LINE_SHIFT to 7 as cache lines are 128 bytes on ThunderX.
MFC after:	1 week
Sponsored by:	ABT Systems Ltd
2016-10-24 13:44:24 +00:00
andrew
75768c2bf4 Add support for the fpu_kern(9) KPI on arm64. It hooks into the existing
VFP code to store the old context, with lazy loading of the new context
when needed.

FPU_KERN_NOCTX is missing as this is unused in the crypto code this has
been tested with, and I am unsure on the requirements of the UEFI
Runtime Services.

Reviewed by:	kib
Obtained from:	ABT Systeems Ltd
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D8276
2016-10-20 09:22:10 +00:00
andrew
bd90949460 Add PCB_FP_USERMASK so we can mask off floating point flags that should
not be sent to userspace, for example the future flag to tell when we are
using floating point in the kernel.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-10-18 13:39:55 +00:00
andrew
4a327f5e89 Fix the build, struct vfpstate needs to be visible to userspace as it's
part of struct pcb which is in a header used in libutil.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-10-15 16:29:06 +00:00
andrew
0058087fc0 Create macros for the MAIR memory attributes. While here add an uncached
memory type, however the VM code still needs to be taught about this.

MFC after:	1 week
Sponsored by:	ABT Systems Ltd
2016-10-14 22:20:14 +00:00
andrew
4d2f7af1cf Rework how we store the VFP registers in the pcb. This will be used when
creating a floating-point context within the kernel without having to move
the stored values in memory.

Sponsored by:	The FreeBSD Foundation
2016-10-14 15:53:48 +00:00
imp
081e8d8587 Fix building on i386 and arm. But 'public domain' headers on the files
with no creative content. Include "lost" changes from git:
o Use /dev/efi instead of /dev/efidev
o Remove redundant NULL checks.

Submitted by: kib@, dim@, zbb@, emaste@
2016-10-13 06:56:23 +00:00
kib
0b0178b3a6 Add a way for the architecture to specify the calling ABI for methods
in the EFI Runtime Services Table.  On amd64, the calling conventions
are MS.

Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
2016-09-21 10:35:44 +00:00
andrew
4a70bd772c Fix the arm64 kernel build when DDB is disabled, debug_monitor.c depends on
DDB, and is unused when it's disabled.

Obtained from:	ABT Systems Ltd
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2016-09-13 15:45:22 +00:00
andrew
d31d501396 Add a pc_clock pcpu field and use it to implement cpu_est_clockrate. This
will allow drivers that manage the clock frequency to communicate this with
the reset of the kernel.

Reported by:	jmcneill
MFC after:	1 week
Sponsored by:	ABT Systems Ltd
2016-09-02 10:13:51 +00:00
andrew
83c9cebd9a Teach the parts of the arm64 pmap that need to iterate over pages to also
iterate over superpages. We don't yet create these, but soon will.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-24 12:32:18 +00:00
kib
e56264ca17 Implement userspace gettimeofday(2) with HPET timecounter.
Right now, userspace (fast) gettimeofday(2) on x86 only works for
RDTSC.  For older machines, like Core2, where RDTSC is not C2/C3
invariant, and which fall to HPET hardware, this means that the call
has both the penalty of the syscall and of the uncached hw behind the
QPI or PCIe connection to the sought bridge.  Nothing can me done
against the access latency, but the syscall overhead can be removed.
System already provides mappable /dev/hpetX devices, which gives
straight access to the HPET registers page.

Add yet another algorithm to the x86 'vdso' timehands. Libc is updated
to handle both RDTSC and HPET.  For HPET, the index of the hpet device
to mmap is passed from kernel to userspace, index might be changed and
libc invalidates its mapping as needed.

Remove cpu_fill_vdso_timehands() KPI, instead require that
timecounters which can be used from userspace, to provide
tc_fill_vdso_timehands{,32}() methods.  Merge i386 and amd64
libc/<arch>/sys/__vdso_gettc.c into one source file in the new
libc/x86/sys location.  __vdso_gettc() internal interface is changed
to move timecounter algorithm detection into the MD code.

Measurements show that RDTSC even with the syscall overhead is faster
than userspace HPET access.  But still, userspace HPET is three-four
times faster than syscall HPET on several Core2 and SandyBridge
machines.

Tested by:	Howard Su <howard0su@gmail.com>
Sponsored by:	The FreeBSD Foundation
MFC after:	1 month
Differential revision:	https://reviews.freebsd.org/D7473
2016-08-17 09:52:09 +00:00
andrew
3987d44b33 Add the ARMv8.1 identification registers to the list we print when booting.
MFC after:	1 week
Sponsored by:	ABT Systems Ltd
2016-08-15 09:23:08 +00:00
andrew
3de878798d Implement promotions and demotions in the arm64 pmap code. For now we don't
promote memory as I am not sure all the demotion cases are handled, however
it is useful to implement pmap_page_set_memattr. This is used, for example,
when mapping uncached memory for bus_dma(9).

pmap_page_set_memattr needs to demote the DMAP region as on ARM we need to
ensure all mappings to the same physical address have the same attributes.

Reviewed by:	kib
Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D6987
2016-08-12 10:29:34 +00:00
andrew
4ad2fb1d62 Add a kernel variable to let the user to select their preferred order
between ACPI and FDT. This will be needed on machines with both, e.g. the
SoftIron Overdrive 3000. The kernel will accept one or more comma separated
values of either 'acpi' or 'fdt'. Any other values are skipped.

To set it the user can either set it on the loader command line, or
in loader.conf e.g. in loader.conf:
kern.cfg.order=acpi,fdt

This will try using ACPI then FDT. If none of the selected options work the
kernel tries to use one to get the serial console, then panics.

Reviewed by:	emaste (earlier version)
Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D7274
2016-08-01 12:17:44 +00:00
andrew
6d69f3b524 Add the fields for the PAR_EL1 register. This is used when performing an
address lookup with the AT instructions.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-01 10:36:58 +00:00
andrew
6b8643488b Add the Data Fault Status Code values to the ESR_ELx registers for when the
fault code is a Data Abort.

Obtained from:	AT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-07-31 18:58:20 +00:00
andrew
ac30983101 Finish removing the non-INTRNG support from sys/arm64.
Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-07-14 17:31:29 +00:00
andrew
8c24a12208 Add memmmap on arm64 so we can mmap /dev/mem and /dev/kmem.
Obtained from:	ABT Systems Ltd
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2016-07-13 23:03:34 +00:00
ache
23a48963e2 Undo r302601, WCHAR_MAX may not be a valid wchar value. 2016-07-12 04:20:44 +00:00
ache
97629b93ec I don't know why unsigned int is choosed for wchar_t here, but WCHAR_MAX
should be <= WINT_MAX. It is bigger, __UINT_MAX > INT32_MAX
2016-07-12 00:37:48 +00:00
wma
2da75cc60b ARM64: fix DMAP calculation
Use arithmetic operators instead of logical. This fixes
    DMAP ranges calculation for ThunderX Dual Socket.

    Obtained from:         Semihalf
    Sponsored by:          Cavium
    Reviewed by:           zbb
    Differential Revision: https://reviews.freebsd.org/D7023
    Approved by:           re (gjb)
2016-06-30 04:58:19 +00:00
andrew
a00165d816 Add the GICv3 ITS intrng driver. As the interface to the interrupt
framework has significantly changed the driver has moved to a new file.
While it shares some code with the existing driver this has been modified
to work better with the intrng framework.

This has been tested on the ThunderX servers in the netperf cluster and has
been used to boot them for other testing, including DTrace and hwpmc.

With this we can use intrng on all supported arm64 platforms I was able to
test on. It is expected we will move to intrng soon, and disable the old
arm64 interrupt framework.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D6437
2016-06-03 10:28:06 +00:00
andrew
659d635514 Define PCI_RES_BUS for NEW_PCIB
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-19 14:00:18 +00:00
andrew
58f81b7be8 Add support for intrng to arm64. As the GICv3 drivers will need to be
updated, and until further testing can be done, this is disabled for now.

It is expected arm64 will switch to this interface, and the old interface
will be removed before 11.0 is released.

Obtained from:	ABT Systems Ltd
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
2016-05-16 10:48:51 +00:00
andrew
297d2b0fcc Add support to the arm64 busdma to handle the cache. For now this is
disabled, however when we enable it it will default to assume memory is
not cache-coherent, unless either the tag was created or the parent was
marked as cache-coherent.

Obtained from:	ABT Systems Ltd
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
2016-05-13 16:03:50 +00:00
andrew
8c3616af7a Call busdma_swi from swi_vm as is done from other architectures.
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-05-11 18:48:47 +00:00
zbb
bc6d6fd49a Fix I/O coherence issues on ThunderX when SMP is disabled
To maintain coherence between cache and DMA memory appropriate
shareability flags need to be set in the PTE regardless of SMP
option.

Reviewed by:	wma
Obtained from:	Semihalf
Sponsored by:	Cavium
Differential Revision: https://reviews.freebsd.org/D6231
2016-05-11 13:23:56 +00:00
br
778cc5a811 Move arm's devmap to some generic place, so it can be used
by other architectures.

Reviewed by:	imp
Differential Revision:	https://reviews.freebsd.org/D6091
Sponsored by:	DARPA, AFRL
Sponsored by:	HEIF5
2016-04-26 11:53:37 +00:00
andrew
b40ffdceea Use the yield instruction in the arm64 cpu_spinwait. This instruction is
a hint to the hardware the software is not performing a task.

Sponsored by:	ABT Systems Ltd
2016-04-25 17:32:08 +00:00
andrew
6d3c93b78f Set the upper limit of the DMAP region to the limit of RAM as was found in
the physmap. This will reduce the likelihood of an issue where we have
device memory mapped in the DMAP. This can only happen if it is within the
same 1G block of normal memory.

Reviewed by:	kib
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D5938
2016-04-14 10:43:28 +00:00
andrew
fcf07e4e7b Document the memory ranges within the kernel region to help with debugging
to track down which region an address is from.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-04-13 11:43:03 +00:00
andrew
66ef37d31b Increase the arm64 kernel address space to 512GB, and the DMAP region to
2TB. The latter can be increased in 512GB chunks by adjusting the lower
address, however more work will be needed to increase the former.

There is still some work needed to only create a DMAP region for the RAM
address space as on ARM architectures all mappings should have the same
memory attributes, and these will be different for device and normal memory.

Reviewed by:	kib
Obtained from:	ABT Systems Ltd
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D5859
2016-04-13 09:44:32 +00:00
andrew
d532565c6d Allow vmparam.h to be included from assembly files on arm64.
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-04-06 14:08:10 +00:00
wma
f5a4347e1c Implement dtrace_getupcstack in ARM64
Allow using DTRACE for performance analysis of userspace
applications - the function call stack can be captured.
This is almost an exact copy of AMD64 solution.

Obtained from:         Semihalf
Sponsored by:          Cavium
Reviewed by:           emaste, gnn, jhibbits
Differential Revision: https://reviews.freebsd.org/D5779
2016-04-06 05:13:36 +00:00
wma
153a55cf7f arm64: pagezero improvement
This change has been provided to improve pagezero call performance.

Submitted by:          Dominik Ermel <der@semihalf.com>
Obtained from:         Semihalf
Sponsored by:          Cavium
Reviewed by:           kib
Differential Revision: https://reviews.freebsd.org/D5741
2016-04-04 07:16:43 +00:00
wma
4ba80483c0 arm64: bzero optimization
This optimization attempts to utylize as wide as possible register store instructions to zero large buffers.
The implementation, if possible, will use 'dc zva' to zero buffer by cache lines.

Speedup: 60x faster memory zeroing

Submitted by:          Dominik Ermel <der@semihalf.com>
Obtained from:         Semihalf
Sponsored by:          Cavium
Reviewed by:           kib
Differential Revision: https://reviews.freebsd.org/D5726
2016-04-04 07:06:20 +00:00
andrew
b45ea0fe80 Add support for 4 level pagetables. The userland address space has been
increased to 256TiB. The kernel address space can also be increased to be
the same size, but this will be performed in a later change.

To help work with an extra level of page tables two new functions have
been added, one to file the lowest level table entry, and one to find the
block/page level. Both of these find the entry for a given pmap and virtual
address.

This has been tested with a combination of buildworld, stress2 tests, and
by using sort to consume a large amount of memory by sorting /dev/zero. No
new issues are known to be present from this change.

Reviewed by:	kib
Obtained from:	ABT Systems Ltd
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D5720
2016-03-31 11:07:24 +00:00
andrew
acaab8fb10 Use the saved program state register to detect when an exception frame is
from userpsace. Previously we could have triggered a panic by trying to
jump to a kernel address from userland as the trap handling code thought we
received an ast in kernel mode.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-03-22 08:36:25 +00:00
andrew
d5c75c2b10 Rename COUNT_IPI to INTR_IPI_COUNT to reduce the diff with intrng.
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-03-18 16:29:58 +00:00
andrew
162f59a4a0 Reduce the diff with intrng by renaming similar functions. This is a noop,
but will help move to use the common interrupt handling code later.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-03-18 16:18:29 +00:00
andrew
df8c050ab1 Remove the invalid L0_BLOCK definition. ARMv8 doesn't support block
translation in the level 0 descriptor.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-03-18 10:01:25 +00:00
bz
983c664612 Force re-routing PCI interrupts (this is for legacy INTx not MSI).
Need this for gem5, but was not needed on real hadrware (yet) as it
was always MSI.

Reviewed by:		andrew, jhb
Discovered by:		andrew
Sponsored by:		DARPA/AFRL
Differential Revision:	https://reviews.freebsd.org/D5494
2016-03-02 15:20:42 +00:00
wma
ad7622cfab Enable SRE_EL2 on ARM64
Enable system register access for EL2. Alpine-V2 is
the first device requiring this to be enabled.
It is also in-sync with Linux initialization code,
and compatible with Alpine-V2 uboot requirements.

Obtained from:         Semihalf
Submitted by:          Michal Stanek <mst@semihalf.com>
Sponsored by:          Annapurna Labs
Approved by:           cognet (mentor)
Reviewed by:           wma
Differential revision: https://reviews.freebsd.org/D5394
2016-03-01 08:15:00 +00:00
skra
19710a8b47 Remove pd_prot and pd_cache members from struct arm_devmap_entry.
The struct is used for definition of static device mappings which
should always have same protection and attributes.
2016-02-17 12:36:24 +00:00
zbb
95137856d1 Introduce bus_bind_intr method for ARM64
It can be used to bind specific interrupt to a particular CPU.
Requires PIC support for interrupts binding.

Reviewed by:   wma
Obtained from: Semihalf
Sponsored by:  Cavium
Differential Revision: https://reviews.freebsd.org/D5122
2016-02-11 11:58:27 +00:00
zbb
95c8add1f2 Minor clean-ups for ARM64 GICv3 and GIC drivers
GICv3:
- move ICC_SGI1R_EL1 definitions to armreg.h and use proper system
  register's names
GIC:
- remove unused functions

Reviewed by:   andrew
Obtained from: Semihalf
Sponsored by:  Cavium
Differential Revision: https://reviews.freebsd.org/D5119
2016-02-11 11:55:37 +00:00
andrew
a718ec0580 Implement kdb_cpu_sync_icache on arm64.
Sponsored by:	ABT Systems Ltd
2016-02-05 15:38:28 +00:00
andrew
5b58d7a3f8 Increase the size of PHYS_AVAIL_SIZE to allow firmware to provide a large
number of physical memory locations we can access. This is the case on
some HiKey boards that may have UEFI reserved memory dispersed through the
physical space.

Sponsored by:	ABT Systems Ltd
2016-02-02 17:59:43 +00:00
andrew
f0c954c605 Implement single stepping on arm64. We need to set the single step bits in
the processor and debug state registers. A flag has been added to the pcb
to tell us when to enable single stepping for a given thread.

Reviewed by:	kib
Sponsored by:	ABT Systems Ltd
Differential Revision:	https://reviews.freebsd.org/D4730
2016-02-02 10:28:56 +00:00