Commit Graph

633 Commits

Author SHA1 Message Date
Ed Schouten
aea6d042a9 Port cloudabi32.ko to FreeBSD/arm64.
This change adds an implementation of a sysent for running CloudABI
armv6 and armv7 binaries on FreeBSD/arm64. It is a somewhat literal copy
of the armv6 version, except that it's been patched up to use the proper
registers.

Just like for cloudabi32.ko on FreeBSD/amd64, we make use of a vDSO that
automatically pads system call parameters to 64-bit value. These are
stored in a buffer on the stack, meaning we need to use copyin() and
copyout() unconditionally.
2017-11-30 17:58:48 +00:00
Pedro F. Giffuni
796df753f4 SPDX: Consider code from Carnegie-Mellon University.
Interesting cases, most likely from CMU Mach sources.
2017-11-30 15:48:35 +00:00
Andrew Turner
b4cc39e785 Fix the direction we move the instruction to retuirn to after exiting the
kernel debugger. We should skip the breakpoint instruction, not execute the
instruction before it.

Pointy-hat to:	andrew
MFC after:	1 week
Sponsored by:	DARPA, AFRL
2017-11-28 16:26:56 +00:00
Andrew Turner
be84f91cf4 When we exit the kernel debugger having entered because of a breakpoint
instruction we need to jump over the instruction. Without this we will
execute the same instruction again and enter into the debugger again.

PR:		223917
Reported by:	emaste
MFC after:	1 week
Sponsored by:	DARPA, AFRL
2017-11-28 11:04:47 +00:00
Konstantin Belousov
dde5602786 Fix index calculation for the page table pages for efirt 1:1 map.
Stop issuing pre-assigned number to enumerate all page table pages,
the assignment is incorrect.  Instead automatically calculate the next
unused index. This index in fact does not serve any purpose except to
be unique to satisfy vm_page_grab() interface, we do not look up the
page by the index later.

Reported and tested by:	emaste
Reviewed by:	andrew
Sponsored by:	The FreeBSD Foundation
MFC after:	2 weeks
PR:	223906
Differential revision:	https://reviews.freebsd.org/D13273
2017-11-28 09:34:43 +00:00
Andrew Turner
896d821e54 Set the kernel file name so the kern.bootfile sysctl is set. This allows
for the creation of kernel.old when running installkernel.

MFC after:	1 week
Sponsored by:	DARPA, AFRL
2017-11-27 16:19:00 +00:00
Ed Schouten
2b6a8dd52c Add a Saved Process Status Register bit for AArch32 execution mode.
The documentation on the Saved Process Status Register (SPSR) is a bit
weird; the M[4] bit is documented separately from M[3:0]. The M[4] bit
can be toggled to switch to 32-bit execution mode. This functionality is
orthogonal to M[3:0].

Change the definition of PSR_M_MASK to no longer include M[4]. Add a new
definition, PSR_AARCH32 that can be used to toggle 32-bit independently.
This bit will be used by the cloudabi32 code to force execution of
userspace code in 32-bit mode.

Reviewed by:	andrew
Differential Revision:	https://reviews.freebsd.org/D13148
2017-11-26 14:56:23 +00:00
Ed Schouten
2d19a20d5d Correct some more places where TO_PTR() should be used.
These were missed in r326228.

MFC after:	2 weeks
2017-11-26 14:53:56 +00:00
Ed Schouten
ee13ffbe03 Use TO_PTR() to convert integers to pointers.
For FreeBSD/arm64's cloudabi32 support, I'm going to need a TO_PTR() in
this place. Also use it for all of the other source files, so that the
difference remains as minimal as possible.

MFC after:	2 weeks
2017-11-26 14:45:56 +00:00
Ed Schouten
7af24ff710 Make 32-bit system calls end up in svc_handler().
The nice thing about ARM64 is that it's pretty elegant to install
separate trap/exception handlers for 32-bit and 64-bit processes. That
said, for all other architectures (e.g., i386 on amd64) we always let
32-bit counterparts go through the regular system call codepath. Let's
do the same on ARM64.

Reviewed by:	andrew
Differential Revision:	https://reviews.freebsd.org/D13146
2017-11-26 14:28:27 +00:00
Andrew Turner
c4501bdfac Make the arm64 pmap_invalidate functions static inline. This fixes building
with DIAGNOSTIC.

PR:		223874
Reported by:	emaste
MFC after:	1 week
2017-11-26 09:29:34 +00:00
Ed Maste
ccf802cf18 Temporarily disable VIMAGE on arm64
Loading a kernel module with a static VNET_DEFINE'd variable (e.g.
if_lagg) currently results in a kernel panic.

PR:		223670
2017-11-24 19:21:21 +00:00
Ed Schouten
09f966caf7 Set CP15BEN in SCTLR to make memory barriers work in 32-bit mode.
Binaries generated by Clang for ARMv6 may contain these instructions:

  MCR p15, 0, <Rd>, c7, c10, 5

These instructions are deprecated as of ARMv7, which is why modern
processors have a way of toggling support for them. On FreeBSD/arm64 we
currently disable support for these instructions, meaning that if 32-bit
executables with these instructions are run, they would crash with
SIGILL. This is likely not what we want.

Reviewed by:	andrew
Differential Revision:	https://reviews.freebsd.org/D13145
2017-11-24 13:51:59 +00:00
Ed Schouten
9dcf90f8ad Add rudimentary support for building FreeBSD/arm64 with COMPAT_FREEBSD32.
Right now I'm using two Raspberry Pi's (2 and 3) to test CloudABI
support for armv6, armv7 and aarch64. It would be nice if I could
restrict this to just a single instance when testing smaller changes.
This is why I'd like to get COMPAT_CLOUDABI32 to work on arm64.

As COMPAT_CLOUDABI32 depends on COMPAT_FREEBSD32, at least for the ELF
loading, this change adds all of the bits necessary to at least build a
kernel with COMPAT_FREEBSD32. All of the machine dependent system calls
are still stubbed out, for the reason that implementations for these are
only useful if actual support for running FreeBSD binaries is added.
This is outside the scope of this work.

Reviewed by:	andrew
Differential Revision:	https://reviews.freebsd.org/D13144
2017-11-24 13:50:53 +00:00
Ed Schouten
814629dd64 Don't let cpu_set_syscall_retval() clobber exec_setregs().
Upon successful completion, the execve() system call invokes
exec_setregs() to initialize the registers of the initial thread of the
newly executed process. What is weird is that when execve() returns, it
still goes through the normal system call return path, clobbering the
registers with the system call's return value (td->td_retval).

Though this doesn't seem to be problematic for x86 most of the times (as
the value of eax/rax doesn't matter upon startup), this can be pretty
frustrating for architectures where function argument and return
registers overlap (e.g., ARM). On these systems, exec_setregs() also
needs to initialize td_retval.

Even worse are architectures where cpu_set_syscall_retval() sets
registers to values not derived from td_retval. On these architectures,
there is no way cpu_set_syscall_retval() can set registers to the way it
wants them to be upon the start of execution.

To get rid of this madness, let sys_execve() return EJUSTRETURN. This
will cause cpu_set_syscall_retval() to leave registers intact. This
makes process execution easier to understand. It also eliminates the
difference between execution of the initial process and successive ones.
The initial call to sys_execve() is not performed through a system call
context.

Reviewed by:	kib, jhibbits
Differential Revision:	https://reviews.freebsd.org/D13180
2017-11-24 07:35:08 +00:00
Andrew Turner
521018d379 Ensure we check the program state set in the trap frame on arm and arm64.
This value may be set by userspace so we need to check it before using it.
If this is not done correctly on exception return the kernel may continue
in kernel mode with all registers set to a userspace controlled value. Fix
this by moving the check into set_mcontext, and also add the missing
sanitisation from the arm64 set_regs.

Discussed with:	security-officer@
MFC after:	3 days
Sponsored by:	DARPA, AFRL
2017-11-23 17:40:40 +00:00
Andrew Turner
a3dff126f9 Add a driver for the EFI RTC. This uses the EFI Runtime Services to query
the system time.

As we seem to only read this time on boot, and this is the only source of
time on many arm64 machines we need to enable this by default there. As
this is not always the case with U-Boot firmware, or when we have been
booted from a non-UEFI environment we only enable the device driver when
the Runtime Services are present and reading the time doesn't result in an
error.

PR:		212185
Reviewed by:	imp, kib
Tested by:	emaste
Relnotes:	yes
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D12650
2017-11-21 17:23:16 +00:00
Andrew Turner
2eb3e51e71 When fpcurthread is not the current thread it may be non-NULL. In this
case another thread has had the VFP unit enabled and will have its state
in the VFP registers along with it stored in memory. As such we don't need
to store the state, but do need to zero the fpcurthread pointer to stop
the VFP driver from using the enable fast path.

Reported by:	emaste
Sponsored by:	DARPA, AFRL
2017-11-21 13:19:38 +00:00
Jeff Roberson
8d6fbbb867 Replace manyinstances of VM_WAIT with blocking page allocation flags
similar to the kernel memory allocator.

This simplifies NUMA allocation because the domain will be known at wait
time and races between failure and sleeping are eliminated.  This also
reduces boilerplate code and simplifies callers.

A wait primitive is supplied for uma zones for similar reasons.  This
eliminates some non-specific VM_WAIT calls in favor of more explicit
sleeps that may be satisfied without new pages.

Reviewed by:	alc, kib, markj
Tested by:	pho
Sponsored by:	Netflix, Dell/EMC Isilon
2017-11-08 02:39:37 +00:00
Andrew Turner
f591046e7d Allocate the ITS translation table with a 64k page alignment. This is the
largest alignment the ITS can require.

This fixes a bug with the ARM Architecture Envelope Model (AEM) where it
only allows 64k pages so will fail to attach the ITS device when this table
is not sufficiently aligned.

Sponsored by:	DARPA, AFRL
2017-10-31 23:17:17 +00:00
Andrew Turner
2ed578e1db As with r325242 use mp_maxid when iterating over CPUs in the GICv3 driver.
Sponsored by:	DARPA, AFRL
2017-10-31 22:12:14 +00:00
Andrew Turner
2bfb5e812b Use mp_maxid when iterating over CPUs as we may have sparse id allocations.
Sponsored by:	DARPA, AFRL
2017-10-31 18:22:21 +00:00
Eitan Adler
a2aef24aa3 Update several more URLs
- Primarily http -> https
- Primarily FreeBSD project URLs
2017-10-29 08:17:03 +00:00
Emmanuel Vadot
7562d7ddbd loader.efi: Make framebuffer commands available for arm64
Move framebuffer.{c,h} to sys/boot/efi/loader and add the efifb
related metadata and pass it to the kernel

Reviewed by:	imp, andrew
Differential Revision:	https://reviews.freebsd.org/D12757
2017-10-25 18:55:04 +00:00
Mark Johnston
5fca1d90c1 Fix the VM_NRESERVLEVEL == 0 build.
Add VM_NRESERVLEVEL guards in the pmaps that implement transparent
superpage promotion using reservations.

Reviewed by:	alc, kib
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D12764
2017-10-23 15:34:05 +00:00
Michal Meloun
904d8c492f Add AT_HWCAP2 ELF auxiliary vector.
- allocate value for new AT_HWCAP2 auxiliary vector on all platforms.
 - expand 'struct sysentvec' by new 'u_long *sv_hwcap2', in exactly
   same way as for AT_HWCAP.

MFC after:	1 month
Reviewed by:	kib
Differential Revision:	https://reviews.freebsd.org/D12699
2017-10-21 12:05:01 +00:00
Bjoern A. Zeeb
8e94025b41 With r181803 on 2008-08-17 23:27:27Z the first VIMAGE commit went into
HEAD.  Enable VIMAGE in GENERIC kernels and some others (where GENERIC does
not exist) on HEAD.

Disable building LINT-VIMAGE with VIMAGE being default.

This should give it a lot more exposure in the run-up to 12 to help
us evaluate whether to keep it on by default or not.
We are also hoping to get better performance testing.
The feature can be disabled using nooptions.

Requested by:		many
Reviewed by:		kristof, emaste, hiren
X-MFC after:		never
Relnotes:		yes
Differential Revision:	https://reviews.freebsd.org/D12639
2017-10-20 21:40:59 +00:00
Mark Johnston
46fcd1af63 Move kernel dump offset tracking into MI code.
All of the kernel dump implementations keep track of the current offset
("dumplo") within the dump device. However, except for textdumps, they
all write the dump sequentially, so we can reduce code duplication by
having the MI code keep track of the current offset. The new
dump_append() API can be used to write at the current offset.

This is needed to implement support for kernel dump compression in the
MI kernel dump code.

Also simplify dump_encrypted_write() somewhat: use dump_write() instead
of duplicating its bounds checks, and get rid of the redundant offset
tracking.

Reviewed by:	cem
Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D11722
2017-10-18 15:38:05 +00:00
Andrew Turner
7a158e826d Support the EFI Runtime Services on arm64. As with amd64 we use the 1:1
mapping. This uses the new common code shared with amd64.

The RTC should only be accessed via EFI. There is no locking around it as
the spec only has this as a requirement for the PC-AT CMOS device.

Reviewed by:	kib, imp
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D12595
2017-10-10 13:05:26 +00:00
Andrew Turner
2e0c6823c4 Add machine/fpu.h on arm64 for compatibility with amd64.
Sponsored by:	DARPA, AFRL
2017-10-10 13:02:58 +00:00
Andrew Turner
f72a72115c Move the pmap_l0_index, etc. macros to pte.h. These will be used by the
EFI Runtime Services code.

Sponsored by:	DARPA, AFRL
2017-10-10 12:54:36 +00:00
Andrew Turner
5cf1f313c4 Add a memory barrier to ensure the atomic write is visible to the other
CPUs before waking them up.

Sponsored by:	DARPA, AFRL
2017-10-02 14:22:35 +00:00
Josh Paetzel
c77037f16f Fix indentation for r323068
PR:	220170
Reported by:	lidl
MFC after:	3 days
Pointyhat to:	jpaetzel
2017-09-19 20:40:05 +00:00
Andrew Turner
ca289945b2 Add the ARMv8.3 ID register fields. These were found in the A-Profile
exploration tools documentation:
https://developer.arm.com/products/architecture/a-profile/exploration-tools

Sponsored by:	DARPA, AFRL
2017-09-15 12:57:34 +00:00
Andrew Turner
bcf2b954c3 Add support for handling undefined instructions in userspace and the
kernel. We can register callbacks to perform the required operation on the
saved registers before returning.

This is initially used to work around a bug in old versions of QEMU that
trigger such an exception when reading from an ID register when it should
load z zero value.

I expect this could be used with other exception types, e.g. to emulate
special register access from userland.

Sponsored by:	DARPA, AFRL
2017-09-14 17:29:51 +00:00
John Baldwin
c2f37b9245 Add AT_HWCAP and AT_EHDRFLAGS on all platforms.
A new 'u_long *sv_hwcap' field is added to 'struct sysentvec'.  A
process ABI can set this field to point to a value holding a mask of
architecture-specific CPU feature flags.  If an ABI does not wish to
supply AT_HWCAP to processes the field can be left as NULL.

The support code for AT_EHDRFLAGS was already present on all systems,
just the #define was not present.  This is a step towards unifying the
AT_* constants across platforms.

Reviewed by:	kib
MFC after:	1 month
Differential Revision:	https://reviews.freebsd.org/D12290
2017-09-14 14:26:55 +00:00
Ilya Bakulin
61df30cfd4 Add MMCCAM-enabled kernel config for arm64
Approved by:	imp (mentor)
Differential Revision:	https://reviews.freebsd.org/D12114
2017-09-11 19:07:42 +00:00
Marcin Wojtas
885a74181c Expand Marvell NIC description in arm64 GENERIC config
Suggested by: emaste
2017-09-11 19:00:53 +00:00
Marcin Wojtas
7ca8a2b385 Enable compilation of Marvell NETA controller with arm64 GENERIC
This patch enables network operation on Marvell Armada 3700 SoC.

Submitted by: Patryk Duda <pdk@semihalf.com>
Obtained from: Semihalf
Sponsored by: Semihalf
Differential Revision: https://reviews.freebsd.org/D12259
2017-09-09 11:56:48 +00:00
Marcin Wojtas
e49e3ec31e Add support for uart_mvebu driver arm64 GENERIC config
This patch enables console output on Armada 3700 SoCs
with kernel GENERIC.

Submitted by: Patryk Duda <pdk@semihalf.com>
Obtained from: Semihalf
Sponsored by: Semihalf
Differential Revision: https://reviews.freebsd.org/D12251
2017-09-09 11:46:34 +00:00
Marcin Wojtas
840d633f12 Enable compilation of Marvell EHCI driver in arm64 GENERIC
Enabled driver can be used on boards equipped with Marvell Armada 3700 SoC.

Submitted by: Patryk Duda <pdk@semihalf.com>
Obtained from: Semihalf
Sponsored by: Semihalf
Differential Revision: https://reviews.freebsd.org/D12256
2017-09-09 11:16:10 +00:00
Marcin Wojtas
705f4b2ceb Enable compilation of Marvell XHCI driver in arm64 GENERIC
Enabled driver can be used on boards equipped with Marvell Armada
3700/7k/8k SoCs.

Submitted by: Patryk Duda <pdk@semihalf.com>
Obtained from: Semihalf
Sponsored by: Semihalf
Differential Revision: https://reviews.freebsd.org/D12253
2017-09-09 10:58:45 +00:00
Andrew Turner
c1649fdb02 Not all CPUs handle reading ID_AA64MMFR2_EL1 (e.g. qemu), disable it for now.
Sponsored by:	DARPA, AFRL
2017-09-08 08:02:06 +00:00
Andrew Turner
f9fc9faa3a Fix the SVE ID field shift.
Sponsored by:	DARPA, AFRL
2017-09-07 19:52:04 +00:00
Andrew Turner
130be885e6 Add the ATS1E1 case to the ID_AA64MMFR1_EL1 decoding.
Sponsored by:	DARPA, AFRL
2017-09-07 19:51:17 +00:00
Andrew Turner
d28b950a4d Use the correct mask when printing undecoded fields from the
ID_AA64MMFR2_EL1 register.

Sponsored by:	DARPA, AFRL
2017-09-07 18:58:55 +00:00
Andrew Turner
d1c2e46d9c Fix a mismerge, make sure PRINT_ID_AA64_MMFR2 has a unique value.
Sponsored by:	DARPA, AFRL
2017-09-07 16:43:12 +00:00
Andrew Turner
1a2e5c004d Fix the value of ID_AA64ISAR1_DPB_SHIFT, the field is bits 3:0.
Sponsored by:	DARPA, AFRL
2017-09-07 16:12:56 +00:00
Andrew Turner
f45dc6945b Add the ARMv8.2 ID register additions and use them to decode the register
values. As not all assemblers understand the new ID_AA64MMFR2_EL1 register
add a macro to access it. This seems to be safe for older CPUs to read this
new register, with them returning zero.

Sponsored by:	DARPA, AFRL
2017-09-07 15:45:56 +00:00
Andrew Turner
0f962c6deb Uppercase the special register names in identcpu to be more consistent with
the other source files.

Sponsored by:	DARPA, AFRL
2017-09-07 15:30:13 +00:00
Andrew Turner
a0f16159bd Make the bit mask of ARMv8 ID registers to print sparse to keep values
close, but without having to change all values when new registers are added.

Sponsored by:	DARPA, AFRL
2017-09-07 15:24:47 +00:00
Andrew Turner
5ad42f79fb Add more ARM Ltd parts to the list of knows CPUs.
Submitted by:	Jon Brawn <jon@brawn.org>
2017-09-07 15:02:57 +00:00
Marcin Wojtas
eedd5eafbe Add Marvell RTC driver to arm64 GENERIC config
Marvell Armada 80x0/70x0 SoC family uses same RTC IP as
Armada 38x. This patch adds necessary files and enable driver in
GENERIC config.

Submitted by: Rafal Kozik <rk@semihalf.com>
Obtained from: Semihalf
Sponsored by: Semihalf
Differential Revision: https://reviews.freebsd.org/D12200
2017-09-05 05:50:01 +00:00
Marcin Wojtas
d7d8ab0316 Add ARM Cortex A72 to CPU list
This change is required to properly detect CPUs
on Marvell Armada 80x0/70x0 SoC family.

Submitted by: Rafal Kozik <rk@semihalf.com>
Reviewed by: andrew, cognet (mentor)
Approved by: cognet (mentor)
Sponsored by: Semihalf
Differential Revision: https://reviews.freebsd.org/D12184
2017-09-03 08:32:33 +00:00
Josh Paetzel
9d0ec2a920 Revert r323087
This needs more thinking out and consensus, and the commit message
was wrong AND there was a typo in the commit.

pointyhat:	jpaetzel
2017-09-01 17:03:48 +00:00
Josh Paetzel
0be04b100c Take options IPSEC out of GENERIC
PR:	220170
Submitted by:	delphij
Reviewed by:	ae, glebius
MFC after:	2 weeks
Differential Revision:	D11806
2017-09-01 15:54:53 +00:00
Andrew Turner
5ea26aec6e Add support for quirks while enabling secondary CPUs. This uses the fdt
compatible string to check if the board is compatible with a given quirk.
It's possible this will be moved later, however as it's currently only used
by the MP code put it there.

So far the only instance of a quirk is when the list of CPUs may be
incorrect. This can happen on virtual machines with a hard coded
devicetree, but where the user may then set the number of CPUs as an
argument. This is the case on the ARM models so include the model specific
compat strings for these, including the spelling mistake found in some of
the OpenplatformPkg dtb files.

Sponsored by:	DARPA, AFRL
2017-08-31 20:48:05 +00:00
Josh Paetzel
3b65550eec Allow kldload tcpmd5
PR:	220170
MFC after:	2 weeks
2017-08-31 20:16:28 +00:00
Andrew Turner
dab076c004 Remove an unneeded call to pmap_invalidate_all. This was never called as
the anyvalid variable is never set.

MFC after:	1 week
Sponsored by:	DARPA, AFRL
2017-08-22 18:20:25 +00:00
Andrew Turner
012faa32f9 Fix a bug in pmap_protect where we invalidate the wrong page. With this we
can now remove an unneeded call to invalidate all entries.

MFC after:	1 week
Sponsored by:	DARPA, AFRL
2017-08-22 17:38:06 +00:00
Andrew Turner
e8e65e28ca Fix a comment on uncommitted work. 2017-08-22 13:53:53 +00:00
Andrew Turner
6683b30c03 Move the l0 pagetable address to struct mdproc. It is a property of the
whole process so should live there.

Sponsored by:	DARPA, AFRL
2017-08-22 13:16:14 +00:00
Andrew Turner
cbf2160e81 Improve the performance of the arm64 thread switching code.
The full system memory barrier around a TLB invalidation is stricter than
required. It needs to wait on accesses to main memory, with just the weaker
store variant before the invalidate. As such use the dsb istst, tlbi, dlb
ish sequence already used in pmap.

The tlbi instruction in this sequence is also unnecessarily using a
broadcast invalidate when it just needs to invalidate the local CPUs TLB.
Switch to a non-broadcast variant of this instruction.

Sponsored by:	DARPA, AFRL
2017-08-21 18:12:32 +00:00
Mark Johnston
01938d3666 Rename mkdumpheader() and group EKCD functions in kern_shutdown.c.
This helps simplify the code in kern_shutdown.c and reduces the number
of globally visible functions.

No functional change intended.

Reviewed by:	cem, def
Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D11603
2017-08-18 04:04:09 +00:00
Mark Johnston
50ef60dabe Factor out duplicated kernel dump code into dump_{start,finish}().
dump_start() and dump_finish() are responsible for writing kernel dump
headers, optionally writing the key when encryption is enabled, and
initializing the initial offset into the dump device.

Also remove the unused dump_pad(), and make some functions static now that
they're only called from kern_shutdown.c.

No functional change intended.

Reviewed by:	cem, def
Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D11584
2017-08-18 03:52:35 +00:00
Ed Maste
e7b993842e arm64: return error instead of panic in unimplemented ptrace ops
We don't need a panic as a reminder that these need to be implemented.

Reported by:	Shawn Webb
MFC after:	3 week
Sponsored by:	The FreeBSD Foundation
2017-08-17 19:16:23 +00:00
Andrew Turner
062c276886 Add support for multiple GICv3 ITS devices. For this we add sc_irq_base
and sc_irq_length to the softc to handle the base number of IRQs available,
make gicv3_get_nirqs return the number of available interrupt IDs, and
limit which CPUs we send interrupts to based on the numa domain.

The last point is only strictly needed on a dual socket ThunderX where we
are unable to send MSI/MSI-X interrupts between sockets.

Sponsored by:	DARPA, AFRL
2017-08-13 18:54:51 +00:00
John Baldwin
992029ba10 Reliably enable debug exceptions on all CPUs.
Previously, debug exceptions were only enabled on the boot CPU if
DDB was enabled in the dbg_monitor_init() function.  APs also called
this function, but since mp_machdep.c doesn't include opt_ddb.h, the
APs ended up calling an empty stub defined in <machine/debug_monitor.h>
instead of the real function.  Also, if DDB was not enabled in the kernel,
the boot CPU would not enable debug exceptions.

Fix this by adding a new dbg_init() function that always clears the OS
lock to enable debug exceptions which the boot CPU and the APs call.
This function also calls dbg_monitor_init() to enable hardware breakpoints
from DDB on all CPUs if DDB is enabled.  Eventually base support for
hardware breakpoints/watchpoints will need to move out of the DDB-only
debug_monitor.c for use by userland debuggers.

Reviewed by:	andrew
Differential Revision:	https://reviews.freebsd.org/D12001
2017-08-12 18:42:54 +00:00
John Baldwin
c9ee3caf19 Don't panic for PT_GETFPREGS.
Only fetch the VFP state from the CPU if the thread whose registers are
being requested is the current thread.  If a stopped thread's registers
are being fetched by a debugger, the saved state in the PCB is already
valid.

Reviewed by:	andrew
MFC after:	1 week
2017-08-12 18:38:18 +00:00
John Baldwin
319e8f3e36 Fix a typo. 2017-08-11 22:47:32 +00:00
Andrew Turner
1f15260790 Mark each cpu in the appropriate cpuset_domain set. This allows devices to
handle cases where they can only run on a single domain.

To allow all devices access to this set we need to move reading the domain
earlier in the boot as it was previously handled in the CPU driver, however
this is too late for the GICv3 ITS driver.

Sponsored by:	DARPA, AFRL
2017-08-05 20:57:34 +00:00
Andrew Turner
49f347f450 Start to teach the GICv3 driver about NUMA. On ThunderX we may have
multiple ITS devices, however we only want a single ITS device to be
configured on each CPU. To fix this only enable ITS when the node matches
the CPUs node.

Sponsored by:	DARPA, AFRL
2017-08-04 13:08:45 +00:00
Andrew Turner
dbba8930ce Read the numa-node-id property from each CPU node. This will initially be
used to support the dual package ThunderX where we need to send MSI/MSI-X
interrupts to the same package as the device the interrupt came from.

Sponsored by:	DARPA, AFRL
2017-08-04 10:33:22 +00:00
Emmanuel Vadot
48ee531892 arm64: Add Allwinner H5 SoC
Allwinner H5 is an H3 (arm32) with Cortex A53 cores.
Add support for it and enable it in GENERIC kernel config

Tested on: OrangePi PC2
2017-08-02 20:19:19 +00:00
Andrew Turner
ce793a529b Pass the last exception trap frame to kdb_trap. This allows show registers
in ddb to show the traps registers, and not the registers from within the
panic call.

Sponsored by:	DARPA, AFRL
2017-07-26 17:39:10 +00:00
Andrew Turner
a2d16bc541 Add support for passing FPU_KERN_NOCTX to fpu_kern_enter on arm64. This
will be used to call into UEFI from the kernel.

Sponsored by:	DARPA, AFRL
2017-07-18 16:36:32 +00:00
Emmanuel Vadot
0166fb5d59 arm64: Add Allwinner I2C controller to GENERIC kernel 2017-07-09 12:53:02 +00:00
Andrew Turner
db7b284bd6 In the arm64 pmap_remove, when removing a full superpage there is no need
to demote it to 512 pages, then remove each of these. We can just remove
the l2 map directly. This is what the intel pmaps already do.

Sponsored by:	DARPA, AFRL
2017-07-02 08:52:08 +00:00
Andrew Turner
567dec3d2b Remove all calls to cpu_dcache_wb_range from the arm64 pmap code. These
were unneeded as we tell the tlb the pagetables are in cached memory. This
gives us a small, but statistically significant improvement over just
removing the PTE_SYNC cases.

While here remove PTE_SYNC, it's now unneeded.

Sponsored by:	DARPA, AFRL
2017-07-01 09:38:52 +00:00
Jason A. Harmening
eb36b1d0bc Clean up MD pollution of bus_dma.h:
--Remove special-case handling of sparc64 bus_dmamap* functions.
  Replace with a more generic mechanism that allows MD busdma
  implementations to generate inline mapping functions by
  defining WANT_INLINE_DMAMAP in <machine/bus_dma.h>.  This
  is currently useful for sparc64, x86, and arm64, which all
  implement non-load dmamap operations as simple wrappers
  around map objects which may be bus- or device-specific.

--Remove NULL-checked bus_dmamap macros.  Implement the
  equivalent NULL checks in the inlined x86 implementation.
  For non-x86 platforms, these checks are a minor pessimization
  as those platforms do not currently allow NULL maps.  NULL
  maps were originally allowed on arm64, which appears to have
  been the motivation behind adding arm[64]-specific barriers
  to bus_dma.h, but that support was removed in r299463.

--Simplify the internal interface used by the bus_dmamap_load*
  variants and move it to bus_dma_internal.h

--Fix some drivers that directly include sys/bus_dma.h
  despite the recommendations of bus_dma(9)

Reviewed by:	kib (previous revision), marius
Differential Revision:	https://reviews.freebsd.org/D10729
2017-07-01 05:35:29 +00:00
Andrew Turner
2da3e34e0a Remove a blank line accidentally added in r320403. 2017-06-30 14:45:43 +00:00
Andrew Turner
355ffcc842 Add parentheses missed in r320388
Sponsored by:	DARPA, AFRL
2017-06-27 16:30:01 +00:00
Andrew Turner
c90baf6817 Some of the atomic_clear_* functions were incorrectly defined to be an
atomic add. Correct these, fixing a NULL-pointer dereference in netgraph.

PR:		220273
MFC after:	3 days
Sponsored by:	DARPA, AFRL
2017-06-27 10:45:13 +00:00
Andrew Turner
8e26c6b62f In _bswap16 and _bswap32 cast constant values to the appropriate type. This is
similar to what is done in the x86 code.

Sponsored by:	DARPA, AFRL
2017-06-26 22:32:52 +00:00
Andrew Turner
e899a0575b Stop calling cpu_dcache_wb_range from PTE_SYNC.
We set the shareability attributes in TCR_EL1 on boot. These tell the
hardware the pagetables are in cached memory so there is no need to flush
the entries from the cache to memory.

This has about 4.2% improvement in system time and 2.7% improvement in
user time for a buildkernel -j48 on a ThunderX.

Keep the old code for now to allow for further comparisons.
2017-06-25 13:22:49 +00:00
Konstantin Belousov
2d88da2f06 Move struct syscall_args syscall arguments parameters container into
struct thread.

For all architectures, the syscall trap handlers have to allocate the
structure on the stack.  The structure takes 88 bytes on 64bit arches
which is not negligible.  Also, it cannot be easily found by other
code, which e.g. caused duplication of some members of the structure
to struct thread already.  The change removes td_dbg_sc_code and
td_dbg_sc_nargs which were directly copied from syscall_args.

The structure is put into the copied on fork part of the struct thread
to make the syscall arguments information correct in the child after
fork.

This move will also allow several more uses shortly.

Reviewed by:	jhb (previous version)
Sponsored by:	The FreeBSD Foundation
MFC after:	3 weeks
X-Differential revision:	https://reviews.freebsd.org/D11080
2017-06-12 21:03:23 +00:00
Konstantin Belousov
43f41dd393 Make struct syscall_args visible to userspace compilation environment
from machine/proc.h, consistently on all architectures.

Reviewed by:	jhb
Sponsored by:	The FreeBSD Foundation
MFC after:	3 weeks
X-Differential revision:	https://reviews.freebsd.org/D11080
2017-06-12 20:53:44 +00:00
Andrew Turner
52453d22f2 Allow the arm64 machine/vfp.h to be included without first including
machine/pcb.h. It he latter is only needed for struct pcb.
2017-06-09 15:47:14 +00:00
Andrew Turner
9a19869a5f Store the read-only thread pointer when scheduling a new thread. This is
not currently set, however we may wish to set it later.
2017-06-09 15:37:17 +00:00
Olivier Houchard
5bb27fe15c - Don't bother flushing the data cache for pages we're about to unmap, there's
no need to.
- Remove pmap_is_current(), pmap_[pte|l3]_valid_cacheable as there were only
used to know if we had to write back pages.
- In pmap_remove_pages(), don't bother invalidating each page in the TLB,
we're about to flush the whole TLB anyway.

This makes make world 8-9% faster on my hardware.

Reviewed by:	andrew
2017-06-02 14:17:14 +00:00
Konstantin Belousov
55b78354f6 Add COMPAT_FREEBSD11 on arm64, the arch is almost tier-1.
Discussed with:	andrew, emaste
Sponsored by:	The FreeBSD Foundation
2017-05-23 13:57:55 +00:00
Hans Petter Selasky
65b017b420 Avoid use of contiguous memory allocations in busdma when possible.
This patch improves the boundary checks in busdma to allow more cases
using the regular page based kernel memory allocator. Especially in
the case of having a non-zero boundary in the parent DMA tag. For
example AMD64 based platforms set the PCI DMA tag boundary to
PCI_DMA_BOUNDARY, 4GB, which before this patch caused contiguous
memory allocations to be preferred when allocating more than PAGE_SIZE
bytes. Even if the required alignment was less than PAGE_SIZE bytes.

This patch also fixes the nsegments check for using kmem_alloc_attr()
when the maximum segment size is less than PAGE_SIZE bytes.

Updated some comments describing the code in question.

Differential Revision:	https://reviews.freebsd.org/D10645
Reviewed by:		kib, jhb, gallatin, scottl
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-05-16 14:21:37 +00:00
Andrew Turner
2438ef7673 Allocate a cacheline when reading or writing to write through memory. The
hardware will still write to memory, however following reads will be from
the cache.

MFC after:	1 week
Sponsored by:	DARPA, AFRL
2017-05-13 13:03:20 +00:00
Andrew Turner
c031b4b01e Add the VM_MEMATTR_WRITE_THROUGH memory type to arm64 and use it to support
VM_MEMATTR_WRITE_COMBINING in the kernel. This fixes a bug where Xorg would
use write back cached memory for its graphics buffers. This would produce
artifacts on the screen as cachelines were written to memory.

MFC after:	1 week
Sponsored by:	DARPA, AFRL
2017-05-13 13:01:15 +00:00
Andrew Turner
bd2b26ff0a Add reclaim_pv_chunk on arm64. This is based on the amd64 code so should
operate similarly, other than not needing the delayed invalidation.

It has been tested with artificial injection of vm_page_alloc failures
while running 'sort /dev/zero'.

Reviewed by:	alc, kib
MFC after:	1 week
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D10574
2017-05-05 10:01:27 +00:00
Andrew Turner
8a2b28d7ba Print all virtual addresses in the show vtop ddb command. The results may
be different with PAN enabled.

MFC after:	1 week
Sponsored by:	DARPA, AFRL
2017-05-03 12:24:31 +00:00
Andrew Turner
574a7c6c47 Call the PSCI reset from cpu_reset on arm64. When rebooting from DDB the
kernel calls this directly so the event handler is not called, meaning
the computer fails to reboot.

Tested by:	cognet
MFC after:	1 week
Sponsored by:	DARPA, AFRL
2017-04-24 11:06:10 +00:00
Andrew Turner
52a680fb22 Restrict the arm64 supervisor all instructions to only allow a zero
immediate value for system calls. We may wish to use other values in the
future for other purposes.

MFC after:	1 week
Sponsored by:	DARPA, AFRL
2017-04-20 15:53:20 +00:00
Andrew Turner
d7635c7a49 Push loading curthread into assembly in the synchronous exception handlers.
This will help investigating the performance impact of moving parts of the
switch statement in do_el0_sync into assembly.

Sponsored by:	DARPA, AFRL
2017-04-20 13:56:30 +00:00
Ed Maste
f0e56c1f62 Remove trailing whitespace from r317061 2017-04-17 18:57:26 +00:00