significant source of cache line contention from vm_page_alloc(). Use
accessors and vm_page_unwire_noq() so that the mechanism can be easily
changed in the future.
Reviewed by: markj
Discussed with: kib, glebius
Tested by: pho (earlier version)
Sponsored by: Netflix, Dell/EMC Isilon
Differential Revision: https://reviews.freebsd.org/D14273
- pmap_enter_object() can be used for mapping of executable pages, so it's
necessary to handle I-cache synchronization within it.
- Fix race in I-cache synchronization in pmap_enter(). The current code firstly
maps given page to target VA and then do I-cache sync on it. This causes
race, because this mapping become visible to other threads, before I-cache
is synced.
Do sync I-cache firstly (by using DMAP VA) and then map it to target VA.
- ARM64 ARM permits implementation of aliased (AIVIVT, VIPT) I-cache, but we
can use different that final VA for flushing it. So we should use full
I-cache flush on affected platforms. For now, and as temporary solution,
use full flush always.
- special fault handling for break-before-make mechanism should be also
applied for instruction translation faults, not only for data translation
faults.
- since arm64_address_translate_...() functions are not atomic,
use these with disabled interrupts.
Uses of mallocarray(9).
The use of mallocarray(9) has rocketed the required swap to build FreeBSD.
This is likely caused by the allocation size attributes which put extra pressure
on the compiler.
Given that most of these checks are superfluous we have to choose better
where to use mallocarray(9). We still have more uses of mallocarray(9) but
hopefully this is enough to bring swap usage to a reasonable level.
Reported by: wosch
PR: 225197
kernel by PHYS_TO_DMAP() as previously present on amd64, arm64, riscv, and
powerpc64. This introduces a new MI macro (PMAP_HAS_DMAP) that can be
evaluated at runtime to determine if the architecture has a direct map;
if it does not (or does) unconditionally and PMAP_HAS_DMAP is either 0 or
1, the compiler can remove the conditional logic.
As part of this, implement PHYS_TO_DMAP() on sparc64 and mips64, which had
similar things but spelled differently. 32-bit MIPS has a partial direct-map
that maps poorly to this concept and is unchanged.
Reviewed by: kib
Suggestions from: marius, alc, kib
Runtime tested on: amd64, powerpc64, powerpc, mips64
This fixes operation on Qualcomm Snapdragon and some other platforms.
During boot time on subsystems initialization we have some amount of
kernel threads created, then scheduler gives CPU time to each thread.
Eventually scheduler returns CPU execution back to thread 0. In this
case writing zero to ttbr0 in cpu_switch leads Qualcomm board to
reboot (asynchronously, CPU continues execution).
Similar to other kernel threads install a valid physical address
(kernel pmap) to user page table base register ttbr0.
Reviewed by: andrew
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D13536
When demoting DMAP pages curthread may be pointing to data within the
page we are demoting. Create a new invalidate that doesn't pin and use
it in the demote case.
As the demote has both interrupts disabled, and is within a critical section
this is safe from having the scheduler from switching to another CPU.
Reported by: loos
Reviewed by: loos
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D13955
Focus on code where we are doing multiplications within malloc(9). None of
these ire likely to overflow, however the change is still useful as some
static checkers can benefit from the allocation attributes we use for
mallocarray.
This initial sweep only covers malloc(9) calls with M_NOWAIT. No good
reason but I started doing the changes before r327796 and at that time it
was convenient to make sure the sorrounding code could handle NULL values.
X-Differential revision: https://reviews.freebsd.org/D13837
allocated with a tag to come from the specified domain if it meets the
other constraints provided by the tag. Automatically create a tag at
the root of each bus specifying the domain local to that bus if
available.
Reviewed by: jhb, kib
Tested by: pho
Sponsored by: Netflix, Dell/EMC Isilon
Differential Revision: https://reviews.freebsd.org/D13545
domains can be done by the _domain() API variants. UMA also supports a
first-touch policy via the NUMA zone flag.
The slab layer is now segregated by VM domains and is precise. It handles
iteration for round-robin directly. The per-cpu cache layer remains
a mix of domains according to where memory is allocated and freed. Well
behaved clients can achieve perfect locality with no performance penalty.
The direct domain allocation functions have to visit the slab layer and
so require per-zone locks which come at some expense.
Reviewed by: Attilio (a slightly older version)
Tested by: pho
Sponsored by: Netflix, Dell/EMC Isilon
We need to handle two cases:
1. One process attacking another process.
2. A process attacking the kernel.
For the first case we clear the branch predictor state on context switch
between different processes. For the second we do this when taking an
instruction abort on a non-userspace address.
To clear the branch predictor state a per-CPU function pointer has been
added. This is set by the new cpu errata code based on if the CPU is
known to be affected.
On Cortex-A57, A72, A73, and A75 we call into the PSCI firmware as newer
versions of this will clear the branch predictor state for us.
It has been reported the ThunderX is unaffected, however the ThunderX2 is
vulnerable. The Qualcomm Falkor core is also affected. As FreeBSD doesn't
yet run on the ThunderX2 or Falkor no workaround is included for these CPUs.
MFC after: 3 days
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D13812
For each we need to walk the MADT to find which we have, then add the
driver as needed. As each may have a child they will each walk the same
table to find these details.
Reviewed by: mmel
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D8720
This adds a new acpi_bus interface with a map_intr method. This is similar
to the Open Firmware map_intr method and allows us to create the needed
mapping from ACPI space to INTRNG space.
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D8617
a mask and value to compare with the Main ID Register. If these match then a
function is called to handle the installation of the erratum workaround.
No errata are currently handled, however this will change soon in a future
commit.
MFC after: 1 week
Sponsored by: DARPA, AFRL
The emac bindings that are landing in Linux 4.15 specify a syscon property
on the emac node that point to /soc/syscon. Use this property if it's
specified, but maintain backwards compatibility with the old method.
The older method is still used for boards that we get .dtb from u-boot, such
as pine64, that did not yet have stable emac bindings.
Tested on: Banana Pi-M3 (a83t)
Tested on: Pine64 (a64)
Reviewed by: manu
Differential Revision: https://reviews.freebsd.org/D13296
platform divergence.
Only architectures which pass arguments in registers (mips)
and platforms which use really weird compilers (any?) would
need to augment the contents of <sys/_stdarg.h>
Convert x86, arm and arm64 architectures to use <sys/_stdarg.h>
This reduces noise when kernel is compiled by newer GCC versions,
such as one used by external toolchain ports.
Reviewed by: kib, andrew(sys/arm and sys/arm64), emaste(partial), erj(partial)
Reviewed by: jhb (sys/dev/pci/* sys/kern/vfs_aio.c and sys/kern/kern_synch.c)
Differential Revision: https://reviews.freebsd.org/D10385
They provide relaxed-ordered atomic access semantic. Due to the
FreeBSD memory model, the operations are syntaxical wrappers around
the volatile accesses. The volatile qualifier is used to ensure that
the access not optimized out and in turn depends on the volatile
semantic as implemented by supported compilers.
The motivation for adding the operation is to help people coming from
other systems or knowing the C11/C++ standards where atomics have
special type and require use of the special access operations. It is
still the case that FreeBSD requires plain load and stores of aligned
integer types to be atomic.
Suggested by: jhb
Reviewed by: alc, jhb
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D13534
This variable should be pure MI except possibly for reading it in MD
dump routines. Its initialization was pure MD in 4.4BSD, but FreeBSD
changed this in r36441 in 1998. There were many imperfections in
r36441. This commit fixes only a small one, to simplify fixing the
others 1 arch at a time. (r47678 added support for
special/early/multiple message buffer initialization which I want in
a more general form, but this was too fragile to use because hacking
on the msgbufp global corrupted it, and was only used for 5 hours in
-current...)
arm64 boot sequence. This will be a virtual address in the kernel space
after the kernel and any modules loaded by loader so we can use this to
find the size of the kernel + modules. We can then add on a level 2 page for
the module data and round up the size to be aligned to a level 2 page.
This allows more than 8 MiB of modules to be loaded by loader, e.g. zfs.ko
and opensolaris.ko.
Reported by: Shawn Webb
MFC after: 1 week
Sponsored by: DARPA, AFRL
The cloudabi32.ko kernel modules can only be loaded on AMD64 and ARM64
by kernels built with COMPAT_FREEBSD32. Even though COMPAT_FREEBSD32
does not support the execution of native FreeBSD executables, do add it
to GENERIC, to make cloudabi32.ko usable.
According to size(1), this makes the kernel image approximately 0.7%
larger.
Reviewed by: andrew, imp, emaste
Differential Revision: https://reviews.freebsd.org/D13311
This change adds an implementation of a sysent for running CloudABI
armv6 and armv7 binaries on FreeBSD/arm64. It is a somewhat literal copy
of the armv6 version, except that it's been patched up to use the proper
registers.
Just like for cloudabi32.ko on FreeBSD/amd64, we make use of a vDSO that
automatically pads system call parameters to 64-bit value. These are
stored in a buffer on the stack, meaning we need to use copyin() and
copyout() unconditionally.
kernel debugger. We should skip the breakpoint instruction, not execute the
instruction before it.
Pointy-hat to: andrew
MFC after: 1 week
Sponsored by: DARPA, AFRL
instruction we need to jump over the instruction. Without this we will
execute the same instruction again and enter into the debugger again.
PR: 223917
Reported by: emaste
MFC after: 1 week
Sponsored by: DARPA, AFRL
Stop issuing pre-assigned number to enumerate all page table pages,
the assignment is incorrect. Instead automatically calculate the next
unused index. This index in fact does not serve any purpose except to
be unique to satisfy vm_page_grab() interface, we do not look up the
page by the index later.
Reported and tested by: emaste
Reviewed by: andrew
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
PR: 223906
Differential revision: https://reviews.freebsd.org/D13273
The documentation on the Saved Process Status Register (SPSR) is a bit
weird; the M[4] bit is documented separately from M[3:0]. The M[4] bit
can be toggled to switch to 32-bit execution mode. This functionality is
orthogonal to M[3:0].
Change the definition of PSR_M_MASK to no longer include M[4]. Add a new
definition, PSR_AARCH32 that can be used to toggle 32-bit independently.
This bit will be used by the cloudabi32 code to force execution of
userspace code in 32-bit mode.
Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D13148
For FreeBSD/arm64's cloudabi32 support, I'm going to need a TO_PTR() in
this place. Also use it for all of the other source files, so that the
difference remains as minimal as possible.
MFC after: 2 weeks
The nice thing about ARM64 is that it's pretty elegant to install
separate trap/exception handlers for 32-bit and 64-bit processes. That
said, for all other architectures (e.g., i386 on amd64) we always let
32-bit counterparts go through the regular system call codepath. Let's
do the same on ARM64.
Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D13146
Binaries generated by Clang for ARMv6 may contain these instructions:
MCR p15, 0, <Rd>, c7, c10, 5
These instructions are deprecated as of ARMv7, which is why modern
processors have a way of toggling support for them. On FreeBSD/arm64 we
currently disable support for these instructions, meaning that if 32-bit
executables with these instructions are run, they would crash with
SIGILL. This is likely not what we want.
Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D13145
Right now I'm using two Raspberry Pi's (2 and 3) to test CloudABI
support for armv6, armv7 and aarch64. It would be nice if I could
restrict this to just a single instance when testing smaller changes.
This is why I'd like to get COMPAT_CLOUDABI32 to work on arm64.
As COMPAT_CLOUDABI32 depends on COMPAT_FREEBSD32, at least for the ELF
loading, this change adds all of the bits necessary to at least build a
kernel with COMPAT_FREEBSD32. All of the machine dependent system calls
are still stubbed out, for the reason that implementations for these are
only useful if actual support for running FreeBSD binaries is added.
This is outside the scope of this work.
Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D13144
Upon successful completion, the execve() system call invokes
exec_setregs() to initialize the registers of the initial thread of the
newly executed process. What is weird is that when execve() returns, it
still goes through the normal system call return path, clobbering the
registers with the system call's return value (td->td_retval).
Though this doesn't seem to be problematic for x86 most of the times (as
the value of eax/rax doesn't matter upon startup), this can be pretty
frustrating for architectures where function argument and return
registers overlap (e.g., ARM). On these systems, exec_setregs() also
needs to initialize td_retval.
Even worse are architectures where cpu_set_syscall_retval() sets
registers to values not derived from td_retval. On these architectures,
there is no way cpu_set_syscall_retval() can set registers to the way it
wants them to be upon the start of execution.
To get rid of this madness, let sys_execve() return EJUSTRETURN. This
will cause cpu_set_syscall_retval() to leave registers intact. This
makes process execution easier to understand. It also eliminates the
difference between execution of the initial process and successive ones.
The initial call to sys_execve() is not performed through a system call
context.
Reviewed by: kib, jhibbits
Differential Revision: https://reviews.freebsd.org/D13180
This value may be set by userspace so we need to check it before using it.
If this is not done correctly on exception return the kernel may continue
in kernel mode with all registers set to a userspace controlled value. Fix
this by moving the check into set_mcontext, and also add the missing
sanitisation from the arm64 set_regs.
Discussed with: security-officer@
MFC after: 3 days
Sponsored by: DARPA, AFRL
the system time.
As we seem to only read this time on boot, and this is the only source of
time on many arm64 machines we need to enable this by default there. As
this is not always the case with U-Boot firmware, or when we have been
booted from a non-UEFI environment we only enable the device driver when
the Runtime Services are present and reading the time doesn't result in an
error.
PR: 212185
Reviewed by: imp, kib
Tested by: emaste
Relnotes: yes
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D12650
case another thread has had the VFP unit enabled and will have its state
in the VFP registers along with it stored in memory. As such we don't need
to store the state, but do need to zero the fpcurthread pointer to stop
the VFP driver from using the enable fast path.
Reported by: emaste
Sponsored by: DARPA, AFRL
similar to the kernel memory allocator.
This simplifies NUMA allocation because the domain will be known at wait
time and races between failure and sleeping are eliminated. This also
reduces boilerplate code and simplifies callers.
A wait primitive is supplied for uma zones for similar reasons. This
eliminates some non-specific VM_WAIT calls in favor of more explicit
sleeps that may be satisfied without new pages.
Reviewed by: alc, kib, markj
Tested by: pho
Sponsored by: Netflix, Dell/EMC Isilon
largest alignment the ITS can require.
This fixes a bug with the ARM Architecture Envelope Model (AEM) where it
only allows 64k pages so will fail to attach the ITS device when this table
is not sufficiently aligned.
Sponsored by: DARPA, AFRL
