Not all interrupt controllers enable IPIs by default as the Arm
GIC specs make it an implementation defined option. As at least two
hypervisors have also previously masked the IPIs on boot.
As we already enable these IPIs on the non-boot CPUs it is expected
this is a safe operation.
Differential Revision: https://reviews.freebsd.org/D26975
This will allow us to allocate an unmapped memory resource, then
later map it with a specific memory attribute.
This is also needed for virtio with the modern PCI attachment.
Reviewed by: kib (via D29723)
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D29694
It is defined as a uint64_t in the UEFI spec. As it's not used as a
pointer by the kernel follow this and define it as the same in the
kernel.
Reviewed by: kib, manu, imp
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D29759
On arm64 we currently use a non-posted write for device memory, however
we should move to use posted writes. This is expected to work on most
hardware, however we will need to support a non-posted option for some
broken hardware.
Reviewed by: imp, manu, bcr (manpage)
Differential Revision: https://reviews.freebsd.org/D29722
It has to be zeroed before committing it to device.
We do that by allocating it with M_ZERO, but there was no
memory barrier or cache flush to ensure its sees it zeroed.
This fixes MSIX on LS1028A SoC.
Submitted by: Kornel Duleba <mindal@semihalf.com>
Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: Alstom Group
Differential Revision: https://reviews.freebsd.org/D30033
Option `FIB_ALGO` gates new modular fib lookup functionality,
enabling more performant routing table lookups and improving
control plane convergence under the load.
Detailed feature description is available in D27401.
Reviewed By: olivier, gnn
Differential Revision: https://reviews.freebsd.org/D28434
These comments were copied from dbg_monitor_enter(), but the intended
modifications weren't made. Update them to reflect what this code
actually does.
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
This is both intuitive and required, as any previous breakpoint settings
may not be applicable to the new process.
Reported by: arichardson
Reviewed by: kib
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D29672
Following the analogous change for amd64 and i386 in 8223717ce6,
ensure that new processes start with these registers inactive.
PR: 254661
Reported by: Michał Górny
Reviewed by: kib, emaste
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D29630
When resetting the VFP state we need to discard any old state so we don't
try to save it on a context switch. Move this first so resetting the pcb
is safe to perform outside a critical section.
Reviewed by: arichardson
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D29401
On some systems (e.g. Lenovo ThinkPad X240, Apple MacBookPro12,1)
the SMBIOS entry point is not found in the <0xFFFFF space.
Follow the SMBIOS spec and use the EFI Configuration Table for
locating the entry point on EFI systems.
Reviewed by: rpokala, dab
MFC after: 1 week
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D29276
Summary:
They're nearly identical, so don't use two copies. Merge the newer
driver into the older one, and move it to a common location.
Add the Semihalf and associated copyrights in addition to mine, since
it's a non-trivial amount of code merged.
Reviewed By: mw
Differential Revision: https://reviews.freebsd.org/D29520
The remote protocol allows for implementations to report more specific
reasons for the break in execution back to the client [1]. This is
entirely optional, so it is only implemented for amd64, arm64, and i386
at the moment.
[1] https://sourceware.org/gdb/current/onlinedocs/gdb/Stop-Reply-Packets.html
Reviewed by: jhb
MFC after: 3 weeks
Sponsored by: NetApp, Inc.
Sponsored by: Klara, Inc.
NetApp PR: 51
Differential Revision: https://reviews.freebsd.org/D29174
Handle the 'z' and 'Z' remote packets for manipulating hardware
watchpoints.
This could be expanded quite easily to support hardware or software
breakpoints as well.
https://sourceware.org/gdb/onlinedocs/gdb/Packets.html
Reviewed by: cem, markj
MFC after: 3 weeks
Sponsored by: NetApp, Inc.
Sponsored by: Klara, Inc.
NetApp PR: 51
Differential Revision: https://reviews.freebsd.org/D29173
Use the new kdb variants. Print more specific error messages.
Reviewed by: jhb, markj
MFC after: 3 weeks
Sponsored by: NetApp, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D29156
Add wrappers around the debug_monitor interface, to be consumed by MI
kernel debugger code. Update dbg_setup_watchpoint() and
dbg_remove_watchpoint() to return specific error codes, not just -1.
Reviewed by: jhb, kib, markj
MFC after: 3 weeks
Sponsored by: NetApp, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D29155
This change serves two purposes.
First, we take advantage of the compiler provided endian definitions to
eliminate some long-standing duplication between the different versions
of this header. __BYTE_ORDER__ has been defined since GCC 4.6, so there
is no need to rely on platform defaults or e.g. __MIPSEB__ to determine
endianness. A new common sub-header is added, but there should be no
changes to the visibility of these definitions.
Second, this eliminates the hand-rolled __bswapNN() routines, again in
favor of the compiler builtins. This was done already for x86 in
e6ff6154d2. The benefit here is that we no longer have to maintain our
own implementations on each arch, and can instead rely on the compiler
to emit appropriate instructions or libcalls, as available. This should
result in equivalent or better code generation. Notably 32-bit arm will
start using the `rev` instruction for these routines, which is available
on armv6+.
PR: 236920
Reviewed by: arichardson, imp
Tested by: bdragon (BE powerpc)
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D29012
Make it easy to define interceptors for new sanitizer runtimes, rather
than assuming KCSAN. Lay a bit of groundwork for KASAN and KMSAN.
When a sanitizer is compiled in, atomic(9) and bus_space(9) definitions
in atomic_san.h are used by default instead of the inline
implementations in the platform's atomic.h. These definitions are
implemented in the sanitizer runtime, which includes
machine/{atomic,bus}.h with SAN_RUNTIME defined to pull in the actual
implementations.
No functional change intended.
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
e4b8deb222 removed the last in-tree uses of PCPU_INC(). Its
potential benefit is also practically nonexistent. Non-x86
platforms already implement it as PCPU_ADD(..., 1), and according
to [0] there are no recent x86 processors for which the 'inc'
instruction provides a performance benefit over the equivalent
memory-operand form of the 'add' instruction. The only remaining
benefit of 'inc' is smaller instruction size, which in this case
is inconsequential given the limited number of per-CPU data consumers.
[0]: https://www.agner.org/optimize/instruction_tables.pdf
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D29308
TCP/IPv6 packets to be forwarded can be laid out with only the Ethernet
header in the first mbuf, and these packets are lost. There was a
previous hack to pullup ICMPv6 packets with such a layout for the
same reason. Generalize, and pullup any IPv6 packets with only the
Ethernet header in the first mbuf. Possibly this should also include
IPv4, but that situation has not been observed to fail.
PR: 254060
Reported by: denis at h3q.com
MFC after: 3 days
On FreeBSD/arm fill_fpregs, fill_dbregs are stubs that zero the reg
struct and return success. set_fpregs and set_dbregs do nothing and
return success.
Provide the same implementation for arm64 COMPAT_FREEBSD32.
Reviewed by: andrew
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D29314
These are not stored in the trapframe so must be cleared explicitly.
This is similar to one of the MIPS changes in 822d2d6ac9.
Reviewed by: andrew
Obtained from: CheriBSD
MFC after: 1 week
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D28711
Implement a driver for the RTC embedded in the RK805/RK808 power
management system used for RK3328 and RK3399 SoCs.
Based on experiments on my RK808, setting the time doesn't alter the
internal/inaccessible sub-second counter, therefore there's no point
in calling clock_schedule().
Based on an earlier revision by andrew.
Reviewed by: manu
Differential Revision: https://reviews.freebsd.org/D22692
Sponsored by: Google
MFC after: 1 week
The ENTRY() macro was modified by commit
28d945204e to add an optional NOP instruction
at the beginning of the function. It is of course an arm64 instruction, so
unsuitable for the 32bits sigcode. So just use EENTRY() instead for
aarch32_sigcode. This should fix receiving signals when running 32bits
binaries on FreeBSD/arm64.
MFC After: 1 week
I noticed that many of the math-related tests were failing on AArch64.
After a lot of debugging, I noticed that the floating point exception flags
were not being reset when starting a new process. This change resets the
VFP inside exec_setregs() to ensure no VFP register state is leaked from
parent processes to children.
This commit also moves the clearing of fpcr that was added in 65618fdda0
from fork() to execve() since that makes more sense: fork() can retain
current register values, but execve() should result in a well-defined
clean state.
Reviewed By: andrew
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D29060
Other kernel sanitizers (KMSAN, KASAN) require interceptors as well, so
put these in a more generic place as a step towards importing the other
sanitizers.
No functional change intended.
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D29103
In 48ba9b2669 we switched from creating level 1 blocks to smaller
level 2 blocks when creating the early arm64 page tables. On issue
was that they had a different meaning for register x7. The former used
it to hold page table attributes, while the latter held just the memory
type. This caused these attributes to be incorrectly shifted.
Fix this by changing the meaning of x7 to hold the block attributes
and fix the only caller that used the old meaning.
Most hardware seems to have handled the bits being off however qemu
failed to boot as reserved bits that should be zero were being set and
qemu fails to clear these when translating from a virtual address to a
physical address.
Sponsored by: Innovate UK
To trace leaf asm functions we can insert a single nop instruction as
the first instruction in a function and trigger off this.
Reviewed by: gnn
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D28132
This reduces the memory mapped to be closer to the minimal memory
needed to enable the MMU.
Reviewed by: mmel
Sponsored by: Innovate UK
Differential Revision:://reviews.freebsd.org/D27765
This flag has been set on startup since 65618fdda0.
However, This causes some of the math-related tests to fail as they report
zero instead of a tiny number. This fixes at least
/usr/tests/lib/msun/ldexp_test and possibly others.
Additionally, setting this flag prevents printf() from printing subnormal
numbers in decimal form.
See also https://www.openwall.com/lists/musl/2021/02/26/1
PR: 253847
Reviewed By: mmel
Differential Revision: https://reviews.freebsd.org/D28938
arm64 has a distinct exception code for single-step, so we can use this
to detect when an unexpected SS trap is encountered, or when an expected
one is not. See db_stop_at_pc().
Reviewed by: markj, jhb
MFC after: 5 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D28942
This value should be kept in sync with updates to kdb_frame->tf_elr,
since it is queried by PC_REGS() in several places.
Reviewed by: markj, jhb
MFC after: 5 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D28943
The main issue is that debug exceptions must to be disabled for the
entire duration that SS bit in MDSCR_EL1 is set. Otherwise, a
single-step exception will be generated immediately. This can occur
before returning from the debugger (when MDSCR is written to) or before
re-entering it after the single-step (when debug exceptions are unmasked
in the exception handler).
Solve this by delaying the unmask to C code for EL1, and avoid unmasking
at all while handling debug exceptions, thus avoiding any recursive
debug traps.
Reviewed by: markj, jhb
MFC after: 5 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D28944
The scheduler uses this topology to try and preserve locality when
migrating threads between CPUs and when performing work stealing.
Ensure that on NUMA systems it will at least take the NUMA topology into
account.
Reviewed by: mmel
Submitted by: Klara, Inc.
Sponsored by: Ampere Computing
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D28579
This macro returns true if a provided virtual address is contained
in the kernel's clean submap.
In CHERI kernels, the buffer cache and transient I/O map are allocated
as separate regions. Abstracting this check reduces the diff relative
to FreeBSD. It is perhaps slightly more readable as well.
Reviewed by: kib
Obtained from: CheriBSD
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D28710
ACPI Sec 5.2.16.5 (SRAT, GIC Interrupt Translation Service (ITS)
Affinity Structure) says:
> The GIC ITS Affinity Structure provides the association between
> a GIC ITS and a proximity domain. This enables the OSPM to
> discover the memory that is closest to the ITS, and use that in
> allocating its management tables and command queue.
Previously the ITS driver was using the proximity domain to
restrict which CPUs can be targeted by an LPI. We keep that logic
just for the original dual socket ThunderX which cannot forward
LPIs between sockets.
We also use the SRAT entry for its intended purpose of attempting
to allocate ITS table structures near the ITS.
Reviewed by: andrew
Sponsored by: Ampere Computing LLC
Differential Revision: https://reviews.freebsd.org/D28340
This follows the behavior on x86 where edge triggered interrupts are
not disabled when executing the handler. Because the ITS is a shared
resource, contention for the command queue lock can be substantial.
Suggested by: gallatin
Reviewed by: andrew
Tested by: gallatin
Sponsored by: Ampere Computing LLC
Differential Revision: https://reviews.freebsd.org/D28709
The motivation is to provide access to these registers from userspace
via ptrace(2) requests PT_GETDBREGS and PT_SETDBREGS.
This change breaks the ABI of these particular requests, but is
justified by the fact that the intended consumers (debuggers) have not
been taught to use them yet. Making this change now enables active
upstream work on lldb to begin using this interface, and take advantage
of the hardware debugging registers available on the platform.
PR: 252860
Reported by: Michał Górny (mgorny@gentoo.org)
Reviewed by: andrew, markj (earlier version)
Tested by: Michał Górny (mgorny@gentoo.org)
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D28415
This is a prerequisite to allowing the use of hardware watchpoints for
userspace debuggers.
This is also a slight departure from the x86 behaviour, since `si_addr`
returns the data address that triggered the watchpoint, not the
address of the instruction that was executed. Otherwise, there is no
straightforward way for the application to determine which watchpoint
was triggered. Make a note of this in the siginfo(3) man page.
Reviewed by: jhb, markj (earlier version)
Tested by: Michał Górny (mgorny@gentoo.org)
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D28561
In particular, we want to disallow setting breakpoints on kernel
addresses from userspace. The control register fields are validated or
ignored as appropriate.
Reviewed by: markj
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D28560
linux_shared_page_init() creates an object and grabs and maps a single
page to back the VDSO. When destroying the VDSO object, we failed to
destroy the mapping and free KVA. Fix this.
Reviewed by: kib
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D28696
