While here move to decimal for the _op and _CR definitions to be used
by a future macro to define the register when the assembler doesn't
know about it.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40890
While here move to decimal for the _op and _CR definitions to be used
by a future macro to define the register when the assembler doesn't
know about it.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40889
While here move to decimal for the _op and _CR definitions to be used
by a future macro to define the register when the assembler doesn't
know about it.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40888
While here move to decimal for the _op and _CR definitions to be used
by a future macro to define the register when the assembler doesn't
know about it.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40887
Eliminate unnecessary TLB invalidations by pmap_kenter(),
pmap_qenter(), and pmap_mapbios() when the old page table entries
were invalid.
While I'm here, correct some nearby whitespace issues.
MFC after: 2 weeks
In order to compile lib32 libraries and other 32-bit code on arm64,
<machine/foo.h> needs to be redirected to an arm header rather
than arm64 when building with -m32. Ifdef the arm64 headers that
are installed in /usr/include/machine and used by user-level software
(including references from /usr/include/*.h) so that if __arm__ is
defined when including the arm64 version, <arm/foo.h> is included
rather than using the rest of the file's contents. Some arm headers
had no arm64 equivalent; headers were added just to do the redirection.
These files use #error if __arm__ is not defined to guard against
confusion. Also add an include/arm Makefile, and modify Makefiles
as needed to install everything, including the arm files in
/usr/include/arm. fenv.h comes from lib/msun/arm/fenv.h.
The new arm64 headers are:
acle-compat.h
cpuinfo.h
sysreg.h
Reviewed by: jrtc27, imp
Differential Revision: https://reviews.freebsd.org/D40944
Removed redundant OP_RN_SP for TYPE_02, since these addressing modes
always use the SP register, never XZR.
Reviewed by: mhorne
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D40588
Neither of the remaining callers to pmap_kremove() warrant inlining.
Those calls rarely occur. In other words, we were optimizing for the
uncommon case.
MFC after: 1 week
pmap_unmapbios() can simply call pmap_kremove_device() rather than
duplicating its code.
While I'm here, add a comment to pmap_kremove_device() explaining its
proper use, and fix a whitespace issue.
MFC after: 1 week
I believe that there are two reasons that the missing TAILQ
initialization operations haven't caused a problem. First, the TAILQ
head's first field is being initialized to zeroes elsewhere. Second,
the first access to the TAILQ head's last field is by
TAILQ_INSERT_HEAD(), which assigns to the last field without reading
it when the first field is NULL.
Reviewed by: kib, markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D41118
Several vm_radix tries are not initialized with vm_radix_init. That
works, for now, since static initialization zeroes the root field
anyway, but if initialization changes, these tries will fail. Add
missing initializer calls.
Reviewed by: alc, kib, markj
Differential Revision: https://reviews.freebsd.org/D40971
Strictly speaking, SYS V Aarch64 ABI does not states that lr should be
cleared, however gdb relies on lr and tryes to unwind stack further.
Reviewed by:
Differential Revision: https://reviews.freebsd.org/D40841
Use of compiler builtin ffs/ctz functions will result in optimized
instruction sequences when possible, and fall back to calling a function
provided by the compiler run-time library. We have slowly shifted our
platforms to take advantage of these builtins in 60645781d6 (arm64),
1c76d3a9fb (arm), 9e319462a0 (powerpc, partial).
Some platforms still rely on the libkern implementations of these
functions provided by libkern, namely riscv, powerpc (ffs*, flsll), and
i386 (ffsll and flsll). These routines are slow, as they perform a
linear search for the bit in question. Even on platforms lacking
dedicated bit-search instructions, such as riscv, the compiler library
will provide better-optimized routines, e.g. by using binary search.
Consolidate all definitions of these functions (whether currently using
builtins or not) to libkern.h. This should result in equivalent or
better performing routines in all cases.
One wart in all of this is the existing HAVE_INLINE_F*** macros, which
we use in a few places to conditionally avoid the slow libkern routines.
These aren't easily removed in one commit. For now, provide these
defines unconditionally, but marked for removal after subsequent
cleanup.
Removal of the now unused libkern routines will follow in the next
commit.
Reviewed by: dougm, imp (previous version)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40698
This will be used by bhyve and will allow the size to change, e.g. for SVE.
Reviewed by: markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40131
In get_kernel_reg_masked we use update_lower_register to get the lower
value of two registers for a given field. It will return the entire
register value with just the single field updated.
Because of this get_kernel_reg_masked needs to use the returned value
directly rather than ORing each field together. Fix this by updating
the mask and returning that from get_kernel_reg_masked.
Reviewed by: markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40846
The setting of VM_NFREEORDER and the comment describing it were copied
from sparc64 where both the page size and the number of page table
entries that fit in a cache line are different from arm64.
Reviewed by: andrew, kib, markj
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D40782
No functional change intended.
Reviewed by: andrew
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Sponsored by: Klara, Inc. (hardware)
Differential Revision: https://reviews.freebsd.org/D40502
This lets consumers fetch the value of a system register and apply a
mask over individual fields. That is, each field in the returned value
will be the "smaller" of the two provided by "mask" and the value saved
in kern_cpu_desc. This will be used by vmm to sanitize host system
register fields.
Reviewed by: andrew
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Sponsored by: Klara, Inc. (hardware)
Differential Revision: https://reviews.freebsd.org/D40500
Since pmap_ps_enabled() is true by default, check it inside of
pmap_promote_l2() instead of at every call site.
Modify pmap_promote_l2() to return true if the promotion succeeded and
false otherwise.
(A similar change was applied to the amd64 pmap in 0d2f98c2f092.)
Reviewed by: kib, markj
Differential Revision: https://reviews.freebsd.org/D40781
Not making it explicit that we're printing values in hex can be
misleading when the number doesn't contain hex-only symbols (a-f). A
good example of this is print_gp_register(), where we print "(func +
offset)"; if the offset doesn't contain a-f symbols, it's not
immediately clear if that value is in decimal or hex. Using '%#' instead
of '0x%' also isn't a better option, it doesn't print '0x' if the value
is 0, and it also messes up column alignment.
Reviewed by: imp, markj
Approved by: markj (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40583
Add disassembly support for the following shifted register instructions:
* mvn
* orn
* orr
* and
* ands
* bic
* bics
* eon
* eor
* tst
According to Arm64 documenation, operational pseuducode of shifted
register instruction must return `UNDEFINED` if shift type is `RESERVED`
('11'). Hence, removed "rsv" from `shift_2` array and add "ror". In case
of shift type is 3 and this type is `RESERVED`, we will return
`undefined`.
Reviewed by: mhorne
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D40386
Replace several sequential searches for a segment that contains a
phyiscal address with a call to a function that does it by binary
search. In vm_page_reclaim_contig_domain_ext, find the first segment
to reclaim from, and reclaim from each subsequent appropriate segment.
Eliminate vm_phys_scan_contig.
Reviewed by: alc, markj
Differential Revision: https://reviews.freebsd.org/D40058
It doesn't seem particularly useful to have it enabled by default, and
it spits several dozen lines into the dmesg on a test system.
Reviewed by: andrew
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D40499
It was used in one place and was added specifically to support dtrace
stack unwinding code. Write an equivalent expression using struct
unwind_state instead. No functional change intended.
Reviewed by: andrew
MFC after: 1 week
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D40538
Stop requiring all of the PTEs to have the accessed bit set for superpage
promotion to occur. Given that change, add support for promotion to
pmap_enter_quick(), which does not set the accessed bit in the PTE that
it creates.
Since the final mapping within a superpage-aligned and sized region of a
memory-mapped file is typically created by a call to pmap_enter_quick(),
we now achieve promotions in circumstances where they did not occur
before, for example, the X server's read-only mapping of libLLVM-15.so.
See also https://www.usenix.org/system/files/atc20-zhu-weixi_0.pdf
Reviewed by: kib, markj
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D40478
This is a mapping from ID register value to offset in struct cpu_desc.
These registers may be needed with future architecture revisions either
by userspace or by bhyve.
Sponsored by: Arm Ltd
We only need this during boot. Allocate the array before starting CPUs
to reduce the memory usage.
Reviewed by: Zach Leaf <zachary.leaf@arm.com>
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40433
To help moving to a dynamically allocated cpu_desc array reduce the
places we use it directly and create a pointer that is passed in to
functions that read it.
Reviewed by: Zach Leaf <zachary.leaf@arm.com>
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40392
To find the redistributor registers use the resource we have already
found and add an offset. This removed the need to create a
per-redistributor resource as it can now be a pointer to the resource
found in attach.
While here check the offset is within the bounds of the resource. Some
ACPI tables list each redistributor as a separate memory range, even
if they are physically contiguous. In this case we may not have each
resource virtually contiguous with neighbouring resources. This can
lead to a data abort when reading past the resource range.
Reviewed by: kevans
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40263
The prior implementation of VM_PAGE_TO_PV_LIST_LOCK() performed a
linear-time search of the vm_phys_segs[] array. However, in contrast to
PHYS_TO_PV_LIST_LOCK(), that search is unnecessary because every (non-
fictitious) vm_page contains the index of the vm_phys_seg in which it
resides.
Change most of the remaining uses of CHANGE_PV_LIST_LOCK_TO_PHYS() and
PHYS_TO_PV_LIST_LOCK() to CHANGE_PV_LIST_LOCK_TO_VM_PAGE() and
VM_PAGE_TO_PV_LIST_LOCK(), respectively.
Collectively, these changes also reduce the size of a GENERIC-NODEBUG
kernel's pmap.
Before:
text data bss dec hex filename
70144 3200 2248 75592 0x12748 pmap.o
After:
text data bss dec hex filename
69192 3200 2248 74640 0x12390 pmap.o
Reviewed by: kib, markj
Differential Revision: https://reviews.freebsd.org/D40306
The sysentvec sv_imgact_try was used by kern_exec() to allow
non-native ABI to fixup shell path according to ABI root directory.
Since the non-native ABI can now specify its root directory directly
to namei() via pwd_altroot() call this facility is not needed anymore.
Differential Revision: https://reviews.freebsd.org/D40092
MFC after: 2 month
The Barndinfo emul_path was used by the Elf image activator to fixup
interpreter file name according to ABI root directory. Since the
non-native ABI can now specify its root directory directly to namei()
via pwd_altroot() call this facility is not needed anymore.
Differential Revision: https://reviews.freebsd.org/D40091
MFC after: 2 month
The global variable "pmap_last_pa" was copied from the amd64 pmap as a
part of commit c15085278c "arm64 pmap: implement per-superpage locks"
but it is neither used nor needed by the arm64 pmap.
This avoids bloating the kernel image when MAXCPU is large.
A follow-up patch for kgdb and other kernel debuggers is needed since
the stoppcbs symbol is now a pointer. Bump __FreeBSD_version so that
debuggers can use osreldate to figure out how to handle stoppcbs.
PR: 269572
MFC after: never
Reviewed by: mjg, emaste
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39806
Add disassembly support for the following shifted register instructions:
* adds
* subs
* sub
* neg
* negs
* cmp
* cmn
The 'Mandatory Tokens' checks are relaxed to allow for the alias
instructions (e.g. cmp) which hard-code one or more registers as xzr.
Reviewed by: mhorne
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D40006
Update the few uppercase fields fields to be consistent with others.
Reviewed by: mhorne
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D39820
The GICv3 redistributor register access macros take arguments that are
used as variables. Add brackets around them as is the style, and for
safety.
Sponsored by: Arm Ltd
When checking if the spinlock count is correct in a data abort we can
panic without printing the registers. These are useful to debug the
abort, e.g. by giving the fault address register.
Reviewed by: kib
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40228
We don't print leading zeros for other registers, so do the same with
ESR_EL1.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40129
These will be used by bhyve to manage the host VFP registers, e.g.
saving the host state before entering a guest.
Reviewed by: markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40130
These will be used by bhyve
Reviewed by: markj
Sponsored by: Arm Ltd
Sponsored by: Innovate UK
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40128
Introduce macro for PHYS_TO_PTE, setting the groundwork for future
support of various Arm VMSA extensions.
For extensions such as 52-bit VA/PA (FEAT_LPA2), the representation of
an address between a PTE and PA are not equivalent. This macro will
allow converting between the different representations.
Currently PHYS_TO_PTE is a NOP. Replace all instances where we go from
PA to PTE with new PHYS_TO_PTE macro.
Reviewed by: markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D39828
Introduce macro for PTE_TO_PHYS, setting the groundwork for future
support of various Arm VMSA extensions.
For extensions such as 52-bit VA/PA (FEAT_LPA2), the representation of
an address between a PTE and PA are not equivalent. This macro will
allow converting between the different representations.
Currently going from PTE to PA is achieved by masking off the upper and
lower attributes. Retain this behaviour but replace all instances with
the new macro instead.
Reviewed by: alc, markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D39827
For all PPIs setup earlier than SI_SUB_SMP, PIC_INIT_SECONDARY ends up
cleaning these up for each AP as it comes online. Once they're online,
we don't currently do anything to make sure they're configured for other
APs. Fix it by using smp_rendezvous for the meaty bits of configuring a
PPI, which will just do single-thread behavior before APs are online but
do the right thing for other CPUs after.
While we're here, make sure redistributor config is correct for other
APs as they come online in gic_v3_init_secondary.
Reported/Tested by: Souradeep Chakrabarti (Microsoft/Hyper-V)
Reviewed by: andrew (before slight refactor)
Differential Revision: https://reviews.freebsd.org/D40112
The kernel doesn't execute this code, it's only ever copied to
userspace. Move it to .rodata as we don't need to modify it.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D39399
!a != !b -> a != b; this part was lifted from NetBSD, and I clearly did
not reconsider that these are bools (I'm guessing they were ints in an
earlier iteration of the NetBSD implementation).
while we're here, it should be easy to see that we've covered all of the
cases but let's add in an __assert_unreachable() to make it easier on
the eyes.
Reported by: jrtc27
Add another undefined instruction handler for compat32 and watch out for
SWP/SWPB instructions.
SWP/SWPB were deprecated in ARMv6 and declared obsolete in ARMv7, but
this implementation is motivated by some proprietary software that still
uses SWP/SWPB. Because it's deprecated, emulation is pushed back behind
a sysctl that defaults to OFF in GENERIC so that it doesn't potentially
adversely affect package builds; it's unknown whether software may test
for a functional swp/swpb instruction with the desire of using it later,
so we err on the side of caution to ensure we don't end up with swp/swpb
use in freebsd/arm packages (which are built on aarch64).
The EMUL_SWP config option may be used to enable emulation by default in
environments where emulation is desired and won't really be turned off.
Reviewed by: andrew, mmel (both earlier version)
Sponsored by: Stormshield
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D39667
To avoid clobbering of any registers by the trampoline code use Linux
way to call signal handlers. I.e., we are out from the kernel right into
the signal handler, put return address from the signal handler into the
link register.
The mysterious NOP is required for some unwinders (e.g. libc++) that
unconditionally subtract one from the result of _Unwind_GetIP() in order
to identify the calling function.
MFC after: 1 week
To allow unwinders to go througth a previous to sigreturn frame we should
properly emulate the trampoline frame record which should points to the
previous frame and set the trampoline frame pointer to the emulated frame
before calling signal handler.
MFC after: 1 week
An Aarch64 sigreturn trampoline frame can't currently be described in
a DWARF .eh_frame section, because Aarch64 does not define a register
number for PC and provide no direct way to encode PC of the previous
frame. Instead, unwinders (libgcc, gdb, libunwind) detect the sigreturn
frame by looking for the sigreturn instruction. If a sigreturn frame is
detected, unwinders restores all the gprs, SP and PC by assuming that
sp points to an rt_sigframe Linux kernel struct
When entering the kernel, the link register (lr) contains the return
address of the previous frame, the exception link register (elr) contains
the address of the next instruction after the one which generated the
exception, i.e., PC.
MFC after: 1 week
The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.
Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix
Much like casueword*, except just a plain old swap. Maintains a similar
interface to casu(9)- return value -1 (fault), 0 (success), or 1 (fail),
and also both ll/sc and LSE variants are implemented.
These will be used to implement 32-bit swp/swpb emulation on aarch64.
Reveiwed by: andrew
Sponsored by: Stormshield
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D39837
Commit 281402e0a5 ("arm64: Shave off two instructions in exceptions")
removed the instruction that set the frame pointer (x29) as it appeared
to be unused.
The frame pointer is used in arm64/db_trace.c:db_stack_trace_cmd() when
unwinding state, and hence still needs to be set.
Add back the instruction to save_registers to properly update frame
pointer.
Reported by: andrew
Sponsored by: Arm Ltd
Added support to distinguish between XZR/WZR and SP/WSP registers.
Add new OP_ flags to indicate if the instruction allows the use of SP
for a given register field. "wSP" and "SP" are removed from w_reg and
x_reg, and helper functions are introduced for this purpose of detecting
the correct name of the x31 register.
mhorne: While here, adjust some whitespace issues from a previous
commit.
Reviewed by: mhorne
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D39862
This existing helper function is preferable to the hand-rolled
calculation of the kstack bounds.
Make some small style improvements while here. Notably, rename every
instance of "r", the return address, to "ra". Tidy the includes in the
affected files.
Reviewed by: jkoshy
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39909
As of version 2.6.0 of the Linux kernel, dev_t is a 32-bit unsigned integer
on all platforms. Prior the 2.6 kernel dev_t type was an unsigned short.
However, since the firs commit of the Linuxulator, mknod syscall get int dev
argument.
Also, there is some confusion here, while the kernel declares a dev_t type
as a 32-bit sized, the user-space dev_t type can be size of 64 bits, e.g.,
in the Glibc library.
To avoid confusion and to help porting of the Linuxulator to other platforms
use explicit l_dev_t for dev argument of mknod syscalls.
This was needed before INTRNG was in place and handling the push of
curthread->td_intr_frame. Since INTRNG now handles this, there is no
longer and need for playing around with the frame inside IPI interrupts.
This was needed when intr_ipi_dispatch() was called by hardware-specific
IPI interrupt routines which didn't save the trap frame. Now all ARM
interrupts pass through INTRNG which will have already saved the trap
frame and disabled preemption.
Remove the conditional trapframe/argument passing to the handlers.
Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D37938
This allows us to support this hardware and, in the future, use clocks
so they are enabled past the initial kernel boot process.
Reviewed by: ray
Differential Revision: https://reviews.freebsd.org/D30103
This commentary was carried over from the x86 version of the same code,
but has actually been inaccurate for a while now. As of FreeBSD 12.x,
all environments are used unless they disable each other. See
39d44f7f15 ("kern_environment: use any provided environments [...]")
for details.
Reviewed by: imp
Differentiala Revision: https://reviews.freebsd.org/D35695
Added disassembly support for each type of str/strb/strh instruction
encoding.
Reviewed by: mhorne
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D39336
The current implementation is wrong, since it unconditionally sets the
amount equal to the <size> field of the instruction. However, when the
<S> bit (scale) is not set, it must be zero.
Also fix a typo, sxts to sxtx, according to the Arm64 documentation.
Reviewed by: mhorne
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D39334
Allow new features to be supported without changing the size of
existing structures.
Reviewed by: kib
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D39777
This function needs to unmap all memory in a given SMMU context. Have
it iterate over all page table entries to find what has been mapped
rather than looking at virtual addresses.
While here use SMMU specific macros.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D39185
This is not managed through the VM subsystem so only needs to hold the
data the SMMU driver needs.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D39184
Now the PCI drivers have a common interface to read the IOMMU xref
and SID create a common function to read it. This fixes an issue where
we will call into an ACPI specific function when booting with FDT when
both are enabled.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D39183
Use the xref from OF_xref_from_node for the smmu xref. We already have
a valid xref ID, there is no need to convert this to a memory address.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D39181
