This entails:
- Marking some obvious candidates for __nosanitizeaddress
- Similar trap frame markings as amd64, for similar reasons
- Shadow map implementation
The shadow map implementation is roughly similar to what was done on
amd64, with some exceptions. Attempting to use available space at
preinit_map_va + PMAP_PREINIT_MAPPING_SIZE (up to the end of that range,
as depicted in the physmap) results in odd failures, so we instead
search the physmap for free regions that we can carve out, fragmenting
the shadow map as necessary to try and fit as much as we need for the
initial kernel map. pmap_bootstrap_san() is thus after
pmap_bootstrap(), which still included some technically reserved areas
of the memory map that needed to be included in the DMAP.
The odd failure noted above may be a bug, but I haven't investigated it
all that much.
Initial work by mhorne with additional fixes from kevans and markj.
Reviewed by: andrew, markj
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D36701
These were kept for ABI reasons. Remove them and bump __FreeBSD_version
so debuggers can be updated to use the new layout.
Reviewed by: jhb
Sponsored by: Arm Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35378
It was previously possible for the fault address register to get
clobbered before it was saved. This small window occurred when an
additional exception was encountered inside the exception handler,
overwriting the previous value.
Commit f29942229d ("Read the arm64 far early in el0 exceptions")
patched this issue, but avoided changing the trapframe since this could
be considered a KBI change in FreeBSD 13.
Revert the above fix and save the fault address in the trapframe
instead. This saves the fault address even earlier in the exception
handling process, and is a more robust and simple fix.
Reviewed by: andrew, jhb, jrtc27
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D38984
For the Exception Syndrome Register, ESR_ELx, the upper 32b were
previously unused, but now may contain additional exception info as of
Armv8.7 (FEAT_LS64).
Extend ESR from u32->u64 in exception handling code to support this. In
addition, also extend Saved Program Status Register SPSR_ELx in the same
way to allow for future extensions.
Reviewed by: andrew
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D38983
sig_atomic_t is defined as a long and thus is 64-bit on arm64. For some
reason its limit was incorrectly specified as a 32-bit number. This had
the unfortunate side effect of causing gnulib to override most of the
definitions in stdint.h. On CheriBSD this breaks all software that uses
gnulib in annoying and hard to debug ways.
Technically updating the limits might be an ABI change, but these
defines are largely unused (the only use in tree is in the libc++ test
suite where it's use an assertion that will fail due to this bug).
Further, since the underlying type remains the same, we're just
increasing the range of values a paranoid program might use.
Reviewed by: andrew, emaste
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D39193
Add macros for offsets of macros we set in the arm64 pcb pcb_x array.
This will simplift reducing the size of this array in a later change.
Sponsored by: Arm Ltd
Move device memory to a weaker type. The new device memory type allows
the system to acknowledge a write to a device before the write has
completed. This is inline with VM_MEMATTR_DEVICE on armv6/armv7.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D38945
To invalidate stage 2 mappings on arm64 we may need to call into the
hypervisor so add a function pointer that bhyve can use to implement
this.
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D37254
On some hardware, we can't clear HCR_EL2.E2H so accesses to the physical
timer hopelessly trap to EL2. Stash off the value of HCR_EL2 and use it
in has_hyp() to avoid this.
Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D38884
On Apple Silicon systems, E2H can't actually be cleared; we're stuck
with it. Check it again when we're setting up CPTR_EL2 and set FPEN
appropriately to avoid later trapping to EL2 on writes to SIMD
registers.
Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D38819
The SPDX-License-Identifier was wrong in the Arm CoreLink CMN-600
driver files. It used the incorrect FreeBSD variant of the BSD-2-Clause
identifier. According to [1] all files should use BSD-2-Clause.
[1] https://tools.spdx.org/app/check_license/
Reported by: emaste
Sponsored by: Arm Ltd
Zero the vttbr_el2 register on each CPU so we can tell if we are
running the host or guest kernel from a hypervisor.
Obtained from: https://github.com/FreeBSD-UPB/freebsd-src (earlier version)
Sponsored by: Innovate UK
Sponsored by: The FreeBSD Foundation
For completeness add accessors for the MIDR field. As the field is
always 0xf on arm64 it is unneeded in the current MICR handling, but
will be used in the vmm module for bhyve.
Obtained from: https://github.com/FreeBSD-UPB/freebsd-src (earlier version)
Sponsored by: The FreeBSD Foundation
With PERTHREAD_SSP configured, the compiler's stack-smashing protection
uses a per-thread canary value instead of a global value. The value is
stored in td->td_md.md_canary; the sp_el0 register always contains a
pointer to that value, and certain functions selected by the compiler
will store the canary value on the stack as a part of the function
prologue (and will verify the copy as part of the epilogue). In
particular, the thread structure may be accessed.
This happens to occur in data_abort(), which leads to the same problem
addressed by commit 2c10be9e06 ("arm64: Handle translation faults for
thread structures"). This commit fixes that directly, by disabling SSP
in data_abort() and a couple of related functions by using a function
attribute. It also moves the update of sp_el0 out of C code in case
the compiler decides to start checking the canary in pmap_switch()
someday.
A different solution might be to move the canary value to the PCB, which
currently lives on the kernel stack and isn't subject to the same
problem as thread structures (if only because guard pages inhibit
superpage promotion). However, there isn't any particular reason the
PCB has to live on the stack today; on amd64 it is embedded in struct
thread, reintroducing the same problem. Keeping the reference canary
value at the top of the stack is also rather dubious since it could be
clobbered by a sufficiently large stack overflow.
A third solution could be to go back to the approach of commit
5aa5420ff2, and modify UMA to use the direct map for thread structures
even if KASAN is enabled. But, transient promotions and demotions in
the direct map are possible too.
Reviewed by: alc, kib, andrew
MFC after: 1 month
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D37255
Add a minimal implementation of cpu_ptrace() for arm64. It is only used to
get/set VFP registers for 32bits binaries, as it is apparently what we use
there, instead of the MI PT_GETFPREGS/PT_SETFPREGS.
PR: 267361
MFC After: 1 week
They are used when ASLR is not applied.
The need for adjusting is due to rtld direct exec mode puts ld-elf.so.1
at the PIE load address, and this address must not conflict with the
default linker' load address for non-PIE binaries. Otherwise rtld in
direct mode cannot activate image. Example of implicit failure is ldd(1)
refusing to run.
Reported by: kp
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D37085
These are 64-bit. Mark them as unsigned long so we don't rely on
undefined behaviour or shift a 32-bit value more than 32 bits.
Sponsored by: Innovate UK
Sponsored by: The FreeBSD Foundation
Add a <sys/_pv_entry.h> intended for use in <machine/pmap.h> to
define struct pv_entry, pv_chunk, and related macros and inline
functions.
Note that powerpc does not yet use this as while the mmu_radix pmap
in powerpc uses the new scheme (albeit with fewer PV entries in a
chunk than normal due to an used pv_pmap field in struct pv_entry),
the Book-E pmaps for powerpc use the older style PV entries without
chunks (and thus require the pv_pmap field).
Suggested by: kib
Reviewed by: kib
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D36685
We'll only be redefining the various bus_* macros, not the definition of
struct bus_space.
Reviewed by: andrew
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D36718
This matches the return type of pmap_mapdev/bios.
Reviewed by: kib, markj
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D36548
The mpidr register is 64 bit on arm64 and 32 bit on arm. Fix this by
extending the arm64 definition to include the top 32 bits.
To preserve KBI when MFCing split the value into two 32 bit values.
This will be cleaned up later only on main.
Reviewed by: bz
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D36346
When the IDC flag is set in the cache type register we don't need to
clean the data cache to the point of unification. Previously we
supported this flag being set only when the DIC flags was also set.
Add a new handler for when this is not the case.
Reviewed by: kib
Sponsored by: The FreeBSD Foundation, Ampere (hardware)
Differential Revision: https://reviews.freebsd.org/D36296
- Define PC_FREEL and _NPCM in terms of _NPCPV rather than via magic
numbers.
- Remove assertions about _NPC* values from pmap.c. This is less
relevant now that PC_FREEL and _NPCM are derived from _NPCPV.
- Add a helper inline function pc_is_full() which uses a loop to check
if pc_map is all zeroes. Use this to replace three places that
check for a full mask assuming there are only 3 entries in pc_map.
Reviewed by: markj
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D36217
We need to be careful to not promote or demote the memory containing
the per-CPU structures as the exception handlers will dereference it
so any time it's invalid may cause recursive exceptions.
Add a new pmap function to set a flag in the pte marking memory that
cannot be promoted or demoted and use it to mark pcpu memory.
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35434
Define PAGE_SIZE and PAGE_MASK based on PAGE_SHIFT. With this we only
need to set one value to change one value to change the page size.
While here remove the unused PAGE_MASK_* macros.
Sponsored by: The FreeBSD Foundation
The devmap variants used vm_offset_t for some reason, and a few places
explicitly cast bus addresses to vm_offset_t. (Probably those casts
along with similar casts for vm_size_t should just be removed and
instead permit the compiler to DTRT.)
Reviewed by: markj
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D35961
Add initial 16k page support on arm64. It is considered experimental,
with no guarantee of compatibility with a userspace or kernel modules
built with the current a 4k page size as code will likely try to pass
in a too small size when working with APIs that take a multiple of a
page, e.g. mmap.
As this is experimental, and because userspace and the kernel need to
have the PAGE_SIZE macro kept in sync there is no kernel option to
enable this. To test a new image should be built with the
PAGE_{SIZE,SHIFT,MASK} macros changed to the 16k versions.
There are currently known issues with loading modules from an old
loader as it can misalign them to load on a non-16k boundary.
Testing has shown good results in kernel workloads that allocate and
free large amounts of memory as only a quarter of the number of calls
into the VM subsystem are needed in the best case.
Reviewed by: markj
Tested by: gallatin
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34793
The field values are only valid when the ID_AA64PFR0_EL1.SVE or
ID_AA64PFR1_EL1.SME vields are non-zero. When this is not the case
the register is reserved as zero so is safe to read, but the SVEver
field will be incorrect so only print the decoded register when
the SVE or SME fields indicate it is valid.
Sponsored by: The FreeBSD Foundation
On arm64 all registers have a name that encodes op0, op1, CRn, CRm, and
op2 that are used to encode the register in the instruction. As some
registers we need to access may not be supportedby older compilers, or
are only supported when specific extensions are enabled support this
alternative form.
Sponsored by: The FreeBSD Foundation
To keep the vfp thread creation code in one place move into vfp.c. This
will also help with adding SVE support as it depends on VFP.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35615
Add support of ARM CMN-600 controller, PMU access functions only.
Add support of PMU counters of ARM CMN-600 controller.
Reviewed by: mhorne
Sponsored By: ARM
Differential Revision: https://reviews.freebsd.org/D32321
Summary:
It can be useful to see a summary of CPU caches on bootup. This is done
for most platforms already, so add this to arm64, in the form of (taken
from Apple M1 pro test):
L1 cache: 192KB (instruction), 128KB (data)
L2 cache: 12288KB (unified)
This is printed out per-CPU, only under bootverbose.
Future refinements could instead determine if a cache level is shared
with other cores (L2 is shared among cores on some SoCs, for instance),
and perform a better calculation to the full true cache sizes. For
instance, it's known that the M1 pro, on which this test was done, has 2
12MB L2 clusters, for a total of 24MB. Seeing each CPU with 12288KB L2
would make one think that there's 12MB * NCPUs, for possibly 120MB
cache, which is incorrect.
Sponsored by: Juniper Networks, Inc.
Reviewed by: #arm64, andrew
Differential Revision: https://reviews.freebsd.org/D35366
To enable it user-space needs to call feenableexcept().
FPE_FLTIDO has been added as the IDF bit can't be mapped to any existing
FPE code.
Reviewed by: andrew@
Differential revision: https://reviews.freebsd.org/D35247
MFC after: 2 weeks
