A hypervisor, e.g. bhyve, will need to know what exception levelthe kernel
was in when it started booting. If it was EL2 we can then enable said
hypervisor.
Store the boot exception level and allow the kernel to later query it.
Obtained from: https://github.com/FreeBSD-UPB/freebsd (earlier version)
Sponsored by: Innovate UK
The requirements of an Address Space ID allocator and a Virtual Machine ID
allocator are similar. Generalise the former code so it can be used with
the latter.
Reviewed by: alc (previous version)
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D23831
On arm64 the stage 1 and stage 2 pte formats are similar enough we can
reuse the pmap code for both. As they are only similar and not identical
we need to know if we are managing stage 1 or stage 2 tables.
Add an enum to store this information and a check to make sure it is
set to stage 1 when we manage stage 1 pte fields.
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D23830
In preperation for adding bhyve support to arm64 we need to split the
stage 1 and stage 2 pte fields to allow future changes to create stage 2
page tables.
MFC after: 1 month
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D23669
This reverts r177661. The change is no longer very useful since
out-of-tree KLDs will be built to target SMP kernels anyway. Moveover
it breaks the KBI in !SMP builds since cpuset_t's layout depends on the
value of MAXCPU, and several kernel interfaces, notably
smp_rendezvous_cpus(), take a cpuset_t as a parameter.
PR: 243711
Reviewed by: jhb, kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D23512
Once all CPUs are online, determine if they all support LSE atomics and
set lse_supported to indicate this. For now the atomic(9)
implementations are still always inlined, though it would be preferable
to create out-of-line functions to avoid text bloat. This was not done
here since big.little systems exist in which some CPUs implement LSE
while others do not, and ifunc resolution must occur well before this
scenario can be detected. It does seem unlikely that FreeBSD will
ever run on such platforms, however, so converting atomic(9) to use
ifuncs is probably a good next step.
Add a LSE_ATOMICS arm64 kernel configuration option to unconditionally
select LSE-based atomic(9) implementations when the target system is
known.
Reviewed by: andrew, kib
MFC after: 1 month
Sponsored by: The FreeBSD Foundation, Amazon (hardware)
Differential Revision: https://reviews.freebsd.org/D23325
These make use of the cas*, ld* and swp instructions added in ARMv8.1.
Testing shows them to be significantly more performant than LL/SC-based
implementations.
No functional change here since the wrappers still unconditionally
select the _llsc variants.
Reviewed by: andrew, kib
MFC after: 1 month
Submitted by: Ali Saidi <alisaidi@amazon.com> (original version)
Differential Revision: https://reviews.freebsd.org/D23324
Add a _llsc suffix for the existing LL/SC-based implementations and add
trivial wrappers. This is in preparation for supporting LSE-based
atomic(9) implementations.
No functional change intended.
Reviewed by: andrew, kib
MFC after: 1 month
Sponsored by: The FreeBSD Foundation, Amazon (hardware)
Differential Revision: https://reviews.freebsd.org/D23323
Parameterize the macros by type width as well as acq/rel semantics.
This makes modifying the implementations much less tedious and
error-prone and makes it easier to support alternate LSE-based
implementations. No functional change intended.
Reviewed by: andrew, kib
MFC after: 1 month
Sponsored by: The FreeBSD Foundation, Amazon (hardware)
Differential Revision: https://reviews.freebsd.org/D23322
These will reportedly be used in future uma changes.
MFC after: 2 weeks
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D23019
- Add all the Cortex-A CPU ID register values I can find.
- Add the Neoverse-N1 ID regiser value [1]
- Sort macros by register value.
PR: 243065
Submitted by: Ali Saidi <alisaidi AT amazon.com> [1]
Sponsored by: DARPA, AFRL (other than [1])
entry's access flag and dirty state, and enable this feature when it's
available.
Ensure that we don't overlook a dirty state update that is concurrent
with a call to pmap_enter(). (Previously, all dirty state updates would
have occurred with the containing pmap's lock held, so a page table entry's
dirty state could not have changed while pmap_enter() held that same lock.)
Reviewed by: andrew, markj
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D22907
On arm64 the eret instruction is used to return from an exception handler.
Some implementations may speculate past this instruction into the next
function. As the user may control many registers in these functions add
a synchronisation barrier sequence after the eret instruction to stop these
CPUs from speculating out of the exception handler.
PR: 242676
Submitted by: Anthony Steinhauser <asteinhauser@google.com> (previous version)
MFC after: 1 week
This code is non-obvious when reading for the first time. To help with
understanding of it add comments explaining what it's doing.
While here use macros from armreg.h rather than magic numbers.
Sponsored by: DARPA, AFRL
This is a 32-bit structure embedded in each vm_page, consisting mostly
of page queue state. The use of a structure makes it easy to store a
snapshot of a page's queue state in a stack variable and use cmpset
loops to update that state without requiring the page lock.
This change merely adds the structure and updates references to atomic
state fields. No functional change intended.
Reviewed by: alc, jeff, kib
Sponsored by: Netflix, Intel
Differential Revision: https://reviews.freebsd.org/D22650
In some cases, like is locked bootstrap or device's inability to boot from
removable media, we cannot use standard boot sequence and is necessary to
boot kernel directly from U-Boot.
Discussed with: jhibbits
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D13861
o Remove All Rights Reserved from my notices
o imp@FreeBSD.org everywhere
o regularize punctiation, eliminate date ranges
o Make sure that it's clear that I don't claim All Rights reserved by listing
All Rights Reserved on same line as other copyright holders (but not
me). Other such holders are also listed last where it's clear.
After discussing with mmel@, it was clear this is insufficient to address
all the needs. mmel@ will commit his original patch, from
https://reviews.freebsd.org/D13861, and the additions needed from r354714
will be made afterward.
Requested by: mmel
Sponsored by: Juniper Networks, Inc.
Update the NetBSD Kernel Concurrency Sanitizer (KCSAN) runtime to work in
the FreeBSD kernel. It is a useful tool for finding data races between
threads executing on different CPUs.
This can be enabled by enabling KCSAN in the kernel config, or by using the
GENERIC-KCSAN amd64 kernel. It works on amd64 and arm64, however the later
needs a compiler change to allow -fsanitize=thread that KCSAN uses.
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D22315
Summary:
Boot arm64 kernel using booti command from U-boot. booti can relocate initrd
image into higher ram addresses, therefore align the initrd load address to 1GiB
and create VA = PA map for it. Create L2 pagetable entries to copy the initrd
image into KVA.
(parts of the code in https://reviews.freebsd.org/D13861 was referred and used
as appropriate)
Submitted by: Siddharth Tuli <siddharthtuli_gmail.com>
Reviewed by: manu
Sponsored by: Juniper Networks, Inc
Differential Revision: https://reviews.freebsd.org/D22255
These are direct copies of the 32 bit functions, adjusted ad needed.
While here fix atomic_fcmpset_16 to use the valid load and store exclusive
instructions.
Sponsored by: DARPA, AFRL
Some userland libraries incude machine/pcb.h and this needs the full
definition of struct debug_monitor_state. To allow this to work move
stuct debug_monitor_state out of the _KERNEL guard.
Sponsored by: DARPA, AFRL
switching. The indirect costs being unnecessary TLB misses that are
incurred when ASIDs are not used. In fact, currently, when we perform a
context switch on one processor, we issue a broadcast TLB invalidation that
flushes the TLB contents on every processor.
Mark all user-space ("ttbr0") page table entries with the non-global flag so
that they are cached in the TLB under their ASID.
Correct an error in pmap_pinit0(). The pointer to the root of the page
table was being initialized to the root of the kernel-space page table
rather than a user-space page table. However, the root of the page table
that was being cached in process 0's md_l0addr field correctly pointed to a
user-space page table. As long as ASIDs weren't being used, this was
harmless, except that it led to some unnecessary page table switches in
pmap_switch(). Specifically, other kernel processes besides process 0 would
have their md_l0addr field set to the root of the kernel-space page table,
and so pmap_switch() would actually change page tables when switching
between process 0 and other kernel processes.
Implement a workaround for Cavium erratum 27456 affecting ThunderX machines.
(I would like to thank andrew@ for providing the code to detect the affected
machines.)
Address integer overflow in the definition of TCR_ASID_16.
Setup TCR according to the PARange and ASIDBits fields from
ID_AA64MMFR0_EL1. Previously, TCR_ASID_16 was unconditionally set.
Modify build_l1_block_pagetable so that lower attributes, such as ATTR_nG,
can be specified as a parameter.
Eliminate some unused code.
Earlier versions were tested to varying degrees by: andrew, emaste, markj
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D21922
Implement get/fill_dbregs on arm64. This is used by ptrace with the
PT_GETDBREGS and PT_SETDBREGS requests. It allows userspace to set hardware
breakpoints.
The struct dbreg is based on Linux to ease adding hardware breakpoint
support to debuggers.
Reviewed by: jhb
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D22195
We may want to mask exceptions when in userspace. This was previously
impossible as threads are created with all exceptions unmasked and
signals expected userspace to mask any. Fix these by copying the
mask state on thread creation and allow exceptions to be masked on
signal return, as long as they don't change.
Sponsored by: DARPA, AFRL
To allow consistent values to be used in both the kernel and userspace
create a function for these to be read from the kernel. They use a newly
created macro with the name of the ID register to read. For now there is
redundant information in the user_regs array as it still holds the CRm and
Op2 values, however this will be fixed in a later change.
This will be used by ptrace to allow hardware breakpoints in userspace.
Sponsored by: DARPA, AFRL
These instructions are used to access the registers described in armreg.h,
and will be used in a future change to create a per-register identification
macro.
Sponsored by: DARPA, AFRL
The debug monitor register state is now stored in a struct and updated
when required. Currently there is only a kernel state, however a
per-process state will be added in a future change.
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D22128
Because of the previous naming scheme the old ID_AA64PFR0_EL1 macro
collided with a potential macro for the register of the same name. To fix
this collision rename these macros.
Sponsored by: DARPA, AFRL
Summary: Add trivial 32-bit arm cores on aarch64 support for gcore. This
doesn't handle fpregs.
Reviewed by: #arm, andrew
Sponsored by: Juniper Networks, Inc
Differential Revision: https://reviews.freebsd.org/D21947
We must also check for large mappings. pmap_page_is_mapped() is
mostly used in assertions, so the problem was not very noticeable.
Reviewed by: alc
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D21824
Many extern struct pcpu <something>__pcpu declarations were
copied/pasted in sources. The issue is that the definition is MD, but
it cannot be provided by machine/pcpu.h due to actual struct pcpu
defined in sys/pcpu.h later than the inclusion of machine/pcpu.h.
This forced the copying when other code needed direct access to
__pcpu. There is no way around it, due to machine/pcpu.h supplying
part of struct pcpu fields.
To work around the problem, add a new machine/pcpu_aux.h header, which
should fill any needed MD definitions after struct pcpu definition is
completed. This allows to remove copies of __pcpu spread around the
source. Also on x86 it makes it possible to remove work arounds like
OFFSETOF_CURTHREAD or clang specific warnings supressions.
Reported and tested by: lwhsu, bcran
Reviewed by: imp, markj (previous version)
Discussed with: jhb
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D21418
doing so adds more flexibility with less redundant code.
Reviewed by: jhb, markj, kib
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D21250
pmap's lock ensures that other operations on the pmap don't observe the
old mapping being broken before the new mapping is established. However,
pmap_kextract() doesn't acquire the kernel pmap's lock, so it may observe
the broken mapping. And, if it does, it returns an incorrect result.
This revision implements a lock-free solution to this problem in
pmap_update_entry() and pmap_kextract() because pmap_kextract() can't
acquire the kernel pmap's lock.
Reported by: andrew, greg_unrelenting.technology
Reviewed by: andrew, markj
Tested by: greg_unrelenting.technology
X-MFC with: r350579
Differential Revision: https://reviews.freebsd.org/D21169
The ARMv8 reference manual only states that the bit is reserved in
this case; following Linux's example, use it instead of a
software-defined bit for the purpose of indicating that a managed
mapping is writable.
Reviewed by: alc, andrew
MFC after: r350004
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D21121
Previously only some of the ID register fields were 64 bit. To allow
for a script to generate these mark them all 64 bit. To allow for their
use in assembly we need to use the UINT64_C macro via a new UL macro
to stop the lines from being too long.
MFC after: 1 week
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D20977
Add HWCAP support for arm64.
defines are the same as in Linux and a userland program can use
elf_aux_info to get the data.
We only save the common denominator for all cores in case the
big and little cluster have different support (this is known to
exists even if we don't support those SoCs in FreeBSD)
Differential Revision: https://reviews.freebsd.org/D17137
Previously the arm64 pmap did no reference or modification tracking;
all mappings were treated as referenced and all read-write mappings
were treated as dirty. This change implements software management
of these attributes.
Dirty bit management is implemented to emulate ARMv8.1's optional
hardware dirty bit modifier management, following a suggestion from alc.
In particular, a mapping with ATTR_SW_DBM set is logically writeable and
is dirty if the ATTR_AP_RW_BIT bit is clear. Mappings with
ATTR_AP_RW_BIT set are write-protected, and a write access will trigger
a permission fault. pmap_fault() handles permission faults for such
mappings and marks the page dirty by clearing ATTR_AP_RW_BIT, thus
mapping the page read-write.
Reviewed by: alc
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D20907
I just happenned to have 3rd party code using 'var' as the output variable
which drew my attention to this. variables defined inside macros should be
prefixed to avoid getting shadowed varable wanrings from clang.
cpufunc, in terms of __builtin_ffs and the like, for arm64
architectures, and use those, rather than the simple libkern
implementations, in building arm64 kernels.
Tested by: greg_unrelenting.technology (earlier version)
Reviewed by: alc
Approved by: kib (mentor)
Differential Revision: https://reviews.freebsd.org/D20250
In all practical situations, the resolver visibility is static.
Requested by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Approved by: so (emaste)
Differential revision: https://reviews.freebsd.org/D20281
Add the infrastructure to allow MD procctl(2) commands, and use it to
introduce amd64 PTI control and reporting. PTI mode cannot be
modified for existing pmap, the knob controls PTI of the new vmspace
created on exec.
Requested by: jhb
Reviewed by: jhb, markj (previous version)
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D19514
Skylake Xeons.
See SDM rev. 68 Vol 3 4.6.2 Protection Keys and the description of the
RDPKRU and WRPKRU instructions.
Reviewed by: markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
Differential revision: https://reviews.freebsd.org/D18893
cpu_icache_sync_range(), except that it sets pcb_onfault to catch any page
fault, as doing cache maintenance operations for non-mapped generates a
data abort, and use it in freebsd32_sysarch(), so that a userland program
attempting to sync the icache with unmapped addresses doesn't crash the
kernel.
Spotted out by: andrew
Replace a call to DELAY(1) with a new cpu_lock_delay() KPI. Currently
cpu_lock_delay() is defined to DELAY(1) on all platforms. However,
platforms with a DELAY() implementation that uses spin locks should
implement a custom cpu_lock_delay() doesn't use locks.
Reviewed by: kib
MFC after: 3 days
While here tag which architecture release fields were added and remove a
field that only existed in very early releases of the ARMv8 spec.
Sponsored by: DARPA, AFRL
All platforms except powerpc use the same values and powerpc shares a
majority of them.
Go ahead and declare AT_NOTELF, AT_UID, and AT_EUID in favor of the
unused AT_DCACHEBSIZE, AT_ICACHEBSIZE, and AT_UCACHEBSIZE for powerpc.
Reviewed by: jhb, imp
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D17397
We should only unmask interrupts when creating a new thread and leave the
other exceptions in teh same state as before creating the thread.
Reported by: jhibbits
Reviewed by: jhibbits
MFC after: 1 month
Sponsored by: https://reviews.freebsd.org/D17497
Tested with ifunc resolvers in the kernel and module with calls from
kernel to kernel, module to kernel, and module to module.
Reviewed by: kib (previous version)
Approved by: re (gjb)
Differential Revision: https://reviews.freebsd.org/D17370
The initial raise in r336519 wasn't enough for using big resolution
(1920 x 1200 for example). Raise it again.
Reported by: bob prohaska <fbsd@www.zefox.net>
Tested by: bob prohaska <fbsd@www.zefox.net>
Approved by: re (gjb@)
This appeared to be required to have EFI RT support and EFI RTC
enabled by default, because there are too many reports of faulting
calls on many different machines. The knob is added to leave the
exceptions unhandled to allow to debug the actual bugs.
Reviewed by: kevans
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Approved by: re (rgrimes)
Differential revision: https://reviews.freebsd.org/D16972
This calls into the Arm Trusted Firmware to enable and disable the
workaround for the Speculative Store Bypass Disable (SSBD) issue, also
known as Spectre Variant 4.
As this may have a large performance overhead, and how exploitable SSBD is
is unknown we follow the Linux lead of allowing the administrator to select
between always on, always off, or only enabled in the kernel, with the
latter being the default.
PR: 228955
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D15819
See the commit log messages for r321378 and r336288 for descriptions of
this functionality.
Reviewed by: alc
Differential Revision: https://reviews.freebsd.org/D16303
Some driver (like efifb) needs to map more than the current L2_SIZE
Raise the size so we can map the framebuffer setup by the bootloader.
Reviewed by: cognet
These changes ensure that reclaim_pv_chunk() can be safely be
executed concurrently by multiple threads.
Reviewed by: alc
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D16304
- Change pcpu zone consumers to use a stride size of PAGE_SIZE.
(defined as UMA_PCPU_ALLOC_SIZE to make future identification easier)
- Allocate page from the correct domain for a given cpu.
- Don't initialize pc_domain to non-zero value if NUMA is not defined
There are some misconceptions surrounding this field. It is the
_VM_ NUMA domain and should only ever correspond to valid domain
values as understood by the VM.
The former slab size of sizeof(struct pcpu) was somewhat arbitrary.
The new value is PAGE_SIZE because that's the smallest granularity
which the VM can allocate a slab for a given domain. If you have
fewer than PAGE_SIZE/8 counters on your system there will be some
memory wasted, but this is obviously something where you want the
cache line to be coming from the correct domain.
Reviewed by: jeff
Sponsored by: Limelight Networks
Differential Revision: https://reviews.freebsd.org/D15933
This is needed for efifb.
arm and ricv pmap (the two arch with arm64 that uses subr_devmap) have very
different implementation so for now only add this for arm64.
Tested with efifb on Pine64 with a few other patches.
Reviewed by: cognet
Differential Revision: https://reviews.freebsd.org/D15294
Its absence meant that GEOM direct dispatch was disabled (the service
routines check the current thread's stack usage to determine whether
to hand off the request to a dedicated thread), and this change is
sufficient to enable direct dispatch by default.
Reviewed by: allanjude
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D15527
band aid until a better solution to find the correct interrupt controller
can be found.
While here fix one place in the GICv3 ITS driver where the offset wasn't
correctly applied.
Sponsored by: DARPA, AFRL
Sponsored by: Cavium (Hardware)
No implementation of fpu_kern_enter() can fail, and it was causing needless
error checking boilerplate and confusion. Change the return code to void to
match reality.
(This trivial change took nine days to land because of the commit hook on
sys/dev/random. Please consider removing the hook or otherwise lowering the
bar -- secteam never seems to have free time to review patches.)
Reported by: Lachlan McIlroy <Lachlan.McIlroy AT isilon.com>
Reviewed by: delphij
Approved by: secteam (delphij)
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D14380
We don't support float in the boot loaders, so don't include
interfaces for float or double in systems headers. In addition, take
the unusual step of spiking double and float to prevent any more
accidental seepage.
kernel by PHYS_TO_DMAP() as previously present on amd64, arm64, riscv, and
powerpc64. This introduces a new MI macro (PMAP_HAS_DMAP) that can be
evaluated at runtime to determine if the architecture has a direct map;
if it does not (or does) unconditionally and PMAP_HAS_DMAP is either 0 or
1, the compiler can remove the conditional logic.
As part of this, implement PHYS_TO_DMAP() on sparc64 and mips64, which had
similar things but spelled differently. 32-bit MIPS has a partial direct-map
that maps poorly to this concept and is unchanged.
Reviewed by: kib
Suggestions from: marius, alc, kib
Runtime tested on: amd64, powerpc64, powerpc, mips64
We need to handle two cases:
1. One process attacking another process.
2. A process attacking the kernel.
For the first case we clear the branch predictor state on context switch
between different processes. For the second we do this when taking an
instruction abort on a non-userspace address.
To clear the branch predictor state a per-CPU function pointer has been
added. This is set by the new cpu errata code based on if the CPU is
known to be affected.
On Cortex-A57, A72, A73, and A75 we call into the PSCI firmware as newer
versions of this will clear the branch predictor state for us.
It has been reported the ThunderX is unaffected, however the ThunderX2 is
vulnerable. The Qualcomm Falkor core is also affected. As FreeBSD doesn't
yet run on the ThunderX2 or Falkor no workaround is included for these CPUs.
MFC after: 3 days
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D13812
a mask and value to compare with the Main ID Register. If these match then a
function is called to handle the installation of the erratum workaround.
No errata are currently handled, however this will change soon in a future
commit.
MFC after: 1 week
Sponsored by: DARPA, AFRL
platform divergence.
Only architectures which pass arguments in registers (mips)
and platforms which use really weird compilers (any?) would
need to augment the contents of <sys/_stdarg.h>
Convert x86, arm and arm64 architectures to use <sys/_stdarg.h>
They provide relaxed-ordered atomic access semantic. Due to the
FreeBSD memory model, the operations are syntaxical wrappers around
the volatile accesses. The volatile qualifier is used to ensure that
the access not optimized out and in turn depends on the volatile
semantic as implemented by supported compilers.
The motivation for adding the operation is to help people coming from
other systems or knowing the C11/C++ standards where atomics have
special type and require use of the special access operations. It is
still the case that FreeBSD requires plain load and stores of aligned
integer types to be atomic.
Suggested by: jhb
Reviewed by: alc, jhb
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D13534
The documentation on the Saved Process Status Register (SPSR) is a bit
weird; the M[4] bit is documented separately from M[3:0]. The M[4] bit
can be toggled to switch to 32-bit execution mode. This functionality is
orthogonal to M[3:0].
Change the definition of PSR_M_MASK to no longer include M[4]. Add a new
definition, PSR_AARCH32 that can be used to toggle 32-bit independently.
This bit will be used by the cloudabi32 code to force execution of
userspace code in 32-bit mode.
Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D13148
The nice thing about ARM64 is that it's pretty elegant to install
separate trap/exception handlers for 32-bit and 64-bit processes. That
said, for all other architectures (e.g., i386 on amd64) we always let
32-bit counterparts go through the regular system call codepath. Let's
do the same on ARM64.
Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D13146
Right now I'm using two Raspberry Pi's (2 and 3) to test CloudABI
support for armv6, armv7 and aarch64. It would be nice if I could
restrict this to just a single instance when testing smaller changes.
This is why I'd like to get COMPAT_CLOUDABI32 to work on arm64.
As COMPAT_CLOUDABI32 depends on COMPAT_FREEBSD32, at least for the ELF
loading, this change adds all of the bits necessary to at least build a
kernel with COMPAT_FREEBSD32. All of the machine dependent system calls
are still stubbed out, for the reason that implementations for these are
only useful if actual support for running FreeBSD binaries is added.
This is outside the scope of this work.
Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D13144
This value may be set by userspace so we need to check it before using it.
If this is not done correctly on exception return the kernel may continue
in kernel mode with all registers set to a userspace controlled value. Fix
this by moving the check into set_mcontext, and also add the missing
sanitisation from the arm64 set_regs.
Discussed with: security-officer@
MFC after: 3 days
Sponsored by: DARPA, AFRL
Move framebuffer.{c,h} to sys/boot/efi/loader and add the efifb
related metadata and pass it to the kernel
Reviewed by: imp, andrew
Differential Revision: https://reviews.freebsd.org/D12757
- allocate value for new AT_HWCAP2 auxiliary vector on all platforms.
- expand 'struct sysentvec' by new 'u_long *sv_hwcap2', in exactly
same way as for AT_HWCAP.
MFC after: 1 month
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D12699
mapping. This uses the new common code shared with amd64.
The RTC should only be accessed via EFI. There is no locking around it as
the spec only has this as a requirement for the PC-AT CMOS device.
Reviewed by: kib, imp
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D12595
kernel. We can register callbacks to perform the required operation on the
saved registers before returning.
This is initially used to work around a bug in old versions of QEMU that
trigger such an exception when reading from an ID register when it should
load z zero value.
I expect this could be used with other exception types, e.g. to emulate
special register access from userland.
Sponsored by: DARPA, AFRL
A new 'u_long *sv_hwcap' field is added to 'struct sysentvec'. A
process ABI can set this field to point to a value holding a mask of
architecture-specific CPU feature flags. If an ABI does not wish to
supply AT_HWCAP to processes the field can be left as NULL.
The support code for AT_EHDRFLAGS was already present on all systems,
just the #define was not present. This is a step towards unifying the
AT_* constants across platforms.
Reviewed by: kib
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12290
values. As not all assemblers understand the new ID_AA64MMFR2_EL1 register
add a macro to access it. This seems to be safe for older CPUs to read this
new register, with them returning zero.
Sponsored by: DARPA, AFRL
Previously, debug exceptions were only enabled on the boot CPU if
DDB was enabled in the dbg_monitor_init() function. APs also called
this function, but since mp_machdep.c doesn't include opt_ddb.h, the
APs ended up calling an empty stub defined in <machine/debug_monitor.h>
instead of the real function. Also, if DDB was not enabled in the kernel,
the boot CPU would not enable debug exceptions.
Fix this by adding a new dbg_init() function that always clears the OS
lock to enable debug exceptions which the boot CPU and the APs call.
This function also calls dbg_monitor_init() to enable hardware breakpoints
from DDB on all CPUs if DDB is enabled. Eventually base support for
hardware breakpoints/watchpoints will need to move out of the DDB-only
debug_monitor.c for use by userland debuggers.
Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D12001
handle cases where they can only run on a single domain.
To allow all devices access to this set we need to move reading the domain
earlier in the boot as it was previously handled in the CPU driver, however
this is too late for the GICv3 ITS driver.
Sponsored by: DARPA, AFRL
--Remove special-case handling of sparc64 bus_dmamap* functions.
Replace with a more generic mechanism that allows MD busdma
implementations to generate inline mapping functions by
defining WANT_INLINE_DMAMAP in <machine/bus_dma.h>. This
is currently useful for sparc64, x86, and arm64, which all
implement non-load dmamap operations as simple wrappers
around map objects which may be bus- or device-specific.
--Remove NULL-checked bus_dmamap macros. Implement the
equivalent NULL checks in the inlined x86 implementation.
For non-x86 platforms, these checks are a minor pessimization
as those platforms do not currently allow NULL maps. NULL
maps were originally allowed on arm64, which appears to have
been the motivation behind adding arm[64]-specific barriers
to bus_dma.h, but that support was removed in r299463.
--Simplify the internal interface used by the bus_dmamap_load*
variants and move it to bus_dma_internal.h
--Fix some drivers that directly include sys/bus_dma.h
despite the recommendations of bus_dma(9)
Reviewed by: kib (previous revision), marius
Differential Revision: https://reviews.freebsd.org/D10729
from machine/proc.h, consistently on all architectures.
Reviewed by: jhb
Sponsored by: The FreeBSD Foundation
MFC after: 3 weeks
X-Differential revision: https://reviews.freebsd.org/D11080
VM_MEMATTR_WRITE_COMBINING in the kernel. This fixes a bug where Xorg would
use write back cached memory for its graphics buffers. This would produce
artifacts on the screen as cachelines were written to memory.
MFC after: 1 week
Sponsored by: DARPA, AFRL
in place. To do per-cpu stats, convert all fields that previously were
maintained in the vmmeters that sit in pcpus to counter(9).
- Since some vmmeter stats may be touched at very early stages of boot,
before we have set up UMA and we can do counter_u64_alloc(), provide an
early counter mechanism:
o Leave one spare uint64_t in struct pcpu, named pc_early_dummy_counter.
o Point counter(9) fields of vmmeter to pcpu[0].pc_early_dummy_counter,
so that at early stages of boot, before counters are allocated we already
point to a counter that can be safely written to.
o For sparc64 that required a whole dummy pcpu[MAXCPU] array.
Further related changes:
- Don't include vmmeter.h into pcpu.h.
- vm.stats.vm.v_swappgsout and vm.stats.vm.v_swappgsin changed to 64-bit,
to match kernel representation.
- struct vmmeter hidden under _KERNEL, and only vmstat(1) is an exclusion.
This is based on benno@'s 4-year old patch:
https://lists.freebsd.org/pipermail/freebsd-arch/2013-July/014471.html
Reviewed by: kib, gallatin, marius, lidl
Differential Revision: https://reviews.freebsd.org/D10156
from the kernel. Make use of this to restrict accessing userspace to just
the functions that explicitly handle crossing the user kernel boundary.
Reported by: kib
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D10371
pagetables. This sets both bits when entering an address we know shouldn't
be executed.
I expect we could mark all userspace pages as Privileged execute-never to
ensure the kernel doesn't branch to one of these addresses.
While here add the ARMv8.1 upper attributes.
Reviewed by: alc, kib (previous version)
MFC after: 1 week
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D10360
The MFC will include a compat definition of smp_no_rendevous_barrier()
that calls smp_no_rendezvous_barrier().
Reviewed by: gnn, kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D10313
GNU toolchain does not recognize LR as standard register alias,
but clang does. Use of #define will work on both. Place the
definition into central machine/asm.h instead of patching every
affected file, as requested by plaftorm maintainers.
Reviews by: andrew, emaste, imp
Differential Revision: https://reviews.freebsd.org/D10307
I fixed this in 1997, but the fix was over-engineered and fragile and
was broken in 2003 if not before. i386 parameters were copied to 8
other arches verbatim, mostly after they stopped working on i386, and
mostly without the large comment saying how the values were chosen on
i386. powerpc has a non-verbatim copy which just changes the uncritical
parameter and seems to add a sign extension bug to it.
Just treat negative offsets as offsets if they are no more negative than
-db_offset_max (default -64K), and remove all the broken parameters.
-64K is not very negative, but it is enough for frame and stack pointer
offsets since kernel stacks are small.
The over-engineering was mainly to go more negative than -64K for the
negative offset format, without affecting printing for more than a
single address.
Addresses in the top 64K of a (full 32-bit or 64-bit) address space
are now printed less well, but there aren't many interesting ones.
For arches that have many interesting ones very near the top (e.g.,
68k has interrupt vectors there), there would be no good limit for
the negative offset format and -64K is a good as anything.
On arm64 use atomics. Then, both arm and arm64 do not need a critical
section around update. Replace all cpus loop by CPU_FOREACH().
This brings arm and arm64 counter(9) implementation closer to current
amd64, but being more RISC-y, arm* version cannot avoid atomics.
Reported by: Alexandre Martins <alexandre.martins@stormshield.eu>
Reviewed by: andrew
Tested by: Alexandre Martins, andrew
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
The types are for the byte offset and page index in vm object. They
are similar to off_t, which is defined as 64bit MI integer. Using MI
definitions will allow to provide consistent MD values of vm
object-related maximum sizes.
Reviewed by: alc
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
VFP code to store the old context, with lazy loading of the new context
when needed.
FPU_KERN_NOCTX is missing as this is unused in the crypto code this has
been tested with, and I am unsure on the requirements of the UEFI
Runtime Services.
Reviewed by: kib
Obtained from: ABT Systeems Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D8276
not be sent to userspace, for example the future flag to tell when we are
using floating point in the kernel.
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
with no creative content. Include "lost" changes from git:
o Use /dev/efi instead of /dev/efidev
o Remove redundant NULL checks.
Submitted by: kib@, dim@, zbb@, emaste@
will allow drivers that manage the clock frequency to communicate this with
the reset of the kernel.
Reported by: jmcneill
MFC after: 1 week
Sponsored by: ABT Systems Ltd
iterate over superpages. We don't yet create these, but soon will.
Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Right now, userspace (fast) gettimeofday(2) on x86 only works for
RDTSC. For older machines, like Core2, where RDTSC is not C2/C3
invariant, and which fall to HPET hardware, this means that the call
has both the penalty of the syscall and of the uncached hw behind the
QPI or PCIe connection to the sought bridge. Nothing can me done
against the access latency, but the syscall overhead can be removed.
System already provides mappable /dev/hpetX devices, which gives
straight access to the HPET registers page.
Add yet another algorithm to the x86 'vdso' timehands. Libc is updated
to handle both RDTSC and HPET. For HPET, the index of the hpet device
to mmap is passed from kernel to userspace, index might be changed and
libc invalidates its mapping as needed.
Remove cpu_fill_vdso_timehands() KPI, instead require that
timecounters which can be used from userspace, to provide
tc_fill_vdso_timehands{,32}() methods. Merge i386 and amd64
libc/<arch>/sys/__vdso_gettc.c into one source file in the new
libc/x86/sys location. __vdso_gettc() internal interface is changed
to move timecounter algorithm detection into the MD code.
Measurements show that RDTSC even with the syscall overhead is faster
than userspace HPET access. But still, userspace HPET is three-four
times faster than syscall HPET on several Core2 and SandyBridge
machines.
Tested by: Howard Su <howard0su@gmail.com>
Sponsored by: The FreeBSD Foundation
MFC after: 1 month
Differential revision: https://reviews.freebsd.org/D7473
promote memory as I am not sure all the demotion cases are handled, however
it is useful to implement pmap_page_set_memattr. This is used, for example,
when mapping uncached memory for bus_dma(9).
pmap_page_set_memattr needs to demote the DMAP region as on ARM we need to
ensure all mappings to the same physical address have the same attributes.
Reviewed by: kib
Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D6987
between ACPI and FDT. This will be needed on machines with both, e.g. the
SoftIron Overdrive 3000. The kernel will accept one or more comma separated
values of either 'acpi' or 'fdt'. Any other values are skipped.
To set it the user can either set it on the loader command line, or
in loader.conf e.g. in loader.conf:
kern.cfg.order=acpi,fdt
This will try using ACPI then FDT. If none of the selected options work the
kernel tries to use one to get the serial console, then panics.
Reviewed by: emaste (earlier version)
Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D7274
framework has significantly changed the driver has moved to a new file.
While it shares some code with the existing driver this has been modified
to work better with the intrng framework.
This has been tested on the ThunderX servers in the netperf cluster and has
been used to boot them for other testing, including DTrace and hwpmc.
With this we can use intrng on all supported arm64 platforms I was able to
test on. It is expected we will move to intrng soon, and disable the old
arm64 interrupt framework.
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D6437
updated, and until further testing can be done, this is disabled for now.
It is expected arm64 will switch to this interface, and the old interface
will be removed before 11.0 is released.
Obtained from: ABT Systems Ltd
Relnotes: yes
Sponsored by: The FreeBSD Foundation
disabled, however when we enable it it will default to assume memory is
not cache-coherent, unless either the tag was created or the parent was
marked as cache-coherent.
Obtained from: ABT Systems Ltd
Relnotes: yes
Sponsored by: The FreeBSD Foundation
To maintain coherence between cache and DMA memory appropriate
shareability flags need to be set in the PTE regardless of SMP
option.
Reviewed by: wma
Obtained from: Semihalf
Sponsored by: Cavium
Differential Revision: https://reviews.freebsd.org/D6231
the physmap. This will reduce the likelihood of an issue where we have
device memory mapped in the DMAP. This can only happen if it is within the
same 1G block of normal memory.
Reviewed by: kib
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D5938
2TB. The latter can be increased in 512GB chunks by adjusting the lower
address, however more work will be needed to increase the former.
There is still some work needed to only create a DMAP region for the RAM
address space as on ARM architectures all mappings should have the same
memory attributes, and these will be different for device and normal memory.
Reviewed by: kib
Obtained from: ABT Systems Ltd
Relnotes: yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D5859
Allow using DTRACE for performance analysis of userspace
applications - the function call stack can be captured.
This is almost an exact copy of AMD64 solution.
Obtained from: Semihalf
Sponsored by: Cavium
Reviewed by: emaste, gnn, jhibbits
Differential Revision: https://reviews.freebsd.org/D5779
This optimization attempts to utylize as wide as possible register store instructions to zero large buffers.
The implementation, if possible, will use 'dc zva' to zero buffer by cache lines.
Speedup: 60x faster memory zeroing
Submitted by: Dominik Ermel <der@semihalf.com>
Obtained from: Semihalf
Sponsored by: Cavium
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D5726
increased to 256TiB. The kernel address space can also be increased to be
the same size, but this will be performed in a later change.
To help work with an extra level of page tables two new functions have
been added, one to file the lowest level table entry, and one to find the
block/page level. Both of these find the entry for a given pmap and virtual
address.
This has been tested with a combination of buildworld, stress2 tests, and
by using sort to consume a large amount of memory by sorting /dev/zero. No
new issues are known to be present from this change.
Reviewed by: kib
Obtained from: ABT Systems Ltd
Relnotes: yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D5720
from userpsace. Previously we could have triggered a panic by trying to
jump to a kernel address from userland as the trap handling code thought we
received an ast in kernel mode.
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
Need this for gem5, but was not needed on real hadrware (yet) as it
was always MSI.
Reviewed by: andrew, jhb
Discovered by: andrew
Sponsored by: DARPA/AFRL
Differential Revision: https://reviews.freebsd.org/D5494
Enable system register access for EL2. Alpine-V2 is
the first device requiring this to be enabled.
It is also in-sync with Linux initialization code,
and compatible with Alpine-V2 uboot requirements.
Obtained from: Semihalf
Submitted by: Michal Stanek <mst@semihalf.com>
Sponsored by: Annapurna Labs
Approved by: cognet (mentor)
Reviewed by: wma
Differential revision: https://reviews.freebsd.org/D5394
It can be used to bind specific interrupt to a particular CPU.
Requires PIC support for interrupts binding.
Reviewed by: wma
Obtained from: Semihalf
Sponsored by: Cavium
Differential Revision: https://reviews.freebsd.org/D5122
number of physical memory locations we can access. This is the case on
some HiKey boards that may have UEFI reserved memory dispersed through the
physical space.
Sponsored by: ABT Systems Ltd
the processor and debug state registers. A flag has been added to the pcb
to tell us when to enable single stepping for a given thread.
Reviewed by: kib
Sponsored by: ABT Systems Ltd
Differential Revision: https://reviews.freebsd.org/D4730
Provide an easy to use framework for ARM64 DDB disassembler.
This commit does not contain full list of instruction opcodes.
Obtained from: Semihalf
Sponsored by: Cavium
Approved by: cognet (mentor)
Reviewed by: zbb, andrew, cognet
Differential revision: https://reviews.freebsd.org/D5114
tree parsing opt-out rather than opt-in. All FDT-based systems as well as
PowerPC systems with real Open Firmware use the CHRP-derived binding that
includes it, which makes SPARC the odd man out here. Making it opt-out
avoids astonishment on new platform bring up.
information on what the core supports. In most cases these will be
identical across most CPUs in the SoC, however there may be the case where,
with a big.LITTLE setup they may differ. In this case we print the
decoded data on all CPUs.
Reviewed by: kib
Sponsored by: ABT Systems Ltd
Differential Revision: https://reviews.freebsd.org/D4725
possible future CPU extentions with larger registers.
jmp_buf's size and alignment are baked into the ABI of third party libraries
and thus are very hard to change later so it is best to waste a small amount
of space now.
Reviewed by: brooks
Differential Revision: https://reviews.freebsd.org/D3956
the #address-cells property set. For this we need to read more data before
the parent interrupt description.
this is only enabled on arm64 for now as it's not quite compliant with the
ePAPR spec. We should use a default of 2 where the #address-cells property
is missing, however this will need further testing across architectures.
Obtained from: ABT Systems Ltd
Sponsored by: SoftIron Inc
Differential Revision: https://reviews.freebsd.org/D4518
clock_gettime(2) on ARMv7 and ARMv8 systems which have architectural
generic timer hardware. It is similar how the RDTSC timer is used in
userspace on x86.
Fix a permission problem where generic timer access from EL0 (or
userspace on v7) was not properly initialized on APs.
For ARMv7, mark the stack non-executable. The shared page is added for
all arms (including ARMv8 64bit), and the signal trampoline code is
moved to the page.
Reviewed by: andrew
Discussed with: emaste, mmel
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D4209
atomic functions where they are almost identical, or have acquire/release
semantics.
While here clean these function up. The cbnz instruction doesn't change
the condition flags so drop cc, however they should have memory added to the
clobber list.
Reviewed by: kib
Sponsored by: ABT Systems Ltd
Differential Revision: https://reviews.freebsd.org/D4318
place physical memory at an address outside the old DMAP range. This is an
issue as we rely on being able to move from PA -> VA using this range.
Obtained from: Patrick Wildt <patrick@bitrig.org> (earlier version)
Sponsored by: ABT Systems Ltd
Differential Revision: https://reviews.freebsd.org/D3885
the dynamic linker copy them, but not relocate them at the new location.
This allows us to run sqlite3 without it crashing.
Sponsored by: ABT Systems Ltd
boot on an SoC that places physical memory at an address past where three
levels of page tables can access in an identity mapping.
Submitted by: Wojciech Macek <wma@semihalf.com>,
Patrick Wildt <patrick@bitrig.org>
Differential Revision: https://reviews.freebsd.org/D3885 (partial)
Differential Revision: https://reviews.freebsd.org/D3744
in unknown state per spec.
Reviewed by: andrew (previous version)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3668
Currently FreeBSD supports only single PIC controller. Some systems
that have more than one (like ThunderX dual-socket) fails to boot.
Disable other PICes until proper handling is implemented in the
generic interrupt code.
Reviewed by: imp
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3682
Increase MAXCPU number to the maximum known value the existing
hardware can support.
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3405
Introduce supprot for SMP to GICv3 and ITS drivers.
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3299
in savectx where it will be used to store the current state however will
pass in a pcb when vfp_save_state expected a thread pointer.
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
This is copied from the amd64 version with minor changes. These should be
merged into a single file as from a quick look there are other copies of
the same file in other parts of the tree.
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
Use Vritual Counter register associated with Generic Timer to
read the cyclecount.
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3134
Add a method to identify CPU based on RAW MIDR value.
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3117
Previous DMAP size was too small for systems with more than 64GB
of RAM. Increase it to 128GB to support ThunderX CRB.
Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3113
This commit adds proper cache and shareability attributes to
the TCR register.
Set memory attributes to Normal, outer and inner cacheable WBWA.
Set shareability to inner and outer shareable when SMP is enabled.
Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3093
ACPI driver requires special functions to be provided by machdep code.
Add temporary stubs to satisfy the compiler when both "pci" and "acpi"
are enabled in the kernel configuration file.
Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3028
and psci to start them. I expect ACPI support to be added later.
This has been tested on qemu with 2 cpus as that is the current value of
MAXCPUS. This is expected to be increased in the future as FreeBSD has
been tested on 48 cores on the Cavium ThunderX hardware.
Partially based on a patch from Robin Randhawa from ARM.
Approved by: ABT Systems Ltd
Relnotes: yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3024
include this file without first including the headers needed for uint32_t
and the like use the __foo type.
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
This commit reworks the code responsible for identification of
the CPUs during runtime.
It is necessary to provide a way for workarounds and erratums
to be applied only for certain HW versions.
The copy of MIDR is now stored in pcpu to provide a fast and
convenient way for assambly code to read it (pcpu is used quite often
so there is a chance it's inside the cache).
The MIDR is also better way of identification than using user-friendly
cpu_desc structure, because it can be compiled into comparision of
single u32 with only one access to the memory - this is crucial
for some erratums which are called from performance-critical
places.
Changes in cpu_identify makes this function safe to be called
on non-boot CPUs.
New function CPU_MATCH was implemented which returns boolean
value based on mathing masked MIDR with chip identification.
Example of usage:
printf("is thunder: %d\n", CPU_MATCH(CPU_IMPL_MASK | CPU_PART_MASK,
CPU_IMPL_CAVIUM, CPU_PART_THUNDER, 0, 0));
printf("is generic: %d\n", CPU_MATCH(CPU_IMPL_MASK | CPU_PART_MASK,
CPU_IMPL_ARM, CPU_PART_FOUNDATION, 0, 0));
Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3030
provide a semantic defined by the C11 fences with corresponding
memory_order.
atomic_thread_fence_acq() gives r | r, w, where r and w are read and
write accesses, and | denotes the fence itself.
atomic_thread_fence_rel() is r, w | w.
atomic_thread_fence_acq_rel() is the combination of the acquire and
release in single operation. Note that reads after the acq+rel fence
could be made visible before writes preceeding the fence.
atomic_thread_fence_seq_cst() orders all accesses before/after the
fence, and the fence itself is globally ordered against other
sequentially consistent atomic operations.
Reviewed by: alc
Discussed with: bde
Sponsored by: The FreeBSD Foundation
MFC after: 3 weeks
Add ARM ITS (Interrupt Translation Services) support required
to bring-up message signalled interrupts on some ARM64 platforms.
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
On other architectures floatingpoint.h is a symlink to
machine/floatingpoint.h which in turn includes machine/ieeefp.h.
Do this on arm64 as well for now.
Sponsored by: The FreeBSD Foundation
register bits. Nothing in base uses these as they are deprecated, however
third-party applications, such as perl, expect some of these functions to
exist.
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
drivers, one for fdt, one for acpi. It then uses this to decide if it will
use fdt or acpi.
The GICv2 (interrupt controller) and Generic Timer drivers have been
updated to handle both cases.
As this is early code we still need FDT to find the kernel console, and
some parts are still missing, including PCI support.
Differential Revision: https://reviews.freebsd.org/D2463
Reviewed by: jhb, jkim, emaste
Obtained from: ABT Systems Ltd
Relnotes: Yes
Sponsored by: The FreeBSD Foundation
Using plain dsb()/dmb() as full system barriers is usually to much.
Adding proper options to those barriers (instead of full system - sy)
will most likely reduce the cost of the instructions and will benefit
in performance improvement.
This commit adds options to barrier macro definitions.
Obtained from: Semihalf
Reviewed by: andrew, ian
Sponsored by: The FreeBSD Foundation
GICv3 allows to distribute interrupts to more than 8 cores served by
the previous GIC revisions. GICv3 introduces additional logic in form
of Re-Distributors associated with particular CPUs to determine
the highest priority interrupts and manage PPIs and LPIs
(Locality-specific Peripheral Interrupts). Interrupts routing is
based on CPUs' affinity numbers. CPU interface was changed to be
accessible via CPU System Registers and this is the preferred
(and supported) method in this driver.
Obtained from: Semihalf
Reviewed by: andrew, emaste, ian, imp
Sponsored by: The FreeBSD Foundation
The x86 busdma subsystem allows using multiple implementations.
By default the classic bounce buffer approach is used, however
on systems with IOMMU it could be in runtime switched to more
efficient hardware accelerated implementation.
This commit adds ARM64 port of the x86 busdma framework and bounce
buffer backend. It is ready to use on IO coherent systems. If the
IO coherency cannot be guaranteed, the cache management operations have
to be added to this code in places marked by /* XXX ARM64TODO (...) */
comments. Also IOMMU support might be added by registering another
busdma implementation like it is already done on the x86.
Reviewed by: andrew, emaste
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
This is only the minimum set of files needed to boot in qemu. As such it is
missing a few things.
The bus_dma code is currently only stub functions with a full implementation
from the development tree to follow.
The gic driver has been copied as the interrupt framework is different. It
is expected the two drivers will be merged by the arm intrng project,
however this will need to be imported into the tree and support for arm64
would need to be added.
This includes code developed by myself, SemiHalf, Ed Maste, and Robin
Randhawa from ARM. This has been funded by the FreeBSD Foundation, with
early development by myself in my spare time with assistance from Robin.
Differential Revision: https://reviews.freebsd.org/D2199
Reviewed by: emaste, imp
Relnotes: yes
Sponsored by: The FreeBSD Foundation
