clock_gettime(2) on ARMv7 and ARMv8 systems which have architectural
generic timer hardware. It is similar how the RDTSC timer is used in
userspace on x86.
Fix a permission problem where generic timer access from EL0 (or
userspace on v7) was not properly initialized on APs.
For ARMv7, mark the stack non-executable. The shared page is added for
all arms (including ARMv8 64bit), and the signal trampoline code is
moved to the page.
Reviewed by: andrew
Discussed with: emaste, mmel
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D4209
unwound through. For this we need the frame pointer (x29) to point to the
location on the stack where we stored the previous frame pointer, and link
register. To simplify this the stack pointer is only adjusted by addition
and subtraction, and not through the use of post increment on loads and
stores.
The updated frame layout is:
+------------+
| lr -- x30 |
+------------+
| fp -- x29 | <-- x29 points at this
+------------+
| Trap frame |
| ... |
| | <-- sp points at this
+------------+
The only difference is the first two items, and setting of x29.
Sponsored by: ABT Systems Ltd
atomic functions where they are almost identical, or have acquire/release
semantics.
While here clean these function up. The cbnz instruction doesn't change
the condition flags so drop cc, however they should have memory added to the
clobber list.
Reviewed by: kib
Sponsored by: ABT Systems Ltd
Differential Revision: https://reviews.freebsd.org/D4318
prints the trap frame, along with the exception syndrome and fault address
registers. Even though esr is 64-bits here it is only 32-bits in hardware
so only print the valid 32-bits.
While here also print esr and far when appropriate after printing the trap
frame.
Sponsored by: ABT Systems Ltd
sysent.
sv_prepsyscall is unused.
sv_sigsize and sv_sigtbl translate signal number from the FreeBSD
namespace into the ABI domain. It is only utilized on i386 for iBCS2
binaries. The issue with this approach is that signals for iBCS2 were
delivered with the FreeBSD signal frame layout, which does not follow
iBCS2. The same note is true for any other potential user if
sv_sigtbl. In other words, if ABI needs signal number translation, it
really needs custom sv_sendsig method instead.
Sponsored by: The FreeBSD Foundation
No functional change as 'struct resource *' and 'struct resource **'
have the same size, but the former is the proper type.
PR: 204768
Submitted by: David Binderman
place physical memory at an address outside the old DMAP range. This is an
issue as we rely on being able to move from PA -> VA using this range.
Obtained from: Patrick Wildt <patrick@bitrig.org> (earlier version)
Sponsored by: ABT Systems Ltd
Differential Revision: https://reviews.freebsd.org/D3885
created for bus_dma_tag_t tag, bounce pages should be allocated
only if needed.
Before the fix, they were allocated always if BUS_DMA_COULD_BOUNCE flag
was set but BUS_DMA_MIN_ALLOC_COMP not. As bounce pages are never freed,
it could cause memory exhaustion when a lot of such tags together with
their maps were created.
Note that there could be more maps in one tag by current design.
However BUS_DMA_MIN_ALLOC_COMP flag is tag's flag. It's set after
bounce pages are allocated. Thus, they are allocated only for first
tag's map which needs them.
Approved by: kib (mentor)
the map has been created via bounce_bus_dmamem_alloc(). In that case
bus_dmamap_unload(9) typically isn't called during normal operation
but still should be during detach, cleanup from failed attach etc.
Submitted by: yongari
MFC after: 3 days
map has been created via bounce_bus_dmamem_alloc(). Even for coherent
DMA - which bus_dmamem_alloc(9) typically is used for -, calling of
bus_dmamap_sync(9) isn't optional.
PR: 188899 (non-original problem)
MFC after: 3 days
This will enable the elimination of a workaround in the USB driver that
artifically allocates buffers twice as big as they need to be (which
actually saves memory for very small buffers on the buggy platforms).
When deciding how to allocate a dma buffer, armv4, armv6, mips, and
x86/iommu all correctly check for the tag alignment <= maxsize as enabling
simple uma/malloc based allocation. Powerpc, sparc64, x86/bounce, and
arm64/bounce were all checking for alignment < maxsize; on those platforms
when alignment was equal to the max size it would fall back to page-based
allocators even for very small buffers.
This change makes all platforms use the <= check. It should be noted that
on all platforms other than arm[v6] and mips, this check is relying on
undocumented behavior in malloc(9) that if you allocate a block of a given
size it will be aligned to the next larger power-of-2 boundary. There is
nothing in the malloc(9) man page that makes that explicit promise (but the
busdma code has been relying on this behavior all along so I guess it works).
Arm and mips code uses the allocator in kern/subr_busdma_buffalloc.c, which
does explicitly implement this promise about size and alignment. Other
platforms probably should switch to the aligned allocator.
the dynamic linker copy them, but not relocate them at the new location.
This allows us to run sqlite3 without it crashing.
Sponsored by: ABT Systems Ltd
Internal busses (thus ECAM access) should be mapped to
all values from 0 to 143.
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D3753
When one tries to allocate a resource with unspecified range,
read already configured BAR values (by UEFI or whatever).
This is necessary to make VNIC VFs working and to allow them to be
properly allocated.
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D3752
From the (now removed) comment:
* It is unclear in some cases if the bit is implementation defined.
* The Foundation Model and QEMU disagree on if the IL bit should
* be set when we are in a data fault from the same EL and the ISV
* bit (bit 24) is also set.
Instead of adding even more special cases just remove the assertion.
Approved by: andrew
Sponsored by: The FreeBSD Foundation
It turns out that it is pretty easy to make CloudABI work on ARM64. We
essentially only need to copy over the sysvec from AMD64 and ensure that
we use ARM64 specific registers.
As there is an overlap between function argument and return registers,
we do need to extend cloudabi64_schedtail() to only set its values if
we're actually forking. Not when we're creating a new thread.
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D3917
boot on an SoC that places physical memory at an address past where three
levels of page tables can access in an identity mapping.
Submitted by: Wojciech Macek <wma@semihalf.com>,
Patrick Wildt <patrick@bitrig.org>
Differential Revision: https://reviews.freebsd.org/D3885 (partial)
Differential Revision: https://reviews.freebsd.org/D3744
in through the stack pointer, however this may have been misaligned
causing some userland applications to crash. A workaround was committed in
r284707 where userland would check if the aux vector was passed using the
old or new ABI and adjust the stack if needed. As 4 months have passed it
is time to move to the new ABI, with the expectation the compat code in csu
and the runtime linker to be removed in the future.
Sponsored by: ABT Systems Ltd
an extra argument to specify the number of 1GiB pages to map. This should
be a nop as we are only mapping a single page, but when we move to use an
extra level of page tables we will be able to map a second block, e.g. if
the kernel was loaded over a 1GiB boundary.
While trying to get multithreading working for CloudABI on aarch64, I
noticed that compare-and-exchange operations in kernelspace would always
fail. It turns out that we don't properly set the return value to 0 when
the compare and exchange succeeds.
Approved by: andrew
Differential Revision: https://reviews.freebsd.org/D3899
- Move the required kernel compiler flags from Makefile.arm64 to kern.mk.
- Build arm64 modules as PIC; non-PIC relocations in .o for shared object
output cannot be handled.
- Do not try to install aarch64 symlink.
- A hack for arm64 to avoid ld -r stage. See the comment for the explanation.
Some functionality is lost, like ctf handling, but hopefully will be
restored after newer linker is available.
Reviewed by: andrew, emaste
Tested by: andrew (on real hardware)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3796
It is decided to go with the shared object file format for modules on
arm64, due to the Aarch64 instruction set details. Combination of the
signed 28-bit offset in the branch instructions encoding together with
the supported memory model of compilers makes the relocatable object
support impossible or at least too hard.
Reviewed by: andrew, emaste
Tested by: andrew (on real hardware)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3796
belong to a vm object, they can't be paged out. Since they can't be paged
out, they are never enqueued in a paging queue. Nonetheless, passing
PQ_INACTIVE to vm_page_unwire() creates the appearance that these pages
are being enqueued in the inactive queue. As of r288122, we can avoid
this false impression by passing PQ_NONE.
Submitted by: kmacy (an earlier version)
Differential Revision: https://reviews.freebsd.org/D1674
reschedule. Right now arm_cpu_intr() does critical_exit() as the last
action, so the impact is not serious.
Remove duplicated interrupt disable in restore_registers macro, when
returning to usermode. The do_ast macro disabled interrupts for us.
Reviewed by: andrew
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3714
We should not call vm_fault(), or send a signal, with interrupts
disabled. MI kernel code is not prepared for such environment, not to
mention that this increases system latency, since code appears to be
executing as being under spinlock.
The FAR register for data aborts is read before the interrupts are
enabled, to avoid its corruption due to nested exception or context
switch.
Add asserts, similar to the checks done by other architectures, about
not taking page faults in non-sleepable contexts, rather than die with
late and somewhat confusing witness diagnostic.
Reviewed by: andrew
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3669
in unknown state per spec.
Reviewed by: andrew (previous version)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3668
Currently FreeBSD supports only single PIC controller. Some systems
that have more than one (like ThunderX dual-socket) fails to boot.
Disable other PICes until proper handling is implemented in the
generic interrupt code.
Reviewed by: imp
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3682
cpu_init_fdt will now release memory allocated for structures
serving CPUs that have failed to init.
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3297
running thread.
It is currently implemented only on amd64 and i386; on these
architectures, it is implemented by raising an NMI on the CPU on which
the target thread is currently running. Unlike stack_save_td(), it may
fail, for example if the thread is running in user mode.
This change also modifies the kern.proc.kstack sysctl to use this function,
so that stacks of running threads are shown in the output of "procstat -kk".
This is handy for debugging threads that are stuck in a busy loop.
Reviewed by: bdrewery, jhb, kib
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D3256
The only operation which is prevented by the hold is the kernel stack
swapout for the faulted thread, which should be fine to allow.
Remove useless checks for NULL curproc or curproc->p_vmspace from the
trap_pfault() wrappers on x86 and powerpc.
Reviewed by: alc (previous version)
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
* Fail when the length passed in is 0
* Remove an unneeded increment of the count on success
* Return ENAMETOOLONG when the input pointer is too long
Sponsored by: ABT Systems Ltd
store should have release semantics and will have due to the dsb above it
so add a comment to explain this. [1]
While here update the code to not reload the current thread, it's already
in a register, we just need to not trash it.
Suggested by: kib [1]
Sponsored by: ABT Systems Ltd
hardware to perform address translation for us. These are useful to help
track down what caused us to enter the debugger.
Sponsored by: ABT Systems Ltd
SoC is used in the HiKey board from 96boards.
Currently on the SD card is working on the HiKey, as such devices 0 and 2
will need to be disabled, for example by adding the following to
loader.conf:
hint.hisi_dwmmc.0.disabled=1
hint.hisi_dwmmc.2.disabled=1
Relnotes: yes (Hikey board booting)
Sponsored by: ABT Systems Ltd
its_cmd_send() can be called by multiple CPUs simultaneously.
After the command is pushed to ITS command ring the completion
status is polled using global pointer to the next free ring slot.
Use copied pointer and provide correct locking to avoid spurious
pointer value when concurrent access occurs.
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3436
be used with any SoC specific drivers, for example a ThunderX nic driver
would use something like the following in files.arm64:
arm64/cavium/thunder_nic.c optional soc_cavm_thunderx thndr_nic
Reviewed by: imp
Sponsored by: ABT Systems Ltd
Differential Revision: https://reviews.freebsd.org/D3479
correctly handle trying to access an invalid address in the debugger.
While here document that the breakpoint handler is supposed to fall
through to the following case.
Sponsored by: ABT Systems Ltd
Increase MAXCPU number to the maximum known value the existing
hardware can support.
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3405
The global variable has been only used for CPU startup ordering
which is not needed anyway.
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3296
Introduce supprot for SMP to GICv3 and ITS drivers.
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3299
An exception with an unknown reasion is the expected result of the
attempted execution of an instruction bit pattern that has no allocated
instruction.
Sponsored by: The FreeBSD Foundation
It seems we get EXCP_UNKNOWN from QEMU when executing zeroed memory.
Print a register dump here and signal illegal instruction. Also print
a register dump for other invalid exceptions, before panic.
Reviewed by: andrew
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3370
vm_offset_t pmap_quick_enter_page(vm_page_t m)
void pmap_quick_remove_page(vm_offset_t kva)
These will create and destroy a temporary, CPU-local KVA mapping of a specified page.
Guarantees:
--Will not sleep and will not fail.
--Safe to call under a non-sleepable lock or from an ithread
Restrictions:
--Not guaranteed to be safe to call from an interrupt filter or under a spin mutex on all platforms
--Current implementation does not guarantee more than one page of mapping space across all platforms. MI code should not make nested calls to pmap_quick_enter_page.
--MI code should not perform locking while holding onto a mapping created by pmap_quick_enter_page
The idea is to use this in busdma, for bounce buffer copies as well as virtually-indexed cache maintenance on mips and arm.
NOTE: the non-i386, non-amd64 implementations of these functions still need review and testing.
Reviewed by: kib
Approved by: kib (mentor)
Differential Revision: http://reviews.freebsd.org/D3013
When doing a data abort from userland it is possible to get
more than one data abort inside the same exception level.
Add an appropriate exception number to allow nesting of
data_abort handler for EL0.
Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3276
in savectx where it will be used to store the current state however will
pass in a pcb when vfp_save_state expected a thread pointer.
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
unwind_frame function to read each stack frame until either the pc or stack
are no longer withing the kernel's address space.
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
This is copied from the amd64 version with minor changes. These should be
merged into a single file as from a quick look there are other copies of
the same file in other parts of the tree.
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
when trying to read data from outside the DMAP region. I expect this panic
to be from within uiomove_fromphys, which needs to grow support to support
such addresses.
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
FreeBSD provides a feature called Adaptive Mutexes, which allows
a thread to spin for a while when the mutex is taken instead of
immediately going to sleep. This causes issues when called from
syscall handler if interrupts are masked. If every other core
also attempts to access the same mutex there is a chance that
all of them are spinning on the same lock at the same time.
If interrupts are disabled, no kernel preemtion can occur and
the system becomes unresponsive.
This patch enables interrupts when syscall is being executed
and masks them as soon as it is completed.
Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3246
Limit the number of supported device IDs to 0x100000
in order to decrease the size of the ITS device table so
that it matches with the HW capabilities.
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3131
Add brief commentary to vendor-specific devid function in ITS
and remove redundant spaces by the way.
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
malloc() should not go to sleep in case of lack of resource while
the kernel thread is holding a non-sleepable lock.
- change malloc() flags to M_NOWAIT in such cases implement
lpi_free_chunk() routine as it will be needed when ITT
allocation fails in its_device_alloc_locked()
- do not increase verbosity of this code since upper layers will
communicate an error if the interrupt setup fails
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3121
It is possible that some HW will use different PCI devids,
hence allow to replace the default domain🚌slot:func schema
by implementing and registering custom function.
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3118
Use Vritual Counter register associated with Generic Timer to
read the cyclecount.
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3134
Add a method to identify CPU based on RAW MIDR value.
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3117
Basing on B.2.3.4:
Synchronization and coherency issues between data and
instruction accesses.
To ensure that modified instructions are visible to all PEs
(Processing Elements) in a shareability domain one need to
perform following sequence:
1. Clean D-cache
2. Ensure the visibility of data cleaned from cache
3. Invalidate I-cache
4. Ensure completion
5. In SMP system PE must issue isb to ensure execution of the
modified instructions
Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3106
Secondary stack calculation is modified to provide
stack_top = secondary_stacks + (cpu_id) * PAGE_SIZE * KSTACK_PAGES
because on ARM64 the stack grows to lower memory addresses.
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3107
Previous DMAP size was too small for systems with more than 64GB
of RAM. Increase it to 128GB to support ThunderX CRB.
Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3113
If KSTACK_PAGES was changed to anything alse than the default,
the value from param.h was taken instead in some places and
the value from KENRCONF in some others. This resulted in
inconsistency which caused corruption in SMP envorinment.
Ensure all places where KSTACK_PAGES are used the opt_kstack_pages.h
is included.
The file opt_kstack_pages.h could not be included in param.h
because was breaking the toolchain compilation.
Reviewed by: kib
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3094
This commit adds proper cache and shareability attributes to
the TCR register.
Set memory attributes to Normal, outer and inner cacheable WBWA.
Set shareability to inner and outer shareable when SMP is enabled.
Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3093
Identify current CPU. This is necessary to setup
affinity registers and to provide support for
runtime chip identification.
Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3095
Call arm_init_secondary before any other PIC-related functions
are called. This is necessary for GICv3 where PIC_INIT_SECONDARY
allocates resources needed for all further operations.
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3066
On ARMv8 IPIs are mapped to 0-15. Incrementing the number by 16
is wrong, because it sets a reserved bit in the IPI register.
This patch removes all "+16" to comply with specs.
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3029
ACPI driver requires special functions to be provided by machdep code.
Add temporary stubs to satisfy the compiler when both "pci" and "acpi"
are enabled in the kernel configuration file.
Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3028
and psci to start them. I expect ACPI support to be added later.
This has been tested on qemu with 2 cpus as that is the current value of
MAXCPUS. This is expected to be increased in the future as FreeBSD has
been tested on 48 cores on the Cavium ThunderX hardware.
Partially based on a patch from Robin Randhawa from ARM.
Approved by: ABT Systems Ltd
Relnotes: yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3024
include this file without first including the headers needed for uint32_t
and the like use the __foo type.
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
This commit reworks the code responsible for identification of
the CPUs during runtime.
It is necessary to provide a way for workarounds and erratums
to be applied only for certain HW versions.
The copy of MIDR is now stored in pcpu to provide a fast and
convenient way for assambly code to read it (pcpu is used quite often
so there is a chance it's inside the cache).
The MIDR is also better way of identification than using user-friendly
cpu_desc structure, because it can be compiled into comparision of
single u32 with only one access to the memory - this is crucial
for some erratums which are called from performance-critical
places.
Changes in cpu_identify makes this function safe to be called
on non-boot CPUs.
New function CPU_MATCH was implemented which returns boolean
value based on mathing masked MIDR with chip identification.
Example of usage:
printf("is thunder: %d\n", CPU_MATCH(CPU_IMPL_MASK | CPU_PART_MASK,
CPU_IMPL_CAVIUM, CPU_PART_THUNDER, 0, 0));
printf("is generic: %d\n", CPU_MATCH(CPU_IMPL_MASK | CPU_PART_MASK,
CPU_IMPL_ARM, CPU_PART_FOUNDATION, 0, 0));
Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3030
provide a semantic defined by the C11 fences with corresponding
memory_order.
atomic_thread_fence_acq() gives r | r, w, where r and w are read and
write accesses, and | denotes the fence itself.
atomic_thread_fence_rel() is r, w | w.
atomic_thread_fence_acq_rel() is the combination of the acquire and
release in single operation. Note that reads after the acq+rel fence
could be made visible before writes preceeding the fence.
atomic_thread_fence_seq_cst() orders all accesses before/after the
fence, and the fence itself is globally ordered against other
sequentially consistent atomic operations.
Reviewed by: alc
Discussed with: bde
Sponsored by: The FreeBSD Foundation
MFC after: 3 weeks
Add ARM ITS (Interrupt Translation Services) support required
to bring-up message signalled interrupts on some ARM64 platforms.
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
