Replace tlb_flush_local() by tlb_flush() as even not global mappings
could be fetched to TLB(s) on other cores by speculative table walk.
From OS point of view, it was not a problem as either such mappings
were not used anymore or they were flushed from TLB(s) when reused.
However, from hardware point of view, it was a problem. Not flushed
mappings could be a target for speculative reads or prefetches (which
might be quite aggresive on ARM cores). As speculative read can fill
cacheline, it can cause a real problem, when physical page is reused,
but mapped with different memory attributes.
Anyhow, it's good to have only valid mappings in TLB(s).
Approved by: kib (mentor)
This structure must be binary compatible regardless of PMAP
version being used. Create reserved section for NEW_PMAP to
make other variables be placed exactly in the same memory
addresses. This fixes kgdb/gdb behavoiur, which uses pcb.h stuctures.
The NEW_PMAP is kernel flag, so it does not propagate to the buildworld,
what makes the tools using pcb.h unable to parse PCB data.
Reviewed by: mmel, kib
Submitted by: Wojciech Macek <wma@semihalf.com>
Obtained from: Semihalf
Sponsored by: Juniper Networks Inc.
Differential Revision: https://reviews.freebsd.org/D4011
Allow manipulation with PSR_A bit on ARMv6+.
Remove declaration of unused functions.
This effectively enables asynchronous aborts on early bootstrap stage,
which previously was not enabled due to an error in enable_interrupts().
PR: 201434
Reported by: Gregory Soutade <soutade at gmail.com>
Approved by: kib (mentor)
function which checks an address for privileged (PL1) write access.
The function is inlined so it does not bring any cost, but makes
function set for checking privileged access complete.
Approved by: kib (mentor)
little-endian configuration for 64-bit variant is supported.
Reviewed by: mmel
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D4113
as of r288992 use it to manage the CCNT.
Use the CNNT for get_cyclecount() instead of binuptime() when device pmu
is compiled in; if it fails to attach, fall back to the former method.
Enable by default for the BeagleBoneBlack configuration.
Optained from: Cambridge/L41
Sponsored by: DARPA/AFRL
Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D3837
In the old days, device drivers passed NULL for the parent tag when creating
a new tag, and on arm platforms that resulted in a global tag representing
overall platform constraints being substituted in the busdma code. Now all
drivers use bus_get_dma_tag() and if there is a need to represent overall
platform constraints they will be inherited from a tag supplied by nexus or
some bus driver in the hierarchy.
The only arm platforms still relying on the old global-tag scheme were some
xscale boards with special PCI-bus constraints. This change provides those
constraints through a tag supplied by the xscale PCI bus driver, and
eliminates the few remaining references to the old global var.
Reviewed by: cognet
Previous code supported only "continuous" code without any kind of
branch instructions. To change that, new function was implemented
which parses current instruction and returns an addres where
the jump might happen (alternative addr).
mdthread structure was extended to support two breakpoints
(one directly below current instruction and the second placed
at the alternative location).
One of them must trigger regardless the instruction has or has not been
executed due to condition field.
Upon cleanup, both software breakpoints are removed.
This implementation parses only the most common instructions
that are present in the code (like 99.99% of all), but there
is a chance there are some left, not covered by the parsing routine.
Parsing is done only for 32-bit instruction, no Thumb nor Thumb-2
support is provided.
Reviewed by: kib
Submitted by: Wojciech Macek <wma@semihalf.com>
Obtained from: Semihalf
Sponsored by: Juniper Networks Inc.
Differential Revision: https://reviews.freebsd.org/D4021
is a dcache invalidate to point of coherency just like dcache_inv_poc(), but
a slightly different version specific to dma operations. Elaborate the
comment about how and why it's different.
Similar to r286787 for x86, this treats userspace buffers the same as unmapped buffers and no longer borrows the UVA for sync operations.
Submitted by: Svatopluk Kraus <onwahe@gmail.com> (earlier revision)
Tested by: Svatopluk Kraus
Differential Revision: https://reviews.freebsd.org/D3869
The bits in the aux control register vary based on the processor type. In
the past we've always just set the 'smp' and "broadcast tlb/cache ops' bits,
which worked fine for the first few SoCs we supported. Now that we support
most of the cortex-a series processors, it's important to get the right bits
set based on the processor type.
Submitted by: Svatopluk Kraus <onwahe@gmail.com>
The interrupts-extended property is a list of controller-specific
interrupt tuples for more than one controller. The decode routine of
every PIC gets called in the pre-INTRNG code (nexus doesn't know which
device instance belongs to which fdt node), so the GIC code has to
check each FDT node it is asked to decode to ensure it is the owner.
Because in the pre-INTRNG world there can only be one instance of a GIC,
it's safe to cache the results of a positive lookup in a static variable
to avoid the expensive lookups on subsequent calls.
Submitted by: Svatopluk Kraus <onwahe@gmail.com>
Differential Revision: https://reviews.freebsd.org/D2345
and armv6 architecures. The primary enhancement over the old design is
support for hierarchical interrupt controllers (such as a gpio driver
which can receive interrupts from a root PIC and act as a PIC itself for
clients interested in handling a change of gpio pin state as an
interrupt). The new code also provides an infrastructure for mapping
interrupts described in metadata in the form of a "controller reference
plus interrupt number" tuple into the simple "0-n" flat numeric space
understood by rman and the bus resource mechanisms.
Use of the new code is enabled by setting the ARM_INTRNG option, and by
making a few simple changes to the platform's support code. In addition
each existing PIC driver needs changes to be ready for INTRNG; this commit
contains the changes for the arm/gic driver, which most armv6 SoCs use, but
it does not enable the new code yet on any platform.
This project has been many years in the making, starting as a GSoC project
by Jakub Klama (jceel@) in 2012. That didn't get committed right away and
the source base evolved out from under it to some degree. In 2014 I rebased
the diffs to then -current and did some enhancements in the area of mapping
interrupt numbers and storing associated fdt data, then the project went
cold again for a while. Eventually Svata Kraus took that work in progress
and did another big round of work on it, removing most of the remaining
rough edges. Finally I took that and made one more pass through it, mostly
disabling the "INTR_SOLO" feature for now, pending further design
discussions on how to most efficiently dispatch a pending interrupt through
more than one layer of PIC. The current code with the INTR_SOLO feature
disabled uses approximate 100 extra cpu cycles for each cascaded PIC the
interrupt has to be passed to, so what's left to do is about efficiency, not
correct operation.
Differential Revision: https://reviews.freebsd.org/D2047
the name the function will have when the new ARM_INTRNG code is integrated,
and doing this rename first will make it easier to toggle the new interrupt
handling code on/off with a config option for debugging.
casuword(9) and others, use LDRT and STRT instructions to access
memory with the privileges of userspace. If the *RT instruction
faults on the kernel address, then additional checks must be done to
not confuse the VM system with invalid kernel-mode faults.
Put ARM on line with other FreeBSD architectures and disallow usermode
buffers which intersect with the kernel address space in advance,
before any accesses are performed. In other words, vm_fault(9) is no
longer called when e.g. suword(9) stores to invalid (i.e. not
userspace) address.
Also, switch ARM to use fueword(9) and casueword(9).
Note: there is a pending patch in D3617, which adds the special
processing for faults from LDRT and STRT. The addition of the
processing is useful for potential other uses of the instructions and
for completeness, but standard userspace accessors are better served
by not allowing such faults beforehand.
Reviewed by: andrew
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3816
MFC after: 2 weeks
pre-VFPv3 processors, since they do require software support code to
handle denormals. For VFPv3 and later, enable flush-to-zero if
hardware does not claim full denormals arithmetic support by VMVFR1_FZ
field in mvfr1 register.
The end result is that we do use correct fpu environment on Cortexes
with VFPv3, while ARM11 (e.g. rpi) is in non-compliant flush-to-zero
mode. At least CPUs without complete hardware implementation of
IEEE 754 do not cause unhandled floating point exception on underflow,
as it was before r288492.
Noted by: ian
Tested by: gjb
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
ARMv6/7:
- Define _SAVE() macro to allow unwind data to be conditionally defined for
ARM assembly code in the kernel.
- Use _SAVE() to provide unwind information for bcopy_page(), and two (of
many) instances of copyin() and copyout().
Reviewed by: andrew, imp
MFC after: 3 days
Sponsored by: University of Cambridge
self-consistent, there is no need in anything but compiler barrier in
the implementation of atomic_thread_fence_*() on ARMv5. Split
implementation of fences for ARMv4/5 and ARMv6; the former use
compiler barriers, the later also perform hardware barriers.
An issue which is fixed by the change is the faults from the CP15
coprocessor accesses in the user mode. This was uncovered by the
pthread_once() changes in r287556.
Reported by: Mattia Rossi <mattia.rossi.mailinglists@gmail.com>
Discussed with: alc, cognet, jhb
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
initial thread stack is not adjusted by the tunable, the stack is
allocated too early to get access to the kernel environment. See
TD0_KSTACK_PAGES for the thread0 stack sizing on i386.
The tunable was tested on x86 only. From the visual inspection, it
seems that it might work on arm and powerpc. The arm
USPACE_SVC_STACK_TOP and powerpc USPACE macros seems to be already
incorrect for the threads with non-default kstack size. I only
changed the macros to use variable instead of constant, since I cannot
test.
On arm64, mips and sparc64, some static data structures are sized by
KSTACK_PAGES, so the tunable is disabled.
Sponsored by: The FreeBSD Foundation
MFC after: 2 week
vm_offset_t pmap_quick_enter_page(vm_page_t m)
void pmap_quick_remove_page(vm_offset_t kva)
These will create and destroy a temporary, CPU-local KVA mapping of a specified page.
Guarantees:
--Will not sleep and will not fail.
--Safe to call under a non-sleepable lock or from an ithread
Restrictions:
--Not guaranteed to be safe to call from an interrupt filter or under a spin mutex on all platforms
--Current implementation does not guarantee more than one page of mapping space across all platforms. MI code should not make nested calls to pmap_quick_enter_page.
--MI code should not perform locking while holding onto a mapping created by pmap_quick_enter_page
The idea is to use this in busdma, for bounce buffer copies as well as virtually-indexed cache maintenance on mips and arm.
NOTE: the non-i386, non-amd64 implementations of these functions still need review and testing.
Reviewed by: kib
Approved by: kib (mentor)
Differential Revision: http://reviews.freebsd.org/D3013
provide a semantic defined by the C11 fences with corresponding
memory_order.
atomic_thread_fence_acq() gives r | r, w, where r and w are read and
write accesses, and | denotes the fence itself.
atomic_thread_fence_rel() is r, w | w.
atomic_thread_fence_acq_rel() is the combination of the acquire and
release in single operation. Note that reads after the acq+rel fence
could be made visible before writes preceeding the fence.
atomic_thread_fence_seq_cst() orders all accesses before/after the
fence, and the fence itself is globally ordered against other
sequentially consistent atomic operations.
Reviewed by: alc
Discussed with: bde
Sponsored by: The FreeBSD Foundation
MFC after: 3 weeks
discrimination between different subarch binaries, at least for mips
and arm. Arm is implemented, mips is still tbd, so not currently
exported. aarch64 does not export this because aarch64 binaries use
different tags and flags than arm.
Differential Revision: https://reviews.freebsd.org/D2611
1. Align to a 64-bit address so 64-bit data will be correctly aligned.
2. Add a comment explaining why.
3. Remove an unneeded value from the struct.
This fixes an issue where the struct may not be correctly aligned on the
stack in the syscall function. This may lead to accesing a 64-bit value
at a non 64-bit. This will raise an exception and panic the kernel.
We have been lucky where on arm and armv6 both clang and gcc correctly
align the data, even without us asking to, however, on armeb with clang to
not be the case. This tells the compiler we really do need this to be
aligned.
Reported and tested by: jmg (on armeb with clang)
MFC after: 1 Week [1, 2]
Perform cache writebacks and invalidations in the correct (inner to outer
or vice versa) order, and add comments that explain that.
Consistantly use 'va' as the variable name for virtual addresses.
Submitted by: Michal Meloun <meloun@miracle.cz>
For consistency with the naming conventions used by the other
implementations kill armv7_sleep and keep armv7_cpu_sleep.
Differential Revision: https://reviews.freebsd.org/D2537
Submitted by: John Wehle
Reviewed by: ian@, andrew@
because the i386 pmap on which the new armv6 pmap is based had it, and in
r281707 pmap_lazyfix() was removed from the i386 pmap.
Discussed with: kib
Submitted by: Michal Meloun (via Svatopluk Kraus)
Offet for the power control register was specified incorrectly (it had
the same value as the prefetch control register.) This change corrects
the offset value to 0xF80, per the ARM PL310 documentation.
Submitted by: Steve Kiernan <stevek@juniper.net>
Obtained from: Juniper Networks, Inc.
Previously we used pmap_kremove(), but with ARM_NEW_PMAP it does the remove
in a way that isn't SMP-coherent (which is appropriate in some circumstances
such as mapping/unmapping sf buffers). With matching enter/remove routines
for device mappings, each low-level implementation can do the right thing.
Reviewed by: Svatopluk Kraus <onwahe@gmail.com>
the PMC_IN_KERNEL() macro definition.
Add missing macros to extract the return address (LR) from the trapframe.
Discussed with: andrew
Obtained from: Cambridge/L41
Sponsored by: DARPA, AFRL
MFC after: 2 weeks
This is pretty much a complete rewrite based on the existing i386 code. The
patches have been circulating for a couple years and have been looked at by
plenty of people, but I'm not putting anybody on the hook as having reviewed
this in any formal sense except myself.
After this has gotten wider testing from the user community, ARM_NEW_PMAP
will become the default and various dregs of the old pmap code will be
removed.
Submitted by: Svatopluk Kraus <onwahe@gmail.com>,
Michal Meloun <meloun@miracle.cz>
Each plaform performs virtual memory split between kernel and user space
and assigns kernel certain amount of memory space. However, is is sometimes
reasonable to change the default values. Such situation may happen on
systems where the demand for kernel buffers is high, many devices occupying
memory etc. This of course comes with the cost of decreasing user space
memory range so shall be used with care. Most embedded systems will not
suffer from this limtation but rather take advantage of this potential
since default behavior is left unchanged.
Submitted by: Wojciech Macek <wma@semihalf.com>
Reviewed by: imp
Obtained from: Semihalf
used by other places that expect to unwind the stack, e.g. dtrace and
stack(9).
As I have written most of this code I'm changing the license to the
standard FreeBSD license. I have received approval from the other
developers who have changed any of the affected code.
Approved by: ian, imp, rpaulo, eadler (all license change)
Switch the cache line size during invalidations/flushes
to be read from CP15 cache type register.
Submitted by: Wojciech Macek <wma@semihalf.com>
Reviewed by: ian, imp
Obtained from: Semihalf
the data the inline functions access together at the start of the bus_space
struct. The start-of part isn't so important, it's the grouping-together
that's the point: now all the most-accessed data should be in one cache line.
Suggested by: cognet
every operation to retrieve the bs_cookie value almost nothing actually uses.
The bus_space struct contains a private data pointer (poorly named bs_cookie,
now renamed to bs_privdata) which is used only by a few old armv4 xscale
implementations. The bus_space functions were all defined to take this
value as the first parameter instead of the bus_space_tag_t, requiring all
the inline macro and function expansions to dereference the tag to pass it
to another function, which never uses it. Now all the functions take the tag
as the first parameter and retrieve the privdata if they need it.
Also fix a couple bus_space_unmap() implementations that were calling
kva_free() instead of pmap_unmapdev().
Discussed with: cognet
that some #ifdef SMP code is also conditional on __ARM_ARCH >= 7; we don't
support SMP on armv6, but some drivers and modules are compiled with it
forced on via the compiler command line.
code in sys/kern/kern_dump.c. Most dumpsys() implementations are nearly
identical and simply redefine a number of constants and helper subroutines;
a generic implementation will make it easier to implement features around
kernel core dumps. This change does not alter any minidump code and should
have no functional impact.
PR: 193873
Differential Revision: https://reviews.freebsd.org/D904
Submitted by: Conrad Meyer <conrad.meyer@isilon.com>
Reviewed by: jhibbits (earlier version)
Sponsored by: EMC / Isilon Storage Division
mostly paves the way for the new pmap code, and shouldn't result in any
noticible behavior differences.
Submitted by: Svatopluk Kraus <onwahe@gmail.com>,
Michal Meloun <meloun@miracle.cz
The ancient gas we've been using interprets .align 0 as align to the
minimum required alignment for the current section. Clang's integrated
assembler interprets it as align to a byte boundary. Fortunately both
assemblers interpret a non-zero value as align to 2^N so just make sure
we have appropriate non-zero values everywhere.
The elftoolchain project includes these additional defines for various
userland programs. Given that arch-specific defines are still interesting
in the context of userland programs reading or writing ELF metadata, they
should be included in top-level ELF headers.
Remove duplicate defines from ARM and MIPS elf headers.
Submitted by: will (initial version)
Reviewed by: imp, will
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D844
are inline functions that handle all the routine maintenance operations
except the flush-all and invalidate-all routines which are required only
during early kernel init.
These inline functions should be very much faster than the old mechanism
that involved jumping through the big cpufuncs table, especially for
common operations such as invalidating a single TLB entry. Note that
nothing is calling these yet, this just is just required infrastructure
for upcoming changes to the pmap-v6 code.
mechanism defined for armv7 (and also present on some armv6 chips including
the arm1176 used on rpi). The information is parsed into a global cpuinfo
structure, which will be used by (upcoming) new cache and tlb maintenance
code to handle cpu-specific variations of the maintence sequences.
Submitted by: Svatopluk Kraus <onwahe@gmail.com>,
Michal Meloun <meloun@miracle.cz
code, passing a 0/1 flag that indicates which type of abort it was. This
sets the stage for unifying the handling of page faults in a single routine.
Submitted by: Svatopluk Kraus <onwahe@gmail.com>,
Michal Meloun <meloun@miracle.cz
If it seems like this is getting out of hand, I quite agree. I wonder if
it's safe, here in the 21st century, to lose the distinction between C and
ASM symbols?
around so that related things are more grouped together, rewrite comments.
No functional changes, this is all so that the functional changes in the
next commit will stand out.
'extra' entry points which are nested within or provide a synonym name
for another function. It's most likely not safe to be messing with the
IP and LR registers at anything other than the primary entry point to a
function. Anywhere beyond initial function entry, those registers may
be in use as scratch or variable registers.
- Eliminate unused irqframe
- Eliminate unused saframe
- Instead of splitting r4-sp storage between the stack and switchframe,
just put all the registers in switchframe and eliminate the un_32 struct.
Submitted by: Svatopluk Kraus <onwahe@gmail.com>,
Michal Meloun <meloun@miracle.cz>
If this feels like deja vu... the last time this was fixed in this file
only ARM_MMU_V6 was fixed, this time it's ARM_ARCH_V6 (and this time I
searched for other occurrances of pj4b in here).
It is automatically set when -fPIC is passed to the compiler.
Reviewed by: dim, kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D1179
and casuword(9), but do not mix value read and indication of fault.
I know (or remember) enough assembly to handle x86 and powerpc. For
arm, mips and sparc64, implement fueword() and casueword() as wrappers
around fuword() and casuword(), which means that the functions cannot
distinguish between -1 and fault.
On architectures where fueword() and casueword() are native, implement
fuword() and casuword() using fueword() and casuword(), to reduce
assembly code duplication.
Sponsored by: The FreeBSD Foundation
Tested by: pho
MFC after: 2 weeks (ia64 needs treating)
registers and use it in the ARMv7 CPU functions.
The sysreg.h file has been checked by hand, however it may contain errors
with the comments on when a register was first introduced. The ARMv7 cpu
functions have been checked by compiling both the previous and this version
and comparing the md5 of the object files.
Submitted by: Svatopluk Kraus <onwahe at gmail.com>
Submitted by: Michal Meloun <meloun at miracle.cz>
Reviewed by: ian, rpaulo
Differential Revision: https://reviews.freebsd.org/D795
In the fdt data we've written for ourselves, the interrupt properties
for GIC interrupts have just been a bare interrupt number. In standard
data that conforms to the published bindings, GIC interrupt properties
contain 3-tuples that describe the interrupt as shared vs private, the
interrupt number within the shared/private address space, and configuration
info such as level vs edge triggered.
The new gic_decode_fdt() function parses both types of data, based on the
#interrupt-cells property. Previously, each platform implemented a decode
routine and put a pointer to it into fdt_pic_table. Now they can just
list this function in their table instead if they use arm/gic.c.
header (Elf_Ehdr) to determine if a particular interpretor wants to
accept it or not. Use this mechanism to filter EABI arm on OABI arm
kernels, and vice versa. This method could also be used to implement
OABI on EABI arm kernels, if desired, or to allow a single mips kernel
to run o32, n32 and n64 binaries.
Differential Revision: https://reviews.freebsd.org/D609
By Richard Earnshaw at ARM
>
>GCC has for a number of years provides a set of pre-defined macros for
>use with determining the ISA and features of the target during
>pre-processing. However, the design was always somewhat cumbersome in
>that each new architecture revision created a new define and then
>removed the previous one. This meant that it was necessary to keep
>updating the support code simply to recognise a new architecture being
>added.
>
>The ACLE specification (ARM C Language Extentions)
>(http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.set.swdev/index.html)
>provides a much more suitable interface and GCC has supported this
>since gcc-4.8.
>
>This patch makes use of the ACLE pre-defines to map to the internal
>feature definitions. To support older versions of GCC a compatibility
>header is provided that maps the traditional pre-defines onto the new
>ACLE ones.
Stop using __FreeBSD_ARCH_armv6__ and switch to __ARM_ARCH >= 6 in the
couple of places in tree. clang already implements ACLE. Add a define
that says we implement version 1.1, even though the implementation
isn't quite complete.
The MD allocators were very common, however there were some minor
differencies. These differencies were all consolidated in the MI allocator,
under ifdefs. The defines from machine/vmparam.h turn on features required
for a particular machine. For details look in the comment in sys/sf_buf.h.
As result no MD code left in sys/*/*/vm_machdep.c. Some arches still have
machine/sf_buf.h, which is usually quite small.
Tested by: glebius (i386), tuexen (arm32), kevlo (arm32)
Reviewed by: kib
Sponsored by: Netflix
Sponsored by: Nginx, Inc.
don't need any #ifdef stuff to use atomic_load/store_64() elsewhere in
the kernel. For armv4 the atomics are trivial to implement for kernel
code (just disable interrupts), less so for user mode, so this only has
the kernel mode implementations for now.
value shared across multiple cores is with atomic_load_64() and
atomic_store_64(), because the normal 64-bit load/store instructions
are not atomic on 32-bit arm. Luckily the ldrexd/strexd instructions
that are atomic are fairly cheap on armv6. Because it's fairly simple
to do, this implements all the ops for 64-bit, not just load/store.
Reviewed by: andrew, cognet
We have functions nested within functions, and places where we start a
function then never end it, we just jump to the middle of something else.
We tried to express this with nested ENTRY()/END() macros (which result
in .fnstart and .fnend directives), but it turns out there's no way to
express that nesting in ARM EHABI unwind info, and newer tools treat
multiple .fnstart directives without an intervening .fnend as an error.
These changes introduce two new macros, EENTRY() and EEND(). EENTRY()
creates a global label you can call/jump to just like ENTRY(), but it
doesn't emit a .fnstart. EEND() is a no-op that just documents the
conceptual endpoint that matches up with the same-named EENTRY().
This is based on patches submitted by Stepan Dyatkovskiy, but I made some
changes and added the EEND() stuff, so blame any problems on me.
Submitted by: Stepan Dyatkovskiy <stpworld@narod.ru>
handling. For statically linked apps this uses the __exidx_start/end
symbols set up by the linker. For dynamically linked apps it finds the
shared object that contains the given address and returns the location and
size of the exidx section in that shared object.
The dl_unwind_find_exidx() name is used by other BSD projects and Android,
and is mentioned in clang 3.5 comments as "the BSD interface" for finding
exidx data. GCC (in libgcc_s) expects the exact same API and functionality
to be provided by a function named __gnu_Unwind_Find_exidx(), so we provide
that with an alias ("strong reference").
Reviewed by: kib@
MFC after: 1 week
