Commit Graph

347 Commits

Author SHA1 Message Date
Nathan Whitehorn
9f4a7eae43 Make using the #address-cells property on the interrupt parent in device
tree parsing opt-out rather than opt-in. All FDT-based systems as well as
PowerPC systems with real Open Firmware use the CHRP-derived binding that
includes it, which makes SPARC the odd man out here. Making it opt-out
avoids astonishment on new platform bring up.
2016-01-02 19:28:35 +00:00
Ian Lepore
002ef20ca8 Add an OF_decode_addr() implementation for arm64.
Discussed with:		andrew
2016-01-02 19:14:19 +00:00
Ian Lepore
69dcb7e771 Make the 'env' directive described in config(5) work on all architectures,
providing compiled-in static environment data that is used instead of any
data passed in from a boot loader.

Previously 'env' worked only on i386 and arm xscale systems, because it
required the MD startup code to examine the global envmode variable and
decide whether to use static_env or an environment obtained from the boot
loader, and set the global kern_envp accordingly.  Most startup code wasn't
doing so.  Making things even more complex, some mips startup code uses an
alternate scheme that involves calling init_static_kenv() to pass an empty
buffer and its size, then uses a series of kern_setenv() calls to populate
that buffer.

Now all MD startup code calls init_static_kenv(), and that routine provides
a single point where envmode is checked and the decision is made whether to
use the compiled-in static_kenv or the values provided by the MD code.

The routine also continues to serve its original purpose for mips; if a
non-zero buffer size is passed the routine installs the empty buffer ready
to accept kern_setenv() values.  Now if the size is zero, the provided buffer
full of existing env data is installed.  A NULL pointer can be passed if the
boot loader provides no env data; this allows the static env to be installed
if envmode is set to do so.

Most of the work here is a near-mechanical change to call the init function
instead of directly setting kern_envp.  A notable exception is in xen/pv.c;
that code was originally installing a buffer full of preformatted env data
along with its non-zero size (like mips code does), which would have allowed
kern_setenv() calls to wipe out the preformatted data.  Now it passes a zero
for the size so that the buffer of data it installs is treated as
non-writeable.
2016-01-02 02:53:48 +00:00
Andrew Turner
5f0a5fefc6 Decode and print the ID_AA64* registers on boot. These registers hold
information on what the core supports. In most cases these will be
identical across most CPUs in the SoC, however there may be the case where,
with a big.LITTLE setup they may differ. In this case we print the
decoded data on all CPUs.

Reviewed by:	kib
Sponsored by:	ABT Systems Ltd
Differential Revision:	https://reviews.freebsd.org/D4725
2015-12-30 17:36:34 +00:00
Andrew Turner
91569f1f38 Increase the size and alignment of the setjmp buffer. This will allow for
possible future CPU extentions with larger registers.

jmp_buf's size and alignment are baked into the ABI of third party libraries
and thus are very hard to change later so it is best to waste a small amount
of space now.

Reviewed by:	brooks
Differential Revision:	https://reviews.freebsd.org/D3956
2015-12-23 15:22:44 +00:00
Andrew Turner
122493cf95 Support the variant of the interrupt-map property where the parent bus has
the #address-cells property set. For this we need to read more data before
the parent interrupt description.

this is only enabled on arm64 for now as it's not quite compliant with the
ePAPR spec. We should use a default of 2 where the #address-cells property
is missing, however this will need further testing across architectures.

Obtained from:	ABT Systems Ltd
Sponsored by:	SoftIron Inc
Differential Revision:	https://reviews.freebsd.org/D4518
2015-12-17 17:00:04 +00:00
Andrew Turner
68fe2d8f70 Add support for MSI interrupts to the gicv2m controller. The allocation
is still quite simplistic, it just increments a counter to use the next
interrupt.

Obtained from:	ABT Systems Ltd
Sponsored by:	SoftIron Inc
2015-12-14 16:07:20 +00:00
Andrew Turner
90257a2b6c Skip restoring more registers when returning from an exception taken in
the kernel. These registers are all callee saved, and as such will be
restored before returning to the exception handler.

Userland still needs these registers to be restored as they may be changed
by the kernel and we don't currently track these places.
2015-12-14 10:06:01 +00:00
Andrew Turner
a38fc8ca6a Sort the list of NICs after the mii options. While here add the msk driver
as it has now been tested.

Sponsored by:	SoftIron Inc
2015-12-11 15:25:47 +00:00
Andrew Turner
4b771f1b99 Add support for the GICv2M extension to the GICv2 interrupt controller.
This is (oddly) specified in the ARM Server Base System Architecture. It
extends the GICv2 to support MSI and MSI-X interrupts, however only the
latter are currently supported.

Only the FDT attachment is currently supported, however the attachment
and core driver are split to help adding ACPI support in the future.

Obtained from:	ABT Systems Ltd
Relnotes:	yes
Sponsored by:	SoftIron Inc
2015-12-10 16:40:38 +00:00
Andrew Turner
6d8433cb00 Remove pic_map_msix from ARM64, it is unneeded as all mappings happen
through pic_map_msi (without the x).

Sponsored by:	ABT Systems Ltd
2015-12-10 15:51:02 +00:00
Andrew Turner
c1d5e7a1f1 Reduce the numbers of levels of indirection in arm64 pcib drivers by making
the MSI & MSI-X handler functions be directly callable from the driver
methods.

Sponsored by:	ABT Systems Ltd
2015-12-10 13:19:30 +00:00
Konstantin Belousov
4d22d07a07 Add support for usermode (vdso-like) gettimeofday(2) and
clock_gettime(2) on ARMv7 and ARMv8 systems which have architectural
generic timer hardware. It is similar how the RDTSC timer is used in
userspace on x86.

Fix a permission problem where generic timer access from EL0 (or
userspace on v7) was not properly initialized on APs.

For ARMv7, mark the stack non-executable. The shared page is added for
all arms (including ARMv8 64bit), and the signal trampoline code is
moved to the page.

Reviewed by:	andrew
Discussed with:	emaste, mmel
Sponsored by:	The FreeBSD Foundation
Differential revision:	https://reviews.freebsd.org/D4209
2015-12-07 12:20:26 +00:00
Andrew Turner
9d4de283d5 Rework the exception entry/return functions to make them valid frames to be
unwound through. For this we need the frame pointer (x29) to point to the
location on the stack where we stored the previous frame pointer, and link
register. To simplify this the stack pointer is only adjusted by addition
and subtraction, and not through the use of post increment on loads and
stores.

The updated frame layout is:

+------------+
| lr -- x30  |
+------------+
| fp -- x29  | <-- x29 points at this
+------------+
| Trap frame |
| ...        |
|            | <-- sp points at this
+------------+

The only difference is the first two items, and setting of x29.

Sponsored by:	ABT Systems Ltd
2015-12-01 12:37:04 +00:00
Andrew Turner
119a353e3d Rework the atomic code to reduce the repetition. This merges some of the
atomic functions where they are almost identical, or have acquire/release
semantics.

While here clean these function up. The cbnz instruction doesn't change
the condition flags so drop cc, however they should have memory added to the
clobber list.

Reviewed by:	kib
Sponsored by:	ABT Systems Ltd
Differential Revision:	https://reviews.freebsd.org/D4318
2015-12-01 12:27:36 +00:00
Andrew Turner
1e888d7870 Print useful information when we hit a data abort we can't handle. This
prints the trap frame, along with the exception syndrome and fault address
registers. Even though esr is 64-bits here it is only 32-bits in hardware
so only print the valid 32-bits.

While here also print esr and far when appropriate after printing the trap
frame.

Sponsored by:	ABT Systems Ltd
2015-12-01 09:52:41 +00:00
Konstantin Belousov
4424a685c2 Shorten conditional branch code.
Reviewed by:	andrew
Sponsored by:	The FreeBSD Foundation
2015-11-30 10:18:15 +00:00
Konstantin Belousov
5cb72aa7ec Remove unneeded instructions.
Reviewed by:	andrew
Sponsored by:	The FreeBSD Foundation
2015-11-30 10:17:13 +00:00
Konstantin Belousov
724f4b62b0 Remove sv_prepsyscall, sv_sigsize and sv_sigtbl members of the struct
sysent.

sv_prepsyscall is unused.

sv_sigsize and sv_sigtbl translate signal number from the FreeBSD
namespace into the ABI domain.  It is only utilized on i386 for iBCS2
binaries.  The issue with this approach is that signals for iBCS2 were
delivered with the FreeBSD signal frame layout, which does not follow
iBCS2.  The same note is true for any other potential user if
sv_sigtbl.  In other words, if ABI needs signal number translation, it
really needs custom sv_sendsig method instead.

Sponsored by:	The FreeBSD Foundation
2015-11-28 08:49:07 +00:00
Ed Maste
20e0b5fcce Correct arm64 gic_v3 sizeof argument
No functional change as 'struct resource *' and 'struct resource **'
have the same size, but the former is the proper type.

PR:		204768
Submitted by:	David Binderman
2015-11-26 21:05:55 +00:00
Andrew Turner
ac4dad9e2f Add support for moving the DMAP range. This is needed as some AMD SoCs
place physical memory at an address outside the old DMAP range. This is an
issue as we rely on being able to move from PA -> VA using this range.

Obtained from:	Patrick Wildt <patrick@bitrig.org> (earlier version)
Sponsored by:	ABT Systems Ltd
Differential Revision:	https://reviews.freebsd.org/D3885
2015-11-24 11:01:43 +00:00
Svatopluk Kraus
eae22c4430 Revert r291142.
The not quite consistent logic for bounce pages allocation is utilizited
by re(4) interface which can hang now.

Approved by:	kib (mentor)
2015-11-23 11:19:00 +00:00
Svatopluk Kraus
6fa7734d6f Fix BUS_DMA_MIN_ALLOC_COMP flag logic. When bus_dmamap_t map is being
created for bus_dma_tag_t tag, bounce pages should be allocated
only if needed.

Before the fix, they were allocated always if BUS_DMA_COULD_BOUNCE flag
was set but BUS_DMA_MIN_ALLOC_COMP not. As bounce pages are never freed,
it could cause memory exhaustion when a lot of such tags together with
their maps were created.

Note that there could be more maps in one tag by current design.
However BUS_DMA_MIN_ALLOC_COMP flag is tag's flag. It's set after
bounce pages are allocated. Thus, they are allocated only for first
tag's map which needs them.

Approved by:	kib (mentor)
2015-11-21 19:55:01 +00:00
Marius Strobl
ec2fbee752 Avoid a NULL pointer dereference in bounce_bus_dmamap_unload() when
the map has been created via bounce_bus_dmamem_alloc(). In that case
bus_dmamap_unload(9) typically isn't called during normal operation
but still should be during detach, cleanup from failed attach etc.

Submitted by:	yongari
MFC after:	3 days
2015-11-21 02:08:47 +00:00
Marius Strobl
8fd47ac11c Avoid a NULL pointer dereference in bounce_bus_dmamap_sync() when the
map has been created via bounce_bus_dmamem_alloc(). Even for coherent
DMA - which bus_dmamem_alloc(9) typically is used for -, calling of
bus_dmamap_sync(9) isn't optional.

PR:		188899 (non-original problem)
MFC after:	3 days
2015-11-20 02:23:35 +00:00
Ed Maste
f51f84a628 Fix typo in message from arm64 ITS workaround 2015-11-09 01:49:25 +00:00
Ed Maste
22a00274ac Sync arm64 GENERIC whitespace/comments with amd64
Sponsored by:	The FreeBSD Foundation
2015-11-08 21:08:31 +00:00
Ed Maste
13d3ad4d71 arm64: add igb(4) to GENERIC
We have em(4) in GENERIC already and so also supporting the related
igb(4) makes sense.

Sponsored by:	The FreeBSD Foundation
2015-11-07 04:46:34 +00:00
Andrew Turner
1df3f63ac5 Mark the arm64 nexus devices to be attached early. This allows multipass
to work correctly. Without it the pass quickly moves to the final pass
before the nexus device attaches.

Sponsored by:	ABT Systems Ltd
2015-11-06 14:36:21 +00:00
Ian Lepore
53f93ed3ff Fix an alignment check that is wrong in half the busdma implementations.
This will enable the elimination of a workaround in the USB driver that
artifically allocates buffers twice as big as they need to be (which
actually saves memory for very small buffers on the buggy platforms).

When deciding how to allocate a dma buffer, armv4, armv6, mips, and
x86/iommu all correctly check for the tag alignment <= maxsize as enabling
simple uma/malloc based allocation.  Powerpc, sparc64, x86/bounce, and
arm64/bounce were all checking for alignment < maxsize; on those platforms
when alignment was equal to the max size it would fall back to page-based
allocators even for very small buffers.

This change makes all platforms use the <= check.  It should be noted that
on all platforms other than arm[v6] and mips, this check is relying on
undocumented behavior in malloc(9) that if you allocate a block of a given
size it will be aligned to the next larger power-of-2 boundary.  There is
nothing in the malloc(9) man page that makes that explicit promise (but the
busdma code has been relying on this behavior all along so I guess it works).

Arm and mips code uses the allocator in kern/subr_busdma_buffalloc.c, which
does explicitly implement this promise about size and alignment.  Other
platforms probably should switch to the aligned allocator.
2015-11-02 23:37:19 +00:00
Andrew Turner
deeaa1c566 Make the arm64_cpu driver quiet as it adds no new information.
Only report the register used to start each cpu in bootverbose.

Sponsored by:	ABT Systems Ltd
2015-11-02 16:43:26 +00:00
Andrew Turner
4dd6ed5ce6 Mark functions as such. This means we call them directly rather than have
the dynamic linker copy them, but not relocate them at the new location.
This allows us to run sqlite3 without it crashing.

Sponsored by:	ABT Systems Ltd
2015-10-27 22:24:57 +00:00
Zbigniew Bodek
db94e32ba7 Fix bus numbering in ThunderX ITS quirk
Internal busses (thus ECAM access) should be mapped to
all values from 0 to 143.

Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D3753
2015-10-25 23:27:08 +00:00
Zbigniew Bodek
71e2c1d4c0 Add support for unspecified ranges on ThunderX system
When one tries to allocate a resource with unspecified range,
read already configured BAR values (by UEFI or whatever).
This is necessary to make VNIC VFs working and to allow them to be
properly allocated.

Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D3752
2015-10-25 23:22:40 +00:00
Ed Maste
5280a92784 arm64: remove exception instruction length assertion
From the (now removed) comment:

 * It is unclear in some cases if the bit is implementation defined.
 * The Foundation Model and QEMU disagree on if the IL bit should
 * be set when we are in a data fault from the same EL and the ISV
 * bit (bit 24) is also set.

Instead of adding even more special cases just remove the assertion.

Approved by:	andrew
Sponsored by:	The FreeBSD Foundation
2015-10-23 15:24:00 +00:00
Jason A. Harmening
a50730587b Remove unclear comment about address truncation in busdma. Add (hopefully much clearer) comment at declaration of PHYS_TO_VM_PAGE().
Noted by:	avg
2015-10-23 12:03:25 +00:00
Ed Schouten
2ff069a7bc Add support for CloudABI on ARM64.
It turns out that it is pretty easy to make CloudABI work on ARM64. We
essentially only need to copy over the sysvec from AMD64 and ensure that
we use ARM64 specific registers.

As there is an overlap between function argument and return registers,
we do need to extend cloudabi64_schedtail() to only set its values if
we're actually forking. Not when we're creating a new thread.

Reviewed by:	kib
Differential Revision:	https://reviews.freebsd.org/D3917
2015-10-22 11:09:25 +00:00
Jason A. Harmening
d394b026f6 Use pmap_quick* functions in arm64 busdma to make bounce buffer synchronization more flexible and avoid borrowing UVAs for userspace buffers. This is mostly equivalent to r286785 and r286787 for x86.
Differential Revision:	https://reviews.freebsd.org/D3870
2015-10-21 19:44:20 +00:00
Ed Maste
598cd16442 arm64: Enable CTF for DTrace support
Sponsored by:	The FreeBSD Foundation
2015-10-21 19:08:16 +00:00
Andrew Turner
80c4b9e575 Use 4 levels of page tables when enabling the MMU. This will allow us to
boot on an SoC that places physical memory at an address past where three
levels of page tables can access in an identity mapping.

Submitted by:   Wojciech Macek <wma@semihalf.com>,
                Patrick Wildt <patrick@bitrig.org>
Differential Revision:	https://reviews.freebsd.org/D3885 (partial)
Differential Revision:	https://reviews.freebsd.org/D3744
2015-10-19 13:20:23 +00:00
Zbigniew Bodek
32f849ca1e Add ThunderX VNIC to arm64/GENERIC kernel
Add vnic to enabled networking cards and enable SR-IOV by the way.

Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
2015-10-18 22:13:21 +00:00
Andrew Turner
22c6adff09 Correctly align the stack. The early csu assumed we passed the aux vector
in through the stack pointer, however this may have been misaligned
causing some userland applications to crash. A workaround was committed in
r284707 where userland would check if the aux vector was passed using the
old or new ABI and adjust the stack if needed. As 4 months have passed it
is time to move to the new ABI, with the expectation the compat code in csu
and the runtime linker to be removed in the future.

Sponsored by:	ABT Systems Ltd
2015-10-18 13:23:21 +00:00
Andrew Turner
6aa751cff1 Replace build_section_pagetable with build_l1_block_pagetable as it takes
an extra argument to specify the number of 1GiB pages to map. This should
be a nop as we are only mapping a single page, but when we move to use an
extra level of page tables we will be able to map a second block, e.g. if
the kernel was loaded over a 1GiB boundary.
2015-10-17 19:52:17 +00:00
Andrew Turner
81b4133adf Rename build_block_pagetable to build_l2_block_pagetable in preperation
for adding support for 4 levels of page tables.

Obtained from:	Patrick Wildt <patrick@bitrig.org>
2015-10-17 14:07:47 +00:00
Konstantin Belousov
153bf0bce0 Add checks for kernel VA accesses to the copyin(9) and related
functions on arm64.

Reviewed by:	andrew
Sponsored by:	The FreeBSD Foundation
Differential revision:	https://reviews.freebsd.org/D3907
2015-10-17 13:20:42 +00:00
Ed Schouten
632b5263c5 Properly set the return value for casueword to 0 upon success.
While trying to get multithreading working for CloudABI on aarch64, I
noticed that compare-and-exchange operations in kernelspace would always
fail. It turns out that we don't properly set the return value to 0 when
the compare and exchange succeeds.

Approved by:	andrew
Differential Revision:	https://reviews.freebsd.org/D3899
2015-10-15 17:50:28 +00:00
Konstantin Belousov
8598392cd1 Build changes that allow the modules on arm64.
- Move the required kernel compiler flags from Makefile.arm64 to kern.mk.
- Build arm64 modules as PIC; non-PIC relocations in .o for shared object
  output cannot be handled.
- Do not try to install aarch64 symlink.
- A hack for arm64 to avoid ld -r stage.  See the comment for the explanation.
  Some functionality is lost, like ctf handling, but hopefully will be
  restored after newer linker is available.

Reviewed by:	andrew, emaste
Tested by:	andrew (on real hardware)
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D3796
2015-10-08 17:42:08 +00:00
Konstantin Belousov
00a53ebe08 Implement in-kernel relocator for the arm64 module linker.
It is decided to go with the shared object file format for modules on
arm64, due to the Aarch64 instruction set details.  Combination of the
signed 28-bit offset in the branch instructions encoding together with
the supported memory model of compilers makes the relocatable object
support impossible or at least too hard.

Reviewed by:	andrew, emaste
Tested by:	andrew (on real hardware)
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D3796
2015-10-08 16:58:01 +00:00
Konstantin Belousov
fbc7b10762 Make the copyright notice in the file to match reality. Use the
recommended FreeBSD license text.

Approved by:	andrew
Discussed with:	emaste
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D3846
2015-10-08 16:46:11 +00:00
Andrew Turner
8bdcc09641 When trying to execute from a misaligned address raise a SIGBUS with the
invalid address alignment code.

Obtained from:	EuroBSDCon
Sponsored by:	ABT Systems Ltd
2015-10-04 21:16:45 +00:00
Konstantin Belousov
b86860001b When asserting IL bit in exception syndrome register, print the raw
register value.

Reviewed by:	andrew
Sponsored by:	The FreeBSD Foundation
2015-10-04 12:52:30 +00:00
Andrew Turner
9439047657 Use pmap_load more consistently. While here try to only load the data once
when we reuse the same data.

Obtained from:	EuroBSDCon Devsummit
Sponsored by:	ABT Systems Ltd
2015-10-01 10:43:40 +00:00
Andrew Turner
7991717238 Add the ENTRY/END entries around the exception handlers.
Obtained from:	EuroBSDCon Devsummit
Sponsored by:	ABT Systems Ltd
2015-10-01 09:44:15 +00:00
Alan Cox
9f86aba61c Exploit r288122 to address a cosmetic issue. Since PV chunk pages don't
belong to a vm object, they can't be paged out.  Since they can't be paged
out, they are never enqueued in a paging queue.  Nonetheless, passing
PQ_INACTIVE to vm_page_unwire() creates the appearance that these pages
are being enqueued in the inactive queue.  As of r288122, we can avoid
this false impression by passing PQ_NONE.

Submitted by:	kmacy (an earlier version)
Differential Revision:	https://reviews.freebsd.org/D1674
2015-09-26 07:18:05 +00:00
Ed Maste
7fc0791608 Remove apostrophe from AP's for consistency (arm64 mp_machdep) 2015-09-25 21:04:31 +00:00
Ed Maste
2ed39a2216 Add pass device to arm64 GENERIC (for smartctl)
Sponsored by:	The FreeBSD Foundation
2015-09-22 21:43:08 +00:00
Konstantin Belousov
37864a5451 Call ast when handling irq from userspace, otherwise we could miss
reschedule.  Right now arm_cpu_intr() does critical_exit() as the last
action, so the impact is not serious.

Remove duplicated interrupt disable in restore_registers macro, when
returning to usermode.  The do_ast macro disabled interrupts for us.

Reviewed by:	andrew
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D3714
2015-09-22 18:30:06 +00:00
Andrew Turner
9063d39e49 Don't restore interrupts when we are about to disable them in the next
instruction.
2015-09-22 16:46:34 +00:00
Konstantin Belousov
d50c68b2f0 Re-check for new ast after ast was handled. We should not return to
usermode with pending asts.

Reviewed by:	andrew
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D3667
2015-09-22 16:29:55 +00:00
Ed Maste
a9ee805d45 arm64 ThunderX PCIe workaround: enumerate only one slot for now
Otherwise an em(4) NIC is detected 32 times.

Submitted by:	wma@semihalf.com
Obtained from:	Semihalf
Differential Revision:	https://reviews.freebsd.org/D3706
2015-09-22 12:56:34 +00:00
Konstantin Belousov
0510aedcd5 Do not execute exception handlers with disabled interrupts.
We should not call vm_fault(), or send a signal, with interrupts
disabled.  MI kernel code is not prepared for such environment, not to
mention that this increases system latency, since code appears to be
executing as being under spinlock.

The FAR register for data aborts is read before the interrupts are
enabled, to avoid its corruption due to nested exception or context
switch.

Add asserts, similar to the checks done by other architectures, about
not taking page faults in non-sleepable contexts, rather than die with
late and somewhat confusing witness diagnostic.

Reviewed by:	andrew
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D3669
2015-09-18 17:09:59 +00:00
Konstantin Belousov
6ced378970 Clear exclusive monitors when handling data aborts, the monitors are
in unknown state per spec.

Reviewed by:	andrew (previous version)
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D3668
2015-09-18 16:52:18 +00:00
Andrew Turner
cf8b6f76a2 Don't read the floating-point registers for now. We will need to enable the
VFP around the read of these instructions as they may raise an exception.

Sponsored by:	ABT Systems Ltd
2015-09-18 13:44:15 +00:00
Zbigniew Bodek
7b1c945350 Block secondary ITS instances from attaching on ARM64
Currently FreeBSD supports only single PIC controller. Some systems
that have more than one (like ThunderX dual-socket) fails to boot.
Disable other PICes until proper handling is implemented in the
generic interrupt code.

Reviewed by:   imp
Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3682
2015-09-16 23:59:45 +00:00
Zbigniew Bodek
dbf66ac207 Release memory for CPUs that fail to init on ARM64
cpu_init_fdt will now release memory allocated for structures
serving CPUs that have failed to init.

Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3297
2015-09-16 23:46:20 +00:00
Ed Maste
9b76bf99d5 arm64: add xhci driver and umass/ukbd to GENERIC for Cavium ThunderX
Sponsored by:	The FreeBSD Foundation
2015-09-15 19:59:35 +00:00
Mark Johnston
610141cebb Add stack_save_td_running(), a function to trace the kernel stack of a
running thread.

It is currently implemented only on amd64 and i386; on these
architectures, it is implemented by raising an NMI on the CPU on which
the target thread is currently running. Unlike stack_save_td(), it may
fail, for example if the thread is running in user mode.

This change also modifies the kern.proc.kstack sysctl to use this function,
so that stacks of running threads are shown in the output of "procstat -kk".
This is handy for debugging threads that are stuck in a busy loop.

Reviewed by:	bdrewery, jhb, kib
Sponsored by:	EMC / Isilon Storage Division
Differential Revision:	https://reviews.freebsd.org/D3256
2015-09-11 03:54:37 +00:00
Konstantin Belousov
1fa6712471 Do not hold the process around the vm_fault() call from the trap()s.
The only operation which is prevented by the hold is the kernel stack
swapout for the faulted thread, which should be fine to allow.

Remove useless checks for NULL curproc or curproc->p_vmspace from the
trap_pfault() wrappers on x86 and powerpc.

Reviewed by:	alc (previous version)
Sponsored by:	The FreeBSD Foundation
MFC after:	2 weeks
2015-09-10 17:46:48 +00:00
Andrew Turner
8fe99e427f Rework copyinstr to:
* Fail when the length passed in is 0
 * Remove an unneeded increment of the count on success
 * Return ENAMETOOLONG when the input pointer is too long

Sponsored by:	ABT Systems Ltd
2015-09-09 11:51:14 +00:00
Andrew Turner
384dd3be15 Add support for pmap_mincore on arm64 by walking the page tables to find
the details for the requested address.

PR:		202307
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2015-09-08 18:41:06 +00:00
Andrew Turner
0a21ab2c4b Use load-acquire semantics while waiting for td_lock to be released. The
store should have release semantics and will have due to the dsb above it
so add a comment to explain this. [1]

While here update the code to not reload the current thread, it's already
in a register, we just need to not trash it.

Suggested by:	kib [1]
Sponsored by:	ABT Systems Ltd
2015-09-07 14:01:18 +00:00
Andrew Turner
f452c30169 When dropping to EL1 ensure we have written to all special registers by
moving the instruction barrier to just before we drop exception level.

Sponsored by:	ABT Systems Ltd
2015-09-07 10:13:14 +00:00
Andrew Turner
2dae850b5e Add ddb show commands to print the special registers and to ask the
hardware to perform address translation for us. These are useful to help
track down what caused us to enter the debugger.

Sponsored by:	ABT Systems Ltd
2015-09-05 17:29:07 +00:00
Andrew Turner
504a0067c7 Add support for the dwc usb in the HiSilicon hi6220 in the HiKey board. For
this we need to force the driver into host mode, as without this the driver
fails to detect any devices.

Relnotes:	yes
Sponsored by:	ABT Systems Ltd
2015-09-01 17:13:04 +00:00
Andrew Turner
c0e5e17237 Add support for the DesignWare MMC hardware in the HiSilicon hi6220. This
SoC is used in the HiKey board from 96boards.

Currently on the SD card is working on the HiKey, as such devices 0 and 2
will need to be disabled, for example by adding the following to
loader.conf:

hint.hisi_dwmmc.0.disabled=1
hint.hisi_dwmmc.2.disabled=1

Relnotes:	yes (Hikey board booting)
Sponsored by:	ABT Systems Ltd
2015-09-01 16:25:12 +00:00
Andrew Turner
bbbde6d844 Add the SOC_HISI_HI6220 option. This will be used to enable drivers for
the HiSilicon hi6220 SoC used in the HiKey 96boards board.

Sponsored by:	ABT Systems Ltd
2015-08-26 18:09:17 +00:00
Zbigniew Bodek
bb2295eb2c Fix race condition in its_cmd_send()
its_cmd_send() can be called by multiple CPUs simultaneously.
After the command is pushed to ITS command ring the completion
status is polled using global pointer to the next free ring slot.
Use copied pointer and provide correct locking to avoid spurious
pointer value when concurrent access occurs.

Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3436
2015-08-26 12:32:46 +00:00
Andrew Turner
75e0dfe336 Add an option to select which SoCs we are building for. It is intended to
be used with any SoC specific drivers, for example a ThunderX nic driver
would use something like the following in files.arm64:

arm64/cavium/thunder_nic.c optional soc_cavm_thunderx thndr_nic

Reviewed by:	imp
Sponsored by:	ABT Systems Ltd
Differential Revision:	https://reviews.freebsd.org/D3479
2015-08-26 11:36:23 +00:00
Andrew Turner
ed33af42ff Add a above the virtio section. 2015-08-26 10:32:23 +00:00
Andrew Turner
7a1ac4a701 Use kdb_jmpbuf and setjmp to handle accessing invalid addresses in the
debugger memory access functions. This allows us to correctly reenter into
the debugger on failure.

Sponsored by:	ABT Systems Ltd
2015-08-25 17:02:28 +00:00
Andrew Turner
84a9a3a344 Use nitems to count the number of items in a static array.
Sponsored by:	ABT Systems Ltd
2015-08-25 16:50:25 +00:00
Andrew Turner
8f7467734a Call into kdb on a data abort when it's enabled. This is needed to
correctly handle trying to access an invalid address in the debugger.

While here document that the breakpoint handler is supposed to fall
through to the following case.

Sponsored by:	ABT Systems Ltd
2015-08-25 16:49:10 +00:00
Andrew Turner
65780a4020 Correctly update the address to write to after storing the stack pointer
in the kernels setjmp function.

Sponsored by:	ABT Systems Ltd
2015-08-25 16:43:32 +00:00
Andrew Turner
9b9266e8e7 Check the frame pointer is within the kernel before accessing it. 2015-08-24 17:28:19 +00:00
Andrew Turner
67905e0aec Add support for pmap_sync_icache on arm64.
Reviewed by:	emaste, imp (both earlier version)
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D3438
2015-08-24 12:01:39 +00:00
Andrew Turner
db36b645a0 Add pages used by the PV entries to minidumps. 2015-08-24 10:55:54 +00:00
Andrew Turner
0987c184a1 Handle the breakpoint instruction in userland by sending a SIGTRAP signal. 2015-08-23 13:18:56 +00:00
Andrew Turner
52afd687c3 Add the kernel support for minidumps on arm64.
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D3318
2015-08-20 12:49:56 +00:00
Andrew Turner
b0d415fc85 Remove pmap_kenter from pmap.h, the function doesn't exist. 2015-08-20 12:07:44 +00:00
Andrew Turner
9a0de24b5e Add pmap_get_tables to get the page tables for a given virtual address. This
will be used for minidump support.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2015-08-20 12:05:42 +00:00
Andrew Turner
5c714b29ff Add the arm64 minidump header. This was missed from r286953. 2015-08-20 11:26:26 +00:00
Zbigniew Bodek
a8d377b809 Increase MAXCPU in ARM64
Increase MAXCPU number to the maximum known value the existing
hardware can support.

Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3405
2015-08-19 11:59:41 +00:00
Zbigniew Bodek
04ae5bbe55 Remove redundant mp_naps from ARM64 secondary CPU start-up code
The global variable has been only used for CPU startup ordering
which is not needed anyway.

Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3296
2015-08-19 11:45:45 +00:00
Zbigniew Bodek
c50231a494 Add SMP support to GICv3 and ITS drivers
Introduce supprot for SMP to GICv3 and ITS drivers.

Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3299
2015-08-19 10:36:36 +00:00
Ed Maste
8d5bb77429 Correct comment typo noted by erik 2015-08-18 11:52:45 +00:00
Ed Maste
5a060174cd Remove register dump from arm64 el0 unknown exception
An exception with an unknown reasion is the expected result of the
attempted execution of an instruction bit pattern that has no allocated
instruction.

Sponsored by:	The FreeBSD Foundation
2015-08-18 10:07:03 +00:00
Andrew Turner
edc2af7807 Add support for bus_space_read_region and bus_space_write_region. This is
needed for the dwc USB controller driver.

Sponsored by:	ABT Systems Ltd
2015-08-14 09:55:25 +00:00
Ed Maste
13d06fd808 Remove arm64 workaround for Clang 3.4 crash 2015-08-13 19:05:18 +00:00
Ed Maste
ccd285e76c arm64: turn unknown el0 exception into a SIGILL
It seems we get EXCP_UNKNOWN from QEMU when executing zeroed memory.
Print a register dump here and signal illegal instruction. Also print
a register dump for other invalid exceptions, before panic.

Reviewed by:	andrew
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D3370
2015-08-13 13:21:00 +00:00
Andrew Turner
b1bacc1cbd Add the CNTHCTL_EL2 register bits missed in r286674 2015-08-12 17:09:57 +00:00
Andrew Turner
f03aa10f47 Set the counter-timer virtual offset to a know value, it may not have been
set by the boot code and are reset to an implementation defined value that
may be unknown.

Sponsored by:	ABT Systems Ltd
2015-08-12 17:06:22 +00:00
Andrew Turner
c8d372efdd Check the correct value in db_validate_address, pmap_extract returns 0 on
failure.

Sponsored by:	ABT Systems Ltd
2015-08-11 12:32:17 +00:00
Pedro F. Giffuni
6bc3fe5f4e Clean out some externally visible "more then" grammar
MFC after:	3 days
2015-08-11 03:12:09 +00:00
Zbigniew Bodek
bfc3978594 Add support for external PCIe (PEM) on Cavium's ThunderX
Reviewed by:   jhb
Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3257
2015-08-08 21:32:03 +00:00
Zbigniew Bodek
6ff9127b86 Add Ethernet and block device drivers to ARM64 GENERIC 2015-08-08 20:56:30 +00:00
Zbigniew Bodek
d943d79a36 Introduce support for internal PCIe for Cavium's ThunderX
This driver supports internal PCIe Root Complex on
Cavium ThunderX Pass 1.1 hardware.

Reviewed by:   andrew, jhb
Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3031
2015-08-08 20:34:55 +00:00
Andrew Turner
756e25bfd7 Fill in dump_avail based on the physical memory from EFI.
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2015-08-06 14:49:23 +00:00
Ed Maste
fc8c856029 Rationalize BSD license on sys/*/include/in_cksum.h
Remove the advertising clause from the Regents of the University of
California's license, per the letter dated July 22, 1999.

Update clause numbering.
2015-08-05 19:05:12 +00:00
Ed Maste
96226a9aa7 Rationalize BSD license on sys/*/include/float.h
Remove the advertising clause from the Regents of the University of
California's license, per the letter dated July 22, 1999.

Update clause numbering.
2015-08-05 17:05:35 +00:00
Jason A. Harmening
713841afb2 Add two new pmap functions:
vm_offset_t pmap_quick_enter_page(vm_page_t m)
void pmap_quick_remove_page(vm_offset_t kva)

These will create and destroy a temporary, CPU-local KVA mapping of a specified page.

Guarantees:
--Will not sleep and will not fail.
--Safe to call under a non-sleepable lock or from an ithread

Restrictions:
--Not guaranteed to be safe to call from an interrupt filter or under a spin mutex on all platforms
--Current implementation does not guarantee more than one page of mapping space across all platforms. MI code should not make nested calls to pmap_quick_enter_page.
--MI code should not perform locking while holding onto a mapping created by pmap_quick_enter_page

The idea is to use this in busdma, for bounce buffer copies as well as virtually-indexed cache maintenance on mips and arm.

NOTE: the non-i386, non-amd64 implementations of these functions still need review and testing.

Reviewed by:	kib
Approved by:	kib (mentor)
Differential Revision:	http://reviews.freebsd.org/D3013
2015-08-04 19:46:13 +00:00
Zbigniew Bodek
4cbca60875 Add missing exception number to EL0 sync. abort on ARM64
When doing a data abort from userland it is possible to get
more than one data abort inside the same exception level.
Add an appropriate exception number to allow nesting of
data_abort handler for EL0.

Reviewed by:   andrew
Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3276
2015-08-03 14:58:46 +00:00
Andrew Turner
f692e32555 Pass the pcb to store the vfp state in to vfp_save_state. This fixes a bug
in savectx where it will be used to store the current state however will
pass in a pcb when vfp_save_state expected a thread pointer.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2015-08-03 11:05:02 +00:00
Andrew Turner
9e2529043b Try to put the CPU into a low power state if we failed to otherwise halt
the system.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2015-07-31 15:54:34 +00:00
Andrew Turner
176739d3f5 Load the stack in stack_save and stack_save_td. This uses the generalised
unwind_frame function to read each stack frame until either the pc or stack
are no longer withing the kernel's address space.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2015-07-31 15:32:32 +00:00
Andrew Turner
36baf858c9 Add support for uma_small_alloc and uma_small_free, and make use of these.
This is copied from the amd64 version with minor changes. These should be
merged into a single file as from a quick look there are other copies of
the same file in other parts of the tree.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2015-07-31 14:17:26 +00:00
Andrew Turner
872df66596 Add memrw. This has had minimal testing, and will likely panic the kernel
when trying to read data from outside the DMAP region. I expect this panic
to be from within uiomove_fromphys, which needs to grow support to support
such addresses.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2015-07-31 13:39:51 +00:00
Andrew Turner
9b8c3c4f0b Add more atomic_swap_* functions.
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2015-07-31 13:34:43 +00:00
Andrew Turner
71d72ea14f Add VIRT_IN_DMAP to check if a virtual address is from the DMAP range.
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2015-07-31 13:32:25 +00:00
Zbigniew Bodek
a2b3dfad08 Apply erratum for mrs ICC_IAR1_EL1 speculative execution on ThunderX
ERRATUM:     22978, 23154
PASS (rev.): 1.0/1.1

Reviewed by:   imp
Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3184
2015-07-31 10:00:45 +00:00
Andrew Turner
8df0053b7a Add enough of pmap_page_set_memattr to run gstat. It still needs to split
the DMAP 1G pages so we set the attributes only on the specified page.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2015-07-30 16:17:44 +00:00
Ed Maste
c547d650eb Add ARM64TODO markers to unimplemented functionality
Reviewed by:	andrew
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D2389
2015-07-30 14:20:36 +00:00
Zbigniew Bodek
9028b18f75 Enable IRQ during syscalls on ARM64
FreeBSD provides a feature called Adaptive Mutexes, which allows
a thread to spin for a while when the mutex is taken instead of
immediately going to sleep. This causes issues when called from
syscall handler if interrupts are masked. If every other core
also attempts to access the same mutex there is a chance that
all of them are spinning on the same lock at the same time.
If interrupts are disabled, no kernel preemtion can occur and
the system becomes unresponsive.

This patch enables interrupts when syscall is being executed
and masks them as soon as it is completed.

Reviewed by:   andrew
Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3246
2015-07-30 13:59:38 +00:00
Zbigniew Bodek
cf89e8c919 Add quirk for ThunderX ITS device table size
Limit the number of supported device IDs to 0x100000
in order to decrease the size of the ITS device table so
that it matches with the HW capabilities.

Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3131
2015-07-29 11:22:19 +00:00
Andrew Turner
4027d3d62a Teach the GICv2 driver about the Qualcomm GICv2 compatible string.
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2015-07-21 18:08:10 +00:00
Zbigniew Bodek
3ed97a1a52 Add some more explanation to r285752
Add brief commentary to vendor-specific devid function in ITS
and remove redundant spaces by the way.

Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
2015-07-21 17:14:24 +00:00
Zbigniew Bodek
9920b3aa95 Don't allow malloc() to wait for resource while holding a lock in ITS
malloc() should not go to sleep in case of lack of resource while
the kernel thread is holding a non-sleepable lock.

- change malloc() flags to M_NOWAIT in such cases implement
  lpi_free_chunk() routine as it will be needed when ITT
  allocation fails in its_device_alloc_locked()
- do not increase verbosity of this code since upper layers will
  communicate an error if the interrupt setup fails

Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3121
2015-07-21 15:28:07 +00:00
Zbigniew Bodek
1fe6a1a25a Add support for vendor specific function for PCI devid acquisition in ITS
It is possible that some HW will use different PCI devids,
hence allow to replace the default domain🚌slot:func schema
by implementing and registering custom function.

Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3118
2015-07-21 14:47:23 +00:00
Zbigniew Bodek
52b584bc15 Implement get_cyclecount() on ARM64
Use Vritual Counter register associated with Generic Timer to
read the cyclecount.

Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3134
2015-07-21 12:50:45 +00:00
Zbigniew Bodek
13aaea2fd7 Improve ARM64 CPU_MATCH
Add a method to identify CPU based on RAW MIDR value.

Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3117
2015-07-21 12:15:00 +00:00
Zbigniew Bodek
b8bbefed30 Fix possible coherency issues between PEs related to I-cache
Basing on B.2.3.4:
Synchronization and coherency issues between data and
instruction accesses.

To ensure that modified instructions are visible to all PEs
(Processing Elements) in a shareability domain one need to
perform following sequence:
    1. Clean D-cache
    2. Ensure the visibility of data cleaned from cache
    3. Invalidate I-cache
    4. Ensure completion
    5. In SMP system PE must issue isb to ensure execution of the
       modified instructions

Reviewed by:   andrew
Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3106
2015-07-17 14:33:47 +00:00
Zbigniew Bodek
ab89029bd0 Fix secondary stacks calculation on ARM64
Secondary stack calculation is modified to provide
stack_top = secondary_stacks + (cpu_id) * PAGE_SIZE * KSTACK_PAGES
because on ARM64 the stack grows to lower memory addresses.

Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3107
2015-07-17 14:08:08 +00:00
Zbigniew Bodek
d5dfc8ad00 Increase DMAP (Direct Map) size on ARM64
Previous DMAP size was too small for systems with more than 64GB
of RAM. Increase it to 128GB to support ThunderX CRB.

Reviewed by:   andrew
Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3113
2015-07-17 13:58:00 +00:00
Zbigniew Bodek
721555e7ee Fix KSTACK_PAGES issue when the default value was changed in KERNCONF
If KSTACK_PAGES was changed to anything alse than the default,
the value from param.h was taken instead in some places and
the value from KENRCONF in some others. This resulted in
inconsistency which caused corruption in SMP envorinment.

Ensure all places where KSTACK_PAGES are used the opt_kstack_pages.h
is included.

The file opt_kstack_pages.h could not be included in param.h
because was breaking the toolchain compilation.

Reviewed by:   kib
Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3094
2015-07-16 10:46:52 +00:00
Zbigniew Bodek
1038d102c4 Set-up proper TCR values for memory related to Translation Table Walking
This commit adds proper cache and shareability attributes to
the TCR register.
Set memory attributes to Normal, outer and inner cacheable WBWA.
Set shareability to inner and outer shareable when SMP is enabled.

Reviewed by:   andrew
Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3093
2015-07-16 10:22:57 +00:00
Zbigniew Bodek
b49baf8065 Add identify_cpu() to ARM64 init_secondary routine
Identify current CPU. This is necessary to setup
affinity registers and to provide support for
runtime chip identification.

Reviewed by:   andrew
Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3095
2015-07-15 09:24:45 +00:00
Andrew Turner
b7fbd410ab Set memory to be inner-sharable. This isn't needed on device memory as the
MMU will ignore the attribute there, howeverit simplifies to code to alwas
set it.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2015-07-14 12:37:47 +00:00
Zbigniew Bodek
d1be8e59e2 Fix secondary PIC initialization order
Call arm_init_secondary before any other PIC-related functions
are called. This is necessary for GICv3 where PIC_INIT_SECONDARY
allocates resources needed for all further operations.

Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3066
2015-07-14 12:02:56 +00:00
Zbigniew Bodek
b7ac293f44 Fix intr_machdep.c for ARM64
On ARMv8 IPIs are mapped to 0-15. Incrementing the number by 16
is wrong, because it sets a reserved bit in the IPI register.
This patch removes all "+16" to comply with specs.

Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3029
2015-07-14 11:59:43 +00:00
Christian Brueffer
f4c1eac7cd Spell crypto correctly. 2015-07-14 10:47:56 +00:00
Zbigniew Bodek
686836faca Add ARM64TODO comments to ACPI PCI stubs
This will make searching for missing functionalities easier.
2015-07-12 18:32:16 +00:00
Zbigniew Bodek
e7c14c38ba Implement stubs for ACPI PCI routines
ACPI driver requires special functions to be provided by machdep code.
Add temporary stubs to satisfy the compiler when both "pci" and "acpi"
are enabled in the kernel configuration file.

Reviewed by:   andrew
Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3028
2015-07-12 17:28:31 +00:00
Andrew Turner
bf1717e566 Clear the carry bit on the saved program state register when asked to
clear the return value, it's used to indicate an error.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2015-07-09 17:26:56 +00:00
Andrew Turner
b2b5507779 Add support for SMP. This uses the FDT data to find the CPUs to start on,
and psci to start them. I expect ACPI support to be added later.

This has been tested on qemu with 2 cpus as that is the current value of
MAXCPUS. This is expected to be increased in the future as FreeBSD has
been tested on 48 cores on the Cavium ThunderX hardware.

Partially based on a patch from Robin Randhawa from ARM.

Approved by:	ABT Systems Ltd
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D3024
2015-07-09 13:23:29 +00:00
Andrew Turner
3ad7e84ef5 Add logging of synchronous exceptions.
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2015-07-09 13:07:12 +00:00
Andrew Turner
7df38eabdc Add the definition of the shareable bits in the pagetables
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2015-07-09 12:56:09 +00:00
Andrew Turner
144aa0b7f5 Clean up the types used in <machine/ucontext.h> on arm64. As some ports
include this file without first including the headers needed for uint32_t
and the like use the __foo type.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2015-07-09 12:51:50 +00:00
Zbigniew Bodek
6c03ba71f8 Rework CPU identification on ARM64
This commit reworks the code responsible for identification of
the CPUs during runtime.
It is necessary to provide a way for workarounds and erratums
to be applied only for certain HW versions.

The copy of MIDR is now stored in pcpu to provide a fast and
convenient way for assambly code to read it (pcpu is used quite often
so there is a chance it's inside the cache).
The MIDR is also better way of identification than using user-friendly
cpu_desc structure, because it can be compiled into comparision of
single u32 with only one access to the memory - this is crucial
for some erratums which are called from performance-critical
places.

Changes in cpu_identify makes this function safe to be called
on non-boot CPUs.

New function CPU_MATCH was implemented which returns boolean
value based on mathing masked MIDR with chip identification.
Example of usage:

printf("is thunder: %d\n", CPU_MATCH(CPU_IMPL_MASK | CPU_PART_MASK,
        CPU_IMPL_CAVIUM, CPU_PART_THUNDER, 0, 0));
printf("is generic: %d\n", CPU_MATCH(CPU_IMPL_MASK | CPU_PART_MASK,
        CPU_IMPL_ARM, CPU_PART_FOUNDATION, 0, 0));

Reviewed by:   andrew
Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3030
2015-07-09 11:32:29 +00:00
Konstantin Belousov
8954a9a4e6 Add the atomic_thread_fence() family of functions with intent to
provide a semantic defined by the C11 fences with corresponding
memory_order.

atomic_thread_fence_acq() gives r | r, w, where r and w are read and
write accesses, and | denotes the fence itself.

atomic_thread_fence_rel() is r, w | w.

atomic_thread_fence_acq_rel() is the combination of the acquire and
release in single operation.  Note that reads after the acq+rel fence
could be made visible before writes preceeding the fence.

atomic_thread_fence_seq_cst() orders all accesses before/after the
fence, and the fence itself is globally ordered against other
sequentially consistent atomic operations.

Reviewed by:	alc
Discussed with:	bde
Sponsored by:	The FreeBSD Foundation
MFC after:	3 weeks
2015-07-08 18:12:24 +00:00
Andrew Turner
6bae05d951 Correctly set __WCHAR_MIN, there is no __UINT_MIN, it's 0.
Sponsored by:	ABT Systems Ltd
2015-07-08 16:18:28 +00:00
Andrew Turner
ded32d88f1 Add support for ipi_all_but_self on arm64.
Obtained from:	ABT Systems Ltd
Sponsored by:	The freeBSD Foundation
2015-07-08 15:32:59 +00:00
Andrew Turner
80ad08a3e9 Add an implementation of savectx that doesn't just call panic.
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2015-07-08 14:07:06 +00:00
Andrew Turner
cb02f6b942 Send the correct signal when vm_fault fails. While here also set the code
and address fields.

Sponsored by:	ABT Systems Ltd
2015-07-08 12:42:44 +00:00
Zbigniew Bodek
1ae9c994c8 Introduce ITS support for ARM64
Add ARM ITS (Interrupt Translation Services) support required
to bring-up message signalled interrupts on some ARM64 platforms.

Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
2015-07-06 18:27:41 +00:00
Andrew Turner
b67d1aad6f Add more tlb invalidations. We currently invalidate when we may not need
to, but with this I can boot on a simulator that models the tlb.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2015-07-06 18:27:18 +00:00
Andrew Turner
5f8583891f Add the kernel functions needed to enable threading.
Sponsored by:	ABT Systems Ltd
2015-07-05 18:16:06 +00:00
George V. Neville-Neil
0661a7c224 Fix up tabs vs. spaces 2015-07-04 20:31:06 +00:00
George V. Neville-Neil
3839369c03 Enable IPSEC in all GENERIC kernels.
Universe and kernel build tests passed 4 July 2015

PR:		128030
Sponsored by:	Rubicon Communications (Netgate)
2015-07-04 17:37:00 +00:00
Andrew Turner
d2676f552e Remove an unneeded define and old comment referencing amd64. 2015-07-02 16:13:29 +00:00
Andrew Turner
b9b3574474 Remove an old comment, the cache is enabled.
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2015-07-02 15:26:40 +00:00
Andrew Turner
40fc1dffc3 Use pmap_load to load table entries. This simplifies finding places where
we access the tables.

Obtained from:	ABT Systems Ltd
Sponsored by:	The fReeBSD Foundation
2015-07-02 15:17:30 +00:00
Andrew Turner
a380ef6a02 Enable kernel debugging on arm64, other than GDB as it fails to build.
Sponsored by:	ABT Systems Ltd
2015-07-02 14:35:30 +00:00
Andrew Turner
c950fb6b67 Fix the logic for when to restore the VFP registers. It should restore
them when a different thread last used them, or when the thread was last
run on a different cpu.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2015-07-01 17:27:44 +00:00
Ruslan Bukin
b78ee15e9f First cut of DTrace for AArch64.
Reviewed by:	andrew, emaste
Sponsored by:	ARM Limited
Differential Revision:	https://reviews.freebsd.org/D2738
2015-07-01 15:51:11 +00:00
Konstantin Belousov
773554f79e Remove sv_sigtbl handling from the arm64 sendsig(). There is no ABI
emulators on arm64.

Reviewed by:	andrew
Review:	https://reviews.freebsd.org/D2889
Sponsored by:	The FreeBSD Foundation
2015-06-29 10:31:12 +00:00
Ed Maste
ae1860cb89 add floatingpoint.h for arm64
On other architectures floatingpoint.h is a symlink to
machine/floatingpoint.h which in turn includes machine/ieeefp.h.
Do this on arm64 as well for now.

Sponsored by:	The FreeBSD Foundation
2015-06-24 14:51:53 +00:00
Andrew Turner
e2def8eae3 Fix the floating-point exception values to line up with the hardware
register bits. Nothing in base uses these as they are deprecated, however
third-party applications, such as perl, expect some of these functions to
exist.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2015-06-24 12:19:49 +00:00
Andrew Turner
617994efc7 Add basic support for ACPI. It splits out the nexus driver to two new
drivers, one for fdt, one for acpi. It then uses this to decide if it will
use fdt or acpi.

The GICv2 (interrupt controller) and Generic Timer drivers have been
updated to handle both cases.

As this is early code we still need FDT to find the kernel console, and
some parts are still missing, including PCI support.

Differential Revision:	https://reviews.freebsd.org/D2463
Reviewed by:	jhb, jkim, emaste
Obtained from:	ABT Systems Ltd
Relnotes:	Yes
Sponsored by:	The FreeBSD Foundation
2015-06-11 15:45:33 +00:00
Ruslan Bukin
3f66bd5036 Split out db_unwind_frame() so it can be used by DTrace.
Sponsored by:	ARM Ltd.
Differential Revision:	https://reviews.freebsd.org/D2741
2015-06-11 12:47:13 +00:00
Zbigniew Bodek
458f2175ca Add options to dmb() and dsb() macros on ARM64
Using plain dsb()/dmb() as full system barriers is usually to much.
Adding proper options to those barriers (instead of full system - sy)
will most likely reduce the cost of the instructions and will benefit
in performance improvement.
This commit adds options to barrier macro definitions.

Obtained from: Semihalf
Reviewed by:   andrew, ian
Sponsored by:  The FreeBSD Foundation
2015-06-09 23:54:20 +00:00
Alan Cox
966272ca33 Retire VM_FREEPOOL_CACHE as the next step in eliminating PG_CACHE pages.
Differential Revision:	https://reviews.freebsd.org/D2712
Reviewed by:	kib
Sponsored by:	EMC / Isilon Storage Division
2015-06-08 04:59:32 +00:00
Andrew Turner
284743a867 Rework exception entry to help with DTrace. We now store the stack pointer
before adjusting it to store any registers. This is needed as DTrace may
need to adjust the kernel stack pointer, and previously the new stack
pointer would have needed to be checked incase it was changed.
2015-06-06 21:52:46 +00:00
Pedro F. Giffuni
cd508278c1 ddb: finish converting boolean values.
The replacement started at r283088 was necessarily incomplete without
replacing boolean_t with bool.  This also involved cleaning some type
mismatches and ansifying old C function declarations.

Pointed out by:	bde
Discussed with:	bde, ian, jhb
2015-05-21 15:16:18 +00:00
Ruslan Bukin
bc88bb2bf3 Add Performance Monitoring Counters support for AArch64.
Family-common and CPU-specific counters implemented.

Supported CPUs: ARM Cortex A53/57/72.

Reviewed by:	andrew, bz, emaste, gnn, jhb
Sponsored by:	ARM Limited
Differential Revision:	https://reviews.freebsd.org/D2555
2015-05-19 15:25:47 +00:00
Edward Tomasz Napierala
ba8f0eb8fc Build GENERIC with RACCT/RCTL support by default. Note that it still
needs to be enabled by adding "kern.racct.enable=1" to /boot/loader.conf.

Differential Revision:	https://reviews.freebsd.org/D2407
Reviewed by:	emaste@, wblock@
MFC after:	1 month
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
2015-05-14 14:03:55 +00:00
Zbigniew Bodek
42cb216a43 Add support for ARM GICv3 interrupt controller used in some ARM64 chips
GICv3 allows to distribute interrupts to more than 8 cores served by
the previous GIC revisions. GICv3 introduces additional logic in form
of Re-Distributors associated with particular CPUs to determine
the highest priority interrupts and manage PPIs and LPIs
(Locality-specific Peripheral Interrupts). Interrupts routing is
based on CPUs' affinity numbers. CPU interface was changed to be
accessible via CPU System Registers and this is the preferred
(and supported) method in this driver.

Obtained from: Semihalf
Reviewed by:   andrew, emaste, ian, imp
Sponsored by:  The FreeBSD Foundation
2015-05-13 18:57:03 +00:00
Zbigniew Bodek
1ca4eb3acc Port x86 busdma to ARM64
The x86 busdma subsystem allows using multiple implementations.
By default the classic bounce buffer approach is used, however
on systems with IOMMU it could be in runtime switched to more
efficient hardware accelerated implementation.

This commit adds ARM64 port of the x86 busdma framework and bounce
buffer backend. It is ready to use on IO coherent systems. If the
IO coherency cannot be guaranteed, the cache management operations have
to be added to this code in places marked by /* XXX ARM64TODO (...) */
comments. Also IOMMU support might be added by registering another
busdma implementation like it is already done on the x86.

Reviewed by:   andrew, emaste
Obtained from: Semihalf
Sponsored by:  The FreeBSD Foundation
2015-05-08 18:47:19 +00:00
Andrew Turner
2c0d026b52 Move the point we attach the ofw driver on arm64 to nexus.c. This will
allow us to have a single place to decide to use ofw or acpi.
2015-05-05 11:13:16 +00:00
Andrew Turner
85ffc58684 Add pmap_mapbios and pmap_unmapbios. These will be needed to support acpi.
Sponsored by:	The FreeBSD Foundation
2015-04-29 15:00:43 +00:00
Ruslan Bukin
a74af3dc69 o Fix bus_space_read_multi functions: correct increment value
o Use these functions for streaming

Tested with ATA (PCI bus) / Gem5 simulator

Reviewed by:	andrew
Sponsored by:	HEIF5
2015-04-28 20:33:31 +00:00
Andrew Turner
1381f830a5 Return NULL on failure from nexus_alloc_resource as it reutrns a pointer. 2015-04-27 15:16:51 +00:00
Zbigniew Bodek
a364e1cf26 Fix wrong cast of pointer to u_int in db_interface.c on ARM64
Fix to commit introduced in: r282078

Pointed out by: andrew
Obtained from:  Semihalf
Sponsored by:   The FreeBSD Foundation
2015-04-27 14:18:07 +00:00
Zbigniew Bodek
988b7f6045 Introduce ddb(4) support for ARM64
Obtained from: Semihalf
Reviewed by:   emaste
Sponsored by:  The FreeBSD Foundation
2015-04-27 09:12:54 +00:00
Andrew Turner
6c2d9cea22 Add the null implementation of stack_save and stack_save_td.
Sponsored by:	The FreeBSD Foundation
2015-04-22 12:24:38 +00:00
Andrew Turner
6dd719a463 Remove the calls to panic from pmap_object_init_pt and pmap_clear_modify.
Sponsored by:	The FreeBSD Foundation
2015-04-22 09:52:51 +00:00
Andrew Turner
9d77aa2a87 Use the address of fsu_intr_fault, not the first two instructions in it as
the address to check if we are handling a trap for fuswintr or suswintr.

Sponsored by:	The FreeBSD Foundation
2015-04-17 12:58:09 +00:00
Ed Maste
8b25059d94 Set MACHINE_ARCH to aarch64 for arm64
Reviewed by:	imp
Sponsored by:	The FreeBSD Foundation
2015-04-16 19:23:08 +00:00
Andrew Turner
731b28fb31 Enter a critical section when storing the vfp registers, we don't want to
be preempted here as this will enter back into this function, but the
hardware could be in an inconsistant state, and the vfp unit will be off
when switced back to this function.

Sponsored by:	The FreeBSD Foundation
2015-04-15 14:30:07 +00:00
Andrew Turner
2db317ca85 Ensure the userland thread and floating-point state has been saved before
copying the pcb. These values may have been changed just before the call
to fork and without a call to cpu_switch, where they would have been saved.

Sponsored by:	The FreeBSD Foundation
2015-04-15 14:18:25 +00:00
Andrew Turner
0cafabf97f Add support for arm64 to loader.efi and boot1.efi
Reviewed by:	emaste
Sponsored by:	The FreeBSD Foundation
2015-04-14 13:55:01 +00:00
Andrew Turner
7d9cff3d23 Add psci for arm64. This is currently used when rebooting and powering off
in qemu.

Sponsored by:	The FreeBSD Foundation
2015-04-13 16:41:33 +00:00
Andrew Turner
e5acd89c78 Bring in the start of the arm64 kernel.
This is only the minimum set of files needed to boot in qemu. As such it is
missing a few things.

The bus_dma code is currently only stub functions with a full implementation
from the development tree to follow.

The gic driver has been copied as the interrupt framework is different. It
is expected the two drivers will be merged by the arm intrng project,
however this will need to be imported into the tree and support for arm64
would need to be added.

This includes code developed by myself, SemiHalf, Ed Maste, and Robin
Randhawa from ARM. This has been funded by the FreeBSD Foundation, with
early development by myself in my spare time with assistance from Robin.

Differential Revision:	https://reviews.freebsd.org/D2199
Reviewed by:	emaste, imp
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
2015-04-13 14:43:10 +00:00
Andrew Turner
8c9db319f6 We will have fueword on arm64, mark as such in machine/param.h.
Sponsored by:	The FreeBSD Foundation
2015-04-07 13:17:28 +00:00
Andrew Turner
46f52b028f Split out the _acq and _rel functions. These were the same, but there is
no need for them to be this strong, we only need to provide one or the
other.

While here replace atomic_load_acq_* and atomic_store_rel_* with a single
instruction version, and fix the definition of atomic_clear_* to point to
the correct functions.

Sponsored by:	The FreeBSD Foundation
2015-04-06 16:27:22 +00:00
Andrew Turner
23f518054b Add space for future expansion [1]. While here clean up a little, spsr is
only 32-bits, and mark the holes in the struct with a pad member.

Suggested by:	kib [1]
Sponsored by:	The FreeBSD Foundation
2015-04-03 10:56:42 +00:00
Andrew Turner
147b9d0418 Add more arm64 machine dependent headers. With this we now have the minimum
set of machine headers needed to build the userland toolchain.

Differential Revision:	https://reviews.freebsd.org/D2148
Reviewed by:	imp
Sponsored by:	The FreeBSD Foundation
2015-03-26 21:10:42 +00:00
Ed Maste
f72c920c5f Renumber clauses to avoid missing 3 2015-03-23 16:04:04 +00:00
Andrew Turner
412042e2ae Add the start of the arm64 machine headers. This is the subset needed to
start getting userland libraries building.

Reviewed by:	imp
Sponsored by:	The FreeBSD Foundation
2015-03-23 11:54:56 +00:00