Commit Graph

1205 Commits

Author SHA1 Message Date
Mark Johnston
8db2e8fd16 Remove the secondary_stacks array in arm64 and riscv kernels.
Instead, dynamically allocate a page for the boot stack of each AP when
starting them up, like we do on x86.  This shrinks the bss by
MAXCPU*KSTACK_PAGES pages, which corresponds to 4MB on arm64 and 256KB
on riscv.

Duplicate the logic used on x86 to free the bootstacks, by using a
sysinit to wait for each AP to switch to a thread before freeing its
stack.

While here, mark some static MD variables as such.

Reviewed by:	kib
MFC after:	1 month
Sponsored by:	Juniper Networks, Klara Inc.
Differential Revision:	https://reviews.freebsd.org/D24158
2020-03-24 18:43:23 +00:00
Brandon Bergren
3069380898 [PowerPC][Book-E] Fix missing load base in elf_cpu_parse_dynamic().
When I implemented MD DYNAMIC parsing, I was originally passing a
linker_file_t so that the MD code could relocate pointers.

However, it turns out this isn't even filled in until later, so it was
always 0.

Just pass the load base (ef->address) directly, as that's really the only
thing we were interested in in the first place.

This fixes a crash on RB800 where it was trying to write to an unmapped
address when updating the GOT.

Reviewed by:	jhibbits
Sponsored by:	Tag1 Consulting, Inc.
Differential Revision:	https://reviews.freebsd.org/D24105
2020-03-18 02:58:18 +00:00
Michal Meloun
711b7264f4 Add the missing brackets to the logical expression.
Reported by:	clang10 via dim
MFC with:	r355755
2020-03-09 13:36:45 +00:00
Andrew Turner
3a1c1a303e Add more are64 special register fields
Obtained from:	https://github.com/FreeBSD-UPB/freebsd
2020-03-06 16:00:35 +00:00
Andrew Turner
db724d9005 Update the hypervisor registers
- Add more registers needed by bhyve [1]
 - Move EL2 registers from armreg.h to hypervisor.h
 - Add the register name to hypervisor.h

Obtained from:	https://github.com/FreeBSD-UPB/freebsd [1]
2020-03-06 14:46:50 +00:00
Andrew Turner
0c4be71a59 Mark the arm64 machdep.h as kernel only
None of this is useful for userspace.

Sponsored by:	Innovate UK
2020-03-05 10:52:16 +00:00
Andrew Turner
0e6627350e Fix the spelling of aliasing.
Sponsored by:	Innovate UK
2020-03-03 15:31:40 +00:00
Andrew Turner
411531cf30 Move the arm64 cache identification to identcpu.c
This allows us to call it on a per-CPU basis and to warn if the details
are different across CPUs.

While here read the L1 I-Cache type and store this for use later by pmap.

Sponsored by:	Innovate UK
2020-03-03 15:25:01 +00:00
Andrew Turner
2923027c15 Fix the spelling of the VIPT cache type field
Sponsored by:	Innovate UK
2020-03-03 12:50:45 +00:00
Andrew Turner
228b87bc31 Store the boot exception level on arm64 so it can be queried later
A hypervisor, e.g. bhyve, will need to know what exception levelthe kernel
was in when it started booting. If it was EL2 we can then enable said
hypervisor.

Store the boot exception level and allow the kernel to later query it.

Obtained from:	https://github.com/FreeBSD-UPB/freebsd (earlier version)
Sponsored by:	Innovate UK
2020-03-03 08:28:16 +00:00
Andrew Turner
80b9f30590 Add a space missed in r358545
Sponsored by:	Innovate UK
2020-03-02 14:36:15 +00:00
Andrew Turner
15f8f7200f Generate the offsets for struct arm64_bootparams and use it in locore.S
This removes one place with hard coded offsets in locore.S

Sponsored by:	Innovate UK
2020-03-02 14:06:50 +00:00
Dimitry Andric
b95cee35d3 Fix the following -Werror warning from clang 10.0.0:
sys/arm64/arm64/identcpu.c:1170:5: error: misleading indentation; statement is not part of the previous 'if' [-Werror,-Wmisleading-indentation]
                                break;
                                ^
sys/arm64/arm64/identcpu.c:1168:4: note: previous statement is here
                        if (fv[j].desc[0] != '\0')
                        ^

The break should be after the if statement, indented one level less.

MFC after:	3 days
Differential Revision: https://reviews.freebsd.org/D23871
2020-02-29 14:11:27 +00:00
Andrew Turner
efc228ae21 Fix the cache type identification
DIC and IDC are supported when the field bits are set.

Sponsored by:	Innovate UK
2020-02-26 15:56:07 +00:00
Pawel Biernacki
7029da5c36 Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many)
r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are
still not MPSAFE (or already are but aren’t properly marked).
Use it in preparation for a general review of all nodes.

This is non-functional change that adds annotations to SYSCTL_NODE and
SYSCTL_PROC nodes using one of the soon-to-be-required flags.

Mark all obvious cases as MPSAFE.  All entries that haven't been marked
as MPSAFE before are by default marked as NEEDGIANT

Approved by:	kib (mentor, blanket)
Commented by:	kib, gallatin, melifaro
Differential Revision:	https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
Andrew Turner
a4c354bdd5 Teach the arm64 ident CPU code to print non-ID registers
Add support for non-ID registers when printing CPU information. This is
used with the cache type register to print details of the cache on boot.

Sponsored by:	Innovate UK
2020-02-26 13:22:23 +00:00
Andrew Turner
249ecb6338 Generalise the arm64 ASID allocator.
The requirements of an Address Space ID allocator and a Virtual Machine ID
allocator are similar. Generalise the former code so it can be used with
the latter.

Reviewed by:	alc (previous version)
Sponsored by:	Innovate UK
Differential Revision:	https://reviews.freebsd.org/D23831
2020-02-26 11:50:24 +00:00
Andrew Turner
23982ffd47 Start to support multiple stages in the arm64 pmap.
On arm64 the stage 1 and stage 2 pte formats are similar enough we can
reuse the pmap code for both. As they are only similar and not identical
we need to know if we are managing stage 1 or stage 2 tables.

Add an enum to store this information and a check to make sure it is
set to stage 1 when we manage stage 1 pte fields.

Sponsored by:	Innovate UK
Differential Revision:	https://reviews.freebsd.org/D23830
2020-02-26 11:47:24 +00:00
Andrew Turner
c32e28d5e7 Add more arm64 CTR_EL0 register fields
While here make the _SIZE macros return the size in bytes, not the log2
of the size

Sponsored by:	Innovate UK
2020-02-26 11:29:03 +00:00
Andrew Turner
d153d023f5 Split out the stage 1 pte bits and add the stage 2 bits
In preperation for adding bhyve support to arm64 we need to split the
stage 1 and stage 2 pte fields to allow future changes to create stage 2
page tables.

MFC after:	1 month
Sponsored by:	Innovate UK
Differential Revision:	https://reviews.freebsd.org/D23669
2020-02-24 16:45:31 +00:00
Olivier Houchard
b0254d9afb debug_monitor: Avoid setting the PSR_D flag for 32bits binaries.
In dbg_monitor_exit(), avoid setting the PSR_D bit if the process is
a 32bits binary. PSR_D is an aarch64-only flags, and for aarch32 processes,
it means "run in big endian".
This should make COMPAT_FREEBSD32 run much better on arm64.
2020-02-24 16:25:11 +00:00
Pawel Biernacki
8eea36ae5b Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (13 of many)
r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are
still not MPSAFE (or already are but aren’t properly marked).
Use it in preparation for a general review of all nodes.

This is non-functional change that adds annotations to SYSCTL_NODE and
SYSCTL_PROC nodes using one of the soon-to-be-required flags.

Approved by:	kib (mentor, blanket)
Differential Revision:	https://reviews.freebsd.org/D23635
2020-02-24 10:45:22 +00:00
Emmanuel Vadot
83198172dd arm64: rockchip: rk808: Only init regulator not enabled
If a regulator is already enabled, do not set its value to the minimum
supported on the board.
This fixes booting on rock64 where we set some regulator to the minimal value
while the IPs needs more based on what the bootloader configured.

MFC after:	1 week
2020-02-24 10:40:35 +00:00
Emmanuel Vadot
d71e2ff2dc arm64: rockchip: rk_i2c: Bump to DELAY to 1000
In polling mode with use DELAY to wait for interrupts. The value was
too low for RK3328.

MFC after:	1 week
2020-02-24 10:38:38 +00:00
Mateusz Guzik
f50a2474b7 arm64: remove no longer needed atomic_load_ptr casts 2020-02-14 23:17:18 +00:00
Ruslan Bukin
667c3fc0f6 Add PCI Express driver for the ARM Neoverse N1 System Development
Platform (N1SDP).

Neoverse N1 is a high-performance ARM microarchitecture designed
by the ARM Holdings for the server market.

The PCI part on N1SDP was shipped untested and suffers from some
integration issues.

For instance accessing to not existing BDFs causes System Error
(SError) exception. To mitigate this, the firmware scans the bus,
catches SErrors and creates a table with valid BDFs. That allows
us to filter-out accesses to invalid BDFs in this driver.

Also the root complex config space (BDF == 0) has an unusual
location in memory map, so remapping accesses to it is required.

Finally, the config space is restricted to 32-bit accesses only.

This was tested on the ARM boxes kindly provided by the ARM Ltd
to the DARPA CHERI Project.

In collaboration with:	andrew
Reviewed by:	andrew
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D23349
2020-02-11 15:12:09 +00:00
Andrew Turner
eb43a03acb Implement the Linux/arm64 VDSO gettimeofday and clock_gettime
These are used by Centos 7 and will be needed for the
emulators/linux_base-c7 port.

Sponsored by:	DARPA, AFRL
2020-02-08 13:35:56 +00:00
Mark Johnston
c3d326fd44 Define MAXCPU consistently between the kernel and KLDs.
This reverts r177661.  The change is no longer very useful since
out-of-tree KLDs will be built to target SMP kernels anyway.  Moveover
it breaks the KBI in !SMP builds since cpuset_t's layout depends on the
value of MAXCPU, and several kernel interfaces, notably
smp_rendezvous_cpus(), take a cpuset_t as a parameter.

PR:		243711
Reviewed by:	jhb, kib
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D23512
2020-02-05 19:08:21 +00:00
Ed Maste
fc7510aef7 linuxulator: implement sendfile
Submitted by:	Bora Özarslan <borako.ozarslan@gmail.com>
Submitted by:	Yang Wang <2333@outlook.jp>
Reviewed by:	markj
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D19917
2020-02-05 16:53:02 +00:00
Andrew Turner
9bf9b8b93a Print useful debug data on unhandled kernel fault on arm64
When panicing because of an unhandled data abort from the kernel it is
useful to know the register state and faulting address to aid debugging.
Print these registers before calling panic.

Sponsored by:	DARPA, AFRL
2020-02-04 12:33:00 +00:00
Mark Johnston
a83c682b36 Dynamically select LSE-based atomic(9)s on arm64.
Once all CPUs are online, determine if they all support LSE atomics and
set lse_supported to indicate this.  For now the atomic(9)
implementations are still always inlined, though it would be preferable
to create out-of-line functions to avoid text bloat.  This was not done
here since big.little systems exist in which some CPUs implement LSE
while others do not, and ifunc resolution must occur well before this
scenario can be detected.  It does seem unlikely that FreeBSD will
ever run on such platforms, however, so converting atomic(9) to use
ifuncs is probably a good next step.

Add a LSE_ATOMICS arm64 kernel configuration option to unconditionally
select LSE-based atomic(9) implementations when the target system is
known.

Reviewed by:	andrew, kib
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation, Amazon (hardware)
Differential Revision:	https://reviews.freebsd.org/D23325
2020-02-03 18:23:50 +00:00
Mark Johnston
920de6a15f Add LSE-based atomic(9) implementations.
These make use of the cas*, ld* and swp instructions added in ARMv8.1.
Testing shows them to be significantly more performant than LL/SC-based
implementations.

No functional change here since the wrappers still unconditionally
select the _llsc variants.

Reviewed by:	andrew, kib
MFC after:	1 month
Submitted by:	Ali Saidi <alisaidi@amazon.com> (original version)
Differential Revision:	https://reviews.freebsd.org/D23324
2020-02-03 18:23:35 +00:00
Mark Johnston
c1fced6800 Add wrappers for arm64 atomics.
Add a _llsc suffix for the existing LL/SC-based implementations and add
trivial wrappers.  This is in preparation for supporting LSE-based
atomic(9) implementations.

No functional change intended.

Reviewed by:	andrew, kib
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation, Amazon (hardware)
Differential Revision:	https://reviews.freebsd.org/D23323
2020-02-03 18:23:14 +00:00
Mark Johnston
3ad6c736cf Provide a single implementation for each of the arm64 atomic(9) ops.
Parameterize the macros by type width as well as acq/rel semantics.
This makes modifying the implementations much less tedious and
error-prone and makes it easier to support alternate LSE-based
implementations.  No functional change intended.

Reviewed by:	andrew, kib
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation, Amazon (hardware)
Differential Revision:	https://reviews.freebsd.org/D23322
2020-02-03 18:22:59 +00:00
Andrew Turner
59606417c4 Remove the GICv3 ITS irq and replace it with an ID
In r357324 most of the use of gi_irq was moved to gi_lpi. Complete this
with the last few places we need the IRQ value and create gi_id for the
per-device value we need.

MFC after:	2 weeks
Sponsored by:	DARPA, AFRL
2020-02-03 14:38:19 +00:00
Andrew Turner
7877018c2c Use a unique name for the GICv3 ITS vmem
When there are multiple GICv3 ITS devices we don't know which vmem is for
which device. Use device_get_nameunit to get a per-device name.

MFC after:	2 weeks
Sponsored by:	DARPA, AFRL
2020-02-03 13:50:55 +00:00
Andrew Turner
a58fc7cb88 Disable the use of the quantum cache in the GICv3 ITS
This uses UMA to allocate space. It causes issues when there are multiple
ITS devices in the system where interrupts are not allocated from a low
address on some interrupt controllers. Disabling the quantum cache fixes
this on the Neoverse N1 SDP.

MFC after:	2 weeks
Sponsored by:	DARPA, AFRL
2020-02-03 13:47:41 +00:00
Mark Johnston
1c29da0279 Reimplement stack capture of running threads on i386 and amd64.
After r355784 the td_oncpu field is no longer synchronized by the thread
lock, so the stack capture interrupt cannot be delievered precisely.
Fix this using a loop which drops the thread lock and restarts if the
wrong thread was sampled from the stack capture interrupt handler.

Change the implementation to use a regular interrupt instead of an NMI.
Now that we drop the thread lock, there is no advantage to the latter.

Simplify the KPIs.  Remove stack_save_td_running() and add a return
value to stack_save_td().  On platforms that do not support stack
capture of running threads, stack_save_td() returns EOPNOTSUPP.  If the
target thread is running in user mode, stack_save_td() returns EBUSY.

Reviewed by:	kib
Reported by:	mjg, pho
Tested by:	pho
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D23355
2020-01-31 15:43:33 +00:00
Andrew Turner
80ba579b00 Call the MAPTI command earlier in the ITS driver
The GICv3 Software Overview suggests when allocating a new MSI/MSI-X
interrupt we need to call MAPD followed by MAPTI. Unfortunately the code
would place a MOVI command between these. This is invalid as it needs
values set by the MAPTI to be present.

Re-order so we allocate a temporary CPU for the interrupt, then use the
MAPTI command to assign the MSI to it.

MFC after:	2 weeks
Sponsored by:	DARPA, AFRL
2020-01-31 11:33:11 +00:00
Andrew Turner
e93448dd69 Only create one ITS configuration table
When there are multiple ITS devices in the system we would allocate a
configuration table for each, however only one table is needed as all the
ITS devices share this.

Allocate a table only when the global table is unset.

While here fix the type of this to be a pointer to a uint8_t array as the
entries are all 8 bits wide.

MFC after:	2 weeks
Sponsored by:	DARPA, AFRL
2020-01-31 10:30:13 +00:00
Andrew Turner
45951bf0a9 Ignore the SMMUv3 and PMCG interrupt controller in the IORT tables
When mapping MSI/MSI-X interrupts throught he Arm IORT ACPI tables we may
need to ignore an interrupt controller even if it is within the bounds the
entry describes. When the SMMUv3 is not using GSIV (non-MSI/MSI-X)
interrupts we need to read the defined field. The Performance Monitoring
Counter Group always ignores the first table entry.

MFC after:	2 weeks
Sponsored by:	DARPA, AFRL
2020-01-31 09:51:38 +00:00
Andrew Turner
9c7eb5ca62 Shift the ITS processor ID after reading it.
When using the processor ID value we mask off the low and high bits that
should be zero. Unfortunatly we don't shift the ID value so it won't be
affected. Add the shift when reading the ID as this will need to align
with the address based target value.

MFC after:	2 weeks
Sponsored by:	DARPA, AFRL
2020-01-30 18:49:19 +00:00
Ganbold Tsagaankhuu
5e1e7e6192 Enable USB3 support for Rockchip RK3328 SoC.
Reviewed by:	manu
2020-01-29 09:36:59 +00:00
Ganbold Tsagaankhuu
87fcb283c5 Add USB3 related clock definitions for Rockchip RK3328 SoC.
Reviewed by:	manu
2020-01-29 08:46:35 +00:00
Mark Johnston
b6cf94ae07 Print missing ID_AA64PFR{0,1}_EL1 register fields.
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D23213
2020-01-23 16:10:38 +00:00
Mark Johnston
32a1f92f2c arm64: Don't enable interrupts in init_secondary().
Doing so can cause deadlocks or panics during boot, if an interrupt
handler accesses uninitialized per-CPU scheduler structures.  This seems
to occur frequently when running under QEMU or AWS.  The idle threads
are set up to release a spinlock section and enable interrupts in
fork_exit(), so there is no need to enable interrupts earlier.

Reviewed by:	kib
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D23328
2020-01-23 16:07:27 +00:00
Mark Johnston
9ddf3d342b Add relocation handling required for -zifunc-noplt to work on arm64.
Static relocations for the immediate operand of a branch instruction
must be applied.

In a patch which implements LSE-based atomic(9) operations using ifuncs,
-zifunc-noplt reduces system CPU usage during a buildkernel by several
percent.

Also fix elf_reloc_internal() to return an error if symbol lookup fails.

Reviewed by:	andrew
MFC after:	2 weeks
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D17392
2020-01-21 17:45:49 +00:00
Emmanuel Vadot
f70961bf86 rk805: Add a regnode_init method
This method will set the desired voltaged based on values in the DTS.
It will not enable the regulator, this is the job of either a consumer
or regnode_set_constraint SYSINIT if the regulator is boot_on or always_on.

Reviewed by:	mmel
MFC after:	1 month
Differential Revision:	https://reviews.freebsd.org/D23216
2020-01-19 19:56:50 +00:00
Kyle Evans
05d7dd739c sysent targets: further cleanup and deduplication
r355473 vastly improved the readability and cleanliness of these Makefiles.
Every single one of them follows the same pattern and duplicates the exact
same logic.

Now that we have GENERATED/SRCS, split SRCS up into the two parameters we'll
use for ${MAKESYSCALLS} rather than assuming a specific ordering of SRCS and
include a common sysent.mk to handle the rest. This makes it less tedious to
make sweeping changes.

Some default values are provided for GENERATED/SYSENT_*; almost all of these
just use a 'syscalls.master' and 'syscalls.conf' in cwd, and they all use
effectively the same filenames with an arbitrary prefix. Most ABIs will be
able to get away with just setting GENERATED_PREFIX and including
^/sys/conf/sysent.mk, while others only need light additions. kern/Makefile
is the notable exception, as it doesn't take a SYSENT_CONF and the generated
files are spread out between ^/sys/kern and ^/sys/sys, but it otherwise fits
the pattern enough to use the common version.

Reviewed by:	brooks, imp
Nice!:		emaste
Differential Revision:	https://reviews.freebsd.org/D23197
2020-01-18 20:37:45 +00:00
John Baldwin
ffbf0e3ce4 Save and restore floating point registers in get/set_mcontext().
arm64 and riscv were only saving and restoring floating point
registers for sendsig() and sys_sigreturn(), but not for getcontext(),
setcontext(), and swapcontext().

While here, remove an always-false check for uap being NULL from
sys_sigreturn().

Reviewed by:	br, mhorne
Sponsored by:	DARPA
Differential Revision:	https://reviews.freebsd.org/D23218
2020-01-17 19:01:59 +00:00
Emmanuel Vadot
bcd380e88b arm64: rockchip: Add RK3399 PWM driver
Add a driver for the pwm controller in the RK3399 SoC

Submitted by:	bdragon (original version)
Reviewed by:	ganbold (previous version)
MFC after:	1 month
Differential Revision:	https://reviews.freebsd.org/D19046
2020-01-16 21:25:13 +00:00
Emmanuel Vadot
3ee778c15e arm64: rockchip: Add new interface for rk_pinctrl
The gpio controller in rockchips SoC in a child of the pinctrl driver
and cannot control pullups and pulldowns.
Use the new fdt_pinctrl interface for accessing pin capabilities and
setting them.
We can now report that every pins is capable of being IN or OUT function
and PULLUP PULLDOWN.
If the pin isn't in gpio mode no changes will be allowed.

Reviewed by:	ganbold (previous version)
MFC after:	1 month
Differential Revision:	https://reviews.freebsd.org/D22849
2020-01-16 21:21:20 +00:00
Kyle Evans
1171c633fb Set .ORDER for makesyscalls generated files
When either makesyscalls.lua or syscalls.master changes, all of the
${GENERATED} targets are now out-of-date. With make jobs > 1, this means we
will run the makesyscalls script in parallel for the same ABI, generating
the same set of output files.

Prior to r356603 , there is a large window for interlacing output for some
of the generated files that we were generating in-place rather than staging
in a temp dir. After that, we still should't need to run the script more
than once per-ABI as the first invocation should update all of them. Add
.ORDER to do so cleanly.

Reviewed by:	brooks
Discussed with:	sjg
Differential Revision:	https://reviews.freebsd.org/D23099
2020-01-10 18:24:17 +00:00
Andrew Turner
f2792e5e55 Add atomic_testandset/clear on arm64.
These will reportedly be used in future uma changes.

MFC after:	2 weeks
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D23019
2020-01-09 10:26:36 +00:00
Emmanuel Vadot
546652fbe3 rk805: Add regnode_status method
This allow consumers to check if the regulator is enable or not.

Reviewed by:	mmel
Differential Revision:	https://reviews.freebsd.org/D23005
2020-01-08 11:30:03 +00:00
Emmanuel Vadot
d194b63134 rk808: Add min/max for the switch regulators
The two switch regulator are always 3.0V.
Add a special case in get_voltage that if min=max we directly
return the value without calculating it.

Reviewed by:	mmel
Differential Revision:	https://reviews.freebsd.org/D23004
2020-01-08 11:29:22 +00:00
Andrew Turner
1b02a76602 Add more Arm arm64 CPU identification values
- Add all the Cortex-A CPU ID register values I can find.
 - Add the Neoverse-N1 ID regiser value [1]
 - Sort macros by register value.

PR:		243065
Submitted by:	Ali Saidi <alisaidi AT amazon.com> [1]
Sponsored by:	DARPA, AFRL (other than [1])
2020-01-06 20:57:59 +00:00
Pawel Biernacki
a1d7296784 sysctl: mark more nodes as MPSAFE
vm.kvm_size and vm.kvm_free are read only and marked as MPSAFE on i386
already. Mark them as that on amd64 and arm64 too to avoid locking Giant.

Reviewed by:	kib (mentor)
Approved by:	kib (mentor)
Differential Revision:	https://reviews.freebsd.org/D23039
2020-01-06 10:52:13 +00:00
Alan Cox
1c3a241032 When a copy-on-write fault occurs, pmap_enter() is called on to replace the
mapping to the old read-only page with a mapping to the new read-write page.
To destroy the old mapping, pmap_enter() must destroy its page table and PV
entries and invalidate its TLB entry.  This change simply invalidates that
TLB entry a little earlier, specifically, on amd64 and arm64, before the PV
list lock is held.

Reviewed by:	kib, markj
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D23027
2020-01-04 19:50:25 +00:00
Andrew Turner
6d26116baa Add the 8 and 16 bit atomic load/store functions with a barrier on arm64.
Reviewed by:	cem
MFC after:	2 weeks
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D22966
2020-01-03 10:03:36 +00:00
Ian Lepore
fce985b0bd Add the xdma framework and pl330 dma drivers to arm and arm64 lint builds. 2020-01-02 22:26:54 +00:00
Emmanuel Vadot
84e404763f arm64: rockchip: Add a module for rk_spi
The spi node doesn't lives under a simple-bus compatible node so we need
OFWBUS_PNP_INFO instead of SIMPLEBUS_PNP_INFO.
2020-01-02 17:44:41 +00:00
Edward Tomasz Napierala
cc50333011 Add basic getcpu(2) support to linuxulator. The purpose of this
syscall is to query the CPU number and the NUMA domain the calling
thread is currently running on.  The third argument is ignored.
It doesn't do anything regarding scheduling - it's literally
just a way to query the current state, without any guarantees
you won't get rescheduled an opcode later.

This unbreaks Java from CentOS 8
(java-11-openjdk-11.0.5.10-0.el8_0.x86_64).

Reviewed by:	kib
MFC after:	2 weeks
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D22972
2019-12-31 22:01:08 +00:00
Alan Cox
b0a0152ab9 Determine whether the MMU hardware is capable of updating a page table
entry's access flag and dirty state, and enable this feature when it's
available.

Ensure that we don't overlook a dirty state update that is concurrent
with a call to pmap_enter().  (Previously, all dirty state updates would
have occurred with the containing pmap's lock held, so a page table entry's
dirty state could not have changed while pmap_enter() held that same lock.)

Reviewed by:	andrew, markj
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D22907
2019-12-30 20:30:31 +00:00
Pawel Biernacki
54666dffa8 linux(4): implement copy_file_range(2)
copy_file_range(2) is implemented natively since r350315, make it available
for Linux binaries too.

Reviewed by:	kib (mentor), trasz (previous version)
Approved by:	kib (mentor)
Differential Revision:	https://reviews.freebsd.org/D22959
2019-12-30 18:11:06 +00:00
Edward Tomasz Napierala
ee0fe82ee2 Implement Linux syslog(2) syscall; just enough to make Linux dmesg(8)
utility work.

MFC after:	2 weeks
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D22465
2019-12-29 15:53:55 +00:00
Emmanuel Vadot
6db3672c08 arm64: rockchip: Add driver for the io domain
This driver configure the registers in the GRF according to the value
of the regulators for the platform.
Some IP can run with either 3.0V or 1.8V, if we don't configure them
correctly according to the external voltage used they will not work.
It's only done at boot time for now and might be needed at runtime for
IP like sdmmc.

Reviewed by:	mmel
Tested On:	RockPro64, Firefly-RK3399 (gonzo), AIO-3288 (mmel)
MFC after:	1 month
Differential Revision:	https://reviews.freebsd.org/D22854
2019-12-28 15:30:50 +00:00
Emmanuel Vadot
b68dfe6949 arm64: rockchip: rk808: Add remaining regulators
The RK808 driver was missing the LDO and switch regulators.
Add support for them.

Reviewed by:	mmel
MFC after:	1 month
Differential Revision:	https://reviews.freebsd.org/D22852
2019-12-28 15:28:39 +00:00
Brandon Bergren
7821a820d0 [PowerPC] Implement Secure-PLT jump table processing for ppc32.
Due to clang and LLD's tendency to use a PLT for builtins, and as they
don't have full support for EABI, we sometimes have to deal with a PLT in
.ko files in a clang-built kernel.

As such, augment the in-kernel linker to support jump table processing.

As there is no particular reason to support lazy binding in kernel modules,
only implement Secure-PLT immediate binding.

As part of these changes, add elf_cpu_parse_dynamic() to the MD API of the
in-kernel linker (except on platforms that use raw object files.)

The new function will allow MD code to act on MD tags in _DYNAMIC.

Use this new function in the PowerPC MD code to ensure BSS-PLT modules using
PLT will be rejected during insertion, and to poison the runtime resolver to
ensure we get a clear panic reason if a call is made to the resolver.

Reviewed by:	jhibbits
Differential Revision:	https://reviews.freebsd.org/D22608
2019-12-24 15:56:24 +00:00
Scott Long
757d4fbaa7 Introduce the concept of busdma tag templates. A template can be allocated
off the stack, initialized to default values, and then filled in with
driver-specific values, all without having to worry about the numerous
other fields in the tag. The resulting template is then passed into
busdma and the normal opaque tag object created.  See the man page for
details on how to initialize a template.

Templates do not support tag filters.  Filters have been broken for many
years, and only existed for an ancient make/model of hardware that had a
quirky DMA engine.  Instead of breaking the ABI/API and changing the
arugment signature of bus_dma_tag_create() to remove the filter arguments,
templates allow us to ignore them, and also significantly reduce the
complexity of creating and managing tags.

Reviewed by:	imp, kib
Differential Revision:	https://reviews.freebsd.org/D22906
2019-12-24 14:48:46 +00:00
Alan Cox
7c237b7c3a Correct a mistakenly inverted condition in r355833.
Noticed by:	kib
X-MFC with:	r355833
2019-12-20 20:46:26 +00:00
Andrew Turner
494278bb26 Stop speculation past an eret instruction
On arm64 the eret instruction is used to return from an exception handler.
Some implementations may speculate past this instruction into the next
function. As the user may control many registers in these functions add
a synchronisation barrier sequence after the eret instruction to stop these
CPUs from speculating out of the exception handler.

PR:		242676
Submitted by:	Anthony Steinhauser <asteinhauser@google.com> (previous version)
MFC after:	1 week
2019-12-19 08:52:16 +00:00
Alan Cox
48ef33180a When pmap_enter_{l2,pde}() are called to create a kernel mapping, they are
incrementing (and decrementing) the ref_count on kernel page table pages.
They should not do this.  Kernel page table pages are expected to have a
fixed ref_count.  Address this problem by refactoring pmap_alloc{_l2,pde}()
and their callers.  This also eliminates some duplicated code from the
callers.

Correctly implement PMAP_ENTER_NOREPLACE in pmap_enter_{l2,pde}() on kernel
mappings.

Reduce code duplication by defining a function, pmap_abort_ptp(), for
handling a common error case.

Handle a possible page table page leak in pmap_copy().  Suppose that we are
determining whether to copy a superpage mapping.  If we abort because there
is already a mapping in the destination pmap at the current address, then
simply decrementing the page table page's ref_count is correct, because the
page table page must have a ref_count > 1.  However, if we abort because we
failed to allocate a PV entry, this might be a just allocated page table
page that has a ref_count = 1, so we should call pmap_abort_ptp().

Simplify error handling in pmap_enter_quick_locked().

Reviewed by:	kib, markj (an earlier)
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D22763
2019-12-18 18:21:39 +00:00
Ian Lepore
a856c60bdd Remove unnecessary MODULE_DEPEND() from imx_i2c.c, and also from rk_i2c
where it got copied to.
2019-12-17 17:03:03 +00:00
Emmanuel Vadot
bc188bd4f8 arm64: rockchip: rk_gpio: Fix pin number
The maxpin counter starts at 0, fix one by one error.
This is still not totally correct for some banks in some SoC that have
fewer pins but this will be dealt with in another commit.

MFC after:	3 days
2019-12-17 10:57:31 +00:00
Emmanuel Vadot
3df0c02623 arm64: rockchip: rk_pinctrl: Fix clear bits in SYSCON_MODIFY
r351187 change the SYSCON_WRITE to SYSCON_MODIFY but didn't changed the
mask variable that used to hold the bitmask in the upper 16 bits of the
register that control which bits are changed. So we ended up clearing
bit from the upper 16bits half which are always 0 after a read.
Use the correct bit mask for bits that we want to clear.

MFC after:	3 days
2019-12-17 10:55:28 +00:00
Jeff Roberson
a94ba188c3 Repeat the spinlock_enter/exit pattern from amd64 on other architectures to
fix an assert violation introduced in r355784.  Without this spinlock_exit()
may see owepreempt and switch before reducing the spinlock count.  amd64
had been optimized to do a single critical enter/exit regardless of the
number of spinlocks which avoided the problem and this optimization had
not been applied elsewhere.

Reported by:	emaste
Suggested by:	rlibby
Discussed with:	jhb, rlibby
Tested by:	manu (arm64)
2019-12-16 20:15:04 +00:00
Edward Tomasz Napierala
b5f20658ee Add compat.linux.emul_path, so it can be set to something other
than "/compat/linux".  Useful when you have several compat directories
with different Linux versions and you don't want to clash with files
installed by linux-c7 packages.

Reviewed by:	bcr (manpages)
MFC after:	2 weeks
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D22574
2019-12-16 20:07:04 +00:00
Alan Cox
68ca966558 Apply a small optimization to pmap_remove_l3_range(). Specifically, hoist a
PHYS_TO_VM_PAGE() operation that always returns the same vm_page_t out of
the loop.  (Since arm64 is configured as VM_PHYSSEG_SPARSE, the
implementation of PHYS_TO_VM_PAGE() is more costly than that of
VM_PHYSSEG_DENSE platforms, like amd64.)

MFC after:	1 week
2019-12-15 22:41:57 +00:00
Michal Meloun
dfd1d0fcab Add driver for Rockchip PCIe root complex found in RK3399 SOC.
Unfortunately, there are some limitations:
- memory aperture of his controller is only 16MiB, so it is nearly
  unusable for graphic cards
- every attempt to generate type 1 config cycle always causes trap.
  These config cycles are disabled now and we don't support cards
  with PCIe switch.
- in some cases, attempt to do config cycle to (probably) not-yet ready
  card also causes trap. This cannot be detected at runtime, but it seems
  like very rare issue.

MFC after:	3 weeks
Differential Revision:  https://reviews.freebsd.org/D22724
2019-12-14 14:56:34 +00:00
Edward Tomasz Napierala
cf69fe66d4 Add sync_file_range(2) implementation to linux(4); it's a thin wrapper
over the usual fsync(2).

This silences some warnings when running "apt-get upgrade".

Reviewed by:	brooks, emaste
MFC after:	2 weeks
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D22371
2019-12-14 13:37:17 +00:00
Edward Tomasz Napierala
0cde2b3239 Regen after r355752.
MFC after:	2 weeks
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D22371
2019-12-14 13:32:37 +00:00
Edward Tomasz Napierala
0610f417a4 Fix definitions for linuxulator's sync_file_range(2).
Reviewed by:	brooks, emaste
MFC after:	2 weeks
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D22371
2019-12-14 13:30:43 +00:00
Andrew Turner
65565c9784 Add comments and macros to the tcr_el1 setting code to help understand it.
This code is non-obvious when reading for the first time. To help with
understanding of it add comments explaining what it's doing.

While here use macros from armreg.h rather than magic numbers.

Sponsored by:	DARPA, AFRL
2019-12-12 18:27:54 +00:00
Emmanuel Vadot
e63adc105d arm64: rockchip: rk_pinctrl: Fix parse_bias for RK3399
Only bank 0 and bank 2 are different than other rockchip SoC, fix this.
While here remove some debug printfs that where added in r355648

MFC after:	3 days
X-MFC-With:	r355648
2019-12-12 13:21:43 +00:00
Emmanuel Vadot
064486c05f arm64: rockchip: rk_pinctrl: Add bias parsing based on the SoC type
Not all rockchip have the same value for pullup/pulldown so add a function
per SoC and call the right one to have the proper value.

MFC after:	3 days
2019-12-12 13:02:22 +00:00
Emmanuel Vadot
6fe4e8bdbd arm64: rk3328: Add the *clk_peri_niu clocks
Those clocks are always enable by default and are not really explained
in the TRM but the reason we had them is that they have the periph clock
as a parent and those parent should never be disable which can happen
if we disable all the childs. The current childs are the sd/emmc/sdio clocks
so the board will hang if we disable them.

MFC after:	1 month
2019-12-11 18:39:05 +00:00
Emmanuel Vadot
946c511ee5 arm64: Add explicit devices for dwmmc variant
We used to include the hisi version if soc_hisi_hi6220 was present,
include the altera version if dwmmc_altera was present and include
the rockchip version if soc_rockchip_rk3328 was present.
Now every version have it's own device directive.
The rockchip version isn't named dwmmc_rockchip because all other
rockchip driver are named rk_XXX.

MFC after:	1 month
2019-12-11 18:36:07 +00:00
Mark Johnston
5cff1f4dc3 Introduce vm_page_astate.
This is a 32-bit structure embedded in each vm_page, consisting mostly
of page queue state.  The use of a structure makes it easy to store a
snapshot of a page's queue state in a stack variable and use cmpset
loops to update that state without requiring the page lock.

This change merely adds the structure and updates references to atomic
state fields.  No functional change intended.

Reviewed by:	alc, jeff, kib
Sponsored by:	Netflix, Intel
Differential Revision:	https://reviews.freebsd.org/D22650
2019-12-10 18:14:50 +00:00
John Baldwin
d8010b1175 Copy out aux args after the argument and environment vectors.
Partially revert r354741 and r354754 and go back to allocating a
fixed-size chunk of stack space for the auxiliary vector.  Keep
sv_copyout_auxargs but change it to accept the address at the end of
the environment vector as an input stack address and no longer
allocate room on the stack.  It is now called at the end of
copyout_strings after the argv and environment vectors have been
copied out.

This should fix a regression in r354754 that broke the stack alignment
for newer Linux amd64 binaries (and probably broke Linux arm64 as
well).

Reviewed by:	kib
Tested on:	amd64 (native, linux64 (only linux-base-c7), and i386)
Sponsored by:	DARPA
Differential Revision:	https://reviews.freebsd.org/D22695
2019-12-09 19:17:28 +00:00
Emmanuel Vadot
670f36540d arm64: Use the kenv provided by loader
Otherwise we have an empty kenv a likely cannot boot.

Submitted by:	kevans
MFC after:	1 month
X-MFC-With:	355487
2019-12-09 15:23:05 +00:00
Michal Meloun
5641eda2f3 Add support for booting kernel directly from U-Boot using booti command.
In some cases, like is locked bootstrap or device's inability to boot from
removable media, we cannot use standard boot sequence and is necessary to
boot kernel directly from U-Boot.

Discussed with:	jhibbits
MFC after:	1 month
Differential Revision:	https://reviews.freebsd.org/D13861
2019-12-07 16:14:23 +00:00
Brooks Davis
af796bfa71 sysent: Reduce duplication and improve readability.
Use the power of variable to avoid spelling out source and generated
files too many times.  The previous Makefiles were hard to read, hard to
edit, and badly formatted.

Reviewed by:	kevans, emaste
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D22714
2019-12-06 23:59:23 +00:00
Luiz Otavio O Souza
8cfe2a7ad3 Add the SPI driver for the Marvell Armada 37x0 SoC.
Interrupt based driver, implements SPI mode and clock configuration.

Tested on espressobin and SG-3200.

Sponsored by:	Rubicon Communications, LLC (Netgate)
2019-12-06 12:55:39 +00:00
Alan Cox
1bef4955bc On a context switch, handle the possibility that the old thread was
preempted after an "ic" or "tlbi" instruction but before it performed a
"dsb" instruction.  The "ic" and "tlbi" instructions have unusual
synchronization requirements.  If the old thread migrates to a new
processor, its completion of a "dsb" instruction on that new processor does
not guarantee that the "ic" or "tlbi" instructions performed on the old
processor have completed.

This issue is not restricted to the kernel.  Since locore.S sets the UCI bit
in SCTLR, user-space programs can perform "ic ivau" instructions (as well as
some forms of the "dc" instruction).

Reviewed by:	andrew, kib, markj, mmel
X-MFC with:	r355145
Differential Revision:	https://reviews.freebsd.org/D22622
2019-12-05 19:25:49 +00:00
Luiz Otavio O Souza
8a82d5675d Add the I2C driver for the Armada 37x0.
This controller is a bit tricky as the STOP condition must be indicated in
the last tranferred byte, some devices will not like the repeated start
behavior of this controller.  A proper fix to this issue is in the works.

This driver works in polling mode, can be used early in the boot (required
in some cases).

Tested on espressobin/SG-1100 and the SG-3200.

Obtained from:	pfSense
Sponsored by:	Rubicon Communications, LLC (Netgate)
2019-12-05 00:56:03 +00:00
Warner Losh
f86e60008b Regularize my copyright notice
o Remove All Rights Reserved from my notices
o imp@FreeBSD.org everywhere
o regularize punctiation, eliminate date ranges
o Make sure that it's clear that I don't claim All Rights reserved by listing
  All Rights Reserved on same line as other copyright holders (but not
  me). Other such holders are also listed last where it's clear.
2019-12-04 16:56:11 +00:00
Li-Wen Hsu
0386b6c8a0 Fix arm64 build after r355373
Sponsored by:	The FreeBSD Foundation
2019-12-04 08:21:54 +00:00
John Baldwin
31174518d2 Use uintptr_t instead of register_t * for the stack base.
- Use ustringp for the location of the argv and environment strings
  and allow destp to travel further down the stack for the stackgap
  and auxv regions.
- Update the Linux copyout_strings variants to move destp down the
  stack as was done for the native ABIs in r263349.
- Stop allocating a space for a stack gap in the Linux ABIs.  This
  used to hold translated system call arguments, but hasn't been used
  since r159992.

Reviewed by:	kib
Tested on:	md64 (amd64, i386, linux64), i386 (i386, linux)
Sponsored by:	DARPA
Differential Revision:	https://reviews.freebsd.org/D22501
2019-12-03 23:17:54 +00:00
Emmanuel Vadot
e213223c9b Remove "all rights reserved" from copyright for the file I own.
Some of the files have both me and Jared McNeill and he gave me
permission to remove it from his files too.
2019-12-03 21:00:45 +00:00