Commit Graph

852 Commits

Author SHA1 Message Date
Emmanuel Vadot
dff9720331 arm64: rockchip: clk: Add support to reparent to clk_composite
All clk_composite type have the possibility to reparent (choosing another
parent to find a better frequency), add the support for that.
2018-05-07 07:29:48 +00:00
Emmanuel Vadot
66a4c42756 arm64: rk3328: Add pll rates tables
Add the known value to be safe for the rk3328 PLLs
2018-05-07 07:28:47 +00:00
Emmanuel Vadot
78d07c93a7 arm64: rk: Add support for setting pll rate
Add support for setting pll rate. On RockChip SoC two kind of plls are
supported, integer mode and fractional mode.
The two modes are intended to support more frequencies for the core plls.
While here change the recalc method as it appears that the datasheet is
wrong on the calculation method.
2018-05-07 07:28:10 +00:00
Emmanuel Vadot
178f57b143 arm64: rockchip: rk3328: Add armclk clock
Add the clock definition for the arm clock.
While here remove the indexes in the clock table as we will need clock
with a 0 index (non-exported clocks).
2018-05-07 07:26:48 +00:00
Mark Johnston
20f85b1ddd Print the dump progress indicator after calling dump_start().
Dumpers may wish to print messages from an initialization hook; this
change ensures that such messages aren't mixed with output from the
generic dump code.

MFC after:	1 week
2018-05-01 17:32:43 +00:00
Emmanuel Vadot
590a3c9619 arm64: rockchip: rk_gpio fix rk_gpio_pin_config32
Pointy Hat to:	 me
2018-04-26 22:15:09 +00:00
Emmanuel Vadot
cbf65cf228 arm64: rockchip: Add gpio controller driver
Add a driver that match on 'rockchip,gpio-bank', this compatible
string is found on almost all RockChip SoC so this driver is compatible
with almost all of the RockChip SoCs.

The only features missing for this driver are :
- Interrupts support
- Debouncing
2018-04-26 21:44:00 +00:00
Emmanuel Vadot
d40a1b91c0 arm64: rockchip: RK3328 CRU Add gpio gates
Add the gates for the gpio controller in order to properly support them.
2018-04-26 21:40:05 +00:00
Emmanuel Vadot
a59a485e79 arm64: rockchip: Rk3328 CRU Fix some offset for gates
Some offset of some clock gates where wrong, correct them so we can
use thoses clocks.

Pointy Hat to:	me
2018-04-26 21:38:59 +00:00
Emmanuel Vadot
be2e72c286 arm64: rockchip: Add pinctrl driver
Add pinctrl driver for RockChip SoCs. This device manage which function
to set on which pin and some other properties like pull up/down, drive
strength etc ...
For now the driver only support RK3328 but it is versatile enough to
add support for other RockChip SoC in the future.
2018-04-26 21:37:38 +00:00
Emmanuel Vadot
49755fc6d2 arm64: rockchip: Add GRF driver
RockChip GRF (General Register Files) is present on almost all RockChip
SoC and is used to control some area of the system like iomuxing, gpio
or usb phy.
We need it to be probed and attached early in the boot process so
subclass syscon_generic and set the pass to BUS_PASS_BUS + BUS_PASS_ORDER_MIDDLE.
2018-04-26 21:35:04 +00:00
Ed Maste
5f2336efb4 arm64 linuxulator: add generated sysent files
From syscalls.master in r333027

Sponsored by:	Turing Robotic Industries Inc.
2018-04-26 18:46:38 +00:00
Ed Maste
2464626e2c Add arm64 Linux syscall table
This is the first step (after the recent refactoring of some common
code) to supporting the Linuxulator on arm64.

Reviewed by:	andrew
Sponsored by:	Turing Robotic Industries Inc.
Differential Revision:	https://reviews.freebsd.org/D15187
2018-04-26 18:38:59 +00:00
Brooks Davis
9c11d8d483 Remove the unused fuwintr() and suiwintr() functions.
Half of implementations always failed (returned (-1)) and they were
previously used in only one place.

Reviewed by:	kib, andrew
Obtained from:	CheriBSD
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D15102
2018-04-17 18:04:28 +00:00
Ruslan Bukin
5819c3eb8a Enable Qualcomm Debug Subsystem (QDSS) block on MSM8916 SoC.
This is required for ARM Coresight operation on Dragonboard 410c.

Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D14987
2018-04-10 12:53:48 +00:00
Oleksandr Tymoshenko
b7dc17589b Fix one more OF_getprop_alloc instance missed in r332310
X-MFC-With:	r332310
2018-04-08 23:17:51 +00:00
Brooks Davis
6469bdcdb6 Move most of the contents of opt_compat.h to opt_global.h.
opt_compat.h is mentioned in nearly 180 files. In-progress network
driver compabibility improvements may add over 100 more so this is
closer to "just about everywhere" than "only some files" per the
guidance in sys/conf/options.

Keep COMPAT_LINUX32 in opt_compat.h as it is confined to a subset of
sys/compat/linux/*.c.  A fake _COMPAT_LINUX option ensure opt_compat.h
is created on all architectures.

Move COMPAT_LINUXKPI to opt_dontuse.h as it is only used to control the
set of compiled files.

Reviewed by:	kib, cem, jhb, jtl
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D14941
2018-04-06 17:35:35 +00:00
Ruslan Bukin
b09de0b34a Add support for the Coresight technology from ARM Ltd.
ARM Coresight is a solution for debug and trace of complex SoC designs.

This includes a collection of drivers for ARM Coresight interconnect
devices within a small Coresight framework.

Supported devices are:
o Embedded Trace Macrocell v4 (ETMv4)
o Funnel
o Dynamic Replicator
o Trace Memory Controller (TMC)
o CPU debug module

Devices are connected to each other internally in SoC and the
configuration of each device endpoints is described in FDT.

Typical trace flow (as found on Qualcomm Snapdragon 410e):
CPU0 -> ETM0 -> funnel1 -> funnel0 -> ETF -> replicator -> ETR -> DRAM
CPU1 -> ETM1 -^
CPU2 -> ETM2 -^
CPU3 -> ETM3 -^

Note that both Embedded Trace FIFO (ETF) and Embedded Trace Router (ETR)
are hardware configurations of TMC.

This is required for upcoming HWPMC tracing support.

This is tested on single-core system only.

Reviewed by:	andrew (partially)
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D14618
2018-04-05 15:45:54 +00:00
Oleksandr Tymoshenko
de14bffe33 Fix arm64 buildkernel target with "nooptions KDB"
Make kdb_trap in breakpoint exception handler conditional. If "options KDB"
is not present just panic with message that debugger is not enabled.

PR:		224653
2018-04-04 01:13:28 +00:00
Ed Maste
fc2a8776a2 Rename assym.s to assym.inc
assym is only to be included by other .s files, and should never
actually be assembled by itself.

Reviewed by:	imp, bdrewery (earlier)
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D14180
2018-03-20 17:58:51 +00:00
Kyle Evans
63ee68c220 EFIRT: SetVirtualAddressMap with 1:1 mapping after exiting boot services
This fixes a problem encountered on the Lenovo Thinkpad X220/Yoga 11e where
runtime services would try to inexplicably jump to other parts of memory
where it shouldn't be when attempting to enumerate EFI vars, causing a
panic.

The virtual mapping is enabled by default and can be disabled by setting
efi_disable_vmap in loader.conf(5).

Reviewed by:	kib (earlier version)
MFC after:	3 weeks
Differential Revision:	https://reviews.freebsd.org/D14677
2018-03-13 17:10:52 +00:00
Andrew Turner
e150585e9e Use the correct address to write back to memory in the GICv3 ITS driver.
This seems to no be needed on supported hardware as they are cache-coherent,
however this may not be the case on all platforms.

Sponsored by:	DARPA, AFRL
2018-03-09 10:34:44 +00:00
Andrew Turner
fd5b330b19 Bump MAXCPUS on arm64. We are starting to see hardware with more than 96
cores so increase it to the same as amd64.

Sponsored by:	DARPA, AFRL
Sponsored by:	Cavium (Hardware)
2018-03-07 13:54:44 +00:00
Andrew Turner
e0fe10600a Create macros for the ACPI interrupt cross references. This is considered a
band aid until a better solution to find the correct interrupt controller
can be found.

While here fix one place in the GICv3 ITS driver where the offset wasn't
correctly applied.

Sponsored by:	DARPA, AFRL
Sponsored by:	Cavium (Hardware)
2018-03-07 13:16:03 +00:00
Andrew Turner
0f71fc4fc4 Restrict the arm64 DMAP region to the 1G blocks where we have at least
one physical page. This is in preparation for limiting it further as this
is needed on some hardware, however testing has shown issues with further
restricting the DMAP and ACPI.

Sponsored by:	DARPA, AFRL
Sponsored by:	Cavium (Hardware)
2018-03-07 09:58:36 +00:00
Andrew Turner
a29169fda3 Register each GICv3 ITS driver with a useful cross reference. We currently
only use the first driver, however this may change in the future and
hardware exists with multiple ITS devices.

Sponsored by:	DARPA, AFRL
Sponsored by:	Cavium (Hardware)
2018-03-05 10:11:30 +00:00
Andrew Turner
c5691e5922 In the ACPI GICv3 attach function call device_get_children to get the list
of children. We expect this to be populated when configuring the secondary
cores.

Sponsored by:	DARPA, AFRL
Sponsored by:	Cavium (Hardware)
2018-03-05 10:09:18 +00:00
Konstantin Belousov
8c8ee2ee1c Unify bulk free operations in several pmaps.
Submitted by:	Yoshihiro Ota
Reviewed by:	markj
MFC after:	2 weeks
Differential revision:	https://reviews.freebsd.org/D13485
2018-03-04 20:53:20 +00:00
Andrew Turner
b53ff85eb2 Move setting the IRQ base and length into the common GICv3 ITS attach
function. This is common across both ACPI and FDT.

Sponsored by:	ABT Systems Ltd
Sponsored by:	Cavium (Hardware)
2018-03-03 13:20:44 +00:00
Andrew Turner
490ae78090 Add the missing GICv3 dev info struct to the ACPI dev info.Previously we
would read from the resource list when querying from this.

Sponsored by:	ABT Systems Ltd
Sponsored by:	Cavium (Hardware)
2018-03-03 13:19:08 +00:00
Andrew Turner
6d9e17fc83 Count the number of children in the GICv3 ACPI attachment. This is needed
later for assigning IRQ vector space.

Sponsored by:	DARPA, AFRL
Sponsored by:	Cavium (Hardware)
2018-03-02 13:04:21 +00:00
Andrew Turner
e9c0572e4c Allow releasing APs to take more time, as long as we are making progress.
On large core count machines this can be slow while all the CPUs update
the online counter.

Sponsored by:	DARPA, AFRL
Sponsored by:	Cavium (Hardware)
2018-02-28 16:03:40 +00:00
Andrew Turner
cf0afdaab6 Count the number of GIC redistributors in the ACPI tables. The GICv3 driver
needs this to allocate memory, and connect the CPUs to the interrupt
controller.

Sponsored by:	DARPA, AFRL
Sponsored by:	Cavium (Hardware)
2018-02-28 15:25:47 +00:00
Emmanuel Vadot
925ee294ea rk_cru: Add missing break 2018-02-26 22:57:52 +00:00
Emmanuel Vadot
2a3d5e3364 rk3328: Add support for this SoC
* rk_cru is a cru driver that needs to be subclassed by
  the real CRU driver
* rk_clk_pll handle the pll type clock on RockChip SoC, it's only read
  only for now.
* rk_clk_composite handle the different composite clock types (with gate,
  with mux etc ...)
* rk_clk_gate handle the RockChip gates
* rk_clk_mux handle the RockChip muxes (unused for now)
* Only clocks for supported devices are supported for now, the rest will be
  added when driver support comes
* The assigned-clock* property are not handled for now so we rely a lot on the
  bootloader to setup some initial values for some clocks.
2018-02-26 21:25:50 +00:00
Olivier Houchard
ed8bce2cd5 In do_ast, make sure the interrupts are enabled before calling ast().
We can reach that point with IRQs disabled, and calling ast() with IRQs 
disabled can lead to a deadlock.
This should fix the freezes on arm64 under load.

Reviewed by:	andrew
2018-02-26 13:12:51 +00:00
Patrick Kelsey
18a7530938 Greatly reduce the number of #ifdefs supporting the TCP_RFC7413 kernel option.
The conditional compilation support is now centralized in
tcp_fastopen.h and tcp_var.h. This doesn't provide the minimum
theoretical code/data footprint when TCP_RFC7413 is disabled, but
nearly all the TFO code should wind up being removed by the optimizer,
the additional footprint in the syncache entries is a single pointer,
and the additional overhead in the tcpcb is at the end of the
structure.

This enables the TCP_RFC7413 kernel option by default in amd64 and
arm64 GENERIC.

Reviewed by:	hiren
MFC after:	1 month
Sponsored by:	Limelight Networks
Differential Revision:	https://reviews.freebsd.org/D14048
2018-02-26 03:03:41 +00:00
Andrew Turner
4f3cad09fe Correctly set the 16kB page size field in the ITS BASER register. Some
new arm64 hardware, e.g. ThunderX2, seems to use this page size so was
failing to attach as the register value read back was incorrect.

While here fix the spelling on shareability.

Sponsored by:	DARPA, AFRL
2018-02-24 10:33:31 +00:00
Conrad Meyer
849ce31a82 Remove unused error return from API that cannot fail
No implementation of fpu_kern_enter() can fail, and it was causing needless
error checking boilerplate and confusion. Change the return code to void to
match reality.

(This trivial change took nine days to land because of the commit hook on
sys/dev/random.  Please consider removing the hook or otherwise lowering the
bar -- secteam never seems to have free time to review patches.)

Reported by:	Lachlan McIlroy <Lachlan.McIlroy AT isilon.com>
Reviewed by:	delphij
Approved by:	secteam (delphij)
Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D14380
2018-02-23 20:15:19 +00:00
Warner Losh
ef1fcaf0f5 Do not include float interfaces when using libsa.
We don't support float in the boot loaders, so don't include
interfaces for float or double in systems headers. In addition, take
the unusual step of spiking double and float to prevent any more
accidental seepage.
2018-02-23 04:04:25 +00:00
Konstantin Belousov
2c0f13aa59 vm_wait() rework.
Make vm_wait() take the vm_object argument which specifies the domain
set to wait for the min condition pass.  If there is no object
associated with the wait, use curthread' policy domainset.  The
mechanics of the wait in vm_wait() and vm_wait_domain() is supplied by
the new helper vm_wait_doms(), which directly takes the bitmask of the
domains to wait for passing min condition.

Eliminate pagedaemon_wait().  vm_domain_clear() handles the same
operations.

Eliminate VM_WAIT and VM_WAITPFAULT macros, the direct functions calls
are enough.

Eliminate several control state variables from vm_domain, unneeded
after the vm_wait() conversion.

Scetched and reviewed by:	jeff
Tested by:	pho
Sponsored by:	The FreeBSD Foundation, Mellanox Technologies
Differential revision:	https://reviews.freebsd.org/D14384
2018-02-20 10:13:13 +00:00
Jeff Roberson
e958ad4cf3 Make v_wire_count a per-cpu counter(9) counter. This eliminates a
significant source of cache line contention from vm_page_alloc().  Use
accessors and vm_page_unwire_noq() so that the mechanism can be easily
changed in the future.

Reviewed by:	markj
Discussed with:	kib, glebius
Tested by:	pho (earlier version)
Sponsored by:	Netflix, Dell/EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D14273
2018-02-12 22:53:00 +00:00
Warner Losh
62bca77843 Move __va_list and related defines to sys/sys/_types.h
__va_list and related defines are identical in all the
ARCH/include/_types.h files. Move them to sys/sys/_types.h

Sponsored by: Netflix
2018-02-12 14:48:20 +00:00
Mark Johnston
ab7c09f121 Use vm_page_unwire_noq() instead of directly modifying page wire counts.
No functional change intended.

Reviewed by:	alc, kib (previous revision)
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D14266
2018-02-08 19:28:51 +00:00
Andrew Turner
faa3fd222a Only promote userspace mappings to superpages. This was dropped in r328510,
however due to the break-before-make requirement on arm64 is is currently
unsafe to promote kernel pages.

Sponsored by:	DARPA, AFRL
2018-02-01 14:26:26 +00:00
Michal Meloun
2b4c1a7ffc Remove #endif forgotten in r328510.
Pointy hat: mmel
2018-01-28 15:33:32 +00:00
Michal Meloun
89b090f1e6 Fix handling of I-cache sync operations
- pmap_enter_object() can be used for mapping of executable pages, so it's
  necessary to handle I-cache synchronization within it.

- Fix race in I-cache synchronization in pmap_enter(). The current code firstly
  maps given page to target VA and then do I-cache sync on it. This causes
  race, because this mapping become visible to other threads, before I-cache
  is synced.
  Do sync I-cache firstly (by using DMAP VA) and then map it to target VA.

- ARM64 ARM permits implementation of aliased (AIVIVT, VIPT) I-cache, but we
  can use different that final VA for flushing it. So we should use full
  I-cache flush on affected platforms. For now, and as temporary solution,
  use full flush always.
2018-01-28 15:02:49 +00:00
Michal Meloun
116a5567dd Fix pmap_fault().
- special fault handling for break-before-make mechanism should be also
  applied for instruction translation faults, not only for data translation
  faults.

- since arm64_address_translate_...() functions are not atomic,
  use these with disabled interrupts.
2018-01-27 09:49:47 +00:00
Pedro F. Giffuni
ac2fffa4b7 Revert r327828, r327949, r327953, r328016-r328026, r328041:
Uses of mallocarray(9).

The use of mallocarray(9) has rocketed the required swap to build FreeBSD.
This is likely caused by the allocation size attributes which put extra pressure
on the compiler.

Given that most of these checks are superfluous we have to choose better
where to use mallocarray(9). We still have more uses of mallocarray(9) but
hopefully this is enough to bring swap usage to a reasonable level.

Reported by:	wosch
PR:		225197
2018-01-21 15:42:36 +00:00
Nathan Whitehorn
9a8196ce19 Remove SFBUF_OPTIONAL_DIRECT_MAP and such hacks, replacing them across the
kernel by PHYS_TO_DMAP() as previously present on amd64, arm64, riscv, and
powerpc64. This introduces a new MI macro (PMAP_HAS_DMAP) that can be
evaluated at runtime to determine if the architecture has a direct map;
if it does not (or does) unconditionally and PMAP_HAS_DMAP is either 0 or
1, the compiler can remove the conditional logic.

As part of this, implement PHYS_TO_DMAP() on sparc64 and mips64, which had
similar things but spelled differently. 32-bit MIPS has a partial direct-map
that maps poorly to this concept and is unchanged.

Reviewed by:		kib
Suggestions from:	marius, alc, kib
Runtime tested on:	amd64, powerpc64, powerpc, mips64
2018-01-19 17:46:31 +00:00
Ruslan Bukin
9d00c86401 Set the base address of translation table 0.
This fixes operation on Qualcomm Snapdragon and some other platforms.

During boot time on subsystems initialization we have some amount of
kernel threads created, then scheduler gives CPU time to each thread.
Eventually scheduler returns CPU execution back to thread 0. In this
case writing zero to ttbr0 in cpu_switch leads Qualcomm board to
reboot (asynchronously, CPU continues execution).

Similar to other kernel threads install a valid physical address
(kernel pmap) to user page table base register ttbr0.

Reviewed by:	andrew
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D13536
2018-01-18 16:20:09 +00:00
Andrew Turner
7680515c07 Add a pmap invalidate that doesn't call sched_pin.
When demoting DMAP pages curthread may be pointing to data within the
page we are demoting. Create a new invalidate that doesn't pin and use
it in the demote case.

As the demote has both interrupts disabled, and is within a critical section
this is safe from having the scheduler from switching to another CPU.

Reported by:	loos
Reviewed by:	loos
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D13955
2018-01-18 10:52:31 +00:00
Pedro F. Giffuni
a67b3b1621 arm: make some use of mallocarray(9).
Focus on code where we are doing multiplications within malloc(9). None of
these ire likely to overflow, however the change is still useful as some
static checkers can benefit from the allocation attributes we use for
mallocarray.

This initial sweep only covers malloc(9) calls with M_NOWAIT. No good
reason but I started doing the changes before r327796 and at that time it
was convenient to make sure the sorrounding code could handle NULL values.

X-Differential revision: https://reviews.freebsd.org/D13837
2018-01-15 21:09:58 +00:00
Jeff Roberson
6f4acaf4c9 Add support for NUMA domains to bus dma tags. This causes all memory
allocated with a tag to come from the specified domain if it meets the
other constraints provided by the tag.  Automatically create a tag at
the root of each bus specifying the domain local to that bus if
available.

Reviewed by:	jhb, kib
Tested by:	pho
Sponsored by:	Netflix, Dell/EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D13545
2018-01-12 23:34:16 +00:00
Jeff Roberson
ab3185d15e Implement NUMA support in uma(9) and malloc(9). Allocations from specific
domains can be done by the _domain() API variants.  UMA also supports a
first-touch policy via the NUMA zone flag.

The slab layer is now segregated by VM domains and is precise.  It handles
iteration for round-robin directly.  The per-cpu cache layer remains
a mix of domains according to where memory is allocated and freed.  Well
behaved clients can achieve perfect locality with no performance penalty.

The direct domain allocation functions have to visit the slab layer and
so require per-zone locks which come at some expense.

Reviewed by:	Attilio (a slightly older version)
Tested by:	pho
Sponsored by:	Netflix, Dell/EMC Isilon
2018-01-12 23:25:05 +00:00
Andrew Turner
7023544aec Workaround Spectre Variant 2 on arm64.
We need to handle two cases:

1. One process attacking another process.
2. A process attacking the kernel.

For the first case we clear the branch predictor state on context switch
between different processes. For the second we do this when taking an
instruction abort on a non-userspace address.

To clear the branch predictor state a per-CPU function pointer has been
added. This is set by the new cpu errata code based on if the CPU is
known to be affected.

On Cortex-A57, A72, A73, and A75 we call into the PSCI firmware as newer
versions of this will clear the branch predictor state for us.

It has been reported the ThunderX is unaffected, however the ThunderX2 is
vulnerable. The Qualcomm Falkor core is also affected. As FreeBSD doesn't
yet run on the ThunderX2 or Falkor no workaround is included for these CPUs.

MFC after:	3 days
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D13812
2018-01-12 14:01:38 +00:00
Andrew Turner
ee42eb7e5c iAdd ACPI attachments the the GIC and GICv3 interrupt controller drivers.
For each we need to walk the MADT to find which we have, then add the
driver as needed. As each may have a child they will each walk the same
table to find these details.

Reviewed by:	mmel
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D8720
2018-01-11 17:23:24 +00:00
Andrew Turner
224c3776f6 Add the start of INTRNG support for ACPI.
This adds a new acpi_bus interface with a map_intr method. This is similar
to the Open Firmware map_intr method and allows us to create the needed
mapping from ACPI space to INTRNG space.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D8617
2018-01-11 17:09:12 +00:00
Michal Meloun
e426794f61 Initialize CONTEXTIDR register on secondary cores by zero,
not with undefined value from X1 register.

MFC after:	1 month
2018-01-11 15:03:01 +00:00
Andrew Turner
4bb409fb8d Add a framework to install CPU errata on arm64. Each erratum can encode
a mask and value to compare with the Main ID Register. If these match then a
function is called to handle the installation of the erratum workaround.

No errata are currently handled, however this will change soon in a future
commit.

MFC after:	1 week
Sponsored by:	DARPA, AFRL
2018-01-09 14:33:05 +00:00
Andrew Turner
dde4c2fc9d Only install the new pagetable pointer into ttbr0_el1 when it differs from
the existing value.

MFC after:	1 week
Sponsored by:	DARPA, AFRL
2018-01-08 11:08:45 +00:00
Andrew Turner
45a91d4538 Move some of the common thread switching code into C. This will help with
future optimisations, e.g. using Address Space IDs (asid).

MFC after:	1 week
Sponsored by:	DARPA, AFRL
2018-01-08 10:23:31 +00:00
Kyle Evans
2defb358ea if_awg: Use syscon prop if it exists
The emac bindings that are landing in Linux 4.15 specify a syscon property
on the emac node that point to /soc/syscon. Use this property if it's
specified, but maintain backwards compatibility with the old method.

The older method is still used for boards that we get .dtb from u-boot, such
as pine64, that did not yet have stable emac bindings.

Tested on:	Banana Pi-M3 (a83t)
Tested on:	Pine64 (a64)
Reviewed by:	manu
Differential Revision:	https://reviews.freebsd.org/D13296
2018-01-04 22:37:15 +00:00
Poul-Henning Kamp
8ba749fbe3 Introduce an architecture-agnostic <sys/_stdarg.h> to reduce
platform divergence.

Only architectures which pass arguments in registers (mips)
and platforms which use really weird compilers (any?) would
need to augment the contents of <sys/_stdarg.h>

Convert x86, arm and arm64 architectures to use <sys/_stdarg.h>
2017-12-25 20:54:00 +00:00
Alexander Kabaev
151ba7933a Do pass removing some write-only variables from the kernel.
This reduces noise when kernel is compiled by newer GCC versions,
such as one used by external toolchain ports.

Reviewed by: kib, andrew(sys/arm and sys/arm64), emaste(partial), erj(partial)
Reviewed by: jhb (sys/dev/pci/* sys/kern/vfs_aio.c and sys/kern/kern_synch.c)
Differential Revision: https://reviews.freebsd.org/D10385
2017-12-25 04:48:39 +00:00
Konstantin Belousov
30d4f9e888 Add atomic_load(9) and atomic_store(9) operations.
They provide relaxed-ordered atomic access semantic.  Due to the
FreeBSD memory model, the operations are syntaxical wrappers around
the volatile accesses.  The volatile qualifier is used to ensure that
the access not optimized out and in turn depends on the volatile
semantic as implemented by supported compilers.

The motivation for adding the operation is to help people coming from
other systems or knowing the C11/C++ standards where atomics have
special type and require use of the special access operations.  It is
still the case that FreeBSD requires plain load and stores of aligned
integer types to be atomic.

Suggested by:	jhb
Reviewed by:	alc, jhb
Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
Differential revision:	https://reviews.freebsd.org/D13534
2017-12-19 09:59:20 +00:00
Bruce Evans
fb3cc1c37d Move instantiation of msgbufp from 9 MD files to subr_prf.c.
This variable should be pure MI except possibly for reading it in MD
dump routines.  Its initialization was pure MD in 4.4BSD, but FreeBSD
changed this in r36441 in 1998.  There were many imperfections in
r36441.  This commit fixes only a small one, to simplify fixing the
others 1 arch at a time.  (r47678 added support for
special/early/multiple message buffer initialization which I want in
a more general form, but this was too fragile to use because hacking
on the msgbufp global corrupted it, and was only used for 5 hours in
-current...)
2017-12-07 07:55:38 +00:00
Andrew Turner
78f23de54c Use the module pointer to find the address we need to map to in the early
arm64 boot sequence. This will be a virtual address in the kernel space
after the kernel and any modules loaded by loader so we can use this to
find the size of the kernel + modules. We can then add on a level 2 page for
the module data and round up the size to be aligned to a level 2 page.

This allows more than 8 MiB of modules to be loaded by loader, e.g. zfs.ko
and opensolaris.ko.

Reported by:	Shawn Webb
MFC after:	1 week
Sponsored by:	DARPA, AFRL
2017-12-04 11:25:34 +00:00
Ed Schouten
03c132654d Make COMPAT_FREEBSD32 part of GENERIC on arm64.
The cloudabi32.ko kernel modules can only be loaded on AMD64 and ARM64
by kernels built with COMPAT_FREEBSD32. Even though COMPAT_FREEBSD32
does not support the execution of native FreeBSD executables, do add it
to GENERIC, to make cloudabi32.ko usable.

According to size(1), this makes the kernel image approximately 0.7%
larger.

Reviewed by:	andrew, imp, emaste
Differential Revision:	https://reviews.freebsd.org/D13311
2017-12-03 19:26:14 +00:00
Ed Schouten
aea6d042a9 Port cloudabi32.ko to FreeBSD/arm64.
This change adds an implementation of a sysent for running CloudABI
armv6 and armv7 binaries on FreeBSD/arm64. It is a somewhat literal copy
of the armv6 version, except that it's been patched up to use the proper
registers.

Just like for cloudabi32.ko on FreeBSD/amd64, we make use of a vDSO that
automatically pads system call parameters to 64-bit value. These are
stored in a buffer on the stack, meaning we need to use copyin() and
copyout() unconditionally.
2017-11-30 17:58:48 +00:00
Pedro F. Giffuni
796df753f4 SPDX: Consider code from Carnegie-Mellon University.
Interesting cases, most likely from CMU Mach sources.
2017-11-30 15:48:35 +00:00
Andrew Turner
b4cc39e785 Fix the direction we move the instruction to retuirn to after exiting the
kernel debugger. We should skip the breakpoint instruction, not execute the
instruction before it.

Pointy-hat to:	andrew
MFC after:	1 week
Sponsored by:	DARPA, AFRL
2017-11-28 16:26:56 +00:00
Andrew Turner
be84f91cf4 When we exit the kernel debugger having entered because of a breakpoint
instruction we need to jump over the instruction. Without this we will
execute the same instruction again and enter into the debugger again.

PR:		223917
Reported by:	emaste
MFC after:	1 week
Sponsored by:	DARPA, AFRL
2017-11-28 11:04:47 +00:00
Konstantin Belousov
dde5602786 Fix index calculation for the page table pages for efirt 1:1 map.
Stop issuing pre-assigned number to enumerate all page table pages,
the assignment is incorrect.  Instead automatically calculate the next
unused index. This index in fact does not serve any purpose except to
be unique to satisfy vm_page_grab() interface, we do not look up the
page by the index later.

Reported and tested by:	emaste
Reviewed by:	andrew
Sponsored by:	The FreeBSD Foundation
MFC after:	2 weeks
PR:	223906
Differential revision:	https://reviews.freebsd.org/D13273
2017-11-28 09:34:43 +00:00
Andrew Turner
896d821e54 Set the kernel file name so the kern.bootfile sysctl is set. This allows
for the creation of kernel.old when running installkernel.

MFC after:	1 week
Sponsored by:	DARPA, AFRL
2017-11-27 16:19:00 +00:00
Ed Schouten
2b6a8dd52c Add a Saved Process Status Register bit for AArch32 execution mode.
The documentation on the Saved Process Status Register (SPSR) is a bit
weird; the M[4] bit is documented separately from M[3:0]. The M[4] bit
can be toggled to switch to 32-bit execution mode. This functionality is
orthogonal to M[3:0].

Change the definition of PSR_M_MASK to no longer include M[4]. Add a new
definition, PSR_AARCH32 that can be used to toggle 32-bit independently.
This bit will be used by the cloudabi32 code to force execution of
userspace code in 32-bit mode.

Reviewed by:	andrew
Differential Revision:	https://reviews.freebsd.org/D13148
2017-11-26 14:56:23 +00:00
Ed Schouten
2d19a20d5d Correct some more places where TO_PTR() should be used.
These were missed in r326228.

MFC after:	2 weeks
2017-11-26 14:53:56 +00:00
Ed Schouten
ee13ffbe03 Use TO_PTR() to convert integers to pointers.
For FreeBSD/arm64's cloudabi32 support, I'm going to need a TO_PTR() in
this place. Also use it for all of the other source files, so that the
difference remains as minimal as possible.

MFC after:	2 weeks
2017-11-26 14:45:56 +00:00
Ed Schouten
7af24ff710 Make 32-bit system calls end up in svc_handler().
The nice thing about ARM64 is that it's pretty elegant to install
separate trap/exception handlers for 32-bit and 64-bit processes. That
said, for all other architectures (e.g., i386 on amd64) we always let
32-bit counterparts go through the regular system call codepath. Let's
do the same on ARM64.

Reviewed by:	andrew
Differential Revision:	https://reviews.freebsd.org/D13146
2017-11-26 14:28:27 +00:00
Andrew Turner
c4501bdfac Make the arm64 pmap_invalidate functions static inline. This fixes building
with DIAGNOSTIC.

PR:		223874
Reported by:	emaste
MFC after:	1 week
2017-11-26 09:29:34 +00:00
Ed Maste
ccf802cf18 Temporarily disable VIMAGE on arm64
Loading a kernel module with a static VNET_DEFINE'd variable (e.g.
if_lagg) currently results in a kernel panic.

PR:		223670
2017-11-24 19:21:21 +00:00
Ed Schouten
09f966caf7 Set CP15BEN in SCTLR to make memory barriers work in 32-bit mode.
Binaries generated by Clang for ARMv6 may contain these instructions:

  MCR p15, 0, <Rd>, c7, c10, 5

These instructions are deprecated as of ARMv7, which is why modern
processors have a way of toggling support for them. On FreeBSD/arm64 we
currently disable support for these instructions, meaning that if 32-bit
executables with these instructions are run, they would crash with
SIGILL. This is likely not what we want.

Reviewed by:	andrew
Differential Revision:	https://reviews.freebsd.org/D13145
2017-11-24 13:51:59 +00:00
Ed Schouten
9dcf90f8ad Add rudimentary support for building FreeBSD/arm64 with COMPAT_FREEBSD32.
Right now I'm using two Raspberry Pi's (2 and 3) to test CloudABI
support for armv6, armv7 and aarch64. It would be nice if I could
restrict this to just a single instance when testing smaller changes.
This is why I'd like to get COMPAT_CLOUDABI32 to work on arm64.

As COMPAT_CLOUDABI32 depends on COMPAT_FREEBSD32, at least for the ELF
loading, this change adds all of the bits necessary to at least build a
kernel with COMPAT_FREEBSD32. All of the machine dependent system calls
are still stubbed out, for the reason that implementations for these are
only useful if actual support for running FreeBSD binaries is added.
This is outside the scope of this work.

Reviewed by:	andrew
Differential Revision:	https://reviews.freebsd.org/D13144
2017-11-24 13:50:53 +00:00
Ed Schouten
814629dd64 Don't let cpu_set_syscall_retval() clobber exec_setregs().
Upon successful completion, the execve() system call invokes
exec_setregs() to initialize the registers of the initial thread of the
newly executed process. What is weird is that when execve() returns, it
still goes through the normal system call return path, clobbering the
registers with the system call's return value (td->td_retval).

Though this doesn't seem to be problematic for x86 most of the times (as
the value of eax/rax doesn't matter upon startup), this can be pretty
frustrating for architectures where function argument and return
registers overlap (e.g., ARM). On these systems, exec_setregs() also
needs to initialize td_retval.

Even worse are architectures where cpu_set_syscall_retval() sets
registers to values not derived from td_retval. On these architectures,
there is no way cpu_set_syscall_retval() can set registers to the way it
wants them to be upon the start of execution.

To get rid of this madness, let sys_execve() return EJUSTRETURN. This
will cause cpu_set_syscall_retval() to leave registers intact. This
makes process execution easier to understand. It also eliminates the
difference between execution of the initial process and successive ones.
The initial call to sys_execve() is not performed through a system call
context.

Reviewed by:	kib, jhibbits
Differential Revision:	https://reviews.freebsd.org/D13180
2017-11-24 07:35:08 +00:00
Andrew Turner
521018d379 Ensure we check the program state set in the trap frame on arm and arm64.
This value may be set by userspace so we need to check it before using it.
If this is not done correctly on exception return the kernel may continue
in kernel mode with all registers set to a userspace controlled value. Fix
this by moving the check into set_mcontext, and also add the missing
sanitisation from the arm64 set_regs.

Discussed with:	security-officer@
MFC after:	3 days
Sponsored by:	DARPA, AFRL
2017-11-23 17:40:40 +00:00
Andrew Turner
a3dff126f9 Add a driver for the EFI RTC. This uses the EFI Runtime Services to query
the system time.

As we seem to only read this time on boot, and this is the only source of
time on many arm64 machines we need to enable this by default there. As
this is not always the case with U-Boot firmware, or when we have been
booted from a non-UEFI environment we only enable the device driver when
the Runtime Services are present and reading the time doesn't result in an
error.

PR:		212185
Reviewed by:	imp, kib
Tested by:	emaste
Relnotes:	yes
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D12650
2017-11-21 17:23:16 +00:00
Andrew Turner
2eb3e51e71 When fpcurthread is not the current thread it may be non-NULL. In this
case another thread has had the VFP unit enabled and will have its state
in the VFP registers along with it stored in memory. As such we don't need
to store the state, but do need to zero the fpcurthread pointer to stop
the VFP driver from using the enable fast path.

Reported by:	emaste
Sponsored by:	DARPA, AFRL
2017-11-21 13:19:38 +00:00
Jeff Roberson
8d6fbbb867 Replace manyinstances of VM_WAIT with blocking page allocation flags
similar to the kernel memory allocator.

This simplifies NUMA allocation because the domain will be known at wait
time and races between failure and sleeping are eliminated.  This also
reduces boilerplate code and simplifies callers.

A wait primitive is supplied for uma zones for similar reasons.  This
eliminates some non-specific VM_WAIT calls in favor of more explicit
sleeps that may be satisfied without new pages.

Reviewed by:	alc, kib, markj
Tested by:	pho
Sponsored by:	Netflix, Dell/EMC Isilon
2017-11-08 02:39:37 +00:00
Andrew Turner
f591046e7d Allocate the ITS translation table with a 64k page alignment. This is the
largest alignment the ITS can require.

This fixes a bug with the ARM Architecture Envelope Model (AEM) where it
only allows 64k pages so will fail to attach the ITS device when this table
is not sufficiently aligned.

Sponsored by:	DARPA, AFRL
2017-10-31 23:17:17 +00:00
Andrew Turner
2ed578e1db As with r325242 use mp_maxid when iterating over CPUs in the GICv3 driver.
Sponsored by:	DARPA, AFRL
2017-10-31 22:12:14 +00:00
Andrew Turner
2bfb5e812b Use mp_maxid when iterating over CPUs as we may have sparse id allocations.
Sponsored by:	DARPA, AFRL
2017-10-31 18:22:21 +00:00
Eitan Adler
a2aef24aa3 Update several more URLs
- Primarily http -> https
- Primarily FreeBSD project URLs
2017-10-29 08:17:03 +00:00
Emmanuel Vadot
7562d7ddbd loader.efi: Make framebuffer commands available for arm64
Move framebuffer.{c,h} to sys/boot/efi/loader and add the efifb
related metadata and pass it to the kernel

Reviewed by:	imp, andrew
Differential Revision:	https://reviews.freebsd.org/D12757
2017-10-25 18:55:04 +00:00
Mark Johnston
5fca1d90c1 Fix the VM_NRESERVLEVEL == 0 build.
Add VM_NRESERVLEVEL guards in the pmaps that implement transparent
superpage promotion using reservations.

Reviewed by:	alc, kib
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D12764
2017-10-23 15:34:05 +00:00
Michal Meloun
904d8c492f Add AT_HWCAP2 ELF auxiliary vector.
- allocate value for new AT_HWCAP2 auxiliary vector on all platforms.
 - expand 'struct sysentvec' by new 'u_long *sv_hwcap2', in exactly
   same way as for AT_HWCAP.

MFC after:	1 month
Reviewed by:	kib
Differential Revision:	https://reviews.freebsd.org/D12699
2017-10-21 12:05:01 +00:00
Bjoern A. Zeeb
8e94025b41 With r181803 on 2008-08-17 23:27:27Z the first VIMAGE commit went into
HEAD.  Enable VIMAGE in GENERIC kernels and some others (where GENERIC does
not exist) on HEAD.

Disable building LINT-VIMAGE with VIMAGE being default.

This should give it a lot more exposure in the run-up to 12 to help
us evaluate whether to keep it on by default or not.
We are also hoping to get better performance testing.
The feature can be disabled using nooptions.

Requested by:		many
Reviewed by:		kristof, emaste, hiren
X-MFC after:		never
Relnotes:		yes
Differential Revision:	https://reviews.freebsd.org/D12639
2017-10-20 21:40:59 +00:00
Mark Johnston
46fcd1af63 Move kernel dump offset tracking into MI code.
All of the kernel dump implementations keep track of the current offset
("dumplo") within the dump device. However, except for textdumps, they
all write the dump sequentially, so we can reduce code duplication by
having the MI code keep track of the current offset. The new
dump_append() API can be used to write at the current offset.

This is needed to implement support for kernel dump compression in the
MI kernel dump code.

Also simplify dump_encrypted_write() somewhat: use dump_write() instead
of duplicating its bounds checks, and get rid of the redundant offset
tracking.

Reviewed by:	cem
Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D11722
2017-10-18 15:38:05 +00:00
Andrew Turner
7a158e826d Support the EFI Runtime Services on arm64. As with amd64 we use the 1:1
mapping. This uses the new common code shared with amd64.

The RTC should only be accessed via EFI. There is no locking around it as
the spec only has this as a requirement for the PC-AT CMOS device.

Reviewed by:	kib, imp
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D12595
2017-10-10 13:05:26 +00:00
Andrew Turner
2e0c6823c4 Add machine/fpu.h on arm64 for compatibility with amd64.
Sponsored by:	DARPA, AFRL
2017-10-10 13:02:58 +00:00
Andrew Turner
f72a72115c Move the pmap_l0_index, etc. macros to pte.h. These will be used by the
EFI Runtime Services code.

Sponsored by:	DARPA, AFRL
2017-10-10 12:54:36 +00:00
Andrew Turner
5cf1f313c4 Add a memory barrier to ensure the atomic write is visible to the other
CPUs before waking them up.

Sponsored by:	DARPA, AFRL
2017-10-02 14:22:35 +00:00
Josh Paetzel
c77037f16f Fix indentation for r323068
PR:	220170
Reported by:	lidl
MFC after:	3 days
Pointyhat to:	jpaetzel
2017-09-19 20:40:05 +00:00
Andrew Turner
ca289945b2 Add the ARMv8.3 ID register fields. These were found in the A-Profile
exploration tools documentation:
https://developer.arm.com/products/architecture/a-profile/exploration-tools

Sponsored by:	DARPA, AFRL
2017-09-15 12:57:34 +00:00
Andrew Turner
bcf2b954c3 Add support for handling undefined instructions in userspace and the
kernel. We can register callbacks to perform the required operation on the
saved registers before returning.

This is initially used to work around a bug in old versions of QEMU that
trigger such an exception when reading from an ID register when it should
load z zero value.

I expect this could be used with other exception types, e.g. to emulate
special register access from userland.

Sponsored by:	DARPA, AFRL
2017-09-14 17:29:51 +00:00
John Baldwin
c2f37b9245 Add AT_HWCAP and AT_EHDRFLAGS on all platforms.
A new 'u_long *sv_hwcap' field is added to 'struct sysentvec'.  A
process ABI can set this field to point to a value holding a mask of
architecture-specific CPU feature flags.  If an ABI does not wish to
supply AT_HWCAP to processes the field can be left as NULL.

The support code for AT_EHDRFLAGS was already present on all systems,
just the #define was not present.  This is a step towards unifying the
AT_* constants across platforms.

Reviewed by:	kib
MFC after:	1 month
Differential Revision:	https://reviews.freebsd.org/D12290
2017-09-14 14:26:55 +00:00
Ilya Bakulin
61df30cfd4 Add MMCCAM-enabled kernel config for arm64
Approved by:	imp (mentor)
Differential Revision:	https://reviews.freebsd.org/D12114
2017-09-11 19:07:42 +00:00
Marcin Wojtas
885a74181c Expand Marvell NIC description in arm64 GENERIC config
Suggested by: emaste
2017-09-11 19:00:53 +00:00
Marcin Wojtas
7ca8a2b385 Enable compilation of Marvell NETA controller with arm64 GENERIC
This patch enables network operation on Marvell Armada 3700 SoC.

Submitted by: Patryk Duda <pdk@semihalf.com>
Obtained from: Semihalf
Sponsored by: Semihalf
Differential Revision: https://reviews.freebsd.org/D12259
2017-09-09 11:56:48 +00:00
Marcin Wojtas
e49e3ec31e Add support for uart_mvebu driver arm64 GENERIC config
This patch enables console output on Armada 3700 SoCs
with kernel GENERIC.

Submitted by: Patryk Duda <pdk@semihalf.com>
Obtained from: Semihalf
Sponsored by: Semihalf
Differential Revision: https://reviews.freebsd.org/D12251
2017-09-09 11:46:34 +00:00
Marcin Wojtas
840d633f12 Enable compilation of Marvell EHCI driver in arm64 GENERIC
Enabled driver can be used on boards equipped with Marvell Armada 3700 SoC.

Submitted by: Patryk Duda <pdk@semihalf.com>
Obtained from: Semihalf
Sponsored by: Semihalf
Differential Revision: https://reviews.freebsd.org/D12256
2017-09-09 11:16:10 +00:00
Marcin Wojtas
705f4b2ceb Enable compilation of Marvell XHCI driver in arm64 GENERIC
Enabled driver can be used on boards equipped with Marvell Armada
3700/7k/8k SoCs.

Submitted by: Patryk Duda <pdk@semihalf.com>
Obtained from: Semihalf
Sponsored by: Semihalf
Differential Revision: https://reviews.freebsd.org/D12253
2017-09-09 10:58:45 +00:00
Andrew Turner
c1649fdb02 Not all CPUs handle reading ID_AA64MMFR2_EL1 (e.g. qemu), disable it for now.
Sponsored by:	DARPA, AFRL
2017-09-08 08:02:06 +00:00
Andrew Turner
f9fc9faa3a Fix the SVE ID field shift.
Sponsored by:	DARPA, AFRL
2017-09-07 19:52:04 +00:00
Andrew Turner
130be885e6 Add the ATS1E1 case to the ID_AA64MMFR1_EL1 decoding.
Sponsored by:	DARPA, AFRL
2017-09-07 19:51:17 +00:00
Andrew Turner
d28b950a4d Use the correct mask when printing undecoded fields from the
ID_AA64MMFR2_EL1 register.

Sponsored by:	DARPA, AFRL
2017-09-07 18:58:55 +00:00
Andrew Turner
d1c2e46d9c Fix a mismerge, make sure PRINT_ID_AA64_MMFR2 has a unique value.
Sponsored by:	DARPA, AFRL
2017-09-07 16:43:12 +00:00
Andrew Turner
1a2e5c004d Fix the value of ID_AA64ISAR1_DPB_SHIFT, the field is bits 3:0.
Sponsored by:	DARPA, AFRL
2017-09-07 16:12:56 +00:00
Andrew Turner
f45dc6945b Add the ARMv8.2 ID register additions and use them to decode the register
values. As not all assemblers understand the new ID_AA64MMFR2_EL1 register
add a macro to access it. This seems to be safe for older CPUs to read this
new register, with them returning zero.

Sponsored by:	DARPA, AFRL
2017-09-07 15:45:56 +00:00
Andrew Turner
0f962c6deb Uppercase the special register names in identcpu to be more consistent with
the other source files.

Sponsored by:	DARPA, AFRL
2017-09-07 15:30:13 +00:00
Andrew Turner
a0f16159bd Make the bit mask of ARMv8 ID registers to print sparse to keep values
close, but without having to change all values when new registers are added.

Sponsored by:	DARPA, AFRL
2017-09-07 15:24:47 +00:00
Andrew Turner
5ad42f79fb Add more ARM Ltd parts to the list of knows CPUs.
Submitted by:	Jon Brawn <jon@brawn.org>
2017-09-07 15:02:57 +00:00
Marcin Wojtas
eedd5eafbe Add Marvell RTC driver to arm64 GENERIC config
Marvell Armada 80x0/70x0 SoC family uses same RTC IP as
Armada 38x. This patch adds necessary files and enable driver in
GENERIC config.

Submitted by: Rafal Kozik <rk@semihalf.com>
Obtained from: Semihalf
Sponsored by: Semihalf
Differential Revision: https://reviews.freebsd.org/D12200
2017-09-05 05:50:01 +00:00
Marcin Wojtas
d7d8ab0316 Add ARM Cortex A72 to CPU list
This change is required to properly detect CPUs
on Marvell Armada 80x0/70x0 SoC family.

Submitted by: Rafal Kozik <rk@semihalf.com>
Reviewed by: andrew, cognet (mentor)
Approved by: cognet (mentor)
Sponsored by: Semihalf
Differential Revision: https://reviews.freebsd.org/D12184
2017-09-03 08:32:33 +00:00
Josh Paetzel
9d0ec2a920 Revert r323087
This needs more thinking out and consensus, and the commit message
was wrong AND there was a typo in the commit.

pointyhat:	jpaetzel
2017-09-01 17:03:48 +00:00
Josh Paetzel
0be04b100c Take options IPSEC out of GENERIC
PR:	220170
Submitted by:	delphij
Reviewed by:	ae, glebius
MFC after:	2 weeks
Differential Revision:	D11806
2017-09-01 15:54:53 +00:00
Andrew Turner
5ea26aec6e Add support for quirks while enabling secondary CPUs. This uses the fdt
compatible string to check if the board is compatible with a given quirk.
It's possible this will be moved later, however as it's currently only used
by the MP code put it there.

So far the only instance of a quirk is when the list of CPUs may be
incorrect. This can happen on virtual machines with a hard coded
devicetree, but where the user may then set the number of CPUs as an
argument. This is the case on the ARM models so include the model specific
compat strings for these, including the spelling mistake found in some of
the OpenplatformPkg dtb files.

Sponsored by:	DARPA, AFRL
2017-08-31 20:48:05 +00:00
Josh Paetzel
3b65550eec Allow kldload tcpmd5
PR:	220170
MFC after:	2 weeks
2017-08-31 20:16:28 +00:00
Andrew Turner
dab076c004 Remove an unneeded call to pmap_invalidate_all. This was never called as
the anyvalid variable is never set.

MFC after:	1 week
Sponsored by:	DARPA, AFRL
2017-08-22 18:20:25 +00:00
Andrew Turner
012faa32f9 Fix a bug in pmap_protect where we invalidate the wrong page. With this we
can now remove an unneeded call to invalidate all entries.

MFC after:	1 week
Sponsored by:	DARPA, AFRL
2017-08-22 17:38:06 +00:00
Andrew Turner
e8e65e28ca Fix a comment on uncommitted work. 2017-08-22 13:53:53 +00:00
Andrew Turner
6683b30c03 Move the l0 pagetable address to struct mdproc. It is a property of the
whole process so should live there.

Sponsored by:	DARPA, AFRL
2017-08-22 13:16:14 +00:00
Andrew Turner
cbf2160e81 Improve the performance of the arm64 thread switching code.
The full system memory barrier around a TLB invalidation is stricter than
required. It needs to wait on accesses to main memory, with just the weaker
store variant before the invalidate. As such use the dsb istst, tlbi, dlb
ish sequence already used in pmap.

The tlbi instruction in this sequence is also unnecessarily using a
broadcast invalidate when it just needs to invalidate the local CPUs TLB.
Switch to a non-broadcast variant of this instruction.

Sponsored by:	DARPA, AFRL
2017-08-21 18:12:32 +00:00
Mark Johnston
01938d3666 Rename mkdumpheader() and group EKCD functions in kern_shutdown.c.
This helps simplify the code in kern_shutdown.c and reduces the number
of globally visible functions.

No functional change intended.

Reviewed by:	cem, def
Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D11603
2017-08-18 04:04:09 +00:00
Mark Johnston
50ef60dabe Factor out duplicated kernel dump code into dump_{start,finish}().
dump_start() and dump_finish() are responsible for writing kernel dump
headers, optionally writing the key when encryption is enabled, and
initializing the initial offset into the dump device.

Also remove the unused dump_pad(), and make some functions static now that
they're only called from kern_shutdown.c.

No functional change intended.

Reviewed by:	cem, def
Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D11584
2017-08-18 03:52:35 +00:00
Ed Maste
e7b993842e arm64: return error instead of panic in unimplemented ptrace ops
We don't need a panic as a reminder that these need to be implemented.

Reported by:	Shawn Webb
MFC after:	3 week
Sponsored by:	The FreeBSD Foundation
2017-08-17 19:16:23 +00:00
Andrew Turner
062c276886 Add support for multiple GICv3 ITS devices. For this we add sc_irq_base
and sc_irq_length to the softc to handle the base number of IRQs available,
make gicv3_get_nirqs return the number of available interrupt IDs, and
limit which CPUs we send interrupts to based on the numa domain.

The last point is only strictly needed on a dual socket ThunderX where we
are unable to send MSI/MSI-X interrupts between sockets.

Sponsored by:	DARPA, AFRL
2017-08-13 18:54:51 +00:00
John Baldwin
992029ba10 Reliably enable debug exceptions on all CPUs.
Previously, debug exceptions were only enabled on the boot CPU if
DDB was enabled in the dbg_monitor_init() function.  APs also called
this function, but since mp_machdep.c doesn't include opt_ddb.h, the
APs ended up calling an empty stub defined in <machine/debug_monitor.h>
instead of the real function.  Also, if DDB was not enabled in the kernel,
the boot CPU would not enable debug exceptions.

Fix this by adding a new dbg_init() function that always clears the OS
lock to enable debug exceptions which the boot CPU and the APs call.
This function also calls dbg_monitor_init() to enable hardware breakpoints
from DDB on all CPUs if DDB is enabled.  Eventually base support for
hardware breakpoints/watchpoints will need to move out of the DDB-only
debug_monitor.c for use by userland debuggers.

Reviewed by:	andrew
Differential Revision:	https://reviews.freebsd.org/D12001
2017-08-12 18:42:54 +00:00
John Baldwin
c9ee3caf19 Don't panic for PT_GETFPREGS.
Only fetch the VFP state from the CPU if the thread whose registers are
being requested is the current thread.  If a stopped thread's registers
are being fetched by a debugger, the saved state in the PCB is already
valid.

Reviewed by:	andrew
MFC after:	1 week
2017-08-12 18:38:18 +00:00
John Baldwin
319e8f3e36 Fix a typo. 2017-08-11 22:47:32 +00:00
Andrew Turner
1f15260790 Mark each cpu in the appropriate cpuset_domain set. This allows devices to
handle cases where they can only run on a single domain.

To allow all devices access to this set we need to move reading the domain
earlier in the boot as it was previously handled in the CPU driver, however
this is too late for the GICv3 ITS driver.

Sponsored by:	DARPA, AFRL
2017-08-05 20:57:34 +00:00
Andrew Turner
49f347f450 Start to teach the GICv3 driver about NUMA. On ThunderX we may have
multiple ITS devices, however we only want a single ITS device to be
configured on each CPU. To fix this only enable ITS when the node matches
the CPUs node.

Sponsored by:	DARPA, AFRL
2017-08-04 13:08:45 +00:00
Andrew Turner
dbba8930ce Read the numa-node-id property from each CPU node. This will initially be
used to support the dual package ThunderX where we need to send MSI/MSI-X
interrupts to the same package as the device the interrupt came from.

Sponsored by:	DARPA, AFRL
2017-08-04 10:33:22 +00:00
Emmanuel Vadot
48ee531892 arm64: Add Allwinner H5 SoC
Allwinner H5 is an H3 (arm32) with Cortex A53 cores.
Add support for it and enable it in GENERIC kernel config

Tested on: OrangePi PC2
2017-08-02 20:19:19 +00:00
Andrew Turner
ce793a529b Pass the last exception trap frame to kdb_trap. This allows show registers
in ddb to show the traps registers, and not the registers from within the
panic call.

Sponsored by:	DARPA, AFRL
2017-07-26 17:39:10 +00:00
Andrew Turner
a2d16bc541 Add support for passing FPU_KERN_NOCTX to fpu_kern_enter on arm64. This
will be used to call into UEFI from the kernel.

Sponsored by:	DARPA, AFRL
2017-07-18 16:36:32 +00:00
Emmanuel Vadot
0166fb5d59 arm64: Add Allwinner I2C controller to GENERIC kernel 2017-07-09 12:53:02 +00:00
Andrew Turner
db7b284bd6 In the arm64 pmap_remove, when removing a full superpage there is no need
to demote it to 512 pages, then remove each of these. We can just remove
the l2 map directly. This is what the intel pmaps already do.

Sponsored by:	DARPA, AFRL
2017-07-02 08:52:08 +00:00
Andrew Turner
567dec3d2b Remove all calls to cpu_dcache_wb_range from the arm64 pmap code. These
were unneeded as we tell the tlb the pagetables are in cached memory. This
gives us a small, but statistically significant improvement over just
removing the PTE_SYNC cases.

While here remove PTE_SYNC, it's now unneeded.

Sponsored by:	DARPA, AFRL
2017-07-01 09:38:52 +00:00
Jason A. Harmening
eb36b1d0bc Clean up MD pollution of bus_dma.h:
--Remove special-case handling of sparc64 bus_dmamap* functions.
  Replace with a more generic mechanism that allows MD busdma
  implementations to generate inline mapping functions by
  defining WANT_INLINE_DMAMAP in <machine/bus_dma.h>.  This
  is currently useful for sparc64, x86, and arm64, which all
  implement non-load dmamap operations as simple wrappers
  around map objects which may be bus- or device-specific.

--Remove NULL-checked bus_dmamap macros.  Implement the
  equivalent NULL checks in the inlined x86 implementation.
  For non-x86 platforms, these checks are a minor pessimization
  as those platforms do not currently allow NULL maps.  NULL
  maps were originally allowed on arm64, which appears to have
  been the motivation behind adding arm[64]-specific barriers
  to bus_dma.h, but that support was removed in r299463.

--Simplify the internal interface used by the bus_dmamap_load*
  variants and move it to bus_dma_internal.h

--Fix some drivers that directly include sys/bus_dma.h
  despite the recommendations of bus_dma(9)

Reviewed by:	kib (previous revision), marius
Differential Revision:	https://reviews.freebsd.org/D10729
2017-07-01 05:35:29 +00:00
Andrew Turner
2da3e34e0a Remove a blank line accidentally added in r320403. 2017-06-30 14:45:43 +00:00
Andrew Turner
355ffcc842 Add parentheses missed in r320388
Sponsored by:	DARPA, AFRL
2017-06-27 16:30:01 +00:00
Andrew Turner
c90baf6817 Some of the atomic_clear_* functions were incorrectly defined to be an
atomic add. Correct these, fixing a NULL-pointer dereference in netgraph.

PR:		220273
MFC after:	3 days
Sponsored by:	DARPA, AFRL
2017-06-27 10:45:13 +00:00
Andrew Turner
8e26c6b62f In _bswap16 and _bswap32 cast constant values to the appropriate type. This is
similar to what is done in the x86 code.

Sponsored by:	DARPA, AFRL
2017-06-26 22:32:52 +00:00
Andrew Turner
e899a0575b Stop calling cpu_dcache_wb_range from PTE_SYNC.
We set the shareability attributes in TCR_EL1 on boot. These tell the
hardware the pagetables are in cached memory so there is no need to flush
the entries from the cache to memory.

This has about 4.2% improvement in system time and 2.7% improvement in
user time for a buildkernel -j48 on a ThunderX.

Keep the old code for now to allow for further comparisons.
2017-06-25 13:22:49 +00:00
Konstantin Belousov
2d88da2f06 Move struct syscall_args syscall arguments parameters container into
struct thread.

For all architectures, the syscall trap handlers have to allocate the
structure on the stack.  The structure takes 88 bytes on 64bit arches
which is not negligible.  Also, it cannot be easily found by other
code, which e.g. caused duplication of some members of the structure
to struct thread already.  The change removes td_dbg_sc_code and
td_dbg_sc_nargs which were directly copied from syscall_args.

The structure is put into the copied on fork part of the struct thread
to make the syscall arguments information correct in the child after
fork.

This move will also allow several more uses shortly.

Reviewed by:	jhb (previous version)
Sponsored by:	The FreeBSD Foundation
MFC after:	3 weeks
X-Differential revision:	https://reviews.freebsd.org/D11080
2017-06-12 21:03:23 +00:00
Konstantin Belousov
43f41dd393 Make struct syscall_args visible to userspace compilation environment
from machine/proc.h, consistently on all architectures.

Reviewed by:	jhb
Sponsored by:	The FreeBSD Foundation
MFC after:	3 weeks
X-Differential revision:	https://reviews.freebsd.org/D11080
2017-06-12 20:53:44 +00:00
Andrew Turner
52453d22f2 Allow the arm64 machine/vfp.h to be included without first including
machine/pcb.h. It he latter is only needed for struct pcb.
2017-06-09 15:47:14 +00:00
Andrew Turner
9a19869a5f Store the read-only thread pointer when scheduling a new thread. This is
not currently set, however we may wish to set it later.
2017-06-09 15:37:17 +00:00
Olivier Houchard
5bb27fe15c - Don't bother flushing the data cache for pages we're about to unmap, there's
no need to.
- Remove pmap_is_current(), pmap_[pte|l3]_valid_cacheable as there were only
used to know if we had to write back pages.
- In pmap_remove_pages(), don't bother invalidating each page in the TLB,
we're about to flush the whole TLB anyway.

This makes make world 8-9% faster on my hardware.

Reviewed by:	andrew
2017-06-02 14:17:14 +00:00
Konstantin Belousov
55b78354f6 Add COMPAT_FREEBSD11 on arm64, the arch is almost tier-1.
Discussed with:	andrew, emaste
Sponsored by:	The FreeBSD Foundation
2017-05-23 13:57:55 +00:00
Hans Petter Selasky
65b017b420 Avoid use of contiguous memory allocations in busdma when possible.
This patch improves the boundary checks in busdma to allow more cases
using the regular page based kernel memory allocator. Especially in
the case of having a non-zero boundary in the parent DMA tag. For
example AMD64 based platforms set the PCI DMA tag boundary to
PCI_DMA_BOUNDARY, 4GB, which before this patch caused contiguous
memory allocations to be preferred when allocating more than PAGE_SIZE
bytes. Even if the required alignment was less than PAGE_SIZE bytes.

This patch also fixes the nsegments check for using kmem_alloc_attr()
when the maximum segment size is less than PAGE_SIZE bytes.

Updated some comments describing the code in question.

Differential Revision:	https://reviews.freebsd.org/D10645
Reviewed by:		kib, jhb, gallatin, scottl
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-05-16 14:21:37 +00:00
Andrew Turner
2438ef7673 Allocate a cacheline when reading or writing to write through memory. The
hardware will still write to memory, however following reads will be from
the cache.

MFC after:	1 week
Sponsored by:	DARPA, AFRL
2017-05-13 13:03:20 +00:00
Andrew Turner
c031b4b01e Add the VM_MEMATTR_WRITE_THROUGH memory type to arm64 and use it to support
VM_MEMATTR_WRITE_COMBINING in the kernel. This fixes a bug where Xorg would
use write back cached memory for its graphics buffers. This would produce
artifacts on the screen as cachelines were written to memory.

MFC after:	1 week
Sponsored by:	DARPA, AFRL
2017-05-13 13:01:15 +00:00
Andrew Turner
bd2b26ff0a Add reclaim_pv_chunk on arm64. This is based on the amd64 code so should
operate similarly, other than not needing the delayed invalidation.

It has been tested with artificial injection of vm_page_alloc failures
while running 'sort /dev/zero'.

Reviewed by:	alc, kib
MFC after:	1 week
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D10574
2017-05-05 10:01:27 +00:00
Andrew Turner
8a2b28d7ba Print all virtual addresses in the show vtop ddb command. The results may
be different with PAN enabled.

MFC after:	1 week
Sponsored by:	DARPA, AFRL
2017-05-03 12:24:31 +00:00
Andrew Turner
574a7c6c47 Call the PSCI reset from cpu_reset on arm64. When rebooting from DDB the
kernel calls this directly so the event handler is not called, meaning
the computer fails to reboot.

Tested by:	cognet
MFC after:	1 week
Sponsored by:	DARPA, AFRL
2017-04-24 11:06:10 +00:00
Andrew Turner
52a680fb22 Restrict the arm64 supervisor all instructions to only allow a zero
immediate value for system calls. We may wish to use other values in the
future for other purposes.

MFC after:	1 week
Sponsored by:	DARPA, AFRL
2017-04-20 15:53:20 +00:00
Andrew Turner
d7635c7a49 Push loading curthread into assembly in the synchronous exception handlers.
This will help investigating the performance impact of moving parts of the
switch statement in do_el0_sync into assembly.

Sponsored by:	DARPA, AFRL
2017-04-20 13:56:30 +00:00
Ed Maste
f0e56c1f62 Remove trailing whitespace from r317061 2017-04-17 18:57:26 +00:00
Gleb Smirnoff
83c9dea1ba - Remove 'struct vmmeter' from 'struct pcpu', leaving only global vmmeter
in place.  To do per-cpu stats, convert all fields that previously were
  maintained in the vmmeters that sit in pcpus to counter(9).
- Since some vmmeter stats may be touched at very early stages of boot,
  before we have set up UMA and we can do counter_u64_alloc(), provide an
  early counter mechanism:
  o Leave one spare uint64_t in struct pcpu, named pc_early_dummy_counter.
  o Point counter(9) fields of vmmeter to pcpu[0].pc_early_dummy_counter,
    so that at early stages of boot, before counters are allocated we already
    point to a counter that can be safely written to.
  o For sparc64 that required a whole dummy pcpu[MAXCPU] array.

Further related changes:
- Don't include vmmeter.h into pcpu.h.
- vm.stats.vm.v_swappgsout and vm.stats.vm.v_swappgsin changed to 64-bit,
  to match kernel representation.
- struct vmmeter hidden under _KERNEL, and only vmstat(1) is an exclusion.

This is based on benno@'s 4-year old patch:
https://lists.freebsd.org/pipermail/freebsd-arch/2013-July/014471.html

Reviewed by:	kib, gallatin, marius, lidl
Differential Revision:	https://reviews.freebsd.org/D10156
2017-04-17 17:34:47 +00:00
Gleb Smirnoff
9ed01c32e0 All these files need sys/vmmeter.h, but now they got it implicitly
included via sys/pcpu.h.
2017-04-17 17:07:00 +00:00
Andrew Turner
71cb533e98 Rather than checking if the top bit in a virtual address is a 0 or 1
compare against VM_MAXUSER_ADDRESS as we should have been doing.

Sponsored by:	DARPA, AFRL
2017-04-13 16:57:02 +00:00
Andrew Turner
ad0b190e82 Set the arm64 Execute-never bits in more places.
We need to set the Execute-never bits when mapping device memory as the
hardware may perform speculative instruction fetches.

Set the Privileged Execute-ever bit on userspace memory to stop the kernel
if it is tricked into executing it.

Reviewed by:	kib
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D10382
2017-04-13 15:03:03 +00:00
Alexander Kabaev
11011dfc94 Use proper fields to check for interrupt trigger mode. 2017-04-13 14:23:27 +00:00
Andrew Turner
d6a0af236d In ARMv8.1 ARM has added a process state bit to disable access to userspace
from the kernel. Make use of this to restrict accessing userspace to just
the functions that explicitly handle crossing the user kernel boundary.

Reported by:	kib
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D10371
2017-04-13 13:46:01 +00:00
Andrew Turner
aec085f4d3 Add SCTLR bits added in ARMv8.1 and ARMv8.2 and start to use them in the
early boot code.

Sponsored by:	DARPA, AFRL
2017-04-13 11:56:27 +00:00
Andrew Turner
56ab86aa3e Start to use the User and Privileged execute-never bits in the arm64
pagetables. This sets both bits when entering an address we know shouldn't
be executed.

I expect we could mark all userspace pages as Privileged execute-never to
ensure the kernel doesn't branch to one of these addresses.

While here add the ARMv8.1 upper attributes.

Reviewed by:	alc, kib (previous version)
MFC after:	1 week
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D10360
2017-04-12 16:28:40 +00:00
Andrew Turner
c577e950d8 Use the unprivileged variant of the load and store instructions most
places possible in the kernel. This forces these functions to fail if
userspace is unable to access a given memory location, even if it is in
the user memory range.

This will simplify adding Privileged Access Never support later.

MFC after:	1 week
Sponsored by:	DARPA, AFRL
2017-04-12 12:34:27 +00:00
Konstantin Belousov
571a4f172b Do not lose dirty bits for removing PROT_WRITE on arm64.
Arm64 pmap interprets accessed writable ptes as modified, since
ARMv8.0 does not track Dirty Bit Modifier in hardware. If writable bit
is removed, page must be marked as dirty for MI VM.

This change is most important for COW, where fork caused losing
content of the dirty pages which were not yet scanned by pagedaemon.

Reviewed by:	alc, andrew
Reported and tested by:	Mark Millard <markmi@dsl-only.net>
PR:	217138, 217239
Sponsored by:	The FreeBSD Foundation
MFC after:	2 weeks
2017-04-10 15:32:26 +00:00
Patrick Kelsey
67d955aab4 Corrected misspelled versions of rendezvous.
The MFC will include a compat definition of smp_no_rendevous_barrier()
that calls smp_no_rendezvous_barrier().

Reviewed by:	gnn, kib
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D10313
2017-04-09 02:00:03 +00:00
Alexander Kabaev
8b609ea571 Define 'lr' as x30 on aarch64
GNU toolchain does not recognize LR as standard register alias,
but clang does. Use of #define will work on both. Place the
definition into central machine/asm.h instead of patching every
affected file, as requested by plaftorm maintainers.

Reviews by: andrew, emaste, imp
Differential Revision:	https://reviews.freebsd.org/D10307
2017-04-07 22:58:28 +00:00
Bruce Evans
f434f3515b Fix printing of negative offsets (typically from frame pointers) again.
I fixed this in 1997, but the fix was over-engineered and fragile and
was broken in 2003 if not before.  i386 parameters were copied to 8
other arches verbatim, mostly after they stopped working on i386, and
mostly without the large comment saying how the values were chosen on
i386.  powerpc has a non-verbatim copy which just changes the uncritical
parameter and seems to add a sign extension bug to it.

Just treat negative offsets as offsets if they are no more negative than
-db_offset_max (default -64K), and remove all the broken parameters.

-64K is not very negative, but it is enough for frame and stack pointer
offsets since kernel stacks are small.

The over-engineering was mainly to go more negative than -64K for the
negative offset format, without affecting printing for more than a
single address.

Addresses in the top 64K of a (full 32-bit or 64-bit) address space
are now printed less well, but there aren't many interesting ones.
For arches that have many interesting ones very near the top (e.g.,
68k has interrupt vectors there), there would be no good limit for
the negative offset format and -64K is a good as anything.
2017-03-26 18:46:35 +00:00
Warner Losh
480c955cee Add 'device iic' to bring in userland I2C driver.
Submitted by: karl@
2017-03-24 22:33:03 +00:00
Ed Schouten
ebfc28088b Stop providing the compat_3_brand.
As of r315860, the ELF image activator works fine for CloudABI without it.

Reviewed by:	kib
MFC after:	2 weeks
2017-03-23 14:12:21 +00:00
Konstantin Belousov
2274ab3d7b Update r315753 with the proper flag name.
Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
2017-03-22 22:28:13 +00:00
Konstantin Belousov
1438fe3cf2 Add a flag BI_BRAND_ONLY_STATIC to specify that the brand only
matches static binaries.

Interpretation of the 'static' there is that the binary must not
specify an interpreter.  In particular, shared objects are matched by
the brand if BI_CAN_EXEC_DYN is also set.

This improves precision of the brand matching, which should eliminate
surprises due to brand ordering.

Revert r315701.

Discussed with and tested by:	ed (previous version)
Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
2017-03-22 22:23:01 +00:00
Ed Schouten
ae2373da91 Set the interpreter path to /nonexistent.
CloudABI executables are statically linked and don't have an
interpreter. Setting the interpreter path to NULL used to work
previously, but r314851 introduced code that checks the string
unconditionally. Running CloudABI executables now causes a null pointer
dereference.

Looking at the rest of imgact_elf.c, it seems various other codepaths
already leaned on the fact that the interpreter path is set. Let's just
go ahead and pick an obviously incorrect interpreter path to appease
imgact_elf.c.

MFC after:	1 week
2017-03-22 07:05:27 +00:00
Andrew Turner
86b5c43667 If ofw_bus_msimap fails don't try to use the invalid MSI/MSI-X parent node.
Sponsored by:	ABT Systems Ltd
2017-03-16 17:49:37 +00:00
Andrew Turner
783b367605 Load the new sp_el0 with interrupts disabled in fork_trampoline. If an
interrupt arrives in fork_trampoline after sp_el0 was written we may then
switch to a new thread, enter userland so change this stack pointer, then
return to this code with the wrong value. This fixes this case by moving
the load of sp_el0 until after interrupts have been disabled.

Reported by:	Mark Millard (markmi@dsl-only.net)
Sponsored by:	ABT Systems Ltd
Differential Revision:	https://reviews.freebsd.org/D9593
2017-02-15 14:56:47 +00:00
Andrew Turner
9c6d6488fa Port the Linux AMX 10G network driver to FreeBSD as axgbe. It is unlikely
we will import a newer version of the Linux code so the linuxkpi was not
used.

This is still missing 10G support, and multicast has not been tested.

Reviewed by:	gnn
Obtained from:	ABT Systems Ltd
Sponsored by:	SoftIron Inc
Differential Revision:	https://reviews.freebsd.org/D8549
2017-02-15 13:56:04 +00:00
Andrew Turner
f17e4f0759 Push reading of ESR_EL1 to assembly. Among other uses this will allow us
to expose this to signal handlers, e.g. for the clang sanitizers.

Sponsored by:	DARPA, AFRL
2017-02-07 18:19:11 +00:00
Andrew Turner
2c40450667 Remove arm64_tlb_flushID_SE, it's unused and may be wrong.
Sponsored by:	ABT Systems Ltd
2017-02-06 17:50:09 +00:00
Konstantin Belousov
3b7a388b3e Update arm and arm64 counters MD bits.
On arm64 use atomics.  Then, both arm and arm64 do not need a critical
section around update.  Replace all cpus loop by CPU_FOREACH().
This brings arm and arm64 counter(9) implementation closer to current
amd64, but being more RISC-y, arm* version cannot avoid atomics.

Reported by:	Alexandre Martins <alexandre.martins@stormshield.eu>
Reviewed by:	andrew
Tested by:	Alexandre Martins, andrew
Sponsored by:	The FreeBSD Foundation
MFC after:	2 weeks
2017-02-06 17:20:37 +00:00
Konstantin Belousov
9fb10d635e Define the vm_ooffset_t and vm_pindex_t types as machine-independend.
The types are for the byte offset and page index in vm object.  They
are similar to off_t, which is defined as 64bit MI integer.  Using MI
definitions will allow to provide consistent MD values of vm
object-related maximum sizes.

Reviewed by:	alc
Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
2017-02-04 12:26:38 +00:00
Olivier Houchard
dc5f9fcdae Implement atomic_fcmpset_* for arm and arm64. 2017-01-28 16:24:06 +00:00
Bjoern A. Zeeb
09d8a96e1c Remove a static function declaration for a function not implemented.
Makes head code compile on 10.3 and cleanup is never wrong.

MFC after:	3 days
2017-01-23 16:40:20 +00:00
Ed Schouten
4423244072 Catch up with changes to structure member names.
Pointer/length pairs are now always named ${name} and ${name}_len.
2017-01-17 22:05:52 +00:00
Sean Bruno
f2d6ace4a6 Migrate e1000 to the IFLIB framework:
- em(4) igb(4) and lem(4)
- deprecate the igb device from kernel configurations
- create a symbolic link in /boot/kernel from if_em.ko to if_igb.ko

Devices tested:
- 82574L
- I218-LM
- 82546GB
- 82579LM
- I350
- I217

Please report problems to freebsd-net@freebsd.org

Partial review from jhb and suggestions on how to *not* brick folks who
originally would have lost their igbX device.

Submitted by:	mmacy@nextbsd.org
MFC after:	2 weeks
Relnotes:	yes
Sponsored by:	Limelight Networks and Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D8299
2017-01-10 03:23:22 +00:00
Jayachandran C.
de5cef6525 Add virtio_pci to GENERIC arm64 conf
virtio_pci was missing from the GENERIC arm64 configuration, while
other virtio devices are present. Adding it will allow us to boot
the GENERIC kernel on QEMU with virtio storage and networking.
2016-12-18 11:15:31 +00:00
Jayachandran C.
b2ef3bae84 Initialize GIC[DR]_IGROUPRn registers for GICv3
In case where GICD_CTLR.DS is 1, the IGROUPR registers are RW in
non-secure state and has to be initialized to 1 for the
corresponding interrupts to be delivered as Group 1 interrupts.

Update gic_v3_dist_init() and gic_v3_redist_init() to initialize
GICD_IGROUPRn and GICR_IGROUPRn respectively to address this. The
registers can be set unconditionally since the writes are ignored
in non-secure state when GICD_CTLR.DS is 0.

This fixes the hang on boot seen when running qemu-system-aarch64
with machine virt,gic-version=3
2016-12-18 08:31:01 +00:00
Andrew Turner
adca4ffc22 Enable ACPI on arm64. It's not yet functional, but it will help keeping the
code building until it is ready.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-12-12 18:13:03 +00:00
Konrad Witaszczyk
480f31c214 Add support for encrypted kernel crash dumps.
Changes include modifications in kernel crash dump routines, dumpon(8) and
savecore(8). A new tool called decryptcore(8) was added.

A new DIOCSKERNELDUMP I/O control was added to send a kernel crash dump
configuration in the diocskerneldump_arg structure to the kernel.
The old DIOCSKERNELDUMP I/O control was renamed to DIOCSKERNELDUMP_FREEBSD11 for
backward ABI compatibility.

dumpon(8) generates an one-time random symmetric key and encrypts it using
an RSA public key in capability mode. Currently only AES-256-CBC is supported
but EKCD was designed to implement support for other algorithms in the future.
The public key is chosen using the -k flag. The dumpon rc(8) script can do this
automatically during startup using the dumppubkey rc.conf(5) variable.  Once the
keys are calculated dumpon sends them to the kernel via DIOCSKERNELDUMP I/O
control.

When the kernel receives the DIOCSKERNELDUMP I/O control it generates a random
IV and sets up the key schedule for the specified algorithm. Each time the
kernel tries to write a crash dump to the dump device, the IV is replaced by
a SHA-256 hash of the previous value. This is intended to make a possible
differential cryptanalysis harder since it is possible to write multiple crash
dumps without reboot by repeating the following commands:
# sysctl debug.kdb.enter=1
db> call doadump(0)
db> continue
# savecore

A kernel dump key consists of an algorithm identifier, an IV and an encrypted
symmetric key. The kernel dump key size is included in a kernel dump header.
The size is an unsigned 32-bit integer and it is aligned to a block size.
The header structure has 512 bytes to match the block size so it was required to
make a panic string 4 bytes shorter to add a new field to the header structure.
If the kernel dump key size in the header is nonzero it is assumed that the
kernel dump key is placed after the first header on the dump device and the core
dump is encrypted.

Separate functions were implemented to write the kernel dump header and the
kernel dump key as they need to be unencrypted. The dump_write function encrypts
data if the kernel was compiled with the EKCD option. Encrypted kernel textdumps
are not supported due to the way they are constructed which makes it impossible
to use the CBC mode for encryption. It should be also noted that textdumps don't
contain sensitive data by design as a user decides what information should be
dumped.

savecore(8) writes the kernel dump key to a key.# file if its size in the header
is nonzero. # is the number of the current core dump.

decryptcore(8) decrypts the core dump using a private RSA key and the kernel
dump key. This is performed by a child process in capability mode.
If the decryption was not successful the parent process removes a partially
decrypted core dump.

Description on how to encrypt crash dumps was added to the decryptcore(8),
dumpon(8), rc.conf(5) and savecore(8) manual pages.

EKCD was tested on amd64 using bhyve and i386, mipsel and sparc64 using QEMU.
The feature still has to be tested on arm and arm64 as it wasn't possible to run
FreeBSD due to the problems with QEMU emulation and lack of hardware.

Designed by:	def, pjd
Reviewed by:	cem, oshogbo, pjd
Partial review:	delphij, emaste, jhb, kib
Approved by:	pjd (mentor)
Differential Revision:	https://reviews.freebsd.org/D4712
2016-12-10 16:20:39 +00:00
George V. Neville-Neil
c2e34f488c This adds a configuration for arm64 users that track CURRENT but
don't need the extra debug facilities.  Copied from the amd64
configuration of the same name.

Submitted by: Nikolai Lifanov
Reviewed by: emaste
MFC after: 2 weeks
2016-12-10 10:00:27 +00:00
Alan Cox
e94965d82e Previously, vm_radix_remove() would panic if the radix trie didn't
contain a vm_page_t at the specified index.  However, with this
change, vm_radix_remove() no longer panics.  Instead, it returns NULL
if there is no vm_page_t at the specified index.  Otherwise, it
returns the vm_page_t.  The motivation for this change is that it
simplifies the use of radix tries in the amd64, arm64, and i386 pmap
implementations.  Instead of performing a lookup before every remove,
the pmap can simply perform the remove.

Reviewed by:	kib, markj
Differential Revision:	https://reviews.freebsd.org/D8708
2016-12-08 04:29:29 +00:00
Andrew Turner
75747c209c Add ACPI support to the arm64 mp code. We use the Multiple APIC Description
Table to find the CPUs to find the CPUs to start. Currently we assume PSCI,
however this assumption is shared with the FDT code.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-12-07 14:35:05 +00:00
Andrew Turner
0afaf183d0 Add the missing gic_common.h include for GIC_BUS_FDT.
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-12-06 15:16:00 +00:00
Andrew Turner
be04b41da2 Create two GIC ivars to find the bus type and GIC hardware version. These
will be used by the gicv2m and ITS ACPI drivers to only attach to the
correct parent.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-12-06 15:12:14 +00:00
Andrew Turner
c417fba9eb Move the common bit manipulation macros from the GICv3 header to the
common GIC header file.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-12-06 13:55:19 +00:00
Andrew Turner
31b892557f Move the stored signal mask later in the jump buf. It was being clobbered
by a later store of a VFP register.

Sponsored by:	ABT Systems Ltd
2016-12-06 13:46:12 +00:00
Andrew Turner
00e61f254c Add the full set of GICD_PIDR* registers
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-12-06 11:41:19 +00:00
Andrew Turner
7abeae41fb Remove an unneeded header. It was used when we had an arm64 specific GIC
driver, however with INTRNG this has been merged into a common arm driver.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-12-05 11:40:54 +00:00
Andrew Turner
d2314cb3b9 Fix the alloc function the ThunderX PCIe driver calls, the previous
function may not exist when FDT is removed from the kernel.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-11-30 14:17:06 +00:00
Andrew Turner
323057e8af We only use the cpu0 variable in the FDT code.
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-11-29 13:32:23 +00:00
Andrew Turner
a70475ca42 Fix ISS_DATA_DFSC_MASK, it should include all bits in the field. 2016-11-28 14:24:07 +00:00
Andrew Turner
dc836c65c8 Fix the TLB conflict abort value. This should be a no-op as we don't use
this value in the code.
2016-11-25 16:04:36 +00:00
Oleksandr Tymoshenko
9153cbff42 [rpi3] Move SOC_BRCM_BCM2837 from UP config to SMP one
Now that BCM283x source are buildable with SMP option it cam be moved to
GENERIC SMP config. SMP itself does not work on RPi3 yet due to lack of
PSCI monitor which is work in progress at the moment
2016-11-24 00:45:52 +00:00
Andrew Turner
925f047fdc Mark cpu_find_cpu0_fdt as FDT only. It's only called when this is set, and
the kernel is using FDT.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-11-23 09:38:10 +00:00
Andrew Turner
07ec266234 Remove the unneeded ofw_cpu_reg function signature, it's not used in this
file.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-11-23 09:33:47 +00:00
Andrew Turner
ab950d2d18 Add the missing DEVMETHOD_ENDs to the arm64 nexus ACPI and FDT drivers.
Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-11-22 19:31:32 +00:00
Andrew Turner
eba1a249df Add support to find the arm64 serial using the ACPI tables. This uses the
Serial Port Console Redirection Table to find the device to use.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-11-21 19:26:58 +00:00
Andrew Turner
f94f8e62ab To allow for an ACPI attachment to the generic PCIe driver split off the
FDT attachment to a new file. A separate ACPI attachment will then be added
to allow arm64 servers with ACPI to use it over FDT.

This should also help with merging this with the ofwpci driver, with
further work needed to remove restrictions this driver places on resource
allocation.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D7319
2016-11-21 18:24:05 +00:00
Andrew Turner
b1895f7270 Include the SMSC LAN91C111 driver, this is found in some of the ARM models. 2016-11-16 11:37:43 +00:00
Bryan Drewery
28323add09 Fix improper use of "its".
Sponsored by:	Dell EMC Isilon
2016-11-08 23:59:41 +00:00
Andrew Turner
0c5434ae58 Pull the common FDT interrupt values into a new header rather than be magic
numbers.

Sponsored by:	DARPA, AFRL
2016-10-26 15:18:08 +00:00
Andrew Turner
2eb8079722 Only release CPUs when they exist.
MFC after:	1 week
Sponsored by:	DARPA, AFRL
2016-10-26 12:30:53 +00:00
Andrew Turner
2e620e70f9 Use nitems to get the correct number of registers to read when dumping
them. Previously this would walk past the end of the array and print
whatever happened to be after the trapframe struct.

MFC after:	1 week
Sponsored by:	DARPA, AFRL
2016-10-26 12:29:56 +00:00
Andrew Turner
c2dd354bf4 Create a new PSCI error code and use it to signal that starting the CPU is
impossible as the PSCI firmware is missing.

Sponsored by:	ABT Systmes Ltd
2016-10-25 14:18:27 +00:00
Andrew Turner
02e1c9861d Import the Cortex String memcpy and memmove into the kernel. On ThunderX
these show a 9-10% reduction in user and system time for a buildworld -j48.

Obtained from:	ABT Systems Ltd
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2016-10-25 14:04:35 +00:00
Andrew Turner
9eb0ccbb54 Increase CACHE_LINE_SHIFT to 7 as cache lines are 128 bytes on ThunderX.
MFC after:	1 week
Sponsored by:	ABT Systems Ltd
2016-10-24 13:44:24 +00:00
Wojciech Macek
7902c8dca8 Driver for PCI Ethernet NIC on Alpine V1 and V2.
Obtained from:         Semihalf
Submitted by:          Michal Stanek <mst@semihalf.com>
Sponsored by:          Annapurna Labs
Reviewed by:           wma
Differential Revision: https://reviews.freebsd.org/D7814
2016-10-20 11:31:11 +00:00
Andrew Turner
6ed982a221 Add support for the fpu_kern(9) KPI on arm64. It hooks into the existing
VFP code to store the old context, with lazy loading of the new context
when needed.

FPU_KERN_NOCTX is missing as this is unused in the crypto code this has
been tested with, and I am unsure on the requirements of the UEFI
Runtime Services.

Reviewed by:	kib
Obtained from:	ABT Systeems Ltd
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D8276
2016-10-20 09:22:10 +00:00
Oleksandr Tymoshenko
3ab1c20e5b Move SOC_BRCM_BCM2837 to GENERIC-UP since does not support SMP 2016-10-19 02:03:30 +00:00
Oleksandr Tymoshenko
232de4a9c2 Retire RPI3 kernel config in favour of GENERIC-UP
Policy for FreeBSD/arm64 kernel config is the same one as for x86
architectures: provide GENERIC kernel bootable on as many systems
as possible. Since there is no SMP support for RPi 3 yet new kernel
config was introduced: GENERIC-UP, which is effectively GENERIC with
SMP option disabled
2016-10-18 23:48:47 +00:00
Andrew Turner
4c247b9719 Add PCB_FP_USERMASK so we can mask off floating point flags that should
not be sent to userspace, for example the future flag to tell when we are
using floating point in the kernel.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-10-18 13:39:55 +00:00
Andrew Turner
d88a17dd8f Fix the build, struct vfpstate needs to be visible to userspace as it's
part of struct pcb which is in a header used in libutil.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-10-15 16:29:06 +00:00
Andrew Turner
403ddccb10 Check we are in a critical section when calling vfp_discard. As we may call
it with a NULL thread pointer only check when it is non-NULL.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-10-15 09:54:22 +00:00
Oleksandr Tymoshenko
745c4aa5e8 Make BRCM2837 port conform FreeBSD/ARM64 guidelines
- Rename SOC_BCM2837 to SOC_BRCM_BCM2837, put it to opt_soc.h
- do not use files.XXX files, just move required sources to
    conf/files.arm64 and make them depend on soc_brcm_bcm2837

Suggested by: andrew
2016-10-14 22:23:03 +00:00
Andrew Turner
401d3029af Create macros for the MAIR memory attributes. While here add an uncached
memory type, however the VM code still needs to be taught about this.

MFC after:	1 week
Sponsored by:	ABT Systems Ltd
2016-10-14 22:20:14 +00:00
John Baldwin
31dc1e9681 Drop support for using mmap() with /dev/kmem.
Using the device pager with /dev/kmem is not stable since KVA mappings
are transient, but the device pager caches the PA associated with a
given offset forever.  Interestingly, mips' implementation of
memmap() already refused requests for /dev/kmem.

Note that kvm_read/kvm_write do not use mmap, but use read and write on
/dev/kmem, so this should not affect libkvm users.

Reviewed by:	kib
MFC after:	2 months
2016-10-14 20:01:07 +00:00
Oleksandr Tymoshenko
65de32210c Remove redundant kern_clocksource.c. It is already in conf/files.arm64
Reported by:	andrew
2016-10-14 17:40:48 +00:00
Andrew Turner
8ff003011d Rework how we store the VFP registers in the pcb. This will be used when
creating a floating-point context within the kernel without having to move
the stored values in memory.

Sponsored by:	The FreeBSD Foundation
2016-10-14 15:53:48 +00:00
Oleksandr Tymoshenko
563193fd0c Add initial Raspberry Pi 3 support
RPI3 kernel config builds kernel compatible with latest upstream device
tree and firmware: https://github.com/raspberrypi/firmware/tree/master/boot
As of today it's 597c662a613df1144a6bc43e5f4505d83bd748ca

Default console is PL01x, so pi3-disable-bt dt overlay should be configured
in config.txt and stock U-Boot should be patched to use proper serial port.

Yet unsupported: SMP, VCHIQ, RNG driver. RNG requires some work due to
upstream device tree incompatibility.

Multiple people contributed to this work over time: db@, loos@, manu@
2016-10-14 03:37:35 +00:00
Andrew Turner
1890f18cbe Move printing the AArch64 ID registers to a new SYSINIT, the previous
location only prints them when booting on SMP with multiple cores.

MFC after:	1 week
Sponsored by:	DARPA, AFRL
2016-10-13 09:06:29 +00:00
Warner Losh
b2a7ac4802 Fix building on i386 and arm. But 'public domain' headers on the files
with no creative content. Include "lost" changes from git:
o Use /dev/efi instead of /dev/efidev
o Remove redundant NULL checks.

Submitted by: kib@, dim@, zbb@, emaste@
2016-10-13 06:56:23 +00:00
Oleksandr Tymoshenko
c38fb7809f INTRNG: Propagate IRQ activation error to API consumer
Keep resource state consistent with INTRNG state - if intr_activate_irq
fails - deactivate resource and propagate error to calling function

Reviewed by:	mmel
2016-10-12 17:10:59 +00:00
Jonathan T. Looney
bd79708dbf In the TCP stack, the hhook(9) framework provides hooks for kernel modules
to add actions that run when a TCP frame is sent or received on a TCP
session in the ESTABLISHED state. In the base tree, this functionality is
only used for the h_ertt module, which is used by the cc_cdg, cc_chd, cc_hd,
and cc_vegas congestion control modules.

Presently, we incur overhead to check for hooks each time a TCP frame is
sent or received on an ESTABLISHED TCP session.

This change adds a new compile-time option (TCP_HHOOK) to determine whether
to include the hhook(9) framework for TCP. To retain backwards
compatibility, I added the TCP_HHOOK option to every configuration file that
already defined "options INET". (Therefore, this patch introduces no
functional change. In order to see a functional difference, you need to
compile a custom kernel without the TCP_HHOOK option.) This change will
allow users to easily exclude this functionality from their kernel, should
they wish to do so.

Note that any users who use a custom kernel configuration and use one of the
congestion control modules listed above will need to add the TCP_HHOOK
option to their kernel configuration.

Reviewed by:	rrs, lstewart, hiren (previous version), sjg (makefiles only)
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D8185
2016-10-12 02:16:42 +00:00
Jung-uk Kim
493deb390b Merge ACPICA 20160930. 2016-10-04 20:27:15 +00:00
Konstantin Belousov
559a7b209a Add a way for the architecture to specify the calling ABI for methods
in the EFI Runtime Services Table.  On amd64, the calling conventions
are MS.

Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
2016-09-21 10:35:44 +00:00
Wojciech Macek
a134b98878 Add support for SPI-mapped MSI interrupts in GICv3.
PIC_SETUP_INTR implementation in GICv3 did not allow
for setting up interrupts without included FDT
description. GICv2m-like MSI interrupts, which map
MSI messages to SPI interrupt lines, may not have
a description in FDT. Add support for such interrupts
by setting the trigger and polarity to the appropriate
values for MSI (edge, high) and get the hardware
IRQ number from the corresponding ISRC.

Obtained from:         Semihalf
Submitted by:          Michal Stanek <mst@semihalf.com>
Sponsored by:          Annapurna Labs
Reviewed by:           wma
Differential Revision: https://reviews.freebsd.org/D7662
2016-09-21 05:22:49 +00:00
Wojciech Macek
3502e7b37c Register GICv3 xref.
This allows other drivers to retrieve interrupt parent node.

Obtained from:         Semihalf
Submitted by:          Michal Stanek <mst@semihalf.com>
Sponsored by:          Annapurna Labs
Reviewed by:           wma, zbb
Differential Revision: https://reviews.freebsd.org/D7568
2016-09-21 05:15:50 +00:00