525 Commits

Author SHA1 Message Date
andrew
96474ac2aa Import the Cortex String memcpy and memmove into the kernel. On ThunderX
these show a 9-10% reduction in user and system time for a buildworld -j48.

Obtained from:	ABT Systems Ltd
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2016-10-25 14:04:35 +00:00
andrew
26ea906520 Increase CACHE_LINE_SHIFT to 7 as cache lines are 128 bytes on ThunderX.
MFC after:	1 week
Sponsored by:	ABT Systems Ltd
2016-10-24 13:44:24 +00:00
wma
ec78fa7f7f Driver for PCI Ethernet NIC on Alpine V1 and V2.
Obtained from:         Semihalf
Submitted by:          Michal Stanek <mst@semihalf.com>
Sponsored by:          Annapurna Labs
Reviewed by:           wma
Differential Revision: https://reviews.freebsd.org/D7814
2016-10-20 11:31:11 +00:00
andrew
75768c2bf4 Add support for the fpu_kern(9) KPI on arm64. It hooks into the existing
VFP code to store the old context, with lazy loading of the new context
when needed.

FPU_KERN_NOCTX is missing as this is unused in the crypto code this has
been tested with, and I am unsure on the requirements of the UEFI
Runtime Services.

Reviewed by:	kib
Obtained from:	ABT Systeems Ltd
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D8276
2016-10-20 09:22:10 +00:00
gonzo
3f40ad67cc Move SOC_BRCM_BCM2837 to GENERIC-UP since does not support SMP 2016-10-19 02:03:30 +00:00
gonzo
2f351dfb10 Retire RPI3 kernel config in favour of GENERIC-UP
Policy for FreeBSD/arm64 kernel config is the same one as for x86
architectures: provide GENERIC kernel bootable on as many systems
as possible. Since there is no SMP support for RPi 3 yet new kernel
config was introduced: GENERIC-UP, which is effectively GENERIC with
SMP option disabled
2016-10-18 23:48:47 +00:00
andrew
bd90949460 Add PCB_FP_USERMASK so we can mask off floating point flags that should
not be sent to userspace, for example the future flag to tell when we are
using floating point in the kernel.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-10-18 13:39:55 +00:00
andrew
4a327f5e89 Fix the build, struct vfpstate needs to be visible to userspace as it's
part of struct pcb which is in a header used in libutil.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-10-15 16:29:06 +00:00
andrew
2a35286daf Check we are in a critical section when calling vfp_discard. As we may call
it with a NULL thread pointer only check when it is non-NULL.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2016-10-15 09:54:22 +00:00
gonzo
8c172582e2 Make BRCM2837 port conform FreeBSD/ARM64 guidelines
- Rename SOC_BCM2837 to SOC_BRCM_BCM2837, put it to opt_soc.h
- do not use files.XXX files, just move required sources to
    conf/files.arm64 and make them depend on soc_brcm_bcm2837

Suggested by: andrew
2016-10-14 22:23:03 +00:00
andrew
0058087fc0 Create macros for the MAIR memory attributes. While here add an uncached
memory type, however the VM code still needs to be taught about this.

MFC after:	1 week
Sponsored by:	ABT Systems Ltd
2016-10-14 22:20:14 +00:00
jhb
f689fd5a63 Drop support for using mmap() with /dev/kmem.
Using the device pager with /dev/kmem is not stable since KVA mappings
are transient, but the device pager caches the PA associated with a
given offset forever.  Interestingly, mips' implementation of
memmap() already refused requests for /dev/kmem.

Note that kvm_read/kvm_write do not use mmap, but use read and write on
/dev/kmem, so this should not affect libkvm users.

Reviewed by:	kib
MFC after:	2 months
2016-10-14 20:01:07 +00:00
gonzo
42e43cc882 Remove redundant kern_clocksource.c. It is already in conf/files.arm64
Reported by:	andrew
2016-10-14 17:40:48 +00:00
andrew
4d2f7af1cf Rework how we store the VFP registers in the pcb. This will be used when
creating a floating-point context within the kernel without having to move
the stored values in memory.

Sponsored by:	The FreeBSD Foundation
2016-10-14 15:53:48 +00:00
gonzo
b966860f8c Add initial Raspberry Pi 3 support
RPI3 kernel config builds kernel compatible with latest upstream device
tree and firmware: https://github.com/raspberrypi/firmware/tree/master/boot
As of today it's 597c662a613df1144a6bc43e5f4505d83bd748ca

Default console is PL01x, so pi3-disable-bt dt overlay should be configured
in config.txt and stock U-Boot should be patched to use proper serial port.

Yet unsupported: SMP, VCHIQ, RNG driver. RNG requires some work due to
upstream device tree incompatibility.

Multiple people contributed to this work over time: db@, loos@, manu@
2016-10-14 03:37:35 +00:00
andrew
75e242c27e Move printing the AArch64 ID registers to a new SYSINIT, the previous
location only prints them when booting on SMP with multiple cores.

MFC after:	1 week
Sponsored by:	DARPA, AFRL
2016-10-13 09:06:29 +00:00
imp
081e8d8587 Fix building on i386 and arm. But 'public domain' headers on the files
with no creative content. Include "lost" changes from git:
o Use /dev/efi instead of /dev/efidev
o Remove redundant NULL checks.

Submitted by: kib@, dim@, zbb@, emaste@
2016-10-13 06:56:23 +00:00
gonzo
043456d8a0 INTRNG: Propagate IRQ activation error to API consumer
Keep resource state consistent with INTRNG state - if intr_activate_irq
fails - deactivate resource and propagate error to calling function

Reviewed by:	mmel
2016-10-12 17:10:59 +00:00
jtl
62030781cd In the TCP stack, the hhook(9) framework provides hooks for kernel modules
to add actions that run when a TCP frame is sent or received on a TCP
session in the ESTABLISHED state. In the base tree, this functionality is
only used for the h_ertt module, which is used by the cc_cdg, cc_chd, cc_hd,
and cc_vegas congestion control modules.

Presently, we incur overhead to check for hooks each time a TCP frame is
sent or received on an ESTABLISHED TCP session.

This change adds a new compile-time option (TCP_HHOOK) to determine whether
to include the hhook(9) framework for TCP. To retain backwards
compatibility, I added the TCP_HHOOK option to every configuration file that
already defined "options INET". (Therefore, this patch introduces no
functional change. In order to see a functional difference, you need to
compile a custom kernel without the TCP_HHOOK option.) This change will
allow users to easily exclude this functionality from their kernel, should
they wish to do so.

Note that any users who use a custom kernel configuration and use one of the
congestion control modules listed above will need to add the TCP_HHOOK
option to their kernel configuration.

Reviewed by:	rrs, lstewart, hiren (previous version), sjg (makefiles only)
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D8185
2016-10-12 02:16:42 +00:00
jkim
7706f244fc Merge ACPICA 20160930. 2016-10-04 20:27:15 +00:00
kib
0b0178b3a6 Add a way for the architecture to specify the calling ABI for methods
in the EFI Runtime Services Table.  On amd64, the calling conventions
are MS.

Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
2016-09-21 10:35:44 +00:00
wma
bfcbca20d8 Add support for SPI-mapped MSI interrupts in GICv3.
PIC_SETUP_INTR implementation in GICv3 did not allow
for setting up interrupts without included FDT
description. GICv2m-like MSI interrupts, which map
MSI messages to SPI interrupt lines, may not have
a description in FDT. Add support for such interrupts
by setting the trigger and polarity to the appropriate
values for MSI (edge, high) and get the hardware
IRQ number from the corresponding ISRC.

Obtained from:         Semihalf
Submitted by:          Michal Stanek <mst@semihalf.com>
Sponsored by:          Annapurna Labs
Reviewed by:           wma
Differential Revision: https://reviews.freebsd.org/D7662
2016-09-21 05:22:49 +00:00
wma
3edc270a7d Register GICv3 xref.
This allows other drivers to retrieve interrupt parent node.

Obtained from:         Semihalf
Submitted by:          Michal Stanek <mst@semihalf.com>
Sponsored by:          Annapurna Labs
Reviewed by:           wma, zbb
Differential Revision: https://reviews.freebsd.org/D7568
2016-09-21 05:15:50 +00:00
wma
e53ee72b9a Add driver for PCIe root complex on Annapurna Alpine platform.
The driver subclasses pci-host-generic and additionally
performs configuration of vendor-specific PCIe registers.

Obtained from:         Semihalf
Submitted by:          Michal Stanek <mst@semihalf.com>
Sponsored by:          Annapurna Labs
Reviewed by:           wma
Differential Revision: https://reviews.freebsd.org/D7571
2016-09-20 11:11:06 +00:00
kib
90ee5f51f2 Consolidate four efi_next_descriptor() definitions.
Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
2016-09-18 17:38:02 +00:00
andrew
15d6496613 Add a missing opt_ddb.h include from places that include debug_moditor.h
to fix the build.

Obtained from:	ABT Systems Ltd
X-MFC with:	r305771
Sponsored by:	The FreeBSD Foundation
2016-09-13 16:20:31 +00:00
andrew
4a70bd772c Fix the arm64 kernel build when DDB is disabled, debug_monitor.c depends on
DDB, and is unused when it's disabled.

Obtained from:	ABT Systems Ltd
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2016-09-13 15:45:22 +00:00
andrew
6a0e9d7faa Add a warning about a known erratum we have observed on ThunderX pass 1.1.
As this is evaluation hardware with only a few users, and there is a lack
of information add a warning when booting on this hardware.

Reported by:	cognet
Obtained from:	ABT Systems Ltd
MFC after:	Instant
Sponsored by:	The FreeBSD Foundation
2016-09-13 14:10:49 +00:00
alc
44f29780e8 Various changes to pmap_ts_referenced()
Move PMAP_TS_REFERENCED_MAX out of the various pmap implementations and
into vm/pmap.h, and describe what its purpose is.  Eliminate the archaic
"XXX" comment about its value.  I don't believe that its exact value, e.g.,
5 versus 6, matters.

Update the arm64 and riscv pmap implementations of pmap_ts_referenced()
to opportunistically update the page's dirty field.

On amd64, use the PDE value already cached in a local variable rather than
dereferencing a pointer again and again.

Reviewed by:	kib, markj
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D7836
2016-09-10 16:49:25 +00:00
andrew
41d6813369 Trap msr/mrs instructions. These are privileged arm64 instructions and
shouldn't normally be used.

Obtained from:	ABT Systems Ltd
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2016-09-08 12:53:01 +00:00
andrew
471921aaf5 Don't panic when we don't handle a userland exception, not all we may see
are currently handled.

Obtained from:	ABT Systems Ltd
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2016-09-08 12:39:03 +00:00
andrew
a66a58c963 When synchronising the instruction and data caches we only need to clean
the data cache to the point of unification. This is the point where the
two caches are unified to a single unified cache so cleaning past here
is just extra unneeded work.

This was noticed when investigating r305545.

Reported by:	bz
Obtained from:	ABT Systems Ltd
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2016-09-07 16:46:54 +00:00
andrew
4bde709fd3 Only call cpu_icache_sync_range when inserting an executable page. If the
page is non-executable the contents of the i-cache are unimportant so this
call is just adding unneeded overhead when inserting pages.

While doing research using gem5 with an O3 pipeline and 1k/32k/1M iTLB/L1
iCache/L2 Bjoern Zeeb (bz@) observed a fairly high rate of calls into
arm64_icache_sync_range() from pmap_enter() along with a high number of
instruction fetches and iTLB/iCache hits.

Limiting the calls to arm64_icache_sync_range() to only executable pages,
we observe the iTLB and iCache Hit going down by about 43%. These numbers
are quite misleading when looked at alone as at the same time instructions
retired were reduced by 19.2% and instruction fetches were reduced by 38.8%.
Overall this reduced the runtime of the test program by 22.4%.

On Juno hardware, in steady-state, running the same test, using the cycle
count to determine runtime, we do see a reduction of up to 28.9% in runtime.

While these numbers certainly depend on the program executed, we expect an
overall performance improvement.

Reported by:	bz
Obtained from:	ABT Systems Ltd
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2016-09-07 16:22:05 +00:00
wma
0da791731a Introduce support for Annapurna Alpine CCU and NB devices
This commit adds drivers for Alpine Cache Coherency Unit
and North Bridge Service whose task is to configure
the system fabric and enable cache coherency.

Obtained from:         Semihalf
Submitted by:          Michal Stanek <mst@semihalf.com>
Sponsored by:          Annapurna Labs
Reviewed by:           wma
Differential Revision: https://reviews.freebsd.org/D7565
2016-09-07 05:34:41 +00:00
jmcneill
fe422ee664 Add generic device-tree cpufreq driver. 2016-09-06 21:18:14 +00:00
andrew
9b7353a566 Enable superpages on arm64 by default. These seem to be stable, having
survived multiple world and kernel builds, and of poudriere building full
package sets.

I have observed a 3% reduction in buildworld times with superpages enabled,
however further testing is needed to see if this is observed in other
workloads.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-09-04 17:50:23 +00:00
markj
fb5804c98d Remove support for idle page zeroing.
Idle page zeroing has been disabled by default on all architectures since
r170816 and has some bugs that make it seemingly unusable. Specifically,
the idle-priority pagezero thread exacerbates contention for the free page
lock, and yields the CPU without releasing it in non-preemptive kernels. The
pagezero thread also does not behave correctly when superpage reservations
are enabled: its target is a function of v_free_count, which includes
reserved-but-free pages, but it is only able to zero pages belonging to the
physical memory allocator.

Reviewed by:	alc, imp, kib
Differential Revision:	https://reviews.freebsd.org/D7714
2016-09-03 20:38:13 +00:00
jmcneill
e25bfb37e5 Add support for Allwinner A64 thermal sensors. 2016-09-03 15:26:00 +00:00
andrew
d31d501396 Add a pc_clock pcpu field and use it to implement cpu_est_clockrate. This
will allow drivers that manage the clock frequency to communicate this with
the reset of the kernel.

Reported by:	jmcneill
MFC after:	1 week
Sponsored by:	ABT Systems Ltd
2016-09-02 10:13:51 +00:00
andrew
afff8d4f96 Fix arm64 superpages bugs in pmap_enter:
* Pass the correct virtual address when demoting a superpage
 * Use the correct l3 table after demoting a superpage
 * Remove an invalid KASSERT hit demoting then promoting a superpage [1]

With this it is believed that  superpages on arm64 is stable.

Reported by:	[1] cognet
Obtained from:	ABT Systems Ltd
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2016-09-01 10:26:06 +00:00
andrew
2d05ee6e7e Also handle instruction traps. We might hit these when the page we are
executing is being promoted to a superpage.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-31 13:58:32 +00:00
andrew
098b8c899e Because we need to use a break-before-make sequence when promoting pages
there is a short period where functions that walk the kernel page table
without locking them may see an invalid entry. One solution would be to add
locking to these functions, however some may be called from locations where
we are unable to sleep.

Until a better solution can be found stop promoting pages in the kernel
pmap so these functions work as expected.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-30 16:45:15 +00:00
andrew
2a1daa1aaa Print both the kernel read and write translation in DDB when asking for
a virtual to physical translation. These may be different, e.g. when a
page is mapped as read-only.

MFC after:	1 month
Sponsored by:	ABT Systems Ltd
2016-08-27 10:30:20 +00:00
andrew
dad1fd7735 Fix an assert, it should check if, when moving from 1 l1 to 512 l2 blocks,
the l2 entry is a block type and not an l3 page.

While here fix the string to correct the level name and add a missing ')'.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-25 14:42:29 +00:00
jmcneill
71e464bcf5 Add support for Allwinner A64.
Reviewed by:	andrew, manu
Relnotes:	yes
2016-08-25 13:59:19 +00:00
andrew
ee05666725 Map coherent memory in a non-coherent dma tag as uncached. This is similar
to what the 32-bit arm code does, with the exception that it always assumes
the tag is non-coherent.

Tested by:	jmcneill
Obtained from:	ABT Systems Ltd
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2016-08-25 10:53:03 +00:00
andrew
34d23267eb Allow superpages on arm64 to be enabled. It is disabled for now to allow
for more testing across more hardware.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-24 13:00:50 +00:00
andrew
8f1e7a3f6c Add support to promote and demote managed superpages. This is the last part
needed before enabling superpages on arm64. This code is based on the amd64
pmap with changes as needed to handle the differences between the two
architectures.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-24 12:57:40 +00:00
andrew
83c9cebd9a Teach the parts of the arm64 pmap that need to iterate over pages to also
iterate over superpages. We don't yet create these, but soon will.

Obtained from:	ABT Systems Ltd
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-08-24 12:32:18 +00:00
ed
c0aa6fd209 Convert pointers obtained from the threadattr_t structure with TO_PTR().
In all of these source files, the userspace pointer size corresponds
with the kernelspace pointer size, meaning that casting directly works.
As I'm planning on making 32-bit execution on 64-bit systems work as
well, use TO_PTR() here as well, so that the changes between source
files remain minimal.
2016-08-24 10:13:18 +00:00