On arm64, testing pc_curpcb != NULL is not correct since pc_curpcb is
set in pmap_switch() while the bootstrap stack is still in use. As a
result, smp_after_idle_runnable() can free the boot stack prematurely.
Take a different approach: use smp_rendezvous() to wait for all APs to
acknowledge an interrupt. Since APs must not enable interrupts until
they've entered the scheduler, i.e., switched off the boot stack, this
provides the right guarantee without depending as much on the
implementation of cpu_throw(). And, this approach applies to all
platforms, so convert x86 and riscv as well.
Reported by: mmel
Tested by: mmel
Reviewed by: kib
Fixes: 8db2e8fd16c4 ("Remove the secondary_stacks array in arm64 and riscv kernels.")
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35435
Replace sigframe sf_extramask by native sigset_t and use it to
store/restore the thread signal mask without conversion to/from
Linux signal mask.
Pointy hat to: dchagin
MFC after: 2 weeks
When some APICs are disabled by tunables, some cpu groups could end up
empty. An empty cpu group causes the system to panic because not all
functions handle them correctly. Additionally, it's wasted time to
handle and inspect empty cpu groups. Therefore, just don't create them.
Reviewed by: kib, avg, cem
Sponsored by: Beckhoff Automation GmbH & Co. KG
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D24927
The signal trampoine-related definitions are used only in the MD part
of code, wherefore moved from everywhere used linux.h to separate MD
headers.
MFC after: 2 weeks
Deduplicate code to iterate over the bpages list in a bus_dmamap_t
freeing bounce pages during bus_dmamap_unload.
Reviewed by: imp
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D34967
While here, use a modern function declaration for smbios_modevent and
vpd_modevent.
Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D34996
Previously it was disabled right before translation was enabled.
This way the disable logic is still executed even when translation
is not be activated, e.g. with hw.iommu.dma=0 tunable set.
On some platforms we need to disable PMR in order for core dump to work.
At the same time it was observed that enabling translation has
a significant impact on network performance.
With this patch PMR can be disabled, with IOMMU translation not being
turned on by appending the following to the loader.conf:
hw.dmar.enable=1
hw.dmar.pmr.disable=1
hw.dmar.dma=0
Sponsored by: Stormshield
Obtained from: Semihalf
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D34907
These files no longer depend on the macros required when these checks
were added.
PR: 263102 (exp-run)
Reviewed by: brooks, imp, emaste
Differential Revision: https://reviews.freebsd.org/D34804
All supported compilers (modern versions of GCC and clang) support
this.
Many places didn't have an #else so would just silently do the wrong
thing. Ancient versions of icc (the original motivation for this) are
no longer a compiler FreeBSD supports.
PR: 263102 (exp-run)
Reviewed by: brooks, imp
Differential Revision: https://reviews.freebsd.org/D34797
PR: 263124
Fixes: 62d09b46ad75 ("x86: Defer LAPIC calibration until after timecounters are available")
Reviewed by: kib, jhb, emaste
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34830
This register set contains the values of the fsbase and gsbase
registers. Note that these registers can already be controlled
individually via ptrace(2) via MD operations, so the main reason for
adding this is to include these register values in core dumps. In
particular, this will enable looking up the value of TLS variables
from core dumps in gdb.
The value of NT_X86_SEGBASES was chosen to match the value of
NT_386_TLS on Linux. The notes serve similar purposes, but FreeBSD
will never dump a note equivalent to NT_386_TLS (which dumps a single
segment descriptor rather than a pair of addresses) and picking a
currently-unused value in the NT_X86_* range could result in a future
conflict.
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D34650
Introduce a helper to fetch the TSC frequency from CPUID when running
under Xen.
Since the TSC can also be initialized early when running as a Xen
guest pull out the call to tsc_init() from the
early_clock_source_init() handlers and place it in clock_init(), as
otherwise all handlers would call tsc_init() anyway.
Reviewed by: markj
Sponsored by: Citrix Systems R&D
Differential revision: https://reviews.freebsd.org/D34581
If we can't determine the TSC frequency using CPU registers, we need to
give a chance for Hyper-V drivers to register a timecounter (during
SI_SUB_HYPERVISOR) since an emulated 8254 might not be available.
Thus, split probe_tsc_freq() into early and late stages, and wait until
the latter to attempt calibration using a reference clock.
Fixes: 84369dd52369 ("x86: Probe the TSC frequency earlier")
Reported and tested by: khng, Shawn Webb
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34444
This lets us use the TSC to implement early DELAY, limiting the use of
the sometimes-unreliable 8254 PIT.
PR: 262155
Reviewed by: emaste
Tested by: emaste, mike tancsa <mike@sentex.net>, Stefan Hegnauer <stefan.hegnauer@gmx.ch>
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34367
These headers originate with the Xen project and shouldn't be mixed with
the main portion of the FreeBSD kernel. Notably they shouldn't be the
target of clean-up commits.
Switch to use the headers in sys/contrib/xen.
Reviewed by: royger
SystemCMOS address space is accessible for system wide.
So install address handler in \_SB space.
Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D33892
All supported Xen instances by FreeBSD provide a local APIC
implementation, so there's no need to replace the native local APIC
implementation anymore.
Leave just the ipi_vectored hook in order to be able to override it
with an implementation based on event channels if the underlying local
APIC is not virtualized by hardware. Note the hook cannot use ifuncs,
because at the point where ifuncs are resolved the kernel doesn't yet
know whether it will benefit from using the optimization.
Sponsored by: Citrix Systems R&D
Reviewed by: kib
Differential revision: https://reviews.freebsd.org/D33917
Instead of using event channels or hypercalls to deal with IPIs and
NMIs.
Using a hardware virtualized APIC should be faster than using any PV
interface, since the VM exit can be avoided.
Xen exposes whether the domain is using hardware assisted x{2}APIC
emulation in a CPUID bit.
Sponsored by: Citrix Systems R&D
It has been reported that on some AWS instances VCPUOP_send_nmi
returns -38 (ENOSYS). The hypercall is only available for HVM guests
in Xen 4.7 and newer. Add a fallback to use the native NMI sending
procedure when VCPUOP_send_nmi is not available, so that the NMI is
not lost.
Reported and Tested by: avg
MFC after: 1 week
Fixes: b2802351c162 ('xen: fix dispatching of NMIs')
Sponsored by: Citrix Systems R&D
Some VM systems announce the frequency of the local APIC via the
CPUID leaf 0x40000010. Using this allows us to boot slightly
faster by avoiding the need for timer calibration.
Reviewed by: markj
Sponsored by: https://www.patreon.com/cperciva
While this CPUID leaf was originally only used by VMWare, other
hypervisors now also use it to announce the TSC frequency to guests.
This speeds up the boot process by 100 ms in EC2 and other systems,
by allowing the early calibration DELAY to be skipped.
Reviewed by: markj
Sponsored by: https://www.patreon.com/cperciva
This allows us to set tsc_is_invariant and select appropriately
fenced versions of RDTSC based on the CPU type.
Reviewed by: markj
Sponsored by: https://www.patreon.com/cperciva
The ACPI spec describes the FADT->Century field as:
The RTC CMOS RAM index to the century of data value (hundred and
thousand year decimals). If this field contains a zero, then the
RTC centenary feature is not supported. If this field has a non-zero
value, then this field contains an index into RTC RAM space that
OSPM can use to program the centenary field.
Use this field to decide whether to program the CENTURY register
of the CMOS RTC device.
Reviewed by: akumar3@isilon.com, dab, vangyzen
MFC after: 1 week
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D33667
MFC after: 1 week
Sponsored by: Dell EMC Isilon
The old bogus Xen versions that would deliver a GPF when writing to
the LAPIC MSR are likely retired, so it's safe to enable x2APIC
unconditionally now if available.
Tested by: avg
Reviewed by: kib
Sponsored by: Citrix Systems R&D
Differential revision: https://reviews.freebsd.org/D33877
When running as a Xen guest it's easier to use an hypercall in order
to do power management operations (power off, power cycle). Do this
for all supported guest types (HVM and PVH). Note that for HVM the
power operation could also be done using ACPI, but there's no reason
to differentiate between PVH and HVM.
While there fix the shutdown handler to properly differentiate between
power cycle and power off requests.
Reported by: Freddy DISSAUX
MFC: 1 week
Sponsored by: Citrix Systems R&D
Prior to this commit, the TSC and local APIC frequencies were calibrated
at boot time by measuring the clocks before and after a one-second sleep.
This was simple and effective, but had the disadvantage of *requiring a
one-second sleep*.
Rather than making two clock measurements (before and after sleeping) we
now perform many measurements; and rather than simply subtracting the
starting count from the ending count, we calculate a best-fit regression
between the target clock and the reference clock (for which the current
best available timecounter is used). While we do this, we keep track
of an estimate of the uncertainty in the regression slope (aka. the ratio
of clock speeds), and stop measuring when we believe the uncertainty is
less than 1 PPM.
In order to avoid the risk of aliasing resulting from the data-gathering
loop synchronizing with (a multiple of) the frequency of the reference
clock, we add some additional spinning depending upon the iteration number.
For numerical stability and simplicity of implementation, we make use of
floating-point arithmetic for the statistical calculations.
On the author's Dell laptop, this reduces the time spent in calibration
from 2000 ms to 29 ms; on an EC2 c5.xlarge instance, it is reduced from
2000 ms to 2.5 ms.
Reviewed by: bde (previous version), kib
MFC after: 1 month
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D33802
- Move busdma_lock_mutex to subr_bus_dma.c.
- Move _busdma_lock_dflt to subr_bus_dma.c. This function was named a
couple of different things previously. It is not a public API but
an internal helper used in place of a NULL pointer. The prototype
is in <sys/bus_dma.h> as not all backends include
<sys/bus_dma_internal.h>.
Reviewed by: kib
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D33694
Move mostly duplicated code in various MD bus_dma backends to support
bounce pages into sys/kern/subr_busdma_bounce.c. This file is
currently #include'd into the backends rather than compiled standalone
since it requires access to internal members of opaque bus_dma
structures such as bus_dmamap_t and bus_dma_tag_t.
Reviewed by: kib
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D33684
Some AMD Geode-based systems end up using the 8254 PIT to calibrate the
TSC during late calibration, which doesn't work because that
timecounter's mask (65535) is much smaller than its frequency (1193182).
Moreover, early calibration is done against the 8254 timer anyway.
Work around the problem by simply using early calibration results if no
high-quality timecounters exist.
PR: 260868
Fixes: 22875f88799e ("x86: Implement deferred TSC calibration")
Reported and tested by: mike@sentex.net, Stefan Hegnauer <stefan.hegnauer@gmx.ch>
Reviewed by: imp, kib
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33730
An earlier version of this code computed the TSC frequency in kHz.
When the code was changed to compute the frequency more accurately,
the variable name was not updated.
Reviewed by: markj
Fixes: 22875f88799e x86: Implement deferred TSC calibration
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D33696
It's possible that the "early" TSC calibration gave us a value which
is known to be exact; in that case, skip the later re-calibration.
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D33695
A recent change introduced a one-off error into a test allowing
coalescing chunks into segments. This fixes that error.
broke a check in _bus_dmamap_addseg on many architectures. This change makes it clear that it is not a particular range that is being boundary-checked, but the proposed union of the two adjacent ranges.
Reported by: se
Reviewed by: se
Fixes: c606ab59e7f9 vm_extern: use standard address checkers everywhere
Differential Revision: https://reviews.freebsd.org/D33715
Define simple functions for alignment and boundary checks and use them
everywhere instead of having slightly different implementations
scattered about. Define them in vm_extern.h and use them where
possible where vm_extern.h is included.
Reviewed by: kib, markj
Differential Revision: https://reviews.freebsd.org/D33685
The introduction of <sched.h> improved compatibility with some 3rd
party software, but caused the configure scripts of some ports to
assume that they were run in a GLIBC compatible environment.
Parts of sched.h were made conditional on -D_WITH_CPU_SET_T being
added to ports, but there still were compatibility issues due to
invalid assumptions made in autoconfigure scripts.
The differences between the FreeBSD version of macros like CPU_AND,
CPU_OR, etc. and the GLIBC versions was in the number of arguments:
FreeBSD used a 2-address scheme (one source argument is also used as
the destination of the operation), while GLIBC uses a 3-adderess
scheme (2 source operands and a separately passed destination).
The GLIBC scheme provides a super-set of the functionality of the
FreeBSD macros, since it does not prevent passing the same variable
as source and destination arguments. In code that wanted to preserve
both source arguments, the FreeBSD macros required a temporary copy of
one of the source arguments.
This patch set allows to unconditionally provide functions and macros
expected by 3rd party software written for GLIBC based systems, but
breaks builds of externally maintained sources that use any of the
following macros: CPU_AND, CPU_ANDNOT, CPU_OR, CPU_XOR.
One contributed driver (contrib/ofed/libmlx5) has been patched to
support both the old and the new CPU_OR signatures. If this commit
is merged to -STABLE, the version test will have to be extended to
cover more ranges.
Ports that have added -D_WITH_CPU_SET_T to build on -CURRENT do
no longer require that option.
The FreeBSD version has been bumped to 1400046 to reflect this
incompatible change.
Reviewed by: kib
MFC after: 2 weeks
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D33451
