The xen_domain_type and HYPERVISOR_shared_info variables are shared by
all Xen architectures, so they should be in common rather than
reimplemented by each architecture.
hvm_start_flags is used by xen_initial_domain() and so needs to be in
common.
Reviewed by: royger
Differential Revision: https://reviews.freebsd.org/D28982
The event channel source code or equivalent is needed on all
architectures. Since much of this is viable to share, get this moved out
of x86-land. Each interrupt interface then needs a distinct back-end
implementation.
Reviewed by: royger
Submitted by: Elliott Mitchell <ehem+freebsd@m5p.com>
Original implementation: Julien Grall <julien@xen.org>, 2014-01-13 17:41:04
Differential Revision: https://reviews.freebsd.org/D30236
Simply moving the interrupt allocation and release functions into files
which belong to the architecture. Since x86 interrupt handling is quite
distinct from other architectures, this is a crucial necessary step.
Identifying the border between x86 and architecture-independent is
actually quite tricky. Similarly, getting the prototypes for the
border right is also quite tricky.
Inspired by the work of Julien Grall <julien@xen.org>,
2015-10-20 09:14:56, but heavily adjusted.
Reviewed by: royger
Differential Revision: https://reviews.freebsd.org/D30936
The x86 PIC interface is very much x86-specific and not used by other
architectures. Since most of xen_intr.c can be shared with other
architectures, the PIC interface needs to be broken off.
Introduce wrappers for calls into the architecture-dependent interrupt
layer. All architectures need roughly the same functionality, but the
interface is slightly different between architectures. Due to the
wrappers being so thin, all of them are implemented as inline in
arch-intr.h.
The original implementation was done by Julien Grall in 2015, but this
has required major updating.
Removal of PVHv1 meant substantial portions disappeared. The original
implementation took care of moving interrupt allocation to
xen_arch_intr.c, but this has required massive rework and was broken
off.
In the original implementation the wrappers were normal functions. Some
had empty stubs in xen_intr.c and were removed.
Reviewed by: royger
Submitted by: Elliott Mitchell <ehem+freebsd@m5p.com>
Original implementation: Julien Grall <julien@xen.org>, 2015-10-20 09:14:56
Differential Revision: https://reviews.freebsd.org/D30909
This value doesn't need to be set in xen_intr_alloc_isrc(). What is
needed is simply to ensure the allocated xenisrc won't appear as free,
even if xi_type is written non-atomically. Since the type is no longer
used to indicate free or not, the calling function should take care of
all non-architecture initialization.
Reviewed by: royger
Differential Revision: https://reviews.freebsd.org/D31188
Scanning the list of interrupts to find an unused entry is rather
inefficient. Instead overlay a free list structure and use a list
instead.
This also has the useful effect of removing the last use of evtchn_type
values outside of xen_intr.c.
Reviewed by: royger
[royger]
- Make avail_list static.
Move the xenisrc structure which needs to be shared between the core Xen
interrupt code and architecture-dependent code into a separate header. A
similar situation exists for the NR_EVENT_CHANNELS constant.
Turn xi_intsrc into a type definition named xi_arch to reflect the new
purpose of being an architectural variable for the interrupt source.
This was originally implemented by Julien Grall, but has been heavily
modified. The core side was renamed "intr-internal.h" and is #include'd
by "arch-intr.h" instead of the other way around. This allows the
architecture to add function definitions which use struct xenisrc.
The original version only moved xi_intsrc into xen_arch_isrc_t. Moving
xi_vector was done by the submitter.
The submitter had also moved xi_activehi and xi_edgetrigger into
xen_arch_isrc_t. Those disappeared with the removal of PVHv1 support.
Copyright note. The current xenisrc structure was introduced at
76acc41fb7 by Justin T. Gibbs. Traces remain, but the strength of
Copyright claims from before 2013 seem pretty weak.
Reviewed by: royger
Submitted by: Elliott Mitchell <ehem+freebsd@m5p.com>, 2021-03-17 19:09:01
Original implementation: Julien Grall <julien@xen.org>, 2015-10-20 09:14:56
Differential Revision: https://reviews.freebsd.org/D30648
[royger]
- Adjust some line lengths
- Fix comment about NR_EVENT_CHANNELS after movement.
- Use #include instead of symlinks.
xen_intr_handle_upcall() has two interfaces. It needs to be called by
the x86 assembly code invoked by the APIC. Second, it needs to be called
as a driver_filter_t for the XenPCI code and for architectures besides
x86.
Unfortunately the driver_filter_t interface was implemented as a wrapper
around the x86-APIC interface. Now create a simple wrapper for the
x86-APIC code, which calls an architecture-independent
xen_intr_handle_upcall().
When called via intr_event_handle(), driver_filter_t functions expect
preemption to be disabled. This removes the need for
critical_enter()/critical_exit() when called this way.
The lapic_eoi() call is only needed on x86 in some cases when invoked
directly as an APIC vector handler.
Additionally driver_filter_t functions have no need to handle interrupt
counters. The intrcnt_add() calling function was reworked to match the
current situation. intrcnt_add() is now only called via one path.
The increment/decrement of curthread->td_intr_nesting_level had
previously been left out. Appears this was mostly harmless, but this
was noticed during implementation and has been added.
CONFIG_X86 is a leftover from use with Linux. While the barrier isn't
needed for FreeBSD on x86, it will be needed for FreeBSD on other
architectures.
Copyright note. xen_intr_intrcnt_add() was introduced at 76acc41fb7
by Justin T. Gibbs. xen_intrcnt_init() was introduced at fd036deac1
by John Baldwin.
sys/x86/xen/xen_arch_intr.c was originally created by Julien Grall in
2015 for the purpose of holding the x86 interrupt interface. Later it
was found xen_intr_handle_upcall() was better earlier, and the x86
interrupt interface better later. As such the filename and header list
belong to Julien Grall, but what those were created for is later.
Reviewed by: royger
Differential Revision: https://reviews.freebsd.org/D30006
Keeping released xenisrcs in a known state simplifies allocation, but
forces the allocation function to maintain that state. This turns into
a problem when trying to allow for interchangeable allocation functions.
Fix this issue by ensuring xenisrcs are always *fully* initialized
during binding.
Reviewed by: royger
There are actually several distinct locking domains in xen_intr.c, but
all were sharing the same lock. Both xen_intr_port_to_isrc[] and the
x86 interrupt structures needed protection. Split these two apart as a
precursor to splitting the architecture portions off the file.
Reviewed by: royger
Differential Revision: https://reviews.freebsd.org/D30726
Locking for allocation was being done in xen_intr_bind_isrc(), but the
unlock was inside xen_intr_alloc_isrc(). While the lock acquisition at
the end of xen_intr_alloc_isrc() was to modify xen_intr_port_to_isrc[],
NOT allocation. Fix this garbled (though working) locking scheme.
Now locking for allocation is strictly in xen_intr_alloc_isrc(), while
locking to modify xen_intr_port_to_isrc[] is in xen_intr_bind_isrc().
Reviewed by: royger
Differential Revision: https://reviews.freebsd.org/D30726
The call structure around xen_intr_alloc_isrc() was rather awful.
Notably finding a structure for reuse is part of allocation, but this
was done outside xen_intr_alloc_isrc(). Move this into
xen_intr_alloc_isrc() so the function handles all allocation steps.
Reviewed by: royger
Differential Revision: https://reviews.freebsd.org/D30726
As "CPUs", IRQs (vector) and virtual IRQs are always positive integers,
adjust the Xen code to use unsigned integers. Several format strings
need adjustment to match. Additionally single-bit bitfields are
boolean.
No functional change expected.
Reviewed by: royger
Part of the series for allowing FreeBSD/ARM to run on Xen. On ARM the
function is a trivial pass-through, other architectures need distinct
implementations.
While implementing XEN_VCPUID() as a call to XEN_CPUID_TO_VCPUID()
works, that involves multiple accesses to the PCPU region. As such make
this a distinct macro. Only callers in machine independent code have
been switched.
Add a wrapper for the x86 PIC interface to use matching the old
prototype.
Partially inspired by the work of Julien Grall <julien@xen.org>,
2015-08-01 09:45:06, but XEN_VCPUID() was redone by Elliott Mitchell on
2022-06-13 12:51:57.
Reviewed by: royger
Submitted by: Elliott Mitchell <ehem+freebsd@m5p.com>
Original implementation: Julien Grall <julien@xen.org>, 2014-04-19 08:57:40
Original implementation: Julien Grall <julien@xen.org>, 2014-04-19 14:32:01
Differential Revision: https://reviews.freebsd.org/D29404
Previously the upper layer handle was being set before the last
potential error condition. The reasoning appears to have been it was
assumed invalid in case of an error being returned. Now ensure it is
invalid until just before a successful return.
Fixes: 76acc41fb7 ("Implement vector callback for PVHVM and unify event channel implementations")
Fixes: 6d54cab1fe ("xen: allow to register event channels without handlers")
Reviewed by: royger
The prior implementation of xen_intr_resume() was wiping
xen_intr_port_to_isrc[] and then rebuilding from the x86 interrupt
table. Rework to instead wipe the channel numbers (->xi_port) and then
scan the table for sources with invalid channels.
This will be slower due to scanning the whole table, but this removes
the dependency on the x86 interrupt code.
Reviewed by: royger
Differential Revision: https://reviews.freebsd.org/D30599
[royger]
Split line over 80 characters.
The portions of xen_rebind_ipi() and xen_rebind_virq() were already
near-identical. While xen_rebind_ipi() should panic() on
single-processor, still having the functionality to invoke seems
harmless.
Meanwhile much of the loop from xen_intr_resume() seemed to want to be
closer to this same code. This pushes related bits closer together.
Reviewed by: royger
Differential Revision: https://reviews.freebsd.org/D30598
Remove these no longer needed headers. Key for making xen_intr.c
machine-independent as they don't exist on other architectures.
Originally this was part of a much larger commit, but was broken off
for submission to the FreeBSD project.
Reviewed by: royger
Submitted by: Elliott Mitchell <ehem+freebsd@m5p.com>
Original implementation: Julien Grall <julien@xen.org>, 2015-10-20 09:14:56
MFC after: 1 week
Now that the atomic macros are always genuinely atomic on x86, they can
be used for synchronization with Xen. A single core VM isn't too
unusual, but actual single core hardware is uncommon.
Replace an open-coding of evtchn_clear_port() with the inline.
Substantially inspired by work done by Julien Grall <julien@xen.org>,
2014-01-13 17:40:58.
Reviewed by: royger
MFC after: 1 week
While unusual, intr_register_source() can return failure. A likely
cause might be another device grabbing from Xen's interrupt range.
This should NOT happen, but could happen due to a bug. As such check
for this and fail if it occurs.
This theoretical situation also effects xen_intr_find_unused_isrc().
There, .is_pic must be tested to ensure such an intrusion doesn't cause
misbehavior.
Reviewed by: royger
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D31995
Consistently use ~0 instead of 0 when clearing xenisrc structures.
0 is a valid event channel number, even though it is reserved by Xen.
Whereas ~0 is guaranteed invalid.
Reviewed by: royger
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D30743
In xen_intr_release_isrc(), the isrc should only be removed if it is
assigned to a valid port. This had been mitigated by using 0 for not
having a port, but this is actually corrupting the table. Fix this bug
as modifying the code would cause this bug to manifest as kernel memory
corruption. Similar issue for the vCPU bitmap masks.
The KASSERT() doesn't need lock protection.
Reviewed by: royger
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D30743
The comparison was wrong. Hopefully this never occurred in the wild,
but now ensure the error message will occur before damage is caused.
This appears non-exploitable as exploitation would require a guest to
force Domain 0 to allocate all event channels, which a guest shouldn't
be able to do.
Adjust the error message to better describe what has occurred.
Reviewed by: royger
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D30743
Appears errors are uncommon since calling xen_intr_release_isrc() on a
xenisrc with xi_close in an undefined state could be bad. Fix this
problematic lurking nasty.
Reviewed by: royger
MFC after: 1 week
The xenpf_dom0_console_t structure can grow as more data is added, and
hence we need to check that the fields we accesses have been filled by
Xen. The only extra field FreeBSD currently uses is the top 32 bits
for the frame buffer physical address.
Note that this field is present in all the versions that make the
information available from the platform hypercall interface, so the
check here is mostly cosmetic, and to remember us that newly added
fields require checking the size of the returned data.
Fixes: 6f80738b22 ('xen: fetch dom0 video console information from Xen')
Sponsored by: Citrix Systems R&D
It's possible for Xen to switch the video mode set by the boot loader,
so that the information passed in the kernel metadata is no longer
valid. Fetch the video mode used by Xen using an hypercall and update
the medatada for the kernel to use the correct video mode.
Sponsored by: Citrix Systems R&D
This is required for a further change that will make use of a field
that was added in version 0x00040d00.
No functional change expected.
Sponsored by: Citrix Systems R&D
Now that we can PVH boot on a non-Xen hypervisor, we shouldn't set
machdep.bootmethod to "XEN". Instead, set it to "PVH"; there are
other ways to discern the hypervisor.
Reviewed by: royger
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D36191
For historical reasons, Xen kernel command lines have options
separated by commas. Every other FreeBSD platform uses whitespace;
this is also necessary in PVH in order to support the Firecracker
VMM. Allow options to be separated by any combination of commas
and whitespace.
Reviewed by: imp
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D36190
The PVH boot protocol, introduced by Xen, is now used by some non-Xen
platforms (e.g. the Firecracker VM) as well. In order to accommodate
these, we use CPUID to detect Xen and only perform Xen-specific setup
when running on that platform.
The "isxen" function duplicates some work done by identcpu.c later in
the boot process; but we need it here since this is the very first C
code which runs when PVH booting (even before hammer_time).
In many places the existing code had
xc_printf(...);
HYPERVISOR_shutdown(SHUTDOWN_crash);
making use of Xen functionality to print a message and shut down; in
the places where this idiom can be reached in the non-xen case, we
replace it idiom with a CRASH(...) macro which calls those in the Xen
case and halts in the non-Xen case.
Reviewed by: royger
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D35801
Version 0 of PVH booting uses a Xen hypercall to retrieve the system
memory map; in version 1 the memory map can be provided via the
start_info structure.
Using the memory map from the version 1 start_info structure allows
FreeBSD to use PVH booting on systems other than Xen, e.g. on the
Firecracker VM.
Reviewed by: royger
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D35800
GCC warns about the mismatched sizes on i386 where vm_paddr_t is 64
bits.
Reviewed by: imp, markj
Differential Revision: https://reviews.freebsd.org/D36750
Do not assume that start_info will always be loaded at the highest
memory address, and instead check the position of all the loaded
elements in order to find the last loaded one, and thus a likely safe
place to use as early boot allocation memory space.
Reported by: markj, cperciva
Sponsored by: Citrix Systems R&D
Reviewed by: markj
Differential revision: https://reviews.freebsd.org/D35628
These headers originate with the Xen project and shouldn't be mixed with
the main portion of the FreeBSD kernel. Notably they shouldn't be the
target of clean-up commits.
Switch to use the headers in sys/contrib/xen.
Reviewed by: royger
All supported Xen instances by FreeBSD provide a local APIC
implementation, so there's no need to replace the native local APIC
implementation anymore.
Leave just the ipi_vectored hook in order to be able to override it
with an implementation based on event channels if the underlying local
APIC is not virtualized by hardware. Note the hook cannot use ifuncs,
because at the point where ifuncs are resolved the kernel doesn't yet
know whether it will benefit from using the optimization.
Sponsored by: Citrix Systems R&D
Reviewed by: kib
Differential revision: https://reviews.freebsd.org/D33917
Instead of using event channels or hypercalls to deal with IPIs and
NMIs.
Using a hardware virtualized APIC should be faster than using any PV
interface, since the VM exit can be avoided.
Xen exposes whether the domain is using hardware assisted x{2}APIC
emulation in a CPUID bit.
Sponsored by: Citrix Systems R&D
It has been reported that on some AWS instances VCPUOP_send_nmi
returns -38 (ENOSYS). The hypercall is only available for HVM guests
in Xen 4.7 and newer. Add a fallback to use the native NMI sending
procedure when VCPUOP_send_nmi is not available, so that the NMI is
not lost.
Reported and Tested by: avg
MFC after: 1 week
Fixes: b2802351c1 ('xen: fix dispatching of NMIs')
Sponsored by: Citrix Systems R&D
This ensures that we have a good reference timecounter for performing
calibration.
Change lapic_setup to avoid configuring the timer when booting, and move
calibration and initial configuration to a new lapic routine,
lapic_calibrate_timer. This calibration will be initiated from
cpu_initclocks(), before an eventtimer is selected.
Reviewed by: kib, jhb
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33206
Clang 13 produces the following warning for hammer_time_xen():
sys/x86/xen/pv.c:183:19: error: the pointer incremented by -2147483648 refers past the last possible element for an array in 64-bit address space containing 256-bit (32-byte) elements (max possible 576460752303423488 elements) [-Werror,-Warray-bounds]
(vm_paddr_t)start_info->modlist_paddr + KERNBASE;
^ ~~~~~~~~
sys/xen/interface/arch-x86/hvm/start_info.h:131:5: note: array 'modlist_paddr' declared here
uint64_t modlist_paddr; /* Physical address of an array of */
^
This is because the expression first casts start_info->modlist_paddr to
struct hvm_modlist_entry * (via vmpaddr_t), and *then* adds KERNBASE,
which is then interpreted as KERNBASE * sizeof(struct
hvm_modlist_entry).
Instead, parenthesize the addition to get the intended result, and cast
it to struct hvm_modlist_entry * afterwards. Also remove the cast to
vmpaddr_t since it is not necessary.
Reviewed by: royger
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D31711
xen_vector_callback_enabled is x86 specific and availability of
per-cpu event channel delivery differs on other architectures.
Introduce a new helper to check if there's support for per-cpu event
channel injection.
Submitted by: Elliott Mitchell <ehem+freebsd@m5p.com>
Reviewed by: royger
Differential Revision: https://reviews.freebsd.org/D29402
Several of x86 enable/disable functions depend upon the xen*domain()
functions. As such the xen*domain() functions need to be declared
before machine/xen-os.h.
Officially declare direct inclusion of machine/xen/xen-os.h verboten as
such will break these functions/macros. Remove one such soon to be
broken inclusion.
Reviewed by: royger
Differential Revision: https://reviews.freebsd.org/D29811
Functions tend to get renamed and unless the developer is careful
often debugging messages are missed. As such using func is far
superior. Replace several instances of hard-coded function names.
Reviewed by: royger
Differential revision: https://reviews.freebsd.org/D29499
Since xen_intr_handle_t is meant to be an opaque handle and the only
use is retrieving the associated struct xenisrc *, directly use it as
the opaque handler.
Also add a wrapper function for converting the other direction. If some
other value becomes appropriate in the future, these two functions will
be the only spots needing modification.
Reviewed by: mhorne, royger
Differential Revision: https://reviews.freebsd.org/D29500
Minor changes are necessary to make this processor-independent, but
moving the file out of x86 and into common is the first step (so
others don't add /more/ x86-isms).
Submitted by: Elliott Mitchell <ehem+freebsd@m5p.com>
Reviewed by: royger
Differential Revision: https://reviews.freebsd.org/D29042
The AP startup extern variable declarations are not longer needed,
since PVHv2 uses the native AP startup path using the lapic. Remove
the declaration and make the variables static to mp_machdep.c
Sponsored by: Citrix Systems R&D
