The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.
Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix
In iommu_gas.c, domain->start_gap points to one of the nodes on either
side of the first free, unallocated range. In iommu_gas_init_domain,
it is initialized to point to the node after the single free
range. Change it to point to the node before that free range, so that,
when 'lowaddr' is within the initial free range, the first allocation
search for free space below 'lowaddr' does not begin and end at an
address above 'lowaddr'. This fixes problems on a machine with Intel
DMAR enabled.
Reported by: jah
Reviewed by: dougm
Tested by: jah
Obtained from: jah
Fixes: commit db151ca0c3 iommu_gas: start space search from 1st free space
MFC after: 1 day
Change the range test in iommu_gas_match_one from '< ubound' to '<=
ubound', and pass a smaller-by-one ubound parameter to it, to avoid
overflow in ubound calculation.
Reported by: andrew
Reviewed by: andrew (previous version)
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D37764
Assertions suggest that the loop in iommu_gas_fini_domain is executed
zero times, so remove it.
Reviewed by: alc, kib
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D37204
Maintain a pointer to an element in the domain map that is left of any
sufficiently large free gap in the tree and start the search for free
space there, rather than at the root of the tree. On find_space, move
that pointer to the leftmost leaf in the subtree of nodes with
free_down greater than or equal to the minimum allocation size before
starting the search for space from that pointer. On removal of a node
with address less than that pointer, update that pointer to point to
the predecessor or successor of the removed node.
In experiments with netperf streaming, this reduces by about 40% the
number of map entries examined in first-fit allocation.
Reviewed by: alc, kib
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D36624
When the node to insert in the rb_tree is known to precede or follow a
particular node, new methods RB_INSERT_PREV and RB_INSERT_NEXT,
defined here, allow the search for where to insert the new node begin
with that particular node, rather than at the root, to save a bit of
time.
Using those methods, instead of RB_INSERT, in managing a tree in
iommu_gas.c, saves a little time.
Reviewed by: kib
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D35516
RB-tree augmentation maintains data in each node of the tree that
represents the product of some associative operator applied to all the
nodes of the subtree rooted at that node. If a node in the tree
changes, augmentation data for the node is updated for that node and
all nodes on the path from that node to the tree root. However,
sometimes, augmenting a node changes no data in that node,
particularly if the associated operation is something involving 'max'
or 'min'. If augmentation changes nothing in a node, then the work of
walking to the tree root from that point is pointless, because
augmentation will change nothing in those nodes either. This change
makes it possible to avoid that wasted work.
Define RB_AUGMENT_CHECK as a macro much like RB_AUGMENT, but which
returns a value 'true' when augmentation changes the augmentation data
of a node, and false otherwise. Change code that unconditionally walks
and augments to the top of tree to code that stops once an
augmentation has no effect. In the case of rebalancing the tree after
insertion or deletion, where previously a node rotated into the path
was inevitably augmented on the march to the tree root, now check to
see if it needs augmentation because the march to the tree root
stopped before reaching it.
Change the augmentation function in iommu_gas.c so that it returns
true/false to indicate whether the augmentation had any effect.
Reviewed by: alc, kib
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D36509
Avoid removing an item in iommu_gas_free_region only to reinsert it,
by avoiding removing an entry that is either first_place or
last_place.
Reviewed by: kib
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D36597
The function removes the range of addresses from GAS. Right now it is
unused.
The initial version of the patch was extracted from the stalled work for
integration of DMAR and bhyve, see D25672. Having the function in the
main tree would allow it to co-evolve with other active changes to the
IOMMU driver.
Requested by: alc
Reviewed by: alc, dougm
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D36001
Since IOMMU map entries store a reference to the domain in which they
reside, there is no need to pass the domain to iommu_gas_free_entry(),
iommu_gas_free_space(), and iommu_gas_free_region().
Push down the acquisition and release of the IOMMU domain lock into
iommu_gas_free_space() and iommu_gas_free_region().
Both of these changes allow for simplifications in the callers of the
functions without really complicating the functions themselves.
Moreover, the latter change eliminates the direct use of the IOMMU
domain lock from the x86-specific DMAR code.
Reviewed by: kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D35995
Replace the DMAR unit's tlb_flush TAILQ by a custom list implementation
that enables dmar_qi_task() to dequeue entries without holding the DMAR
lock.
Reviewed by: kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D35951
Eliminate a possible case of use-after-free in an error handling path
after a mapping failure. Specifically, eliminate IOMMU_MAP_ENTRY_QI_NF
and instead perform the IOTLB invalidation synchronously. Otherwise,
when iommu_domain_unload_entry() is called and told not to free the
IOMMU map entry, the caller could free the entry before dmar_qi_task()
is finished with it.
Reviewed by: kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D35878
In the extremely unlikely case that the iommu_gas_map_region() call in
bus_dma_iommu_load_ident() failed, we would attempt to unmap the failed
entry twice, first in iommu_gas_map_region(), and a second time in the
caller. Once is enough, and twice is problematic because it leads to a
second RB_REMOVE call on the same tree node. Like it or not, RB_TREE
does not handle that possibility.
Reviewed by: kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D35869
Eliminate the unroll_entry field from struct iommu_map_entry, shrinking
the struct by 16 bytes on 64-bit architectures.
Reviewed by: kib
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D35769
Merge lowermatch and uppermatch into find_space. Eliminate uppermatch
recursion. Merge match_insert into match_one and eliminate some
redundant calculation. Move some initialization out of find_space and
into map (and out from under a lock).
Reviewed by: kib (previous version), alc
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D35440
As of 19bb5a7244, the IOMMU_MF_CANSPLIT case in iommu_gas_match_one()
must take into account the specified offset. Otherwise, the recently
changed end calculation in iommu_gas_match_insert() could produce an
end address that crosses the specified boundary by one page.
Reviewed by: dougm
MFC with: 19bb5a7244
Introduce fine-grained locking on the dmamap's list of map entries,
replacing the use of the domain lock. This is not the most significant
source of lock contention, but it is the easiest to address.
Reviewed by: kib
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D35557
The loop iteration in iommu_gas_lowermatch checks the bound
a->common->lowaddr twice per loop iteration. Rewrite to test only once
per iteration. Do not worry about passing to iommu_gas_match_one a
range wholly beyond lowaddr. Since that function checks the upper end
of the range against lowaddr, it'll get rejected there.
Reviewed by: alc
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D35522
If iommu_gas_find_space() ever called iommu_gas_uppermatch(), and it
succeeded in allocating space, then the subsequent KASSERT would be
triggered. Change that KASSERT to accept either success or ENOMEM.
MFC after: 1 week
Change the recursive implementation to one that uses parent pointers
to walk back up the rb-tree, to slightly improve performance.
Reviewed by: alc, kib
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D35486
Reverse the order of the search for a free space in lowermatch, to
make it a first-fit search. Iommu_gas_match_one always allocates from
the beginning of the free gap discovered from searching the tree, so
the current code isn't really allocating in a reverse first-fit
anyway, and making the search first-fit reduces the number of iommu
page table pages that are used.
Reported by: alc
Reviewed by: alc, kib
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D35458
Let the caller to iommu_map pass the size parameter without rounding
it up to a multiple of page size. Let iommu_map round it up when
necessary, which is not all of the time, so that in some cases less
space is reserved.
Reviewed by: alc, kib (previous version)
Tested by: pho, br
Discussed with: andrew
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D35424
iommu_gas_lowermatch looks right, then left, then right again in its
search for free space. Change to a more straightforward last-fit
search that touches fewer tree nodes and improves performance.
Reported by: wxzhu@rice.edu
Reviewed by: alc, kib
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D35439
In iommu_gas_lowermatch and iommu_gas_uppermatch, a subtree search is
quickly terminated if the largest available free space in the subtree
is below a limit, where that limit is related to the size of the
allocation request. However, that limit is too small; it does not
account for both of the guard pages that will surround the allocated
space, but only for one of them. Consequently, it permits the search
to proceed through nodes that cannot produce a successful allocation
for all the requested space. Fix that limit to improve search
performance.
Reviewed by: alc, kib
Submitted by: Weixi Zhu (wxzhu@rice.edu)
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D35414
iommu_bus_dmamap_load_something1 includes code for handling the
possibility of splitting a buffer that is needlessly complex.
Simplify it.
Reviewed by: alc, kib
MFC after: 3 weeks
Tested by: pho (previous revisions)
Differential Revision: https://reviews.freebsd.org/D35232
If iommu_gas_match_one has to adjust for a boundary crossing, its
check against maxaddr includes 'offset' in its calculation, to ensure
that the allocated memory does not exceed the max address. However, if
there's no boundary crossing adjustment, then the maxaddr check
disregards 'offset'. Fix that.
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D34978
Define simple functions for alignment and boundary checks and use them
everywhere instead of having slightly different implementations
scattered about. Define them in vm_extern.h and use them where
possible where vm_extern.h is included.
Reviewed by: kib, markj
Differential Revision: https://reviews.freebsd.org/D33685
For some of them, used only when KTR or KMSAN are configured, apply
__unused attribute directly.
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Sanitizer instrumentation of course cannot automatically update shadow
state when devices write to host memory. KMSAN thus hooks into busdma,
both to update shadow state after a device write, and to verify that the
kernel does not publish uninitalized bytes to devices.
To implement this, when KMSAN is configured, each dmamap embeds a memory
descriptor describing the region currently loaded into the map.
bus_dmamap_sync() uses the operation flags to determine whether to
validate the loaded region or to mark it as initialized in the shadow
map.
Note that in cases where the amount of data written is less than the
buffer size, the entire buffer is marked initialized even when it is
not. For example, if a NIC writes a 128B packet into a 2KB buffer, the
entire buffer will be marked initialized, but subsequent accesses past
the first 128 bytes are likely caused by bugs.
Reviewed by: kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31338
These ones were unambiguous cases where the Foundation was the only
listed copyright holder (in the associated license block).
Sponsored by: The FreeBSD Foundation
PCI memory address space is shared between memory-mapped devices (MMIO)
and host memory (which may be remapped by an IOMMU). Device accesses to
an address within a memory aperture in a PCIe root port will be treated
as peer-to-peer and not forwarded to an IOMMU. To avoid this, reserve
the address space of the root port's memory apertures in the address
space used by the IOMMU for remapping.
Reviewed by: kib, tychon
Discussed with: Anton Rang <rang@acm.org>
Tested by: tychon
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D27503
The entry->flags field is initialized in iommu_gas_init_domain().
Reviewed by: kib
Sponsored by: Innovate DSbD
Differential Revision: https://reviews.freebsd.org/D27235
This is needed on arm64 for the interface between iommu framework
and iommu controller drivers.
Reviewed by: kib
Sponsored by: Innovate DSbD
Differential Revision: https://reviews.freebsd.org/D27229
o Add iommu_unmap_msi() to release the msi GAS entry.
o Provide default implementations for iommu init/deinit methods.
Reviewed by: kib
Sponsored by: Innovate DSbD
Differential Revision: https://reviews.freebsd.org/D26906
This allows the interrupt controller driver only need a small change to
create a map for the page the device will write to raise an interrupt.
Submitted by: andrew
Reviewed by: kib
Sponsored by: Innovate DSbD
Differential Revision: https://reviews.freebsd.org/D26705
