2005-01-07 02:29:27 +00:00
|
|
|
/*-
|
2013-06-13 21:05:38 +00:00
|
|
|
* Copyright (c) 2002-2005, 2009, 2013 Jeffrey Roberson <jeff@FreeBSD.org>
|
2005-07-16 09:51:52 +00:00
|
|
|
* Copyright (c) 2004, 2005 Bosko Milekic <bmilekic@FreeBSD.org>
|
|
|
|
* All rights reserved.
|
2002-03-19 09:11:49 +00:00
|
|
|
*
|
|
|
|
* Redistribution and use in source and binary forms, with or without
|
|
|
|
* modification, are permitted provided that the following conditions
|
|
|
|
* are met:
|
|
|
|
* 1. Redistributions of source code must retain the above copyright
|
|
|
|
* notice unmodified, this list of conditions, and the following
|
|
|
|
* disclaimer.
|
|
|
|
* 2. Redistributions in binary form must reproduce the above copyright
|
|
|
|
* notice, this list of conditions and the following disclaimer in the
|
|
|
|
* documentation and/or other materials provided with the distribution.
|
|
|
|
*
|
|
|
|
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
|
|
|
|
* IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
|
|
|
|
* OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
|
|
|
|
* IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
|
|
|
|
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
|
|
|
|
* NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
|
|
|
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
|
|
|
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
|
|
|
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
|
|
|
|
* THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
|
|
*
|
|
|
|
* $FreeBSD$
|
|
|
|
*
|
|
|
|
*/
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This file includes definitions, structures, prototypes, and inlines that
|
|
|
|
* should not be used outside of the actual implementation of UMA.
|
|
|
|
*/
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Here's a quick description of the relationship between the objects:
|
|
|
|
*
|
Bring in mbuma to replace mballoc.
mbuma is an Mbuf & Cluster allocator built on top of a number of
extensions to the UMA framework, all included herein.
Extensions to UMA worth noting:
- Better layering between slab <-> zone caches; introduce
Keg structure which splits off slab cache away from the
zone structure and allows multiple zones to be stacked
on top of a single Keg (single type of slab cache);
perhaps we should look into defining a subset API on
top of the Keg for special use by malloc(9),
for example.
- UMA_ZONE_REFCNT zones can now be added, and reference
counters automagically allocated for them within the end
of the associated slab structures. uma_find_refcnt()
does a kextract to fetch the slab struct reference from
the underlying page, and lookup the corresponding refcnt.
mbuma things worth noting:
- integrates mbuf & cluster allocations with extended UMA
and provides caches for commonly-allocated items; defines
several zones (two primary, one secondary) and two kegs.
- change up certain code paths that always used to do:
m_get() + m_clget() to instead just use m_getcl() and
try to take advantage of the newly defined secondary
Packet zone.
- netstat(1) and systat(1) quickly hacked up to do basic
stat reporting but additional stats work needs to be
done once some other details within UMA have been taken
care of and it becomes clearer to how stats will work
within the modified framework.
From the user perspective, one implication is that the
NMBCLUSTERS compile-time option is no longer used. The
maximum number of clusters is still capped off according
to maxusers, but it can be made unlimited by setting
the kern.ipc.nmbclusters boot-time tunable to zero.
Work should be done to write an appropriate sysctl
handler allowing dynamic tuning of kern.ipc.nmbclusters
at runtime.
Additional things worth noting/known issues (READ):
- One report of 'ips' (ServeRAID) driver acting really
slow in conjunction with mbuma. Need more data.
Latest report is that ips is equally sucking with
and without mbuma.
- Giant leak in NFS code sometimes occurs, can't
reproduce but currently analyzing; brueffer is
able to reproduce but THIS IS NOT an mbuma-specific
problem and currently occurs even WITHOUT mbuma.
- Issues in network locking: there is at least one
code path in the rip code where one or more locks
are acquired and we end up in m_prepend() with
M_WAITOK, which causes WITNESS to whine from within
UMA. Current temporary solution: force all UMA
allocations to be M_NOWAIT from within UMA for now
to avoid deadlocks unless WITNESS is defined and we
can determine with certainty that we're not holding
any locks when we're M_WAITOK.
- I've seen at least one weird socketbuffer empty-but-
mbuf-still-attached panic. I don't believe this
to be related to mbuma but please keep your eyes
open, turn on debugging, and capture crash dumps.
This change removes more code than it adds.
A paper is available detailing the change and considering
various performance issues, it was presented at BSDCan2004:
http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf
Please read the paper for Future Work and implementation
details, as well as credits.
Testing and Debugging:
rwatson,
brueffer,
Ketrien I. Saihr-Kesenchedra,
...
Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
|
|
|
* Kegs contain lists of slabs which are stored in either the full bin, empty
|
2002-03-19 09:11:49 +00:00
|
|
|
* bin, or partially allocated bin, to reduce fragmentation. They also contain
|
|
|
|
* the user supplied value for size, which is adjusted for alignment purposes
|
Bring in mbuma to replace mballoc.
mbuma is an Mbuf & Cluster allocator built on top of a number of
extensions to the UMA framework, all included herein.
Extensions to UMA worth noting:
- Better layering between slab <-> zone caches; introduce
Keg structure which splits off slab cache away from the
zone structure and allows multiple zones to be stacked
on top of a single Keg (single type of slab cache);
perhaps we should look into defining a subset API on
top of the Keg for special use by malloc(9),
for example.
- UMA_ZONE_REFCNT zones can now be added, and reference
counters automagically allocated for them within the end
of the associated slab structures. uma_find_refcnt()
does a kextract to fetch the slab struct reference from
the underlying page, and lookup the corresponding refcnt.
mbuma things worth noting:
- integrates mbuf & cluster allocations with extended UMA
and provides caches for commonly-allocated items; defines
several zones (two primary, one secondary) and two kegs.
- change up certain code paths that always used to do:
m_get() + m_clget() to instead just use m_getcl() and
try to take advantage of the newly defined secondary
Packet zone.
- netstat(1) and systat(1) quickly hacked up to do basic
stat reporting but additional stats work needs to be
done once some other details within UMA have been taken
care of and it becomes clearer to how stats will work
within the modified framework.
From the user perspective, one implication is that the
NMBCLUSTERS compile-time option is no longer used. The
maximum number of clusters is still capped off according
to maxusers, but it can be made unlimited by setting
the kern.ipc.nmbclusters boot-time tunable to zero.
Work should be done to write an appropriate sysctl
handler allowing dynamic tuning of kern.ipc.nmbclusters
at runtime.
Additional things worth noting/known issues (READ):
- One report of 'ips' (ServeRAID) driver acting really
slow in conjunction with mbuma. Need more data.
Latest report is that ips is equally sucking with
and without mbuma.
- Giant leak in NFS code sometimes occurs, can't
reproduce but currently analyzing; brueffer is
able to reproduce but THIS IS NOT an mbuma-specific
problem and currently occurs even WITHOUT mbuma.
- Issues in network locking: there is at least one
code path in the rip code where one or more locks
are acquired and we end up in m_prepend() with
M_WAITOK, which causes WITNESS to whine from within
UMA. Current temporary solution: force all UMA
allocations to be M_NOWAIT from within UMA for now
to avoid deadlocks unless WITNESS is defined and we
can determine with certainty that we're not holding
any locks when we're M_WAITOK.
- I've seen at least one weird socketbuffer empty-but-
mbuf-still-attached panic. I don't believe this
to be related to mbuma but please keep your eyes
open, turn on debugging, and capture crash dumps.
This change removes more code than it adds.
A paper is available detailing the change and considering
various performance issues, it was presented at BSDCan2004:
http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf
Please read the paper for Future Work and implementation
details, as well as credits.
Testing and Debugging:
rwatson,
brueffer,
Ketrien I. Saihr-Kesenchedra,
...
Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
|
|
|
* and rsize is the result of that. The Keg also stores information for
|
2002-03-19 09:11:49 +00:00
|
|
|
* managing a hash of page addresses that maps pages to uma_slab_t structures
|
|
|
|
* for pages that don't have embedded uma_slab_t's.
|
|
|
|
*
|
|
|
|
* The uma_slab_t may be embedded in a UMA_SLAB_SIZE chunk of memory or it may
|
|
|
|
* be allocated off the page from a special slab zone. The free list within a
|
2013-06-13 21:05:38 +00:00
|
|
|
* slab is managed with a bitmask. For item sizes that would yield more than
|
|
|
|
* 10% memory waste we potentially allocate a separate uma_slab_t if this will
|
|
|
|
* improve the number of items per slab that will fit.
|
2002-03-19 09:11:49 +00:00
|
|
|
*
|
|
|
|
* The only really gross cases, with regards to memory waste, are for those
|
|
|
|
* items that are just over half the page size. You can get nearly 50% waste,
|
|
|
|
* so you fall back to the memory footprint of the power of two allocator. I
|
|
|
|
* have looked at memory allocation sizes on many of the machines available to
|
|
|
|
* me, and there does not seem to be an abundance of allocations at this range
|
|
|
|
* so at this time it may not make sense to optimize for it. This can, of
|
|
|
|
* course, be solved with dynamic slab sizes.
|
|
|
|
*
|
Bring in mbuma to replace mballoc.
mbuma is an Mbuf & Cluster allocator built on top of a number of
extensions to the UMA framework, all included herein.
Extensions to UMA worth noting:
- Better layering between slab <-> zone caches; introduce
Keg structure which splits off slab cache away from the
zone structure and allows multiple zones to be stacked
on top of a single Keg (single type of slab cache);
perhaps we should look into defining a subset API on
top of the Keg for special use by malloc(9),
for example.
- UMA_ZONE_REFCNT zones can now be added, and reference
counters automagically allocated for them within the end
of the associated slab structures. uma_find_refcnt()
does a kextract to fetch the slab struct reference from
the underlying page, and lookup the corresponding refcnt.
mbuma things worth noting:
- integrates mbuf & cluster allocations with extended UMA
and provides caches for commonly-allocated items; defines
several zones (two primary, one secondary) and two kegs.
- change up certain code paths that always used to do:
m_get() + m_clget() to instead just use m_getcl() and
try to take advantage of the newly defined secondary
Packet zone.
- netstat(1) and systat(1) quickly hacked up to do basic
stat reporting but additional stats work needs to be
done once some other details within UMA have been taken
care of and it becomes clearer to how stats will work
within the modified framework.
From the user perspective, one implication is that the
NMBCLUSTERS compile-time option is no longer used. The
maximum number of clusters is still capped off according
to maxusers, but it can be made unlimited by setting
the kern.ipc.nmbclusters boot-time tunable to zero.
Work should be done to write an appropriate sysctl
handler allowing dynamic tuning of kern.ipc.nmbclusters
at runtime.
Additional things worth noting/known issues (READ):
- One report of 'ips' (ServeRAID) driver acting really
slow in conjunction with mbuma. Need more data.
Latest report is that ips is equally sucking with
and without mbuma.
- Giant leak in NFS code sometimes occurs, can't
reproduce but currently analyzing; brueffer is
able to reproduce but THIS IS NOT an mbuma-specific
problem and currently occurs even WITHOUT mbuma.
- Issues in network locking: there is at least one
code path in the rip code where one or more locks
are acquired and we end up in m_prepend() with
M_WAITOK, which causes WITNESS to whine from within
UMA. Current temporary solution: force all UMA
allocations to be M_NOWAIT from within UMA for now
to avoid deadlocks unless WITNESS is defined and we
can determine with certainty that we're not holding
any locks when we're M_WAITOK.
- I've seen at least one weird socketbuffer empty-but-
mbuf-still-attached panic. I don't believe this
to be related to mbuma but please keep your eyes
open, turn on debugging, and capture crash dumps.
This change removes more code than it adds.
A paper is available detailing the change and considering
various performance issues, it was presented at BSDCan2004:
http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf
Please read the paper for Future Work and implementation
details, as well as credits.
Testing and Debugging:
rwatson,
brueffer,
Ketrien I. Saihr-Kesenchedra,
...
Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
|
|
|
* Kegs may serve multiple Zones but by far most of the time they only serve
|
|
|
|
* one. When a Zone is created, a Keg is allocated and setup for it. While
|
|
|
|
* the backing Keg stores slabs, the Zone caches Buckets of items allocated
|
|
|
|
* from the slabs. Each Zone is equipped with an init/fini and ctor/dtor
|
|
|
|
* pair, as well as with its own set of small per-CPU caches, layered above
|
|
|
|
* the Zone's general Bucket cache.
|
|
|
|
*
|
2007-05-09 22:53:34 +00:00
|
|
|
* The PCPU caches are protected by critical sections, and may be accessed
|
|
|
|
* safely only from their associated CPU, while the Zones backed by the same
|
|
|
|
* Keg all share a common Keg lock (to coalesce contention on the backing
|
|
|
|
* slabs). The backing Keg typically only serves one Zone but in the case of
|
|
|
|
* multiple Zones, one of the Zones is considered the Master Zone and all
|
|
|
|
* Zone-related stats from the Keg are done in the Master Zone. For an
|
|
|
|
* example of a Multi-Zone setup, refer to the Mbuf allocation code.
|
2002-03-19 09:11:49 +00:00
|
|
|
*/
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This is the representation for normal (Non OFFPAGE slab)
|
|
|
|
*
|
|
|
|
* i == item
|
|
|
|
* s == slab pointer
|
|
|
|
*
|
|
|
|
* <---------------- Page (UMA_SLAB_SIZE) ------------------>
|
|
|
|
* ___________________________________________________________
|
|
|
|
* | _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ___________ |
|
|
|
|
* ||i||i||i||i||i||i||i||i||i||i||i||i||i||i||i| |slab header||
|
|
|
|
* ||_||_||_||_||_||_||_||_||_||_||_||_||_||_||_| |___________||
|
|
|
|
* |___________________________________________________________|
|
|
|
|
*
|
|
|
|
*
|
|
|
|
* This is an OFFPAGE slab. These can be larger than UMA_SLAB_SIZE.
|
|
|
|
*
|
|
|
|
* ___________________________________________________________
|
|
|
|
* | _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ |
|
|
|
|
* ||i||i||i||i||i||i||i||i||i||i||i||i||i||i||i||i||i||i||i| |
|
|
|
|
* ||_||_||_||_||_||_||_||_||_||_||_||_||_||_||_||_||_||_||_| |
|
|
|
|
* |___________________________________________________________|
|
|
|
|
* ___________ ^
|
|
|
|
* |slab header| |
|
|
|
|
* |___________|---*
|
|
|
|
*
|
|
|
|
*/
|
|
|
|
|
|
|
|
#ifndef VM_UMA_INT_H
|
|
|
|
#define VM_UMA_INT_H
|
|
|
|
|
|
|
|
#define UMA_SLAB_SIZE PAGE_SIZE /* How big are our slabs? */
|
|
|
|
#define UMA_SLAB_MASK (PAGE_SIZE - 1) /* Mask to get back to the page */
|
|
|
|
#define UMA_SLAB_SHIFT PAGE_SHIFT /* Number of bits PAGE_MASK */
|
|
|
|
|
1. Prior to r214782, UMA did not support multipage allocations before
uma_startup2() was called. Thus, setting the variable "booted" to true in
uma_startup() was ok on machines with UMA_MD_SMALL_ALLOC defined, because
any allocations made after uma_startup() but before uma_startup2() could be
satisfied by uma_small_alloc(). Now, however, some multipage allocations
are necessary before uma_startup2() just to allocate zone structures on
machines with a large number of processors. Thus, a Boolean can no longer
effectively describe the state of the UMA allocator. Instead, make "booted"
have three values to describe how far initialization has progressed. This
allows multipage allocations to continue using startup_alloc() until
uma_startup2(), but single-page allocations may begin using
uma_small_alloc() after uma_startup().
2. With the aforementioned change, only a modest increase in boot pages is
necessary to boot UMA on a large number of processors.
3. Retire UMA_MD_SMALL_ALLOC_NEEDS_VM. It has only been used between
r182028 and r204128.
Reviewed by: attilio [1], nwhitehorn [3]
Tested by: sbruno
2011-05-21 17:43:43 +00:00
|
|
|
#define UMA_BOOT_PAGES 64 /* Pages allocated for startup */
|
2002-03-19 09:11:49 +00:00
|
|
|
|
2013-04-08 19:10:45 +00:00
|
|
|
/* Max waste percentage before going to off page slab management */
|
|
|
|
#define UMA_MAX_WASTE 10
|
2002-03-19 09:11:49 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* I doubt there will be many cases where this is exceeded. This is the initial
|
|
|
|
* size of the hash table for uma_slabs that are managed off page. This hash
|
|
|
|
* does expand by powers of two. Currently it doesn't get smaller.
|
|
|
|
*/
|
|
|
|
#define UMA_HASH_SIZE_INIT 32
|
|
|
|
|
|
|
|
/*
|
|
|
|
* I should investigate other hashing algorithms. This should yield a low
|
|
|
|
* number of collisions if the pages are relatively contiguous.
|
|
|
|
*/
|
|
|
|
|
2013-06-13 21:05:38 +00:00
|
|
|
#define UMA_HASH(h, s) ((((uintptr_t)s) >> UMA_SLAB_SHIFT) & (h)->uh_hashmask)
|
2002-03-19 09:11:49 +00:00
|
|
|
|
|
|
|
#define UMA_HASH_INSERT(h, s, mem) \
|
|
|
|
SLIST_INSERT_HEAD(&(h)->uh_slab_hash[UMA_HASH((h), \
|
2009-12-05 17:45:56 +00:00
|
|
|
(mem))], (s), us_hlink)
|
2002-03-19 09:11:49 +00:00
|
|
|
#define UMA_HASH_REMOVE(h, s, mem) \
|
|
|
|
SLIST_REMOVE(&(h)->uh_slab_hash[UMA_HASH((h), \
|
2009-12-05 17:45:56 +00:00
|
|
|
(mem))], (s), uma_slab, us_hlink)
|
2002-03-19 09:11:49 +00:00
|
|
|
|
|
|
|
/* Hash table for freed address -> slab translation */
|
|
|
|
|
|
|
|
SLIST_HEAD(slabhead, uma_slab);
|
|
|
|
|
|
|
|
struct uma_hash {
|
|
|
|
struct slabhead *uh_slab_hash; /* Hash table for slabs */
|
|
|
|
int uh_hashsize; /* Current size of the hash table */
|
|
|
|
int uh_hashmask; /* Mask used during hashing */
|
|
|
|
};
|
|
|
|
|
2010-03-17 21:18:28 +00:00
|
|
|
/*
|
|
|
|
* align field or structure to cache line
|
|
|
|
*/
|
2010-03-22 22:39:32 +00:00
|
|
|
#if defined(__amd64__)
|
|
|
|
#define UMA_ALIGN __aligned(CACHE_LINE_SIZE)
|
|
|
|
#else
|
2010-03-18 20:30:25 +00:00
|
|
|
#define UMA_ALIGN
|
2010-03-22 22:39:32 +00:00
|
|
|
#endif
|
2010-03-17 21:18:28 +00:00
|
|
|
|
2002-03-19 09:11:49 +00:00
|
|
|
/*
|
|
|
|
* Structures for per cpu queues.
|
|
|
|
*/
|
|
|
|
|
|
|
|
struct uma_bucket {
|
|
|
|
LIST_ENTRY(uma_bucket) ub_link; /* Link into the zone */
|
2003-09-19 06:26:45 +00:00
|
|
|
int16_t ub_cnt; /* Count of free items. */
|
|
|
|
int16_t ub_entries; /* Max items. */
|
|
|
|
void *ub_bucket[]; /* actual allocation storage */
|
2010-03-22 22:39:32 +00:00
|
|
|
};
|
2002-03-19 09:11:49 +00:00
|
|
|
|
|
|
|
typedef struct uma_bucket * uma_bucket_t;
|
|
|
|
|
|
|
|
struct uma_cache {
|
|
|
|
uma_bucket_t uc_freebucket; /* Bucket we're freeing to */
|
|
|
|
uma_bucket_t uc_allocbucket; /* Bucket to allocate from */
|
2013-04-09 17:43:48 +00:00
|
|
|
uint64_t uc_allocs; /* Count of allocations */
|
|
|
|
uint64_t uc_frees; /* Count of frees */
|
2010-03-17 21:18:28 +00:00
|
|
|
} UMA_ALIGN;
|
2002-03-19 09:11:49 +00:00
|
|
|
|
|
|
|
typedef struct uma_cache * uma_cache_t;
|
|
|
|
|
Bring in mbuma to replace mballoc.
mbuma is an Mbuf & Cluster allocator built on top of a number of
extensions to the UMA framework, all included herein.
Extensions to UMA worth noting:
- Better layering between slab <-> zone caches; introduce
Keg structure which splits off slab cache away from the
zone structure and allows multiple zones to be stacked
on top of a single Keg (single type of slab cache);
perhaps we should look into defining a subset API on
top of the Keg for special use by malloc(9),
for example.
- UMA_ZONE_REFCNT zones can now be added, and reference
counters automagically allocated for them within the end
of the associated slab structures. uma_find_refcnt()
does a kextract to fetch the slab struct reference from
the underlying page, and lookup the corresponding refcnt.
mbuma things worth noting:
- integrates mbuf & cluster allocations with extended UMA
and provides caches for commonly-allocated items; defines
several zones (two primary, one secondary) and two kegs.
- change up certain code paths that always used to do:
m_get() + m_clget() to instead just use m_getcl() and
try to take advantage of the newly defined secondary
Packet zone.
- netstat(1) and systat(1) quickly hacked up to do basic
stat reporting but additional stats work needs to be
done once some other details within UMA have been taken
care of and it becomes clearer to how stats will work
within the modified framework.
From the user perspective, one implication is that the
NMBCLUSTERS compile-time option is no longer used. The
maximum number of clusters is still capped off according
to maxusers, but it can be made unlimited by setting
the kern.ipc.nmbclusters boot-time tunable to zero.
Work should be done to write an appropriate sysctl
handler allowing dynamic tuning of kern.ipc.nmbclusters
at runtime.
Additional things worth noting/known issues (READ):
- One report of 'ips' (ServeRAID) driver acting really
slow in conjunction with mbuma. Need more data.
Latest report is that ips is equally sucking with
and without mbuma.
- Giant leak in NFS code sometimes occurs, can't
reproduce but currently analyzing; brueffer is
able to reproduce but THIS IS NOT an mbuma-specific
problem and currently occurs even WITHOUT mbuma.
- Issues in network locking: there is at least one
code path in the rip code where one or more locks
are acquired and we end up in m_prepend() with
M_WAITOK, which causes WITNESS to whine from within
UMA. Current temporary solution: force all UMA
allocations to be M_NOWAIT from within UMA for now
to avoid deadlocks unless WITNESS is defined and we
can determine with certainty that we're not holding
any locks when we're M_WAITOK.
- I've seen at least one weird socketbuffer empty-but-
mbuf-still-attached panic. I don't believe this
to be related to mbuma but please keep your eyes
open, turn on debugging, and capture crash dumps.
This change removes more code than it adds.
A paper is available detailing the change and considering
various performance issues, it was presented at BSDCan2004:
http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf
Please read the paper for Future Work and implementation
details, as well as credits.
Testing and Debugging:
rwatson,
brueffer,
Ketrien I. Saihr-Kesenchedra,
...
Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
|
|
|
/*
|
|
|
|
* Keg management structure
|
|
|
|
*
|
|
|
|
* TODO: Optimize for cache line size
|
|
|
|
*
|
|
|
|
*/
|
|
|
|
struct uma_keg {
|
2013-06-20 19:08:12 +00:00
|
|
|
struct mtx_padalign uk_lock; /* Lock for the keg */
|
Bring in mbuma to replace mballoc.
mbuma is an Mbuf & Cluster allocator built on top of a number of
extensions to the UMA framework, all included herein.
Extensions to UMA worth noting:
- Better layering between slab <-> zone caches; introduce
Keg structure which splits off slab cache away from the
zone structure and allows multiple zones to be stacked
on top of a single Keg (single type of slab cache);
perhaps we should look into defining a subset API on
top of the Keg for special use by malloc(9),
for example.
- UMA_ZONE_REFCNT zones can now be added, and reference
counters automagically allocated for them within the end
of the associated slab structures. uma_find_refcnt()
does a kextract to fetch the slab struct reference from
the underlying page, and lookup the corresponding refcnt.
mbuma things worth noting:
- integrates mbuf & cluster allocations with extended UMA
and provides caches for commonly-allocated items; defines
several zones (two primary, one secondary) and two kegs.
- change up certain code paths that always used to do:
m_get() + m_clget() to instead just use m_getcl() and
try to take advantage of the newly defined secondary
Packet zone.
- netstat(1) and systat(1) quickly hacked up to do basic
stat reporting but additional stats work needs to be
done once some other details within UMA have been taken
care of and it becomes clearer to how stats will work
within the modified framework.
From the user perspective, one implication is that the
NMBCLUSTERS compile-time option is no longer used. The
maximum number of clusters is still capped off according
to maxusers, but it can be made unlimited by setting
the kern.ipc.nmbclusters boot-time tunable to zero.
Work should be done to write an appropriate sysctl
handler allowing dynamic tuning of kern.ipc.nmbclusters
at runtime.
Additional things worth noting/known issues (READ):
- One report of 'ips' (ServeRAID) driver acting really
slow in conjunction with mbuma. Need more data.
Latest report is that ips is equally sucking with
and without mbuma.
- Giant leak in NFS code sometimes occurs, can't
reproduce but currently analyzing; brueffer is
able to reproduce but THIS IS NOT an mbuma-specific
problem and currently occurs even WITHOUT mbuma.
- Issues in network locking: there is at least one
code path in the rip code where one or more locks
are acquired and we end up in m_prepend() with
M_WAITOK, which causes WITNESS to whine from within
UMA. Current temporary solution: force all UMA
allocations to be M_NOWAIT from within UMA for now
to avoid deadlocks unless WITNESS is defined and we
can determine with certainty that we're not holding
any locks when we're M_WAITOK.
- I've seen at least one weird socketbuffer empty-but-
mbuf-still-attached panic. I don't believe this
to be related to mbuma but please keep your eyes
open, turn on debugging, and capture crash dumps.
This change removes more code than it adds.
A paper is available detailing the change and considering
various performance issues, it was presented at BSDCan2004:
http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf
Please read the paper for Future Work and implementation
details, as well as credits.
Testing and Debugging:
rwatson,
brueffer,
Ketrien I. Saihr-Kesenchedra,
...
Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
|
|
|
struct uma_hash uk_hash;
|
|
|
|
|
|
|
|
LIST_HEAD(,uma_zone) uk_zones; /* Keg's zones */
|
|
|
|
LIST_HEAD(,uma_slab) uk_part_slab; /* partially allocated slabs */
|
|
|
|
LIST_HEAD(,uma_slab) uk_free_slab; /* empty slab list */
|
|
|
|
LIST_HEAD(,uma_slab) uk_full_slab; /* full slabs */
|
|
|
|
|
2013-04-09 17:43:48 +00:00
|
|
|
uint32_t uk_align; /* Alignment mask */
|
|
|
|
uint32_t uk_pages; /* Total page count */
|
|
|
|
uint32_t uk_free; /* Count of items free in slabs */
|
2013-06-26 00:57:38 +00:00
|
|
|
uint32_t uk_reserve; /* Number of reserved items. */
|
2013-04-09 17:43:48 +00:00
|
|
|
uint32_t uk_size; /* Requested size of each item */
|
|
|
|
uint32_t uk_rsize; /* Real size of each item */
|
|
|
|
uint32_t uk_maxpages; /* Maximum number of pages to alloc */
|
Bring in mbuma to replace mballoc.
mbuma is an Mbuf & Cluster allocator built on top of a number of
extensions to the UMA framework, all included herein.
Extensions to UMA worth noting:
- Better layering between slab <-> zone caches; introduce
Keg structure which splits off slab cache away from the
zone structure and allows multiple zones to be stacked
on top of a single Keg (single type of slab cache);
perhaps we should look into defining a subset API on
top of the Keg for special use by malloc(9),
for example.
- UMA_ZONE_REFCNT zones can now be added, and reference
counters automagically allocated for them within the end
of the associated slab structures. uma_find_refcnt()
does a kextract to fetch the slab struct reference from
the underlying page, and lookup the corresponding refcnt.
mbuma things worth noting:
- integrates mbuf & cluster allocations with extended UMA
and provides caches for commonly-allocated items; defines
several zones (two primary, one secondary) and two kegs.
- change up certain code paths that always used to do:
m_get() + m_clget() to instead just use m_getcl() and
try to take advantage of the newly defined secondary
Packet zone.
- netstat(1) and systat(1) quickly hacked up to do basic
stat reporting but additional stats work needs to be
done once some other details within UMA have been taken
care of and it becomes clearer to how stats will work
within the modified framework.
From the user perspective, one implication is that the
NMBCLUSTERS compile-time option is no longer used. The
maximum number of clusters is still capped off according
to maxusers, but it can be made unlimited by setting
the kern.ipc.nmbclusters boot-time tunable to zero.
Work should be done to write an appropriate sysctl
handler allowing dynamic tuning of kern.ipc.nmbclusters
at runtime.
Additional things worth noting/known issues (READ):
- One report of 'ips' (ServeRAID) driver acting really
slow in conjunction with mbuma. Need more data.
Latest report is that ips is equally sucking with
and without mbuma.
- Giant leak in NFS code sometimes occurs, can't
reproduce but currently analyzing; brueffer is
able to reproduce but THIS IS NOT an mbuma-specific
problem and currently occurs even WITHOUT mbuma.
- Issues in network locking: there is at least one
code path in the rip code where one or more locks
are acquired and we end up in m_prepend() with
M_WAITOK, which causes WITNESS to whine from within
UMA. Current temporary solution: force all UMA
allocations to be M_NOWAIT from within UMA for now
to avoid deadlocks unless WITNESS is defined and we
can determine with certainty that we're not holding
any locks when we're M_WAITOK.
- I've seen at least one weird socketbuffer empty-but-
mbuf-still-attached panic. I don't believe this
to be related to mbuma but please keep your eyes
open, turn on debugging, and capture crash dumps.
This change removes more code than it adds.
A paper is available detailing the change and considering
various performance issues, it was presented at BSDCan2004:
http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf
Please read the paper for Future Work and implementation
details, as well as credits.
Testing and Debugging:
rwatson,
brueffer,
Ketrien I. Saihr-Kesenchedra,
...
Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
|
|
|
|
|
|
|
uma_init uk_init; /* Keg's init routine */
|
|
|
|
uma_fini uk_fini; /* Keg's fini routine */
|
|
|
|
uma_alloc uk_allocf; /* Allocation function */
|
|
|
|
uma_free uk_freef; /* Free routine */
|
|
|
|
|
2013-02-26 23:35:27 +00:00
|
|
|
u_long uk_offset; /* Next free offset from base KVA */
|
|
|
|
vm_offset_t uk_kva; /* Zone base KVA */
|
Bring in mbuma to replace mballoc.
mbuma is an Mbuf & Cluster allocator built on top of a number of
extensions to the UMA framework, all included herein.
Extensions to UMA worth noting:
- Better layering between slab <-> zone caches; introduce
Keg structure which splits off slab cache away from the
zone structure and allows multiple zones to be stacked
on top of a single Keg (single type of slab cache);
perhaps we should look into defining a subset API on
top of the Keg for special use by malloc(9),
for example.
- UMA_ZONE_REFCNT zones can now be added, and reference
counters automagically allocated for them within the end
of the associated slab structures. uma_find_refcnt()
does a kextract to fetch the slab struct reference from
the underlying page, and lookup the corresponding refcnt.
mbuma things worth noting:
- integrates mbuf & cluster allocations with extended UMA
and provides caches for commonly-allocated items; defines
several zones (two primary, one secondary) and two kegs.
- change up certain code paths that always used to do:
m_get() + m_clget() to instead just use m_getcl() and
try to take advantage of the newly defined secondary
Packet zone.
- netstat(1) and systat(1) quickly hacked up to do basic
stat reporting but additional stats work needs to be
done once some other details within UMA have been taken
care of and it becomes clearer to how stats will work
within the modified framework.
From the user perspective, one implication is that the
NMBCLUSTERS compile-time option is no longer used. The
maximum number of clusters is still capped off according
to maxusers, but it can be made unlimited by setting
the kern.ipc.nmbclusters boot-time tunable to zero.
Work should be done to write an appropriate sysctl
handler allowing dynamic tuning of kern.ipc.nmbclusters
at runtime.
Additional things worth noting/known issues (READ):
- One report of 'ips' (ServeRAID) driver acting really
slow in conjunction with mbuma. Need more data.
Latest report is that ips is equally sucking with
and without mbuma.
- Giant leak in NFS code sometimes occurs, can't
reproduce but currently analyzing; brueffer is
able to reproduce but THIS IS NOT an mbuma-specific
problem and currently occurs even WITHOUT mbuma.
- Issues in network locking: there is at least one
code path in the rip code where one or more locks
are acquired and we end up in m_prepend() with
M_WAITOK, which causes WITNESS to whine from within
UMA. Current temporary solution: force all UMA
allocations to be M_NOWAIT from within UMA for now
to avoid deadlocks unless WITNESS is defined and we
can determine with certainty that we're not holding
any locks when we're M_WAITOK.
- I've seen at least one weird socketbuffer empty-but-
mbuf-still-attached panic. I don't believe this
to be related to mbuma but please keep your eyes
open, turn on debugging, and capture crash dumps.
This change removes more code than it adds.
A paper is available detailing the change and considering
various performance issues, it was presented at BSDCan2004:
http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf
Please read the paper for Future Work and implementation
details, as well as credits.
Testing and Debugging:
rwatson,
brueffer,
Ketrien I. Saihr-Kesenchedra,
...
Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
|
|
|
uma_zone_t uk_slabzone; /* Slab zone backing us, if OFFPAGE */
|
|
|
|
|
2013-04-09 17:43:48 +00:00
|
|
|
uint16_t uk_slabsize; /* Slab size for this keg */
|
|
|
|
uint16_t uk_pgoff; /* Offset to uma_slab struct */
|
|
|
|
uint16_t uk_ppera; /* pages per allocation from backend */
|
|
|
|
uint16_t uk_ipers; /* Items per slab */
|
|
|
|
uint32_t uk_flags; /* Internal flags */
|
2013-04-08 19:10:45 +00:00
|
|
|
|
|
|
|
/* Least used fields go to the last cache line. */
|
|
|
|
const char *uk_name; /* Name of creating zone. */
|
|
|
|
LIST_ENTRY(uma_keg) uk_link; /* List of all kegs */
|
Bring in mbuma to replace mballoc.
mbuma is an Mbuf & Cluster allocator built on top of a number of
extensions to the UMA framework, all included herein.
Extensions to UMA worth noting:
- Better layering between slab <-> zone caches; introduce
Keg structure which splits off slab cache away from the
zone structure and allows multiple zones to be stacked
on top of a single Keg (single type of slab cache);
perhaps we should look into defining a subset API on
top of the Keg for special use by malloc(9),
for example.
- UMA_ZONE_REFCNT zones can now be added, and reference
counters automagically allocated for them within the end
of the associated slab structures. uma_find_refcnt()
does a kextract to fetch the slab struct reference from
the underlying page, and lookup the corresponding refcnt.
mbuma things worth noting:
- integrates mbuf & cluster allocations with extended UMA
and provides caches for commonly-allocated items; defines
several zones (two primary, one secondary) and two kegs.
- change up certain code paths that always used to do:
m_get() + m_clget() to instead just use m_getcl() and
try to take advantage of the newly defined secondary
Packet zone.
- netstat(1) and systat(1) quickly hacked up to do basic
stat reporting but additional stats work needs to be
done once some other details within UMA have been taken
care of and it becomes clearer to how stats will work
within the modified framework.
From the user perspective, one implication is that the
NMBCLUSTERS compile-time option is no longer used. The
maximum number of clusters is still capped off according
to maxusers, but it can be made unlimited by setting
the kern.ipc.nmbclusters boot-time tunable to zero.
Work should be done to write an appropriate sysctl
handler allowing dynamic tuning of kern.ipc.nmbclusters
at runtime.
Additional things worth noting/known issues (READ):
- One report of 'ips' (ServeRAID) driver acting really
slow in conjunction with mbuma. Need more data.
Latest report is that ips is equally sucking with
and without mbuma.
- Giant leak in NFS code sometimes occurs, can't
reproduce but currently analyzing; brueffer is
able to reproduce but THIS IS NOT an mbuma-specific
problem and currently occurs even WITHOUT mbuma.
- Issues in network locking: there is at least one
code path in the rip code where one or more locks
are acquired and we end up in m_prepend() with
M_WAITOK, which causes WITNESS to whine from within
UMA. Current temporary solution: force all UMA
allocations to be M_NOWAIT from within UMA for now
to avoid deadlocks unless WITNESS is defined and we
can determine with certainty that we're not holding
any locks when we're M_WAITOK.
- I've seen at least one weird socketbuffer empty-but-
mbuf-still-attached panic. I don't believe this
to be related to mbuma but please keep your eyes
open, turn on debugging, and capture crash dumps.
This change removes more code than it adds.
A paper is available detailing the change and considering
various performance issues, it was presented at BSDCan2004:
http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf
Please read the paper for Future Work and implementation
details, as well as credits.
Testing and Debugging:
rwatson,
brueffer,
Ketrien I. Saihr-Kesenchedra,
...
Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
|
|
|
};
|
2009-01-25 09:11:24 +00:00
|
|
|
typedef struct uma_keg * uma_keg_t;
|
Bring in mbuma to replace mballoc.
mbuma is an Mbuf & Cluster allocator built on top of a number of
extensions to the UMA framework, all included herein.
Extensions to UMA worth noting:
- Better layering between slab <-> zone caches; introduce
Keg structure which splits off slab cache away from the
zone structure and allows multiple zones to be stacked
on top of a single Keg (single type of slab cache);
perhaps we should look into defining a subset API on
top of the Keg for special use by malloc(9),
for example.
- UMA_ZONE_REFCNT zones can now be added, and reference
counters automagically allocated for them within the end
of the associated slab structures. uma_find_refcnt()
does a kextract to fetch the slab struct reference from
the underlying page, and lookup the corresponding refcnt.
mbuma things worth noting:
- integrates mbuf & cluster allocations with extended UMA
and provides caches for commonly-allocated items; defines
several zones (two primary, one secondary) and two kegs.
- change up certain code paths that always used to do:
m_get() + m_clget() to instead just use m_getcl() and
try to take advantage of the newly defined secondary
Packet zone.
- netstat(1) and systat(1) quickly hacked up to do basic
stat reporting but additional stats work needs to be
done once some other details within UMA have been taken
care of and it becomes clearer to how stats will work
within the modified framework.
From the user perspective, one implication is that the
NMBCLUSTERS compile-time option is no longer used. The
maximum number of clusters is still capped off according
to maxusers, but it can be made unlimited by setting
the kern.ipc.nmbclusters boot-time tunable to zero.
Work should be done to write an appropriate sysctl
handler allowing dynamic tuning of kern.ipc.nmbclusters
at runtime.
Additional things worth noting/known issues (READ):
- One report of 'ips' (ServeRAID) driver acting really
slow in conjunction with mbuma. Need more data.
Latest report is that ips is equally sucking with
and without mbuma.
- Giant leak in NFS code sometimes occurs, can't
reproduce but currently analyzing; brueffer is
able to reproduce but THIS IS NOT an mbuma-specific
problem and currently occurs even WITHOUT mbuma.
- Issues in network locking: there is at least one
code path in the rip code where one or more locks
are acquired and we end up in m_prepend() with
M_WAITOK, which causes WITNESS to whine from within
UMA. Current temporary solution: force all UMA
allocations to be M_NOWAIT from within UMA for now
to avoid deadlocks unless WITNESS is defined and we
can determine with certainty that we're not holding
any locks when we're M_WAITOK.
- I've seen at least one weird socketbuffer empty-but-
mbuf-still-attached panic. I don't believe this
to be related to mbuma but please keep your eyes
open, turn on debugging, and capture crash dumps.
This change removes more code than it adds.
A paper is available detailing the change and considering
various performance issues, it was presented at BSDCan2004:
http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf
Please read the paper for Future Work and implementation
details, as well as credits.
Testing and Debugging:
rwatson,
brueffer,
Ketrien I. Saihr-Kesenchedra,
...
Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
|
|
|
|
2013-06-13 21:05:38 +00:00
|
|
|
/*
|
|
|
|
* Free bits per-slab.
|
|
|
|
*/
|
|
|
|
#define SLAB_SETSIZE (PAGE_SIZE / UMA_SMALLEST_UNIT)
|
|
|
|
BITSET_DEFINE(slabbits, SLAB_SETSIZE);
|
Bring in mbuma to replace mballoc.
mbuma is an Mbuf & Cluster allocator built on top of a number of
extensions to the UMA framework, all included herein.
Extensions to UMA worth noting:
- Better layering between slab <-> zone caches; introduce
Keg structure which splits off slab cache away from the
zone structure and allows multiple zones to be stacked
on top of a single Keg (single type of slab cache);
perhaps we should look into defining a subset API on
top of the Keg for special use by malloc(9),
for example.
- UMA_ZONE_REFCNT zones can now be added, and reference
counters automagically allocated for them within the end
of the associated slab structures. uma_find_refcnt()
does a kextract to fetch the slab struct reference from
the underlying page, and lookup the corresponding refcnt.
mbuma things worth noting:
- integrates mbuf & cluster allocations with extended UMA
and provides caches for commonly-allocated items; defines
several zones (two primary, one secondary) and two kegs.
- change up certain code paths that always used to do:
m_get() + m_clget() to instead just use m_getcl() and
try to take advantage of the newly defined secondary
Packet zone.
- netstat(1) and systat(1) quickly hacked up to do basic
stat reporting but additional stats work needs to be
done once some other details within UMA have been taken
care of and it becomes clearer to how stats will work
within the modified framework.
From the user perspective, one implication is that the
NMBCLUSTERS compile-time option is no longer used. The
maximum number of clusters is still capped off according
to maxusers, but it can be made unlimited by setting
the kern.ipc.nmbclusters boot-time tunable to zero.
Work should be done to write an appropriate sysctl
handler allowing dynamic tuning of kern.ipc.nmbclusters
at runtime.
Additional things worth noting/known issues (READ):
- One report of 'ips' (ServeRAID) driver acting really
slow in conjunction with mbuma. Need more data.
Latest report is that ips is equally sucking with
and without mbuma.
- Giant leak in NFS code sometimes occurs, can't
reproduce but currently analyzing; brueffer is
able to reproduce but THIS IS NOT an mbuma-specific
problem and currently occurs even WITHOUT mbuma.
- Issues in network locking: there is at least one
code path in the rip code where one or more locks
are acquired and we end up in m_prepend() with
M_WAITOK, which causes WITNESS to whine from within
UMA. Current temporary solution: force all UMA
allocations to be M_NOWAIT from within UMA for now
to avoid deadlocks unless WITNESS is defined and we
can determine with certainty that we're not holding
any locks when we're M_WAITOK.
- I've seen at least one weird socketbuffer empty-but-
mbuf-still-attached panic. I don't believe this
to be related to mbuma but please keep your eyes
open, turn on debugging, and capture crash dumps.
This change removes more code than it adds.
A paper is available detailing the change and considering
various performance issues, it was presented at BSDCan2004:
http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf
Please read the paper for Future Work and implementation
details, as well as credits.
Testing and Debugging:
rwatson,
brueffer,
Ketrien I. Saihr-Kesenchedra,
...
Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
|
|
|
|
2013-06-13 21:05:38 +00:00
|
|
|
/*
|
|
|
|
* The slab structure manages a single contiguous allocation from backing
|
|
|
|
* store and subdivides it into individually allocatable items.
|
|
|
|
*/
|
|
|
|
struct uma_slab {
|
Bring in mbuma to replace mballoc.
mbuma is an Mbuf & Cluster allocator built on top of a number of
extensions to the UMA framework, all included herein.
Extensions to UMA worth noting:
- Better layering between slab <-> zone caches; introduce
Keg structure which splits off slab cache away from the
zone structure and allows multiple zones to be stacked
on top of a single Keg (single type of slab cache);
perhaps we should look into defining a subset API on
top of the Keg for special use by malloc(9),
for example.
- UMA_ZONE_REFCNT zones can now be added, and reference
counters automagically allocated for them within the end
of the associated slab structures. uma_find_refcnt()
does a kextract to fetch the slab struct reference from
the underlying page, and lookup the corresponding refcnt.
mbuma things worth noting:
- integrates mbuf & cluster allocations with extended UMA
and provides caches for commonly-allocated items; defines
several zones (two primary, one secondary) and two kegs.
- change up certain code paths that always used to do:
m_get() + m_clget() to instead just use m_getcl() and
try to take advantage of the newly defined secondary
Packet zone.
- netstat(1) and systat(1) quickly hacked up to do basic
stat reporting but additional stats work needs to be
done once some other details within UMA have been taken
care of and it becomes clearer to how stats will work
within the modified framework.
From the user perspective, one implication is that the
NMBCLUSTERS compile-time option is no longer used. The
maximum number of clusters is still capped off according
to maxusers, but it can be made unlimited by setting
the kern.ipc.nmbclusters boot-time tunable to zero.
Work should be done to write an appropriate sysctl
handler allowing dynamic tuning of kern.ipc.nmbclusters
at runtime.
Additional things worth noting/known issues (READ):
- One report of 'ips' (ServeRAID) driver acting really
slow in conjunction with mbuma. Need more data.
Latest report is that ips is equally sucking with
and without mbuma.
- Giant leak in NFS code sometimes occurs, can't
reproduce but currently analyzing; brueffer is
able to reproduce but THIS IS NOT an mbuma-specific
problem and currently occurs even WITHOUT mbuma.
- Issues in network locking: there is at least one
code path in the rip code where one or more locks
are acquired and we end up in m_prepend() with
M_WAITOK, which causes WITNESS to whine from within
UMA. Current temporary solution: force all UMA
allocations to be M_NOWAIT from within UMA for now
to avoid deadlocks unless WITNESS is defined and we
can determine with certainty that we're not holding
any locks when we're M_WAITOK.
- I've seen at least one weird socketbuffer empty-but-
mbuf-still-attached panic. I don't believe this
to be related to mbuma but please keep your eyes
open, turn on debugging, and capture crash dumps.
This change removes more code than it adds.
A paper is available detailing the change and considering
various performance issues, it was presented at BSDCan2004:
http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf
Please read the paper for Future Work and implementation
details, as well as credits.
Testing and Debugging:
rwatson,
brueffer,
Ketrien I. Saihr-Kesenchedra,
...
Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
|
|
|
uma_keg_t us_keg; /* Keg we live in */
|
|
|
|
union {
|
|
|
|
LIST_ENTRY(uma_slab) _us_link; /* slabs in zone */
|
|
|
|
unsigned long _us_size; /* Size of allocation */
|
|
|
|
} us_type;
|
|
|
|
SLIST_ENTRY(uma_slab) us_hlink; /* Link for hash table */
|
2013-04-09 17:43:48 +00:00
|
|
|
uint8_t *us_data; /* First item */
|
2013-06-13 21:05:38 +00:00
|
|
|
struct slabbits us_free; /* Free bitmask. */
|
|
|
|
#ifdef INVARIANTS
|
|
|
|
struct slabbits us_debugfree; /* Debug bitmask. */
|
|
|
|
#endif
|
2013-04-09 17:43:48 +00:00
|
|
|
uint16_t us_freecount; /* How many are free? */
|
|
|
|
uint8_t us_flags; /* Page flags see uma.h */
|
2013-06-13 21:05:38 +00:00
|
|
|
uint8_t us_pad; /* Pad to 32bits, unused. */
|
Bring in mbuma to replace mballoc.
mbuma is an Mbuf & Cluster allocator built on top of a number of
extensions to the UMA framework, all included herein.
Extensions to UMA worth noting:
- Better layering between slab <-> zone caches; introduce
Keg structure which splits off slab cache away from the
zone structure and allows multiple zones to be stacked
on top of a single Keg (single type of slab cache);
perhaps we should look into defining a subset API on
top of the Keg for special use by malloc(9),
for example.
- UMA_ZONE_REFCNT zones can now be added, and reference
counters automagically allocated for them within the end
of the associated slab structures. uma_find_refcnt()
does a kextract to fetch the slab struct reference from
the underlying page, and lookup the corresponding refcnt.
mbuma things worth noting:
- integrates mbuf & cluster allocations with extended UMA
and provides caches for commonly-allocated items; defines
several zones (two primary, one secondary) and two kegs.
- change up certain code paths that always used to do:
m_get() + m_clget() to instead just use m_getcl() and
try to take advantage of the newly defined secondary
Packet zone.
- netstat(1) and systat(1) quickly hacked up to do basic
stat reporting but additional stats work needs to be
done once some other details within UMA have been taken
care of and it becomes clearer to how stats will work
within the modified framework.
From the user perspective, one implication is that the
NMBCLUSTERS compile-time option is no longer used. The
maximum number of clusters is still capped off according
to maxusers, but it can be made unlimited by setting
the kern.ipc.nmbclusters boot-time tunable to zero.
Work should be done to write an appropriate sysctl
handler allowing dynamic tuning of kern.ipc.nmbclusters
at runtime.
Additional things worth noting/known issues (READ):
- One report of 'ips' (ServeRAID) driver acting really
slow in conjunction with mbuma. Need more data.
Latest report is that ips is equally sucking with
and without mbuma.
- Giant leak in NFS code sometimes occurs, can't
reproduce but currently analyzing; brueffer is
able to reproduce but THIS IS NOT an mbuma-specific
problem and currently occurs even WITHOUT mbuma.
- Issues in network locking: there is at least one
code path in the rip code where one or more locks
are acquired and we end up in m_prepend() with
M_WAITOK, which causes WITNESS to whine from within
UMA. Current temporary solution: force all UMA
allocations to be M_NOWAIT from within UMA for now
to avoid deadlocks unless WITNESS is defined and we
can determine with certainty that we're not holding
any locks when we're M_WAITOK.
- I've seen at least one weird socketbuffer empty-but-
mbuf-still-attached panic. I don't believe this
to be related to mbuma but please keep your eyes
open, turn on debugging, and capture crash dumps.
This change removes more code than it adds.
A paper is available detailing the change and considering
various performance issues, it was presented at BSDCan2004:
http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf
Please read the paper for Future Work and implementation
details, as well as credits.
Testing and Debugging:
rwatson,
brueffer,
Ketrien I. Saihr-Kesenchedra,
...
Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
|
|
|
};
|
|
|
|
|
2013-06-13 21:05:38 +00:00
|
|
|
#define us_link us_type._us_link
|
|
|
|
#define us_size us_type._us_size
|
Bring in mbuma to replace mballoc.
mbuma is an Mbuf & Cluster allocator built on top of a number of
extensions to the UMA framework, all included herein.
Extensions to UMA worth noting:
- Better layering between slab <-> zone caches; introduce
Keg structure which splits off slab cache away from the
zone structure and allows multiple zones to be stacked
on top of a single Keg (single type of slab cache);
perhaps we should look into defining a subset API on
top of the Keg for special use by malloc(9),
for example.
- UMA_ZONE_REFCNT zones can now be added, and reference
counters automagically allocated for them within the end
of the associated slab structures. uma_find_refcnt()
does a kextract to fetch the slab struct reference from
the underlying page, and lookup the corresponding refcnt.
mbuma things worth noting:
- integrates mbuf & cluster allocations with extended UMA
and provides caches for commonly-allocated items; defines
several zones (two primary, one secondary) and two kegs.
- change up certain code paths that always used to do:
m_get() + m_clget() to instead just use m_getcl() and
try to take advantage of the newly defined secondary
Packet zone.
- netstat(1) and systat(1) quickly hacked up to do basic
stat reporting but additional stats work needs to be
done once some other details within UMA have been taken
care of and it becomes clearer to how stats will work
within the modified framework.
From the user perspective, one implication is that the
NMBCLUSTERS compile-time option is no longer used. The
maximum number of clusters is still capped off according
to maxusers, but it can be made unlimited by setting
the kern.ipc.nmbclusters boot-time tunable to zero.
Work should be done to write an appropriate sysctl
handler allowing dynamic tuning of kern.ipc.nmbclusters
at runtime.
Additional things worth noting/known issues (READ):
- One report of 'ips' (ServeRAID) driver acting really
slow in conjunction with mbuma. Need more data.
Latest report is that ips is equally sucking with
and without mbuma.
- Giant leak in NFS code sometimes occurs, can't
reproduce but currently analyzing; brueffer is
able to reproduce but THIS IS NOT an mbuma-specific
problem and currently occurs even WITHOUT mbuma.
- Issues in network locking: there is at least one
code path in the rip code where one or more locks
are acquired and we end up in m_prepend() with
M_WAITOK, which causes WITNESS to whine from within
UMA. Current temporary solution: force all UMA
allocations to be M_NOWAIT from within UMA for now
to avoid deadlocks unless WITNESS is defined and we
can determine with certainty that we're not holding
any locks when we're M_WAITOK.
- I've seen at least one weird socketbuffer empty-but-
mbuf-still-attached panic. I don't believe this
to be related to mbuma but please keep your eyes
open, turn on debugging, and capture crash dumps.
This change removes more code than it adds.
A paper is available detailing the change and considering
various performance issues, it was presented at BSDCan2004:
http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf
Please read the paper for Future Work and implementation
details, as well as credits.
Testing and Debugging:
rwatson,
brueffer,
Ketrien I. Saihr-Kesenchedra,
...
Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* The slab structure for UMA_ZONE_REFCNT zones for whose items we
|
|
|
|
* maintain reference counters in the slab for.
|
|
|
|
*/
|
|
|
|
struct uma_slab_refcnt {
|
2013-06-13 21:05:38 +00:00
|
|
|
struct uma_slab us_head; /* slab header data */
|
|
|
|
uint32_t us_refcnt[0]; /* Actually larger. */
|
Bring in mbuma to replace mballoc.
mbuma is an Mbuf & Cluster allocator built on top of a number of
extensions to the UMA framework, all included herein.
Extensions to UMA worth noting:
- Better layering between slab <-> zone caches; introduce
Keg structure which splits off slab cache away from the
zone structure and allows multiple zones to be stacked
on top of a single Keg (single type of slab cache);
perhaps we should look into defining a subset API on
top of the Keg for special use by malloc(9),
for example.
- UMA_ZONE_REFCNT zones can now be added, and reference
counters automagically allocated for them within the end
of the associated slab structures. uma_find_refcnt()
does a kextract to fetch the slab struct reference from
the underlying page, and lookup the corresponding refcnt.
mbuma things worth noting:
- integrates mbuf & cluster allocations with extended UMA
and provides caches for commonly-allocated items; defines
several zones (two primary, one secondary) and two kegs.
- change up certain code paths that always used to do:
m_get() + m_clget() to instead just use m_getcl() and
try to take advantage of the newly defined secondary
Packet zone.
- netstat(1) and systat(1) quickly hacked up to do basic
stat reporting but additional stats work needs to be
done once some other details within UMA have been taken
care of and it becomes clearer to how stats will work
within the modified framework.
From the user perspective, one implication is that the
NMBCLUSTERS compile-time option is no longer used. The
maximum number of clusters is still capped off according
to maxusers, but it can be made unlimited by setting
the kern.ipc.nmbclusters boot-time tunable to zero.
Work should be done to write an appropriate sysctl
handler allowing dynamic tuning of kern.ipc.nmbclusters
at runtime.
Additional things worth noting/known issues (READ):
- One report of 'ips' (ServeRAID) driver acting really
slow in conjunction with mbuma. Need more data.
Latest report is that ips is equally sucking with
and without mbuma.
- Giant leak in NFS code sometimes occurs, can't
reproduce but currently analyzing; brueffer is
able to reproduce but THIS IS NOT an mbuma-specific
problem and currently occurs even WITHOUT mbuma.
- Issues in network locking: there is at least one
code path in the rip code where one or more locks
are acquired and we end up in m_prepend() with
M_WAITOK, which causes WITNESS to whine from within
UMA. Current temporary solution: force all UMA
allocations to be M_NOWAIT from within UMA for now
to avoid deadlocks unless WITNESS is defined and we
can determine with certainty that we're not holding
any locks when we're M_WAITOK.
- I've seen at least one weird socketbuffer empty-but-
mbuf-still-attached panic. I don't believe this
to be related to mbuma but please keep your eyes
open, turn on debugging, and capture crash dumps.
This change removes more code than it adds.
A paper is available detailing the change and considering
various performance issues, it was presented at BSDCan2004:
http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf
Please read the paper for Future Work and implementation
details, as well as credits.
Testing and Debugging:
rwatson,
brueffer,
Ketrien I. Saihr-Kesenchedra,
...
Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
typedef struct uma_slab * uma_slab_t;
|
|
|
|
typedef struct uma_slab_refcnt * uma_slabrefcnt_t;
|
2009-01-25 09:11:24 +00:00
|
|
|
typedef uma_slab_t (*uma_slaballoc)(uma_zone_t, uma_keg_t, int);
|
|
|
|
|
|
|
|
struct uma_klink {
|
|
|
|
LIST_ENTRY(uma_klink) kl_link;
|
|
|
|
uma_keg_t kl_keg;
|
|
|
|
};
|
|
|
|
typedef struct uma_klink *uma_klink_t;
|
|
|
|
|
2002-03-19 09:11:49 +00:00
|
|
|
/*
|
|
|
|
* Zone management structure
|
|
|
|
*
|
|
|
|
* TODO: Optimize for cache line size
|
|
|
|
*
|
|
|
|
*/
|
|
|
|
struct uma_zone {
|
2013-06-20 19:08:12 +00:00
|
|
|
struct mtx_padalign uz_lock; /* Lock for the zone */
|
|
|
|
struct mtx_padalign *uz_lockptr;
|
|
|
|
const char *uz_name; /* Text name of the zone */
|
Bring in mbuma to replace mballoc.
mbuma is an Mbuf & Cluster allocator built on top of a number of
extensions to the UMA framework, all included herein.
Extensions to UMA worth noting:
- Better layering between slab <-> zone caches; introduce
Keg structure which splits off slab cache away from the
zone structure and allows multiple zones to be stacked
on top of a single Keg (single type of slab cache);
perhaps we should look into defining a subset API on
top of the Keg for special use by malloc(9),
for example.
- UMA_ZONE_REFCNT zones can now be added, and reference
counters automagically allocated for them within the end
of the associated slab structures. uma_find_refcnt()
does a kextract to fetch the slab struct reference from
the underlying page, and lookup the corresponding refcnt.
mbuma things worth noting:
- integrates mbuf & cluster allocations with extended UMA
and provides caches for commonly-allocated items; defines
several zones (two primary, one secondary) and two kegs.
- change up certain code paths that always used to do:
m_get() + m_clget() to instead just use m_getcl() and
try to take advantage of the newly defined secondary
Packet zone.
- netstat(1) and systat(1) quickly hacked up to do basic
stat reporting but additional stats work needs to be
done once some other details within UMA have been taken
care of and it becomes clearer to how stats will work
within the modified framework.
From the user perspective, one implication is that the
NMBCLUSTERS compile-time option is no longer used. The
maximum number of clusters is still capped off according
to maxusers, but it can be made unlimited by setting
the kern.ipc.nmbclusters boot-time tunable to zero.
Work should be done to write an appropriate sysctl
handler allowing dynamic tuning of kern.ipc.nmbclusters
at runtime.
Additional things worth noting/known issues (READ):
- One report of 'ips' (ServeRAID) driver acting really
slow in conjunction with mbuma. Need more data.
Latest report is that ips is equally sucking with
and without mbuma.
- Giant leak in NFS code sometimes occurs, can't
reproduce but currently analyzing; brueffer is
able to reproduce but THIS IS NOT an mbuma-specific
problem and currently occurs even WITHOUT mbuma.
- Issues in network locking: there is at least one
code path in the rip code where one or more locks
are acquired and we end up in m_prepend() with
M_WAITOK, which causes WITNESS to whine from within
UMA. Current temporary solution: force all UMA
allocations to be M_NOWAIT from within UMA for now
to avoid deadlocks unless WITNESS is defined and we
can determine with certainty that we're not holding
any locks when we're M_WAITOK.
- I've seen at least one weird socketbuffer empty-but-
mbuf-still-attached panic. I don't believe this
to be related to mbuma but please keep your eyes
open, turn on debugging, and capture crash dumps.
This change removes more code than it adds.
A paper is available detailing the change and considering
various performance issues, it was presented at BSDCan2004:
http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf
Please read the paper for Future Work and implementation
details, as well as credits.
Testing and Debugging:
rwatson,
brueffer,
Ketrien I. Saihr-Kesenchedra,
...
Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
|
|
|
|
|
|
|
LIST_ENTRY(uma_zone) uz_link; /* List of all zones in keg */
|
2013-06-18 04:50:20 +00:00
|
|
|
LIST_HEAD(,uma_bucket) uz_buckets; /* full buckets */
|
2002-03-19 09:11:49 +00:00
|
|
|
|
2009-01-25 09:11:24 +00:00
|
|
|
LIST_HEAD(,uma_klink) uz_kegs; /* List of kegs. */
|
|
|
|
struct uma_klink uz_klink; /* klink for first keg. */
|
|
|
|
|
|
|
|
uma_slaballoc uz_slab; /* Allocate a slab from the backend. */
|
2002-03-19 09:11:49 +00:00
|
|
|
uma_ctor uz_ctor; /* Constructor for each allocation */
|
|
|
|
uma_dtor uz_dtor; /* Destructor */
|
|
|
|
uma_init uz_init; /* Initializer for each item */
|
2013-06-17 03:43:47 +00:00
|
|
|
uma_fini uz_fini; /* Finalizer for each item. */
|
|
|
|
uma_import uz_import; /* Import new memory to cache. */
|
|
|
|
uma_release uz_release; /* Release memory from cache. */
|
|
|
|
void *uz_arg; /* Import/release argument. */
|
Bring in mbuma to replace mballoc.
mbuma is an Mbuf & Cluster allocator built on top of a number of
extensions to the UMA framework, all included herein.
Extensions to UMA worth noting:
- Better layering between slab <-> zone caches; introduce
Keg structure which splits off slab cache away from the
zone structure and allows multiple zones to be stacked
on top of a single Keg (single type of slab cache);
perhaps we should look into defining a subset API on
top of the Keg for special use by malloc(9),
for example.
- UMA_ZONE_REFCNT zones can now be added, and reference
counters automagically allocated for them within the end
of the associated slab structures. uma_find_refcnt()
does a kextract to fetch the slab struct reference from
the underlying page, and lookup the corresponding refcnt.
mbuma things worth noting:
- integrates mbuf & cluster allocations with extended UMA
and provides caches for commonly-allocated items; defines
several zones (two primary, one secondary) and two kegs.
- change up certain code paths that always used to do:
m_get() + m_clget() to instead just use m_getcl() and
try to take advantage of the newly defined secondary
Packet zone.
- netstat(1) and systat(1) quickly hacked up to do basic
stat reporting but additional stats work needs to be
done once some other details within UMA have been taken
care of and it becomes clearer to how stats will work
within the modified framework.
From the user perspective, one implication is that the
NMBCLUSTERS compile-time option is no longer used. The
maximum number of clusters is still capped off according
to maxusers, but it can be made unlimited by setting
the kern.ipc.nmbclusters boot-time tunable to zero.
Work should be done to write an appropriate sysctl
handler allowing dynamic tuning of kern.ipc.nmbclusters
at runtime.
Additional things worth noting/known issues (READ):
- One report of 'ips' (ServeRAID) driver acting really
slow in conjunction with mbuma. Need more data.
Latest report is that ips is equally sucking with
and without mbuma.
- Giant leak in NFS code sometimes occurs, can't
reproduce but currently analyzing; brueffer is
able to reproduce but THIS IS NOT an mbuma-specific
problem and currently occurs even WITHOUT mbuma.
- Issues in network locking: there is at least one
code path in the rip code where one or more locks
are acquired and we end up in m_prepend() with
M_WAITOK, which causes WITNESS to whine from within
UMA. Current temporary solution: force all UMA
allocations to be M_NOWAIT from within UMA for now
to avoid deadlocks unless WITNESS is defined and we
can determine with certainty that we're not holding
any locks when we're M_WAITOK.
- I've seen at least one weird socketbuffer empty-but-
mbuf-still-attached panic. I don't believe this
to be related to mbuma but please keep your eyes
open, turn on debugging, and capture crash dumps.
This change removes more code than it adds.
A paper is available detailing the change and considering
various performance issues, it was presented at BSDCan2004:
http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf
Please read the paper for Future Work and implementation
details, as well as credits.
Testing and Debugging:
rwatson,
brueffer,
Ketrien I. Saihr-Kesenchedra,
...
Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
|
|
|
|
2013-04-09 17:43:48 +00:00
|
|
|
uint32_t uz_flags; /* Flags inherited from kegs */
|
|
|
|
uint32_t uz_size; /* Size inherited from kegs */
|
2010-03-17 21:18:28 +00:00
|
|
|
|
2013-06-17 03:43:47 +00:00
|
|
|
volatile u_long uz_allocs UMA_ALIGN; /* Total number of allocations */
|
|
|
|
volatile u_long uz_fails; /* Total number of alloc failures */
|
|
|
|
volatile u_long uz_frees; /* Total number of frees */
|
2013-04-09 17:43:48 +00:00
|
|
|
uint64_t uz_sleeps; /* Total number of alloc sleeps */
|
2013-11-19 10:05:53 +00:00
|
|
|
uint16_t uz_count; /* Amount of items in full bucket */
|
|
|
|
uint16_t uz_count_min; /* Minimal amount of items there */
|
Bring in mbuma to replace mballoc.
mbuma is an Mbuf & Cluster allocator built on top of a number of
extensions to the UMA framework, all included herein.
Extensions to UMA worth noting:
- Better layering between slab <-> zone caches; introduce
Keg structure which splits off slab cache away from the
zone structure and allows multiple zones to be stacked
on top of a single Keg (single type of slab cache);
perhaps we should look into defining a subset API on
top of the Keg for special use by malloc(9),
for example.
- UMA_ZONE_REFCNT zones can now be added, and reference
counters automagically allocated for them within the end
of the associated slab structures. uma_find_refcnt()
does a kextract to fetch the slab struct reference from
the underlying page, and lookup the corresponding refcnt.
mbuma things worth noting:
- integrates mbuf & cluster allocations with extended UMA
and provides caches for commonly-allocated items; defines
several zones (two primary, one secondary) and two kegs.
- change up certain code paths that always used to do:
m_get() + m_clget() to instead just use m_getcl() and
try to take advantage of the newly defined secondary
Packet zone.
- netstat(1) and systat(1) quickly hacked up to do basic
stat reporting but additional stats work needs to be
done once some other details within UMA have been taken
care of and it becomes clearer to how stats will work
within the modified framework.
From the user perspective, one implication is that the
NMBCLUSTERS compile-time option is no longer used. The
maximum number of clusters is still capped off according
to maxusers, but it can be made unlimited by setting
the kern.ipc.nmbclusters boot-time tunable to zero.
Work should be done to write an appropriate sysctl
handler allowing dynamic tuning of kern.ipc.nmbclusters
at runtime.
Additional things worth noting/known issues (READ):
- One report of 'ips' (ServeRAID) driver acting really
slow in conjunction with mbuma. Need more data.
Latest report is that ips is equally sucking with
and without mbuma.
- Giant leak in NFS code sometimes occurs, can't
reproduce but currently analyzing; brueffer is
able to reproduce but THIS IS NOT an mbuma-specific
problem and currently occurs even WITHOUT mbuma.
- Issues in network locking: there is at least one
code path in the rip code where one or more locks
are acquired and we end up in m_prepend() with
M_WAITOK, which causes WITNESS to whine from within
UMA. Current temporary solution: force all UMA
allocations to be M_NOWAIT from within UMA for now
to avoid deadlocks unless WITNESS is defined and we
can determine with certainty that we're not holding
any locks when we're M_WAITOK.
- I've seen at least one weird socketbuffer empty-but-
mbuf-still-attached panic. I don't believe this
to be related to mbuma but please keep your eyes
open, turn on debugging, and capture crash dumps.
This change removes more code than it adds.
A paper is available detailing the change and considering
various performance issues, it was presented at BSDCan2004:
http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf
Please read the paper for Future Work and implementation
details, as well as credits.
Testing and Debugging:
rwatson,
brueffer,
Ketrien I. Saihr-Kesenchedra,
...
Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
|
|
|
|
2015-12-20 02:05:33 +00:00
|
|
|
/* The next two fields are used to print a rate-limited warnings. */
|
2012-12-07 22:27:13 +00:00
|
|
|
const char *uz_warning; /* Warning to print on failure */
|
|
|
|
struct timeval uz_ratecheck; /* Warnings rate-limiting */
|
|
|
|
|
2016-02-03 23:30:17 +00:00
|
|
|
struct task uz_maxaction; /* Task to run when at limit */
|
2015-12-20 02:05:33 +00:00
|
|
|
|
2002-03-19 09:11:49 +00:00
|
|
|
/*
|
|
|
|
* This HAS to be the last item because we adjust the zone size
|
|
|
|
* based on NCPU and then allocate the space for the zones.
|
|
|
|
*/
|
2015-04-24 17:03:53 +00:00
|
|
|
struct uma_cache uz_cpu[1]; /* Per cpu caches */
|
2002-03-19 09:11:49 +00:00
|
|
|
};
|
|
|
|
|
2003-09-19 08:37:44 +00:00
|
|
|
/*
|
|
|
|
* These flags must not overlap with the UMA_ZONE flags specified in uma.h.
|
|
|
|
*/
|
2009-01-25 09:11:24 +00:00
|
|
|
#define UMA_ZFLAG_MULTI 0x04000000 /* Multiple kegs in the zone. */
|
|
|
|
#define UMA_ZFLAG_DRAINING 0x08000000 /* Running zone_drain. */
|
2013-06-26 00:57:38 +00:00
|
|
|
#define UMA_ZFLAG_BUCKET 0x10000000 /* Bucket zone. */
|
2005-07-16 02:23:41 +00:00
|
|
|
#define UMA_ZFLAG_INTERNAL 0x20000000 /* No offpage no PCPU. */
|
|
|
|
#define UMA_ZFLAG_FULL 0x40000000 /* Reached uz_maxpages */
|
|
|
|
#define UMA_ZFLAG_CACHEONLY 0x80000000 /* Don't ask VM for buckets. */
|
2002-03-19 09:11:49 +00:00
|
|
|
|
2013-06-26 00:57:38 +00:00
|
|
|
#define UMA_ZFLAG_INHERIT \
|
|
|
|
(UMA_ZFLAG_INTERNAL | UMA_ZFLAG_CACHEONLY | UMA_ZFLAG_BUCKET)
|
2009-01-25 09:11:24 +00:00
|
|
|
|
2013-06-17 03:43:47 +00:00
|
|
|
static inline uma_keg_t
|
|
|
|
zone_first_keg(uma_zone_t zone)
|
|
|
|
{
|
2013-06-20 19:08:12 +00:00
|
|
|
uma_klink_t klink;
|
2013-06-17 03:43:47 +00:00
|
|
|
|
2013-06-20 19:08:12 +00:00
|
|
|
klink = LIST_FIRST(&zone->uz_kegs);
|
|
|
|
return (klink != NULL) ? klink->kl_keg : NULL;
|
2013-06-17 03:43:47 +00:00
|
|
|
}
|
|
|
|
|
2010-03-17 21:18:28 +00:00
|
|
|
#undef UMA_ALIGN
|
|
|
|
|
2005-08-04 10:03:53 +00:00
|
|
|
#ifdef _KERNEL
|
2002-03-19 09:11:49 +00:00
|
|
|
/* Internal prototypes */
|
2013-04-09 17:43:48 +00:00
|
|
|
static __inline uma_slab_t hash_sfind(struct uma_hash *hash, uint8_t *data);
|
2015-04-01 12:42:26 +00:00
|
|
|
void *uma_large_malloc(vm_size_t size, int wait);
|
2002-03-19 09:11:49 +00:00
|
|
|
void uma_large_free(uma_slab_t slab);
|
|
|
|
|
|
|
|
/* Lock Macros */
|
|
|
|
|
2009-01-25 09:11:24 +00:00
|
|
|
#define KEG_LOCK_INIT(k, lc) \
|
2002-04-29 23:45:41 +00:00
|
|
|
do { \
|
|
|
|
if ((lc)) \
|
2009-01-25 09:11:24 +00:00
|
|
|
mtx_init(&(k)->uk_lock, (k)->uk_name, \
|
|
|
|
(k)->uk_name, MTX_DEF | MTX_DUPOK); \
|
2002-04-29 23:45:41 +00:00
|
|
|
else \
|
2009-01-25 09:11:24 +00:00
|
|
|
mtx_init(&(k)->uk_lock, (k)->uk_name, \
|
2002-04-29 23:45:41 +00:00
|
|
|
"UMA zone", MTX_DEF | MTX_DUPOK); \
|
|
|
|
} while (0)
|
2013-06-20 19:08:12 +00:00
|
|
|
|
2009-01-25 09:11:24 +00:00
|
|
|
#define KEG_LOCK_FINI(k) mtx_destroy(&(k)->uk_lock)
|
|
|
|
#define KEG_LOCK(k) mtx_lock(&(k)->uk_lock)
|
|
|
|
#define KEG_UNLOCK(k) mtx_unlock(&(k)->uk_lock)
|
2013-06-20 19:08:12 +00:00
|
|
|
|
|
|
|
#define ZONE_LOCK_INIT(z, lc) \
|
|
|
|
do { \
|
|
|
|
if ((lc)) \
|
|
|
|
mtx_init(&(z)->uz_lock, (z)->uz_name, \
|
|
|
|
(z)->uz_name, MTX_DEF | MTX_DUPOK); \
|
|
|
|
else \
|
|
|
|
mtx_init(&(z)->uz_lock, (z)->uz_name, \
|
|
|
|
"UMA zone", MTX_DEF | MTX_DUPOK); \
|
|
|
|
} while (0)
|
|
|
|
|
|
|
|
#define ZONE_LOCK(z) mtx_lock((z)->uz_lockptr)
|
|
|
|
#define ZONE_TRYLOCK(z) mtx_trylock((z)->uz_lockptr)
|
|
|
|
#define ZONE_UNLOCK(z) mtx_unlock((z)->uz_lockptr)
|
|
|
|
#define ZONE_LOCK_FINI(z) mtx_destroy(&(z)->uz_lock)
|
2002-03-19 09:11:49 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Find a slab within a hash table. This is used for OFFPAGE zones to lookup
|
|
|
|
* the slab structure.
|
|
|
|
*
|
|
|
|
* Arguments:
|
|
|
|
* hash The hash table to search.
|
|
|
|
* data The base page of the item.
|
|
|
|
*
|
|
|
|
* Returns:
|
|
|
|
* A pointer to a slab if successful, else NULL.
|
|
|
|
*/
|
|
|
|
static __inline uma_slab_t
|
2013-04-09 17:43:48 +00:00
|
|
|
hash_sfind(struct uma_hash *hash, uint8_t *data)
|
2002-03-19 09:11:49 +00:00
|
|
|
{
|
|
|
|
uma_slab_t slab;
|
|
|
|
int hval;
|
|
|
|
|
|
|
|
hval = UMA_HASH(hash, data);
|
|
|
|
|
|
|
|
SLIST_FOREACH(slab, &hash->uh_slab_hash[hval], us_hlink) {
|
2013-04-09 17:43:48 +00:00
|
|
|
if ((uint8_t *)slab->us_data == data)
|
2002-03-19 09:11:49 +00:00
|
|
|
return (slab);
|
|
|
|
}
|
|
|
|
return (NULL);
|
|
|
|
}
|
|
|
|
|
2002-09-18 08:26:30 +00:00
|
|
|
static __inline uma_slab_t
|
|
|
|
vtoslab(vm_offset_t va)
|
|
|
|
{
|
|
|
|
vm_page_t p;
|
|
|
|
|
|
|
|
p = PHYS_TO_VM_PAGE(pmap_kextract(va));
|
2013-09-17 07:35:26 +00:00
|
|
|
return ((uma_slab_t)p->plinks.s.pv);
|
2002-09-18 08:26:30 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
static __inline void
|
|
|
|
vsetslab(vm_offset_t va, uma_slab_t slab)
|
|
|
|
{
|
|
|
|
vm_page_t p;
|
|
|
|
|
2004-11-26 15:04:26 +00:00
|
|
|
p = PHYS_TO_VM_PAGE(pmap_kextract(va));
|
2013-08-10 17:36:42 +00:00
|
|
|
p->plinks.s.pv = slab;
|
2002-09-18 08:26:30 +00:00
|
|
|
}
|
|
|
|
|
2002-11-01 01:01:27 +00:00
|
|
|
/*
|
|
|
|
* The following two functions may be defined by architecture specific code
|
|
|
|
* if they can provide more effecient allocation functions. This is useful
|
|
|
|
* for using direct mapped addresses.
|
|
|
|
*/
|
2015-04-01 12:42:26 +00:00
|
|
|
void *uma_small_alloc(uma_zone_t zone, vm_size_t bytes, uint8_t *pflag,
|
|
|
|
int wait);
|
|
|
|
void uma_small_free(void *mem, vm_size_t size, uint8_t flags);
|
2005-08-04 10:03:53 +00:00
|
|
|
#endif /* _KERNEL */
|
2002-11-01 01:01:27 +00:00
|
|
|
|
2002-03-19 09:11:49 +00:00
|
|
|
#endif /* VM_UMA_INT_H */
|