d/freebsd-nq
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
85ec5cbae2
freebsd-nq/module/zfs
History
Matthew Ahrens 85ec5cbae2
Include scatter_chunk_waste in arc_size 
The ARC caches data in scatter ABD's, which are collections of pages,
which are typically 4K.  Therefore, the space used to cache each block
is rounded up to a multiple of 4K.  The ABD subsystem tracks this wasted
memory in the `scatter_chunk_waste` kstat.  However, the ARC's `size` is
not aware of the memory used by this round-up, it only accounts for the
size that it requested from the ABD subsystem.

Therefore, the ARC is effectively using more memory than it is aware of,
due to the `scatter_chunk_waste`.  This impacts observability, e.g.
`arcstat` will show that the ARC is using less memory than it
effectively is.  It also impacts how the ARC responds to memory
pressure.  As the amount of `scatter_chunk_waste` changes, it appears to
the ARC as memory pressure, so it needs to resize `arc_c`.

If the sector size (`1<<ashift`) is the same as the page size (or
larger), there won't be any waste.  If the (compressed) block size is
relatively large compared to the page size, the amount of
`scatter_chunk_waste` will be small, so the problematic effects are
minimal.

However, if using 512B sectors (`ashift=9`), and the (compressed) block
size is small (e.g. `compression=on` with the default `volblocksize=8k`
or a decreased `recordsize`), the amount of `scatter_chunk_waste` can be
very large.  On a production system, with `arc_size` at a constant 50%
of memory, `scatter_chunk_waste` has been been observed to be 10-30% of
memory.

This commit adds `scatter_chunk_waste` to `arc_size`, and adds a new
`waste` field to `arcstat`.  As a result, the ARC's memory usage is more
observable, and `arc_c` does not need to be adjusted as frequently.

Reviewed-by: Pavel Zakharov <pavel.zakharov@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Wilson <gwilson@delphix.com>
Reviewed-by: Ryan Moeller <ryan@iXsystems.com>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Closes #10701
2020-08-17 20:04:04 -07:00
..
abd.c Add gang ABD child to parent gang ABD 2020-07-24 21:09:20 -07:00
aggsum.c Reduce number of atomic_add() calls in aggsum 2020-02-06 13:21:06 -08:00
arc.c Include scatter_chunk_waste in arc_size 2020-08-17 20:04:04 -07:00
blkptr.c Add include files for prototypes 2020-06-18 12:21:25 -07:00
bplist.c
bpobj.c Rename refcount.h to zfs_refcount.h 2020-07-29 16:35:33 -07:00
bptree.c Rename refcount.h to zfs_refcount.h 2020-07-29 16:35:33 -07:00
bqueue.c
btree.c Fix typo in btree.c 2020-08-17 15:25:37 -07:00
dataset_kstats.c
dbuf_stats.c Mark functions as static 2020-06-18 12:20:38 -07:00
dbuf.c Make use of ZFS_DEBUG consistent within kmod sources 2020-07-25 20:07:44 -07:00
ddt_zap.c Refactor dnode dirty context from dbuf_dirty 2020-02-26 16:09:17 -08:00
ddt.c Remove dead code 2020-06-18 12:21:18 -07:00
dmu_diff.c
dmu_object.c Add include files for prototypes 2020-06-18 12:21:25 -07:00
dmu_objset.c Remove duplicate include of sys/zfeature.h in dmu_objset.c 2020-07-31 09:04:45 -07:00
dmu_recv.c filesystem_limit/snapshot_limit is incorrectly enforced against root 2020-07-11 17:18:02 -07:00
dmu_redact.c dmu_objset_from_ds must be called with dp_config_rwlock held 2020-03-12 10:55:02 -07:00
dmu_send.c Replace sprintf()->snprintf() and strcpy()->strlcpy() 2020-06-07 11:42:12 -07:00
dmu_traverse.c
dmu_tx.c Refactor dnode dirty context from dbuf_dirty 2020-02-26 16:09:17 -08:00
dmu_zfetch.c
dmu.c Mark functions as static 2020-06-18 12:20:38 -07:00
dnode_sync.c Make use of ZFS_DEBUG consistent within kmod sources 2020-07-25 20:07:44 -07:00
dnode.c Prevent race condition in dnode_dest (#10101) 2020-03-12 10:25:56 -07:00
dsl_bookmark.c Fix typos 2020-06-09 21:24:09 -07:00
dsl_crypt.c Mark functions as static 2020-06-18 12:20:38 -07:00
dsl_dataset.c FreeBSD: fix the build with Clang 11 2020-08-17 15:40:17 -07:00
dsl_deadlist.c Fix i/o error handling of livelists and zap iteration 2020-08-05 10:22:09 -07:00
dsl_deleg.c
dsl_destroy.c Fix i/o error handling of livelists and zap iteration 2020-08-05 10:22:09 -07:00
dsl_dir.c Fix i/o error handling of livelists and zap iteration 2020-08-05 10:22:09 -07:00
dsl_pool.c Mark functions as static 2020-06-18 12:20:38 -07:00
dsl_prop.c Replace sprintf()->snprintf() and strcpy()->strlcpy() 2020-06-07 11:42:12 -07:00
dsl_scan.c Add device rebuild feature 2020-07-03 11:05:50 -07:00
dsl_synctask.c
dsl_userhold.c Replace sprintf()->snprintf() and strcpy()->strlcpy() 2020-06-07 11:42:12 -07:00
edonr_zfs.c Add include files for prototypes 2020-06-18 12:21:25 -07:00
fm.c Enable zpool events tunables and tests on FreeBSD 2020-02-18 11:22:56 -08:00
gzip.c Add include files for prototypes 2020-06-18 12:21:25 -07:00
hkdf.c
lz4.c Prefix zfs internal endian checks with _ZFS 2020-07-28 13:02:49 -07:00
lzjb.c Add include files for prototypes 2020-06-18 12:21:25 -07:00
Makefile.in Add device rebuild feature 2020-07-03 11:05:50 -07:00
metaslab.c Use zfs_dbgmsg to log metaslab_load/unload 2020-08-12 10:10:50 -07:00
mmp.c Add zfs_multihost_interval tunable handler for FreeBSD 2020-06-23 13:32:42 -07:00
multilist.c Make use of ZFS_DEBUG consistent within kmod sources 2020-07-25 20:07:44 -07:00
objlist.c
pathname.c
range_tree.c Clarify error message when a range-tree double-add occurs 2020-08-07 14:13:13 -07:00
refcount.c Rename refcount.h to zfs_refcount.h 2020-07-29 16:35:33 -07:00
rrwlock.c Rename refcount.h to zfs_refcount.h 2020-07-29 16:35:33 -07:00
sa.c Mark functions as static 2020-06-18 12:20:38 -07:00
sha256.c Add include files for prototypes 2020-06-18 12:21:25 -07:00
skein_zfs.c Add include files for prototypes 2020-06-18 12:21:25 -07:00
spa_boot.c Add include files for prototypes 2020-06-18 12:21:25 -07:00
spa_checkpoint.c Refactor dnode dirty context from dbuf_dirty 2020-02-26 16:09:17 -08:00
spa_config.c FreeBSD: fallback to /boot/ to look for zpool.cache 2020-08-17 14:43:47 -07:00
spa_errlog.c
spa_history.c Make spa_history_zone platform-dependent in kernel 2020-03-02 09:43:30 -08:00
spa_log_spacemap.c
spa_misc.c Add device rebuild feature 2020-07-03 11:05:50 -07:00
spa.c Fix i/o error handling of livelists and zap iteration 2020-08-05 10:22:09 -07:00
space_map.c Rename refcount.h to zfs_refcount.h 2020-07-29 16:35:33 -07:00
space_reftree.c
THIRDPARTYLICENSE.cityhash
THIRDPARTYLICENSE.cityhash.descrip
txg.c Use boot_ncpus in place of max_ncpus in taskq_create 2020-05-20 10:07:21 -07:00
uberblock.c
unique.c
vdev_cache.c Replace ASSERTV macro with compiler annotation 2019-12-05 12:37:00 -08:00
vdev_indirect_births.c
vdev_indirect_mapping.c Replace ASSERTV macro with compiler annotation 2019-12-05 12:37:00 -08:00
vdev_indirect.c Rename refcount.h to zfs_refcount.h 2020-07-29 16:35:33 -07:00
vdev_initialize.c Rename refcount.h to zfs_refcount.h 2020-07-29 16:35:33 -07:00
vdev_label.c Add device rebuild feature 2020-07-03 11:05:50 -07:00
vdev_mirror.c Add device rebuild feature 2020-07-03 11:05:50 -07:00
vdev_missing.c
vdev_queue.c Add device rebuild feature 2020-07-03 11:05:50 -07:00
vdev_raidz_math_aarch64_neon_common.h FreeBSD: fix the build with Clang 11 2020-08-17 15:40:17 -07:00
vdev_raidz_math_aarch64_neon.c
vdev_raidz_math_aarch64_neonx2.c
vdev_raidz_math_avx2.c FreeBSD: fix the build with Clang 11 2020-08-17 15:40:17 -07:00
vdev_raidz_math_avx512bw.c Refactor ccompile.h to not include system headers 2020-07-25 20:09:50 -07:00
vdev_raidz_math_avx512f.c FreeBSD: fix the build with Clang 11 2020-08-17 15:40:17 -07:00
vdev_raidz_math_impl.h Fix const-correctness in raidz math 2020-02-03 10:52:41 -08:00
vdev_raidz_math_powerpc_altivec_common.h FreeBSD: fix the build with Clang 11 2020-08-17 15:40:17 -07:00
vdev_raidz_math_powerpc_altivec.c Prefix zfs internal endian checks with _ZFS 2020-07-28 13:02:49 -07:00
vdev_raidz_math_scalar.c
vdev_raidz_math_sse2.c FreeBSD: fix the build with Clang 11 2020-08-17 15:40:17 -07:00
vdev_raidz_math_ssse3.c Refactor ccompile.h to not include system headers 2020-07-25 20:09:50 -07:00
vdev_raidz_math.c FreeBSD: Fixes required to build ZFS on PowerPC 2020-07-25 11:00:23 -07:00
vdev_raidz.c Fix typos 2020-06-09 21:24:09 -07:00
vdev_rebuild.c Add missed thread_exit() to vdev_{autotrim,rebuild}_thread 2020-08-05 10:17:07 -07:00
vdev_removal.c Trim L2ARC 2020-06-09 10:15:08 -07:00
vdev_root.c Enable splitting mirrors with indirect vdevs 2020-05-06 10:32:28 -07:00
vdev_trim.c Add missed thread_exit() to vdev_{autotrim,rebuild}_thread 2020-08-05 10:17:07 -07:00
vdev.c Fix error handling of vdev_top_zap 2020-07-29 17:04:34 -07:00
zap_leaf.c Refactor dnode dirty context from dbuf_dirty 2020-02-26 16:09:17 -08:00
zap_micro.c Rename refcount.h to zfs_refcount.h 2020-07-29 16:35:33 -07:00
zap.c Rename refcount.h to zfs_refcount.h 2020-07-29 16:35:33 -07:00
zcp_get.c Add include files for prototypes 2020-06-18 12:21:25 -07:00
zcp_global.c
zcp_iter.c
zcp_set.c Support setting user properties in a channel program 2020-02-14 13:41:42 -08:00
zcp_synctask.c filesystem_limit/snapshot_limit is incorrectly enforced against root 2020-07-11 17:18:02 -07:00
zcp.c filesystem_limit/snapshot_limit is incorrectly enforced against root 2020-07-11 17:18:02 -07:00
zfeature.c Replace ASSERTV macro with compiler annotation 2019-12-05 12:37:00 -08:00
zfs_byteswap.c Mark functions as static 2020-06-18 12:20:38 -07:00
zfs_fm.c
zfs_fuid.c Rename refcount.h to zfs_refcount.h 2020-07-29 16:35:33 -07:00
zfs_ioctl.c Release onexit/events with any missed zfsdev_state 2020-08-13 15:03:23 -07:00
zfs_log.c Add prototypes 2020-06-18 12:21:32 -07:00
zfs_onexit.c Remove deduplicated send/receive code 2020-04-23 10:06:57 -07:00
zfs_quota.c File incorrectly zeroed when receiving incremental stream that toggles -L 2020-06-09 10:41:01 -07:00
zfs_ratelimit.c
zfs_replay.c Simplify FreeBSD's locking requirements in zfs_replay.c 2020-01-22 17:55:56 -08:00
zfs_rlock.c Add a "try" operation for range locks 2020-07-06 11:53:31 -07:00
zfs_sa.c Add convenience wrappers for common uio usage 2020-06-14 10:09:55 -07:00
zil.c Mark functions as static 2020-06-18 12:20:38 -07:00
zio_checksum.c Mark functions as static 2020-06-18 12:20:38 -07:00
zio_compress.c zio_decompress_data always ASSERTs successful decompression 2019-12-10 15:51:58 -08:00
zio_inject.c Replace ASSERTV macro with compiler annotation 2019-12-05 12:37:00 -08:00
zio.c Mark functions as static 2020-06-18 12:20:38 -07:00
zle.c Add include files for prototypes 2020-06-18 12:21:25 -07:00
zrlock.c Remove dead code 2020-06-18 12:21:18 -07:00
zthr.c Introduce names for ZTHRs 2020-07-29 09:43:33 -07:00
zvol.c Fix typos 2020-06-09 21:24:09 -07:00