freebsd with flexible iflib nic queues
08d08ebba2
As described in Issue #458 and #258, unlinking large amounts of data can cause the threads in the zio free wait queue to start spinning. Reducing the number of z_fr_iss threads from a fixed value of 100 to 1 per cpu signficantly reduces contention on the taskq spinlock and improves throughput. Instrumenting the taskq code showed that __taskq_dispatch() can spend a long time holding tq->tq_lock if there are a large number of threads in the queue. It turns out the time spent in wake_up() scales linearly with the number of threads in the queue. When a large number of short work items are dispatched, as seems to be the case with unlink, the worker threads drain the queue faster than the dispatcher can fill it. They then all pile into the work wait queue to wait for new work items. So if 100 threads are in the queue, wake_up() takes about 100 times as long, and the woken threads have to spin until the dispatcher releases the lock. Reducing the number of threads helps with the symptoms, but doesn't get to the root of the problem. It would seem that wake_up() shouldn't scale linearly in time with queue depth, particularly if we are only trying to wake up one thread. In that vein, I tried making all of the waiting processes exclusive to prevent the scheduler from iterating over the entire list, but I still saw the linear time scaling. So further investigation is needed, but in the meantime reducing the thread count is an easy workaround. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue #258 Issue #458 |
||
|---|---|---|
| cmd | ||
| config | ||
| dracut | ||
| etc | ||
| include | ||
| lib | ||
| man | ||
| module | ||
| patches | ||
| scripts | ||
| udev | ||
| .gitignore | ||
| AUTHORS | ||
| autogen.sh | ||
| ChangeLog | ||
| configure | ||
| configure.ac | ||
| COPYING | ||
| COPYRIGHT | ||
| DISCLAIMER | ||
| Makefile.am | ||
| Makefile.in | ||
| META | ||
| OPENSOLARIS.LICENSE | ||
| PKGBUILD-zfs-modules.in | ||
| PKGBUILD-zfs.in | ||
| README.markdown | ||
| zfs_config.h.in | ||
| zfs-modules.spec.in | ||
| zfs-script-config.sh.in | ||
| ZFS.RELEASE | ||
| zfs.spec.in | ||
Native ZFS for Linux! ZFS is an advanced file system and volume manager which was originally developed for Solaris. It has been successfully ported to FreeBSD and now there is a functional Linux ZFS kernel port too. The port currently includes a fully functional and stable SPA, DMU, and ZVOL with a ZFS Posix Layer (ZPL) on the way!
$ ./configure
$ make pkg
Full documentation for building, configuring, and using ZFS can be found at: http://zfsonlinux.org