Similar to previous change, the ** paradigm is a bit
problematic for asynchronous routines that could fail.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ife7748280482356c4c51a796817b71cd7bc7e479
Reviewed-on: https://review.gerrithub.io/391483
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Using the ** paradigm is a bit problematic for asynchronous
routines that could fail. Currently we were inconsistent in
that some error paths would zero the pointer while others
did not. So make this just a plain pointer, which simplifies
the API and its implementation.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I67147931c6e8350896a4505022a6a314655de3d3
Reviewed-on: https://review.gerrithub.io/391482
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
As part of clarifying the API and preparing for some
future changes, rename the following functions:
spdk_bs_md_create_blob => spdk_bs_create_blob
spdk_bs_md_open_blob => spdk_bs_open_blob
spdk_bs_md_delete_blob => spdk_bs_delete_blob
spdk_bs_md_iter_first => spdk_bs_iter_first
spdk_bs_md_iter_next => spdk_bs_iter_next
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I86bf792717b68379484a6108396bb891fe1c221e
Reviewed-on: https://review.gerrithub.io/391031
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Require braces around all conditional statements, e.g.:
if (cond)
statement();
becomes:
if (cond) {
statement();
}
This is the style used through most of the SPDK code, but several
exceptions crept in over time. Add the astyle option to make sure we
are consistent.
Change-Id: I5a71980147fe8dfb471ff42e8bc06db2124a1a7f
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/390914
Reviewed-by: <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Disambiguate the log components from the trace functionality
(include/spdk/trace.h).
The internal spdk_trace_flag structure and related functions will be
renamed in a later commit - this is just a find and replace on
SPDK_TRACE_* and SPDK_LOG_REGISTER_TRACE_FLAG().
Change-Id: I617bd5a9fbe35ffb44ae6020b292658c094a0ad6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/376421
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This patch make the cache buffer shift of blobfs configurable. We
can configure the cache buffer shift according workload, it will
help to improve the performance.
Change-Id: Ied1c2e5d6fd9eaa7aa0759c03c654fcf2e77aa23
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
Reviewed-on: https://review.gerrithub.io/383381
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Add newlines to the debug logs.
Also dump the mismatched bstype via SPDK_TRACEDUMP() rather than as a
string, since it may not be null terminated.
Change-Id: Icad567373c56f24aea550903a1370c3734465472
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/382855
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Rather than using char arrays to represent blobstore types, just use the
publicly-defined struct spdk_bs_type.
Change-Id: I8bbfd95085e16f9d53106c176038e067ff1266d7
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/382853
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Introducing bstype as a way to identify and verify
blobstore type.
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: I50267b5408625be10fe0c146ae329016d5509b4a
Reviewed-on: https://review.gerrithub.io/380476
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Mark the file as deleted in the function spdk_fs_delete_file()
when the referance is not 0, and delete the file when it is closed,
make the behaviour as "unlink".
Change-Id: Ia934bb73c82c48fdbab79dbe4b56296a73abc01e
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
Reviewed-on: https://review.gerrithub.io/374944
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
This matches the name to the behavior and prepares for addition of a new
log macro for "info" log level.
Change-Id: I94ccd49face4309d3368e399528776ab140748c4
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/375833
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This is just a convenience and replaces the common practice
of passing -1.
Change-Id: Id96734307ebf52ef0ee7dba0e7ac89602b2b5b1a
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/374520
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I4797e3900eb425f8cddec1543a4612d4d98e07a6
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/372990
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
From the exising code, the comparison should + 1.
Suppose offset = 2^19, we still can be fit into
tree with level =0, since there will be 64 buffer,
each with size 2^18, so we do not need to
increse the tree level.
Also a unit test is added to demonstrate this.
Change-Id: I95d3542b0881aa7bb661bc57bc789cc4ef4e7509
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/372396
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
The code can be merged into cache_free_buffers
Change-Id: I5da59ae5c5ea2b91d4aecd15a611ad6731bd4c56
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/372993
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Change-Id: I9134ffcb9713e0613de61d3178940390291dc287
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/372621
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I0705e9f203e3cb905bf2e6a7bfc72bbb424a6180
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/372644
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Some macros depend on others, so we change to
change it and it will avoid potential wrong change
only one value
Change-Id: I2490064a19adc5cbd03a2c231d5f14fbf5b95792
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/372217
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
we will retry the alloc_cache_memory_buffer outside the function if
it fail to allocate one cache buffer.
Change-Id: I276173e449e700ceb7703f68828153f566249ae2
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
Reviewed-on: https://review.gerrithub.io/371140
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Patch afe860ae deferred freeing the io_device. However, for nvme, the
io_device context (spdk_nvme_ctrlr) is still being destructed before
io_channels are destroyed, causing segfaults on hotremove.
This patch defers io_device context destruction and fixes nvme
hotremove.
Fixes: afe860aeb1 ("channel: Correctly defer unregisters if channels exist")
Fixes: 5533c3d208 ("util: defer put_io_channel")
Change-Id: I7af699174cac0c6c6a6faa2cc65418c47347eb9a
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/370459
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
If the spdk_filesystem object is created but the initial
request allocation fails, correctly clean up. This
addresses a static analysis failure.
Change-Id: If47a380d8804a279ec92d2aee3de7d86bb5c67fa
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/369901
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
During RocksDB testing with MySQL, we found cases
where blobfs would try to update and sync the length
xattr on the underlying blob while an existing update
was already in progress. This was primarily driven
by RocksDB performing appends and syncs on the log
file from multiple writer threads.
The simplest way to fix this is to just process one
sync_request at a time. There could be a tiny bit
of inefficiency here if multiple threads are appending
and syncing a file in parallel - we can look at some
additional optimizations if we find a case where that
is noticeable.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Reported-by: Changpeng Liu <changpeng.liu@intel.com>
Tested-by: Changpeng Liu <changpeng.liu@intel.com>
Change-Id: I7ab7814494d365bae8716efd0b828337286cc7b7
Reviewed-on: https://review.gerrithub.io/369490
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
If a file is extended and then the new region is immediately
read without being written, there as a math error in the
buffer cache.
Change-Id: Ibd2bbe98c734f98df43eada799ed62de4081964b
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/367265
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The math works out so that index is always in range here, but add an
assert anyway to catch potential mistakes in the future and to placate
scan-build.
Change-Id: I09ad37dde56bc315543b3c142d91ca9a49e9b32c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/364530
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Synchronous blobfs channels may allocate requests from
the synchronous thread, but free the request from the
async thread - especially for code that is shared between
sync and async modes. So add a lock to the request list
for sync channels only.
Note this only affects metadata path (i.e. sync, delete, open)
and not I/O path.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I932c500807c2f459c697fab2ffd91a88b88b0c87
Reviewed-on: https://review.gerrithub.io/362964
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
The file's lock does not need to be held across
calls to this function as the file is not dereferenced
at all.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Idf0a52f2528521f12e45963bb0ab1f414b37380f
Reviewed-on: https://review.gerrithub.io/363138
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
- rename spdk_malloc_socket to spdk_dma_malloc_socket
- rename spdk_malloc to spdk_dma_malloc
- rename spdk_zmalloc to spdk_dma_zmalloc
- rename spdk_realloc to spdk_dma_realloc
- rename spdk_free to spdk_dma_free
Change-Id: I52a11b7a4243281f9c56f503e826fd7c4a1fd883
Signed-off-by: John Meneghini <johnm@netapp.com>
Reviewed-on: https://review.gerrithub.io/362604
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This wasn't used anywhere and we currently believe there
are superior software-only techniques for controlling
quality of service.
Change-Id: Icdadd5870ed0629b338c307d2619bbc242c3e7a3
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/362065
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Keep a global counter of the number of opened
blobfs instances. Allocate the cache pool when the
first instance is opened, and free the cache pool
when the last instance closes.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I99db2b587b738badcd2a54c9f5dc6ec8370ce22c
Reviewed-on: https://review.gerrithub.io/362606
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This is no longer used anywhere. For the places where we previously
used it, we've since found alternate solutions that do not
require it.
Change-Id: I738a80b95ef50348ce1c14969a3812b0a625b3fd
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/362064
Tested-by: <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This isn't strictly necessary, but follow the
pattern for sync and I/O devices.
Change-Id: I34b1bd6f9da06008e419f55ee9810022f5fed3d5
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/361674
Tested-by: <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This allows us to no longer use the unique=true flag as well
as assign sync channels different queue depths.
Change-Id: I0b3129a3139bf39d745c2a4dc1d9f402c0a8faa8
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/361673
Tested-by: <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This allows us to have a different queue depth
on I/O channels than on metadata/sync channels
Change-Id: I30dfc95af550082efffec4b12ed6c4cd4b8c11cf
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/361672
Tested-by: <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This temporarily increases the size of the sync channel
to 512 (like all of the other channels). This will be
fixed in a later patch.
Change-Id: I679f1cb7c9f7afdad0acf9a3a26f9632f60cc510
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/361670
Tested-by: <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The channel memory isn't allocated by these
libraries, so they can't free it.
Change-Id: I30909fa4e77bc5a41b45230f04ba5fe75b172dbf
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This will need to be configured globally for all channels.
Change-Id: I773252f220373617f8d09d1f24243db8095cf8a4
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This patch address the issue:
https://github.com/spdk/spdk/issues/151.
For cache_append_no_cache in cache_ut testcase,
there is resource contention for buffer among two
threads in the following two functions.
Thread 0: cache_free_buffers
Thread1: __file_flush_done
When the thread1 execuctes __file_flush_done,
it calls the call back: __sem_post defined in
following statement in spdk_file_sync
_file_sync(file, channel, __sem_post, &channel->sem);
Thus Thread 0 will execute next function
cache_buffers, and it frees the buffer.
Then Thread 1 continues executing the remaining statements
in __file_flush_done with the assert function, and touches
the space already freed.
So it will be safe to move ahead the next buffer check.
Change-Id: Ic007b3481f4e3a17d47eeca5c9c802001949a5ab
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
The code will locks clear to put those definitions into tree.h header
Change-Id: Ib1a34f19d9849acd7ea979eb0a6e153b0e8e39de
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
This avoids corner case where a buffer gets allocated on the 100th
try.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: If65053d539d458d9a53c8850bbb4cbe4ee84f604
This is the initial commit for "blobfs", a lightweight
filesystem built on top of the SPDK blobstore.
Also included in this patch:
1) a shim for using SPDK bdevs as the backing store for
SPDK blobstore/blobfs
2) documentation for using blobfs as the storage engine
with RocksDB
3) scripts for running a set of workloads and collecting
profiling data with RocksDB and blobfs
See doc/blobfs/getting_started.md included in this commit
for more details on blobfs, including some of the current
limitations.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I2a6d3d4b87236730051228ed62c0c04e04c42c73
