This also means that the RocksDB threads don't need
to be SPDK threads any more.
Change-Id: Icadd6dd446958ebf470ad8ab8239f5390942eb87
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449465
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
In the next step, this won't be a channel at all.
Change-Id: Ia8fe4da5b0b283e8dfc5c6477b84cfdd346d89a0
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449464
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
spdk_dma_*malloc() is about to be deprecated.
Change-Id: Ib9fb94463a69ffd9660ab2c7fbe02826a7e06741
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449796
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Our blobfs allocates fixed number of reqs on three different channels
(the default size is 512), if the req cannot be allocated, our blobfs
related function will do nothing.
With this debug info added, we can easily know the issues.
If the fixed number of requests are not enough, we need to change the
design, using malloc/free together with the fixed number of requests.
During high presure test, I think that if the reqs are not allocated,
it will cause the sync, flush related issues.
Change-Id: Ied8cd5197c4f87b861bd7dc09edd5a095fdcc802
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448596
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reason: We need to detect this issue. Otherwise, it will
cause null pointer issue later
Change-Id: I4d9aec0570e0b46274ebf3b9642b4727049a4a3b
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448594
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
The caller function needs to judge whether the buf is
allocated. Putting assert here is easy for debugging,
but not be friendly for real use.
Change-Id: I3c8e66bc95cbd2f9bc87f26335f249c32e339da4
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448697
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
In function spdk_fs_file_stat_async, the
stat.size = f->append_pos >= f->length ? f->append_pos : f->length;
but in spdk_file_get_length, we return f->length.
So generally, it should all use the same method to return the file
length, and this patch will fix this issue.
Change-Id: Idb9aa9e737711fcd48ac0075c7f7ffed825fe3b0
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/c/439627
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Previously the alignment was to 4k due to page size of blobstore being minmal
size to write to blobs. Now that io_unit is used, alignment can be changed to that.
Change-Id: Ib48182e299dad9d9587ae20b97afe683d53933cb
Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.gerrithub.io/425788
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Change-Id: I40d8532494f75f8175aa4730e723c4d0905a486c
Reviewed-on: https://review.gerrithub.io/425356
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This is a string name used for debugging only.
Change-Id: I9827f0e6c83be7bc13951c7b5f0951ce6c2a1ece
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/424127
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Change-Id: I066085a0606d64a0d95ab2d28340aa35d83efdf7
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/423504
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Change-Id: I9fe730e5a34608dbc7def3a5c19a891fbcc0d9e9
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/423195
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
When testing blobfs, the allocation of cache buffer may fail
without any error information, users may get segment fault
error after that.
Change-Id: I6dc8ee4b93c6a9109aba193f599197caa4c4c383
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/422943
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Though the buffer can be allocated in the normal case,
however we still need to check it.
Change-Id: I483c7bd083c24590295e44473674c7b8ed30e9da
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/420462
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Some functions should be executed togther, so
group it into a single function which will avoid
the potential issues.
Change-Id: Ie57be20aa76d9bece82b5534c3c5fc6c9faeebad
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/420233
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
We should check the value of error info. If it
is error, we should not do the read anymore.
Change-Id: Ie3527766779a1f4ed98dde7d9881fe66720badc4
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/420335
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
If it fails to allocate the buf, we should conduct
the error handling.
Change-Id: If36b2dd84236052c777f5007ecbf7295e1c1d3eb
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/420334
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The two values are all default values.
Change-Id: I117eb217b71c3f079a4450c6f716f9bb70762399
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/420333
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reason:It will be helpful for the debug, and
for some places we need to handle no memory
case.
Change-Id: Id221a856b1f65fe7b946dec246bb53532a29db78
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/419923
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Seth Howell <seth.howell5141@gmail.com>
Reviewed-by: John Kariuki <John.K.Kariuki@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Change-Id: I6babd4cf990bf19b510db88bdfb0ca81e29d9252
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/414700
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Madhu Pai <mpai@netapp.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Intermittent failures with the RocksDB tests pointed to
corruption in the MANIFEST file. Further debug showed that
the MANIFEST file would be corrupted when its cache was
evicted. Blobfs was partly handling append after cache
eviction - it would write the data to the correct position
within the blob. But the sync path would not write the
updated file length xattr when the cache had been evicted.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ie5cf273f8d6511548146a756a7b63c73fee12101
Reviewed-on: https://review.gerrithub.io/407232
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
strncpy is going to be added to the list of banned functions because
it does not guarantee strings are null terminated.
Change-Id: I14e2ce65401bbdb77260627e94aa936797c20ce0
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/407020
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
To support resize operations during I/O, we will need
to send messages to each thread to quiesce I/O while
the resize operation is in progress to guard against
the cluster map memory changing while another thread
is accessing the cluster map.
Therefore, spdk_blob_resize needs to be asynchronous.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ida037334739b4b80a1dbc76e8f1c70bca8b73582
Reviewed-on: https://review.gerrithub.io/404616
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I30675083dcb7dda47ac62eaf124614033ff9c587
Reviewed-on: https://review.gerrithub.io/401702
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
These new names are much more clear and are aligned with other
functions such as spdk_blob_close.
Keep the old names around for now but deprecate them. We will
remove them in next release.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Idc60fd0b19fa2a8b0247a1f5835774d342e721f9
Reviewed-on: https://review.gerrithub.io/400884
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
If the file have cache data and not flushed, the file length will be
not correct.
Change-Id: Idde23e158c1d010e67579b5d6a8d87d3cfbfed2d
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
Reviewed-on: https://review.gerrithub.io/393784
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
This makes spdk_file_write consistent with _file_sync.
The lock should not be held across calls to send_request.
Change-Id: I2a2987bbfe2688b520901267aa5a197bd55565a8
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
Reviewed-on: https://review.gerrithub.io/393599
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Similar to previous change, the ** paradigm is a bit
problematic for asynchronous routines that could fail.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ife7748280482356c4c51a796817b71cd7bc7e479
Reviewed-on: https://review.gerrithub.io/391483
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Using the ** paradigm is a bit problematic for asynchronous
routines that could fail. Currently we were inconsistent in
that some error paths would zero the pointer while others
did not. So make this just a plain pointer, which simplifies
the API and its implementation.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I67147931c6e8350896a4505022a6a314655de3d3
Reviewed-on: https://review.gerrithub.io/391482
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
As part of clarifying the API and preparing for some
future changes, rename the following functions:
spdk_bs_md_create_blob => spdk_bs_create_blob
spdk_bs_md_open_blob => spdk_bs_open_blob
spdk_bs_md_delete_blob => spdk_bs_delete_blob
spdk_bs_md_iter_first => spdk_bs_iter_first
spdk_bs_md_iter_next => spdk_bs_iter_next
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I86bf792717b68379484a6108396bb891fe1c221e
Reviewed-on: https://review.gerrithub.io/391031
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Require braces around all conditional statements, e.g.:
if (cond)
statement();
becomes:
if (cond) {
statement();
}
This is the style used through most of the SPDK code, but several
exceptions crept in over time. Add the astyle option to make sure we
are consistent.
Change-Id: I5a71980147fe8dfb471ff42e8bc06db2124a1a7f
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/390914
Reviewed-by: <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Disambiguate the log components from the trace functionality
(include/spdk/trace.h).
The internal spdk_trace_flag structure and related functions will be
renamed in a later commit - this is just a find and replace on
SPDK_TRACE_* and SPDK_LOG_REGISTER_TRACE_FLAG().
Change-Id: I617bd5a9fbe35ffb44ae6020b292658c094a0ad6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/376421
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This patch make the cache buffer shift of blobfs configurable. We
can configure the cache buffer shift according workload, it will
help to improve the performance.
Change-Id: Ied1c2e5d6fd9eaa7aa0759c03c654fcf2e77aa23
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
Reviewed-on: https://review.gerrithub.io/383381
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Add newlines to the debug logs.
Also dump the mismatched bstype via SPDK_TRACEDUMP() rather than as a
string, since it may not be null terminated.
Change-Id: Icad567373c56f24aea550903a1370c3734465472
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/382855
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Rather than using char arrays to represent blobstore types, just use the
publicly-defined struct spdk_bs_type.
Change-Id: I8bbfd95085e16f9d53106c176038e067ff1266d7
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/382853
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Introducing bstype as a way to identify and verify
blobstore type.
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: I50267b5408625be10fe0c146ae329016d5509b4a
Reviewed-on: https://review.gerrithub.io/380476
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Mark the file as deleted in the function spdk_fs_delete_file()
when the referance is not 0, and delete the file when it is closed,
make the behaviour as "unlink".
Change-Id: Ia934bb73c82c48fdbab79dbe4b56296a73abc01e
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
Reviewed-on: https://review.gerrithub.io/374944
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
This matches the name to the behavior and prepares for addition of a new
log macro for "info" log level.
Change-Id: I94ccd49face4309d3368e399528776ab140748c4
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/375833
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This is just a convenience and replaces the common practice
of passing -1.
Change-Id: Id96734307ebf52ef0ee7dba0e7ac89602b2b5b1a
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/374520
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I4797e3900eb425f8cddec1543a4612d4d98e07a6
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/372990
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
From the exising code, the comparison should + 1.
Suppose offset = 2^19, we still can be fit into
tree with level =0, since there will be 64 buffer,
each with size 2^18, so we do not need to
increse the tree level.
Also a unit test is added to demonstrate this.
Change-Id: I95d3542b0881aa7bb661bc57bc789cc4ef4e7509
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/372396
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
