uint32_t supports at most 2TB at most, we need to handle
the larger blobstores, fix this overflow problem.
Signed-off-by: Sochin Jiang <jiangxiaoqing.sochin@bytedance.com>
Change-Id: I27950eb759e9cb9ad48fa4aa8dd1976b4e852832
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4075
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
We still need to be able to explicitly set specific
bits in the cluster array during initialization and
loading (especially recovery), so we use a bit_array
during load, and then convert it to a bit_pool just
before calling the user's cmopletion callback.
This gives a roughly 300% improvement over baseline
on a benchmark which does continuous resize operations.
The benefit is primarily from saving the lowest free
bit rather than having to always start at bit 0. We
may be able to further improve this by saving extents
in the bit pool as well, although after this patch,
the benchmark shows other hot spots different from the
bit search.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Idb1d75d8348bc50560b1f42d49dbe4d79d024619
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3975
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
The ctx allocation was duplicated after both bs_alloc
calling sites, so this reduces the code a bit. This change
also enables some future changes involving the used_clusters
bit array.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I4ea98f079dbe385654e9cb9c0c58a1926a990c9e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3973
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
This will allow for some additional simplifications
in future patches.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ie421ad35f8c0efbb775fbe6bf85799af515264ef
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3972
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
This prepares for some future patches.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: If63c83f76e839b796c58200ddb0ca2137fbc4288
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3971
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Have it both find and set the lowest available cluster
bit index.
This will temporarily hurt the performance for cluster
allocation, since it will always search starting at
bit index 0. But upcoming changes in this patch set
will fix that again by using a new spdk_bit_pool object
that will do allocations much more efficiently than the
current implementation here.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Iad199c9166b82cb9a31597a080f5a28823849e60
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3970
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Currently if we create a blob of 128 clusters, we
lock/unlock the used_cluster_mutex 128 times - once
for each cluster. Same when those clusters are released
when the blob is deleted. Batching these lock/unlock
operations is very easy and gives a noticeable
efficiency improvement.
My local benchmark (1GiB ramdisk, 4KB cluster size,
128 clusters/blob) creates enough blobs to fill the
blobstore and then continuously deletes and recreates
them. Performance increases 20% on that benchmark
with this patch.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ic503accf1ca1ab1af7254b4067771d956f52014d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4069
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Some of the functions were only referenced directly.
There is no need to use void* or pass any bserrno,
in some cases.
Let's be explicit.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ib26dda7068965838f38dad856ea1e456fd87a655
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4061
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This looks like a major omission on persist path.
Especially visible for cases where blobstore was not
reloaded between blob creations/deletion.
Added writing out zeroes to md_pages that contained
truncated extents (resized down).
After zeroes are writen out, md_pages for those extents
are released. In case of blob deletion, extents are
resized down to 0 so all extent pages are released.
Fixes#1590
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I9a2a1190e3f1f3b5d1bb806191c1fe4d27df7780
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4051
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
We will use it earlier in this file in a future patch.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I554f2073185d466bd0b4e98bdeec721f763c1b44
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3969
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
When claiming clusters as part of blobstore initialization
or recovery, just call spdk_bit_array_set directly rather
than going through the bs_claim_cluster function. We will
be modifying how runtime cluster allocation works so need
to separate the two use cases. This code is very small so
inlining it has minimal code impact.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Iaaa1c817e57b4a2eea62eb4683407364bac1fcc0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3966
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
These functions were added during FTL development and
are more efficient than the roll-your-own implementations
blobstore had previously.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ie09e5c305e6e171af0258e805f2aac3b88822b5e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3965
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
The md_page alignment is not really required for md_page
buffers.
Allocating 4k aligned buffers all the time, causes memory
to be heavily fragmented. Due to DPDK keeping track of the
allocation in the same DMA region as the allocation themselves.
Removing this alignment requirement will help DPDK when searching
for the right part of memory in the heap.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reported-by: Mike Cui
Change-Id: If2f4ca2be38d432d5740f6145b5e0ff46237806b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3853
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
We only create one spdk_blob object for a given blob, and just
increase the ref_count if it is opened multiple times. bs_open_blob
would do the lookup for existing opened blobs.
But if the blob is opened again, before the previous open operation
has completed, we would end up with two spdk_blob objects for the same
blob.
Solution is to do another lookup when the open operation completes.
If we find the blob, free the one we just finished opening and return
the existing one instead.
Also added unit test that failed on the existing code but passes now
with this patch.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Reported-by: Mike Cui
Change-Id: I00c3a913b413deddf06f0b63f7a669efb2b5658f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3855
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
There is a fatal bug that could easily cause data corruption when using
thin-provisioned blobs. In blob_request_submit_rw_iov(), we first get
lba by calling blob_calculate_lba_and_lba_count(),
blob_calculate_lba_and_lba_count() calculates different lbas according to
the return of bs_io_unit_is_allocated(). Later, we call bs_io_unit_is_allocated()
again to judge whether the specific cluster is allocated, the problem is it may
have be allocated here while not be allocated when calling blob_calculate_lba_and_lba_count()
before. To ensure the correctness of lba, we can do lba recalculation when
bs_io_unit_is_allocated() returns true, or make
blob_calculate_lba_and_lba_count() return the result of
bs_io_unit_is_allocated(), use the second solution in this patch.
By configuring more than one cpu core, md thread will run in a separate
SPDK thread, this data corruption scenario could be easily reproduced
by running fio verify in VMs using thin-provisioned Lvols as block
devices.
Signed-off-by: Sochin Jiang <jiangxiaoqing.sochin@bytedance.com>
Change-Id: I099865ff291ea42d5d49b693cc53f64b60881684
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3318
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
This can speed up the check for whether a blob is already open
significantly.
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Change-Id: If32b0b1f168fcdb58e61df6281d7b7520725a195
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2781
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
blob_insert_cluster_msg() will release the already claimed
md page(intended for extent page) if the corresponding extent page
is found to be allocated. But later blob_insert_extent() may fails,
and this cause blob_insert_cluster_cpl() to release the same md page again,
this could be wrong if this specific md page is clamied by others, thus
cause data corruption. So, put it to zero after released in
blob_insert_cluster_msg().
Signed-off-by: Sochin Jiang <jiangxiaoqing.sochin@bytedance.com>
Change-Id: I46eba79b24b1950318002dcb27cb51b01ca566ec
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3152
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This function was removed from the public API last release
when the map file was introduced, but I didn't clean up
the name at that time.
Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: I3101723b504531ce2c51dba2feb063511dd32684
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2443
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
A wrong parameter is passed to memset when loading blob->active.clusters,
this leads to an unpredictable wrong lba value using thin provision LVOLs
while submitting IO requests, thus causes EIO error using QEMU vms.
Signed-off-by: Sochin Jiang <jiangxiaoqing.sochin@bytedance.com>
Change-Id: Iecea80cfa58f7a025603430d666fd9cd4d3fea8b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2431
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Hitting only the static functions from the above libraries
with the spdk_ prefix.
Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: Ic6df38dfbeb53f0b1c30d350921f7216acba3170
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2362
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Also, while we are here, consolidate setting SO_SUFFIX to one spot.
Previously, it was possible for a library to slip through
without an SO version.
Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: I4db5fa5839502d266c6259892e5719b05134518c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2361
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Added blobid and metadata page number to the log.
Previously only number within particular blobs md chain
was displayed.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I8e881c5824c9d2eadca9f3ac8ee2ac9ffc0e5cae
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2058
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Blobid and md_page is claimed as first step of blob creation.
If blob creation failed, both should returned to be used by
other blobs.
This caused multiple reports of:
"Metadata page 1 crc mismatch"
when loading blobstore due to md_pages not actually containing
the written out md pages.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I495452c578d879f749281cebf8975eb2c1c7f79a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2057
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Operation of locating right lba from cluster map
is done on I/O path. Instead of division and multiplication,
perform bit shift operation.
Bit shift is only used when pages per cluster is power of 2.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ic3ed7ec0a82867a8a4bc6391785b9d40c800aacb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1724
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This is called on hot path for I/O, inline it.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Iec40033eac19f2c66c2984623acb5e157a5ffe05
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1723
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
We have an unofficial naming convention that the
spdk_ namespace is reserved for public API functions only.
This patch is attempting to bring the blob library into compliance
with that naming convention.
Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: Ie298e41d1b741dae01744826c208378ee60f9d0a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1700
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Broadcom CI
Contidion previous to this should already verify that
md page is not an extent page.
All extent pages are not part of the chain (sequence_num == 0),
and their location (ctx->cur_page) cannot be the root of
md chain (page->id).
Yet during development it could appen, so adding assert
to verify further that the md page is not extent page.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I6d5dc2ae965f8f9a388cd1c8e186145f8ca91db4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1667
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Before this patch reading out the extent pages during
blobstore replay was serialized. Only issuing reads for next
extent page when previous operation finished.
This was done by continously calling _spdk_bs_load_replay_extent_page_cpl()
and decreasing ctx->num_extent_pages.
This patch changes spdk_bs_sequence_* to spdk_bs_batch_*.
All the reads are submitted at once, and only when all of them
finish we proceed to next valid md chain.
Goal of this change is improving efficiency and readability.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I807cdb98166e04706fedb494363f5776e3151827
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1540
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This variable currently holds single extent page.
Further patch will utilize it to use multiple.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: If86ffd57cecf5d3bfd0812a767c784d7bf503fb6
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1538
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This field does not hold actual pages, but just md page numbers
which hold the extent pages.
Rename as prepartation to adding new one that will hold actual
extent pages.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I1fb85a58c92a93b968e1fad22e421252399e9281
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1537
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This patch adds more ways to back off when parts of
blob persist fails.
Otherwise the process would proceed as if nothing happened.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I7cff73e1dc3066d0c822d1e3dac4bd35e27cd54a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1263
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
After opening the blob for deletion, in _spdk_bs_delete_open_cpl(),
the blob is removed from list of blobs in blobstore.
This is to prevent future _spdk_blob_lookup()s from referencing
blob while it is deleted.
In usual blob deletion path, next step is proceeding with deletion
of the blob by reducing its size to 0 and syncing the blob.
Changes from this point forward are persisted.
Meanwhile in special case of deleting snapshot which has single clone
on it, before above occurs additional steps are performed.
Each of the blobs are opened and their attributes changed.
Failures on those steps are fully recoverable on any errors,
and in such case blob should be added back to the bs list of blobs.
Original code had condition on how many references there were
to blob being deleted, which is incorrect.
Any error on that path should clean up after itself (revert attributes
and close blobs) and re-add the blob.
This change is tested with blob_delete_snapshot_power_failure() UT,
by adding error path in persist - which triggers error in aforementioned
blob delete code path.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I926e7cbf3cb86170c69f31231399535859f290dd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/985
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
When creating snapshot, 'original' blob will end up being a thin provisioned clone.
Before that first thin_provisioned 'newblob' is created during this process.
If the first md sync for 'newblob' fails, it means that only valid references to
clusters are still only present in 'original' blob. The 'newblob' can be safely
cleaned up.
Unfortunetly 'newblob' inherited some of 'original' blob properties before sync.
Cluster maps were already swaped in current cleanup code. But during blob close
of 'newblob' - persist blob code expects clusters to be 0 only for thin_provisioned
blobs. If original blob was thick, then it triggers an assert within persist code.
This patch makes sure to set thin_provision to 'newblob', to align with its creation.
Added asserts to verify that clusters maps are 0's, which should be the case
as I/O to origblob is frozen.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I5420617792aefe8a3ef4e5989b2056504cdd1850
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1394
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This will help with making the _spdk_blob_persist_write_extent_pages()
batch all writes of extent pages.
No functional change occurs with this patch, this is just refactor
for future change.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I8c93b1d6473db660f7ad5e04c8ec9f3331b2055c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/986
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
super->clean value signifies if blobstore was unloaded
cleanly.
If it was not, then during bs_load the _spdk_bs_recover()
procedure if called.
Meanwhile bs->clean is always set to 1 after load, causing very first
blob_persist to also re-write super block with the super->clean
set to 0. To signify that md has changed and possibly trigger
the recovery if clean bs unload does not occur.
When the re-write of super block succeeds the bs->clean is set to 0,
because further re-writes of super block are not needed on next
blob persist.
This patch resolves issue when:
1) reading super block fails - execution should backoff, to prevent
writing an empty buffer as super block !
2) writing super->clean = 0 to the super block fails - execution
again should fail, and bs->clean should not be set to 0. It will
cause next persist to attempt re-write again.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ia07cc5c6c107310059b50886edb7283c176b9169
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1164
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
In general it is not possible to delete snapshot when
there are clones on top of it.
There is special case when there is just a single clone
on top that snapshot.
In such case the clone is 'merged' with snapshot.
Unallocated clusters in clone, are filled with the ones
in snapshot (if allocated there).
Similar behavior should have occurred for extent pages.
This patch adds the implementation for moving EP from
snapshot to clone along with UT.
The UT exposes the issue by allowing delete_blob
to proceed beyond just unrecoverable snapshot blob.
Fixes#1291
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ib2824c5737021f8e8d9b533a4cd245c12e6fe9fa
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1163
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This will allow us to keep track of compatibility issues on a
per-library basis.
Change-Id: Ib0c796adb1efe1570212a503ed660bef6f142b6e
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1067
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Further part of the series will hold array of md pages
in the ctx. Callers of _spdk_bs_load_replay_md_parse_page()
will make select which page to parse.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I3fb70660672ba74bdb338eb1233409103903b215
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/983
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This is refactoring change for future patches.
struct spdk_bs_load_ctx will contain array of pages
instead of single one. Having to change just single
line for selection of page will make it easier to
read next patches in series.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: If3dc1e7da7e61c7b4866307d859e55131a32d38b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/982
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
It is possible for multiple blob persists to affect one another.
Either by blob->state changes or blob mutable data.
Safe way to prevent that is to queue up the persists.
Next persist will be executed only after previous one completes.
Fixes#1170Fixes#960
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Iaf95d9238510100b629050bc0d5c2c96c982a60c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/776
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
_spdk_blob_persist_check_dirty() function will be
called in subsequent patch at the end of persist
in _spdk_blob_persist_complete() to proceed
with any queued up persists.
Please see following patch for this.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ieeb334e23cde329743647f728e70dd60333c224a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/872
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
With recent changes to extent on-disk metadata format,
new format (Extent Pages) is not backwards compatible.
Meanwhile old format (Extent RLE) is backwards
compatible with older SPDK applications.
Summing up:
Blobstore created pre SPDK 20.01 can only use Extent RLE.
Blobstore created starting with SPDK 20.01 can use both,
Extent Pages and Extent RLE specified by use_extent_table opts.
When use_extent_table is set to true, invalid flag for it is set.
SPDK application pre 20.01, will not load such blob.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: If14ebd03f19eb581d71dcb46191e099336655189
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/483220
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This was observed after running nighly tests on previous patch.
As part of it, autopackage.sh compiles SPDK
without debug flag set. Exposing the uninitialized var here.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Iedb1641f3c0d4a21f293c81cd4fcf35c6d1c7ae5
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482893
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Extent table and extent page descriptors are now
set to be default way clusters are serialized on disk.
With this patch UT are ran with and without
extent table.
Changed two asserts in test, since amount is dependent on
which type of serialization is used.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ica58fce6a4effd014d7dd40ee26edd0fa3196d0f
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/481901
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
