Added blobid and metadata page number to the log.
Previously only number within particular blobs md chain
was displayed.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I8e881c5824c9d2eadca9f3ac8ee2ac9ffc0e5cae
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2058
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Blobid and md_page is claimed as first step of blob creation.
If blob creation failed, both should returned to be used by
other blobs.
This caused multiple reports of:
"Metadata page 1 crc mismatch"
when loading blobstore due to md_pages not actually containing
the written out md pages.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I495452c578d879f749281cebf8975eb2c1c7f79a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2057
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Operation of locating right lba from cluster map
is done on I/O path. Instead of division and multiplication,
perform bit shift operation.
Bit shift is only used when pages per cluster is power of 2.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ic3ed7ec0a82867a8a4bc6391785b9d40c800aacb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1724
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This is called on hot path for I/O, inline it.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Iec40033eac19f2c66c2984623acb5e157a5ffe05
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1723
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
We have an unofficial naming convention that the
spdk_ namespace is reserved for public API functions only.
This patch is attempting to bring the blob library into compliance
with that naming convention.
Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: Ie298e41d1b741dae01744826c208378ee60f9d0a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1700
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Broadcom CI
Contidion previous to this should already verify that
md page is not an extent page.
All extent pages are not part of the chain (sequence_num == 0),
and their location (ctx->cur_page) cannot be the root of
md chain (page->id).
Yet during development it could appen, so adding assert
to verify further that the md page is not extent page.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I6d5dc2ae965f8f9a388cd1c8e186145f8ca91db4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1667
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Before this patch reading out the extent pages during
blobstore replay was serialized. Only issuing reads for next
extent page when previous operation finished.
This was done by continously calling _spdk_bs_load_replay_extent_page_cpl()
and decreasing ctx->num_extent_pages.
This patch changes spdk_bs_sequence_* to spdk_bs_batch_*.
All the reads are submitted at once, and only when all of them
finish we proceed to next valid md chain.
Goal of this change is improving efficiency and readability.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I807cdb98166e04706fedb494363f5776e3151827
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1540
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This variable currently holds single extent page.
Further patch will utilize it to use multiple.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: If86ffd57cecf5d3bfd0812a767c784d7bf503fb6
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1538
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This field does not hold actual pages, but just md page numbers
which hold the extent pages.
Rename as prepartation to adding new one that will hold actual
extent pages.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I1fb85a58c92a93b968e1fad22e421252399e9281
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1537
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This patch adds more ways to back off when parts of
blob persist fails.
Otherwise the process would proceed as if nothing happened.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I7cff73e1dc3066d0c822d1e3dac4bd35e27cd54a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1263
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
After opening the blob for deletion, in _spdk_bs_delete_open_cpl(),
the blob is removed from list of blobs in blobstore.
This is to prevent future _spdk_blob_lookup()s from referencing
blob while it is deleted.
In usual blob deletion path, next step is proceeding with deletion
of the blob by reducing its size to 0 and syncing the blob.
Changes from this point forward are persisted.
Meanwhile in special case of deleting snapshot which has single clone
on it, before above occurs additional steps are performed.
Each of the blobs are opened and their attributes changed.
Failures on those steps are fully recoverable on any errors,
and in such case blob should be added back to the bs list of blobs.
Original code had condition on how many references there were
to blob being deleted, which is incorrect.
Any error on that path should clean up after itself (revert attributes
and close blobs) and re-add the blob.
This change is tested with blob_delete_snapshot_power_failure() UT,
by adding error path in persist - which triggers error in aforementioned
blob delete code path.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I926e7cbf3cb86170c69f31231399535859f290dd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/985
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
When creating snapshot, 'original' blob will end up being a thin provisioned clone.
Before that first thin_provisioned 'newblob' is created during this process.
If the first md sync for 'newblob' fails, it means that only valid references to
clusters are still only present in 'original' blob. The 'newblob' can be safely
cleaned up.
Unfortunetly 'newblob' inherited some of 'original' blob properties before sync.
Cluster maps were already swaped in current cleanup code. But during blob close
of 'newblob' - persist blob code expects clusters to be 0 only for thin_provisioned
blobs. If original blob was thick, then it triggers an assert within persist code.
This patch makes sure to set thin_provision to 'newblob', to align with its creation.
Added asserts to verify that clusters maps are 0's, which should be the case
as I/O to origblob is frozen.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I5420617792aefe8a3ef4e5989b2056504cdd1850
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1394
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This will help with making the _spdk_blob_persist_write_extent_pages()
batch all writes of extent pages.
No functional change occurs with this patch, this is just refactor
for future change.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I8c93b1d6473db660f7ad5e04c8ec9f3331b2055c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/986
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
super->clean value signifies if blobstore was unloaded
cleanly.
If it was not, then during bs_load the _spdk_bs_recover()
procedure if called.
Meanwhile bs->clean is always set to 1 after load, causing very first
blob_persist to also re-write super block with the super->clean
set to 0. To signify that md has changed and possibly trigger
the recovery if clean bs unload does not occur.
When the re-write of super block succeeds the bs->clean is set to 0,
because further re-writes of super block are not needed on next
blob persist.
This patch resolves issue when:
1) reading super block fails - execution should backoff, to prevent
writing an empty buffer as super block !
2) writing super->clean = 0 to the super block fails - execution
again should fail, and bs->clean should not be set to 0. It will
cause next persist to attempt re-write again.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ia07cc5c6c107310059b50886edb7283c176b9169
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1164
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
In general it is not possible to delete snapshot when
there are clones on top of it.
There is special case when there is just a single clone
on top that snapshot.
In such case the clone is 'merged' with snapshot.
Unallocated clusters in clone, are filled with the ones
in snapshot (if allocated there).
Similar behavior should have occurred for extent pages.
This patch adds the implementation for moving EP from
snapshot to clone along with UT.
The UT exposes the issue by allowing delete_blob
to proceed beyond just unrecoverable snapshot blob.
Fixes#1291
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ib2824c5737021f8e8d9b533a4cd245c12e6fe9fa
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1163
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Further part of the series will hold array of md pages
in the ctx. Callers of _spdk_bs_load_replay_md_parse_page()
will make select which page to parse.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I3fb70660672ba74bdb338eb1233409103903b215
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/983
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This is refactoring change for future patches.
struct spdk_bs_load_ctx will contain array of pages
instead of single one. Having to change just single
line for selection of page will make it easier to
read next patches in series.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: If3dc1e7da7e61c7b4866307d859e55131a32d38b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/982
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
It is possible for multiple blob persists to affect one another.
Either by blob->state changes or blob mutable data.
Safe way to prevent that is to queue up the persists.
Next persist will be executed only after previous one completes.
Fixes#1170Fixes#960
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Iaf95d9238510100b629050bc0d5c2c96c982a60c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/776
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
_spdk_blob_persist_check_dirty() function will be
called in subsequent patch at the end of persist
in _spdk_blob_persist_complete() to proceed
with any queued up persists.
Please see following patch for this.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ieeb334e23cde329743647f728e70dd60333c224a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/872
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
With recent changes to extent on-disk metadata format,
new format (Extent Pages) is not backwards compatible.
Meanwhile old format (Extent RLE) is backwards
compatible with older SPDK applications.
Summing up:
Blobstore created pre SPDK 20.01 can only use Extent RLE.
Blobstore created starting with SPDK 20.01 can use both,
Extent Pages and Extent RLE specified by use_extent_table opts.
When use_extent_table is set to true, invalid flag for it is set.
SPDK application pre 20.01, will not load such blob.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: If14ebd03f19eb581d71dcb46191e099336655189
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/483220
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This was observed after running nighly tests on previous patch.
As part of it, autopackage.sh compiles SPDK
without debug flag set. Exposing the uninitialized var here.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Iedb1641f3c0d4a21f293c81cd4fcf35c6d1c7ae5
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482893
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Extent table and extent page descriptors are now
set to be default way clusters are serialized on disk.
With this patch UT are ran with and without
extent table.
Changed two asserts in test, since amount is dependent on
which type of serialization is used.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ica58fce6a4effd014d7dd40ee26edd0fa3196d0f
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/481901
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
ctx->extent_page signifies if page was allocated
for insertion.
1) It is possible for a thread to claim extent page
on its own thread, and put it in ctx->extent_page.
If conflicting thread allocates another ctx->extent_page,
then it should be freed. This does not mean failure
to insert cluster. As different threads could have
been trying to allocate different clusters,
so condition on line 6716 does not cover it.
If so then it shouldn't be an issue to release
the claimed ctx->extent_page and proceed with updating the
extent page which originally won the race.
NOTE: if clusters were conflicting, then extent_page is
freed in _spdk_blob_insert_cluster_cpl().
2) At this point of _spdk_blob_insert_cluster_msg()
we already verified that there already is
extent page allocated at "*extent_page".
In such case ctx->extent_page will be 0,
and should not be used.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Id5b57c88248890eee60d2e7dbecbd984c98b561b
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482867
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
sz is set to number of clusters that should be have been
in particular unallocated EP (remaining_clusters_in_et
up to SPDK_EXTENTS_PER_EP).
The cluster array should be set to 0 only in region
between original size (cluster_array_size) and new
total size (active.num_clusters).
It was incorrectly using sz in the memset.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ic43e89c17d53e9529e3ed0349aeb4fb7dc6593f2
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482858
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Previously part of function assumed that cluster count
1)means number of clusters in EP and another 2) that it is
following the active.num_clusters (akin to extent_rle).
This was incosistent and showed when using multiple
extent pages to serialize metadata.
This patch changes it to only go with 1), so it is clear
that it means number clusters within particular EP.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I155104cabc127ed47df04434032fb01e08948e13
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482848
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Following changes are done in this patch:
1) _spdk_blob_serialize_extent_table now persists
at least single extent page. When num_extent_pages == 0.
2) Minimum valid size of ET descriptor is even without
extent_pages. This is a case when there are no EP,
but we still want to persist num_clusters in ET.
3) Taking above points, redone the loop for serializing
extent pages.
4) Make sure to mark blob dirty if any new extent pages
were allocted.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I81dc6cf2de2722bb49927ed42f4b9f31292f78c5
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482847
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Changed assert checking if cluster array is allocated
when loading extent pages. This is true only for
the first extent page being loaded, of course after that
the cluster array can be already allocated.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I0f96294ede5a12ffd6bca73cbeadba8d94a35bac
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482857
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Size of a blob (thus size of clusters array in mutable data)
is known from extent table descriptor.
Extent pages were read sequentially in order they were
placed in extent table. This meant that cluster
array could have been filled up from beginning to end.
Yet reading extent pages in any other order,
would result in incorrect placement of clusters.
This patch adds first cluster index that is contained within
each extent page. This will allow to read/write
multiple extent pages in parallel, since
we will know where in clusters array to put the cluster idxs.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ib6b9332111cd93f990d057dc60624152907dd87f
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482701
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This is more adequate name, since this value if first read from
Extent Table descriptor. Then decreased when iterating over entries in
extent table and extent pages are read.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ib188c524b8488b38d4de063a9970dcfdf49c9acd
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482600
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
When replaying md chain for a blob, extent table
descriptor can be read. When it is present, all allocated pages
it points to are now being put into extent_pages array in ctx.
If multiple extent table descriptors are in single md chain,
the array is expanded accordingly.
After replaying single md chain is done, replay extent pages
starting from last one. Replaying extent pages, is similar to
extent_rle in that each allocated cluster is claimed and
number of free clusters in blobstore decreased.
When all extent pages are read, return to
_spdk_bs_load_replay_md_cpl() for continuing replaying
next valid md chain.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I4573226aff7d7b1bcdfd188518235c8d4b68a4c3
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/481621
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
_spdk_bs_load_replay_md_parse_page() is only used in
replay path during blobstore load.
Next patch will expand the load ctx with array of
extent pages to be read. It is filled out when reading
in-chain metadata of extent table descriptors.
Passing the load ctx here will make it simpler to
fill out the array when processing extent table.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: If96e6670560c8c4a3610f33ece14c354d7d5da39
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482412
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
When EXTENT_TABLE descriptor is found when parsing metadata
that means there can be extent pages to read.
If extent page was not allocated, number of clusters can be
increased depending on the num_clusters_in_et.
Unallocated extent page contains either SPDK_EXTENTS_PER_EP
or remainder of num_clusters_in_et worth of clusters.
Depending which is less.
Added decreasing fo num_clusters_in_et to parsing
extent pages as well.
While here, remove ctx->seq = seq assignment as that is
done at beginning of blob load.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I57f54634b908ffb406f3e91e15841b7f36fd6de6
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/476429
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Add new serialization of changed extent pages before persisting md.
Iterate over active extent pages (not array !). When they are
allocated but not yet present on disk - write them out.
All extent pages in clean mutable data are assumed to be written out
already.
So there are two cases here:
1) Active mutable array is larger than clean
All allocated extent pages should be written out.
2) Cluster allocation created new extent page
Blob has to be thin provisioned and persist was called
as part of cluster allocation. New extent page needs to be
written out and EXTENT_TABLE allocated.
Iteration is done over num_extent_pages instead of extent_pages_array_size,
to prevent writting out too many extent pages when size of blob was
made smaller. The two values come back in sync at the end of persist
either way.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I780819fd7f3c44e4cf5d71c188c642536d3cc320
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/479851
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Right now output from _spdk_bs_cluster_to_extent_page()
is used to determine whether the exten_table is used at all.
If NULL pointer was returned this meant that extent table
was not allocated, even if the code might suggest just
checking if we overran the array.
To make it more obvious, the _spdk_bs_cluster_to_extent_page()
now only asserts the extent_table_id.
blob->use_extent_table is now always used to determine the
serialization path.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I9d2630645213539bae5cd1d72e5f9b878f53c2bc
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482599
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This patch add single EXTENT_PAGE updates on cluster allocations.
There are three possible outcomes after inserting a cluster:
1) blob uses EXTENT_RLE
Proceed to usual sync_md.
2) blob uses EXTENT_TABLE and extent page was not yet written out
Update the active mutable data to contain the claimed md page,
write out the EXTENT_PAGE and sync_md to update EXTENT_TABLE.
3) blob uses EXTENT_TABLE and extent page was previously written out
Only serialize that single EXTENT_PAGE and write out the updated
cluster map for it.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ia057b074ad1466c0e1eb9c186d09d6e944d93d03
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/470015
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Similar to EXTENT_RLE, this descriptor holds LBA of clusters.
Difference is that EXTENT is kept in separate md pages,
and only single EXTENT will be updated on cluster allocation.
This patch adds the EXTENT processing, which is not used
until following patch.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ifbac23db7ca3e7c8c91cee01018f20071f0d5160
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/470014
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Added claiming the extent page.
Which is then followed by updates in updates
of mutable data on md thread.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: If511564f812685381c48924310105a4cb6f63cd1
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/479850
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Functions to claim and release md pages were added.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I1c8ddc13c8a5806fb874e5c34dae2a327e1ff248
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482011
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
With this patch extent pages array will change it size accordingly
to size of the blob. Similar to clusters, only resizing up
is done on blob resize. Shrinking is done on persisting the blob.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Id7f7c81efbd96af414fce9fc4045cbb476cc93a6
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/479962
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Extent Pages claim and insertion can be asynchronous
when cluster allocation happens due to writing to a new cluster.
In such case lowest free cluster and lowest free md page
is claimed, and message is passed to md_thread.
Where inserting both into the arrays and md_sycn happens.
This patch adds parameters to pass the Extent Page offset
in such case.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I46d8ace9cd5abc0bfe48174c2f2ec218145b9c75
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/479849
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Added new descriptor SPDK_MD_DESCRIPTOR_TYPE_EXTENT_TABLE.
Extent Table will hold md page offsets for new Extent Page descriptor.
Entries in Extent Table are run-length encoded 0's as unallocated
Extent Page descriptors.
Additionally total number of clusters is persisted in each Extent
Table descriptor. This is because there is no guarantee that
last Extent Page of a blob will be allocated.
Even if number of Extents per Extent Page is always the same,
Extent Page can hold less Extents than that.
This patch does not add more metadata on disk right now.
Only added descriptor parsing/serialization and applicable fields
to store it in run time.
Following patches are going to implement TODO's added in this patch.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Iac5d8f00ddfc655c507bc26d69d7adf8495074e9
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/466920
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Lets get it removed ! :)
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I91b994a883a642d87ecc8c152c801b8a7676f33a
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482010
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This is an additional option that can be passed when creating
a blob.
When opts->enable_extent_pages is set to false (current default),
only EXTENT_RLE should be persisted on sync.
During blob load, when EXTENT_RLE is present in md,
blob->extent_rle_found is set to true.
When opts->enable_extent_pages is set to true,
only EXTENT_TABLE and EXTENT_PAGES should be persisted on sync.
During blob load, when EXTENT_TABLE is present in md,
blob->extent_table_found is set to true.
It is possible to find neither EXTENT_* descriptor when loading a blob.
This means that blob length is 0 and EXTENT_RLE was supposed to be used.
Yet none were persisted due to lack of clusters.
In such case blob->use_extent_table is set to true after finishing
blob load.
When parsing metadata ends, if extent_table_found is set - then
support for extent_table is enabled. All other cases disable it.
At this time path for Extent Pages is not implemented, so it should
not be used.
Later in the series, it will become the default path for serialization.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I2146da6130a0645e686ab02a3b5d2d86a7d35a1f
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/479853
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Replaying md through _spdk_bs_load_replay_md_cpl() starts with
md page 0 in search of first valid md page starting a chain
for particular blob.
When it is found, next pages read are from the current pages
`next` page - next in chain.
After whole chain is read, it goes back to first page in chain
and starts search for next valid chain from there.
This patch adds separation between reading particular chain,
and moving to the next one.
Moving on to the next one happens in _spdk_bs_load_replay_md_chain_cpl().
Further in the series, extent pages will be added in the metadata.
Those are not within any particular blobs chain of metadata,
but spread out over the md region.
It is not enough to read all md and read extent pages.
In case of power failure, only extent pages known to be valid
are the ones which are pointed to by some valid md chain.
In futher patches, a step will be added after reading particular
valid md chain to go read extent pages pointed by it.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I6e7cd64af66ce5db0abd2ad5962d604ac2b30994
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/481900
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Moved finishing of unloading to separate function,
which is now called on every failure and success when unloading
the blobstore.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I34539b78c5cc63a6fe5891014cba89b9eb62d4df
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482009
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Before this change it was possible to fail at
writing out some of used md pages.
bserrno output of those was not verified.
This patch adds it at every step.
With that two function don't need (and never needed)
to pass the bserrno:
_spdk_bs_load_write_used_md()
spdk_bs_load_complete()
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I1a61763f03665ba1b00e5949ef0cf37eefaaf08f
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482008
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This is simplification of load path.
seq is save in ctx already, no need to pass it to the function.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ief0ddc1826c461adbad71ba1a3897c510ec2a971
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482007
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Assigning seq to ctx was done very late in the process.
To keep future functions lean and without the seq,
it is assigned immidietly after starting.
Only functions in load path that require separate
seq argument are those passed directly to read/write
device operations.
Rest of them can just use spdk_bs_load_ctx.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I2bd610dc4c7b4a7b0c3de92391922475c514326a
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/481899
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
No functional change is done in this patch.
Most of the functions already translate cb_arg to ctx and
use it, but then just pass cb_arg.
This will make it clear that it is ctx that is passed around.
Along with simplifying some of changes in next patch,
where arguments of functions will be cut down just to
the ctx.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: If7d8ed38dc92175d867a2231ab2ebd4f2499efcd
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482006
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
This patch creates new _spdk_blob_persist_generate_new_md()
function that is responsible for generation of new metadata
from current state of blob.
Functionality so far is unchanged.
This is preparation for later in the series where new
extent pages will be written out to disk before metadata pages.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I84158cb8316a881a6170ac37e151a60aaa9d7369
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/479848
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
