With this patch extent pages array will change it size accordingly
to size of the blob. Similar to clusters, only resizing up
is done on blob resize. Shrinking is done on persisting the blob.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Id7f7c81efbd96af414fce9fc4045cbb476cc93a6
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/479962
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Extent Pages claim and insertion can be asynchronous
when cluster allocation happens due to writing to a new cluster.
In such case lowest free cluster and lowest free md page
is claimed, and message is passed to md_thread.
Where inserting both into the arrays and md_sycn happens.
This patch adds parameters to pass the Extent Page offset
in such case.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I46d8ace9cd5abc0bfe48174c2f2ec218145b9c75
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/479849
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Added new descriptor SPDK_MD_DESCRIPTOR_TYPE_EXTENT_TABLE.
Extent Table will hold md page offsets for new Extent Page descriptor.
Entries in Extent Table are run-length encoded 0's as unallocated
Extent Page descriptors.
Additionally total number of clusters is persisted in each Extent
Table descriptor. This is because there is no guarantee that
last Extent Page of a blob will be allocated.
Even if number of Extents per Extent Page is always the same,
Extent Page can hold less Extents than that.
This patch does not add more metadata on disk right now.
Only added descriptor parsing/serialization and applicable fields
to store it in run time.
Following patches are going to implement TODO's added in this patch.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Iac5d8f00ddfc655c507bc26d69d7adf8495074e9
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/466920
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Lets get it removed ! :)
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I91b994a883a642d87ecc8c152c801b8a7676f33a
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482010
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Since further patches will be adding new descriptors
that are related to cluster layout throughout the blobstore,
add description for existing descriptor too.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I722eb633445685789d5185ed59dfc910f76b109f
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/481724
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This is an additional option that can be passed when creating
a blob.
When opts->enable_extent_pages is set to false (current default),
only EXTENT_RLE should be persisted on sync.
During blob load, when EXTENT_RLE is present in md,
blob->extent_rle_found is set to true.
When opts->enable_extent_pages is set to true,
only EXTENT_TABLE and EXTENT_PAGES should be persisted on sync.
During blob load, when EXTENT_TABLE is present in md,
blob->extent_table_found is set to true.
It is possible to find neither EXTENT_* descriptor when loading a blob.
This means that blob length is 0 and EXTENT_RLE was supposed to be used.
Yet none were persisted due to lack of clusters.
In such case blob->use_extent_table is set to true after finishing
blob load.
When parsing metadata ends, if extent_table_found is set - then
support for extent_table is enabled. All other cases disable it.
At this time path for Extent Pages is not implemented, so it should
not be used.
Later in the series, it will become the default path for serialization.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I2146da6130a0645e686ab02a3b5d2d86a7d35a1f
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/479853
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Replaying md through _spdk_bs_load_replay_md_cpl() starts with
md page 0 in search of first valid md page starting a chain
for particular blob.
When it is found, next pages read are from the current pages
`next` page - next in chain.
After whole chain is read, it goes back to first page in chain
and starts search for next valid chain from there.
This patch adds separation between reading particular chain,
and moving to the next one.
Moving on to the next one happens in _spdk_bs_load_replay_md_chain_cpl().
Further in the series, extent pages will be added in the metadata.
Those are not within any particular blobs chain of metadata,
but spread out over the md region.
It is not enough to read all md and read extent pages.
In case of power failure, only extent pages known to be valid
are the ones which are pointed to by some valid md chain.
In futher patches, a step will be added after reading particular
valid md chain to go read extent pages pointed by it.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I6e7cd64af66ce5db0abd2ad5962d604ac2b30994
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/481900
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Moved finishing of unloading to separate function,
which is now called on every failure and success when unloading
the blobstore.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I34539b78c5cc63a6fe5891014cba89b9eb62d4df
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482009
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Before this change it was possible to fail at
writing out some of used md pages.
bserrno output of those was not verified.
This patch adds it at every step.
With that two function don't need (and never needed)
to pass the bserrno:
_spdk_bs_load_write_used_md()
spdk_bs_load_complete()
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I1a61763f03665ba1b00e5949ef0cf37eefaaf08f
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482008
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This is simplification of load path.
seq is save in ctx already, no need to pass it to the function.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ief0ddc1826c461adbad71ba1a3897c510ec2a971
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482007
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Assigning seq to ctx was done very late in the process.
To keep future functions lean and without the seq,
it is assigned immidietly after starting.
Only functions in load path that require separate
seq argument are those passed directly to read/write
device operations.
Rest of them can just use spdk_bs_load_ctx.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I2bd610dc4c7b4a7b0c3de92391922475c514326a
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/481899
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
No functional change is done in this patch.
Most of the functions already translate cb_arg to ctx and
use it, but then just pass cb_arg.
This will make it clear that it is ctx that is passed around.
Along with simplifying some of changes in next patch,
where arguments of functions will be cut down just to
the ctx.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: If7d8ed38dc92175d867a2231ab2ebd4f2499efcd
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482006
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
This patch creates new _spdk_blob_persist_generate_new_md()
function that is responsible for generation of new metadata
from current state of blob.
Functionality so far is unchanged.
This is preparation for later in the series where new
extent pages will be written out to disk before metadata pages.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I84158cb8316a881a6170ac37e151a60aaa9d7369
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/479848
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
On blob load, read in the saved clear_method option. If
BLOB_CLEAR_WITH_DEFAULT was passed in, use the setting stored
in metadata previously. If something other than the default was
specified, ignore stored value and used what was passed in. If
ignoring a stored value, print a warning.
Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: Ia0c81fa0adc175dfaeb74c06e1ac91dc6b27e9ab
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/472209
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Accept a clear method option on blob create by adding clear_method
to the opts structure passed in to _spdk_bs_create_blob(). Store
these 2 bits in md_ro_flags so that earlier versions without an
understanding of these bits can not alter metadata.
The new metadata values will be used later in the series.
Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I5440645ca20b426778d13b2e544b65dc2b3b83c7
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/472204
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
In current version, immidietly following parsing all
metadata pages an action is taken inform of loading the back_bs_dev.
Patches later in the series will add more metadata in form of
extent pages, which have to be read separetly from usual
blob metadata pages.
This patch add separation between the two steps,
so later a device read can be put between.
Additionally, _spdk_blob_load_final() when no snapshot was present
passed bserrno which was always 0. This patch just sets 0 directly
there as no errors occured at that point.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I4a77527f90bb1de12f972591067b7a50926f39c9
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/476427
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Just to make all sizes consistent and less error prone.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Id0a21bbd45954a0f2317e0eefd3725f1542ef04f
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/479961
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Fixed size of check in _spdk_bs_snapshot_newblob_open_cpl().
Rest are just to make all consistent and more error prone.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I5a23a7795f1e598c1cfd6d17ce37b367f2f34df8
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/479960
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
The _spdk_bs_page_to_lba() [without 'md'] is only for translating the
pages on the blobstore to lba they are at. Those pages start at
the begining of the device and cover all of it. Thus simple
math is enough to translate those.
It is used to calculate lba_count for set of pages as well.
Meanwhile there are 'md_pages' which are the same pages as for
the above, but their count start at bs->md_start.
Which is right after super_block and couple pages for bit masks.
This patch creates new _spdk_bs_md_page_to_lba() that is more
explicit in what page number is passed. Hopefully avoiding
confusion when reading which page number refers to which
'type' of page.
Exception to the that is _spdk_bs_dump_read_md_page(), where
blobstore is not actually loaded (md_start from super block
is not copied to bs structure).
Additionaly providing assert to catch errors on debug builds.
Making the check in _spdk_blob_load_cpl() for max_md_lba obsolete.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I66bbca55b5ca3d6794c462d50177e6037ddbefa6
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/479017
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Always when creating a snapshot, new blob is created.
That blob is explcitly set as thin provisioned with size of
the original blob in _spdk_bs_snapshot_origblob_open_cpl().
Thus it should always contain empty cluster map,
as API user has to interaction with it yet.
As sanity check for debug builds, verification if all clusters
are 0's is added.
This empty cluster map is later swapped into the original blob
in _spdk_bs_snapshot_swap_cluster_maps().
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I4b935c0cf08917e9ad7b9bbedac4781890626eec
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/478974
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Refactor blob loading when snapshot is present.
All paths now go through _spdk_blob_load_final().
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ifc927de6800501cdf62dba8d73e950af2a46d568
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/479143
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Since all error paths for blob load are now the same,
they can go through common function to handle
freeing and calling the original cb.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ib3afc7e62b6f9c872bb1d5f72ef61170aee966d7
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/479142
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Originally the code was suposed to determine if
loading the blob succeeded, based on passing the
cb_arg.
This breaks the logic of always getting the cb_arg in
cb_fn, and basing the success on bserrno.
In order to fix this, cb_fn always gets the passed
cb_arg. Meanwhile the cb_fn (_spdk_bs_open_blob_cpl(),
now checks the bserrno to determine failure.
In addition since _spdk_bs_open_blob() was the original
caller allocating the blob structure,
the _spdk_bs_open_blob_cpl() is now responsible for freeing it.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ic7eb09f05e04b08dc54fc43243fd576f493cbeb2
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/479141
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
The sequence was saved into the load context much later
into the loading, instead of right when ctx is allocated.
This will come in handy in later patches that refer to
sequence earlier (in error paths).
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ibe513dbd919f36874fcde763fc96d46973b60446
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/479140
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
When serializing extents, run-length encoding is supposed to
1) RLE all sequential LBAs
2) RLE zero LBAs (unallocated)
There is one special case, with sequential LBAs that start
with 0 LBA. This is RLE as 1) case, but results in descriptor
matching case 2). Which causes loss of allocated clusters.
This requires following conditions to be met:
- blobstore has just a single cluster reserved for MD
- blob is thin provisioned
- first allocation occurs on cluster_num=1
For last part to be true, very first write for blob has to be
issued to LBA between cluster_size and 2*cluster_size.
Causing allocation of second cluster in blobstore and assiging
it LBA equal to number of LBAs per cluster.
To fix this, case 1) disallows to RLE zeroes.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I136282407966310c882ca97c960e9a71c442c469
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/475494
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
In prep for storing a clear_method in the blob metadata:
* Set the default to DEFAULT and let the switch statement choose
UNMAP
* Use switch statements to make it clearer which method we are
using and why. (ie previously we set the default to UNMAP and
then had an UNMAP || DEFAULT condition to choose UNMAP.
Later in the patch series it will become clearer why this makes
sense.
Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I216cb97fd8eaa772437a36c2c7a47e66618bbfbd
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/472202
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Originally serializing extent_rle was always done as last step.
There was no need to update the buffer pointer,
since it went unused.
Next patches in series expand serialization to new descriptors,
so here the assumption is removed and buf/buf_sz is updated.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I7ccfb500d64e4276359cc98c5587c6301272d728
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/468232
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This patch simplifies return path when returning from
serialization of extent_rle.
Both paths will share more code in upcoming patch.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ibb0ebcfe4377fe09709345d580d54050b61d3c88
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/468231
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
In future patches new type of extents will be added,
for compatibility the current extent type will be still
handled in the code.
To signify the difference between those two types,
current type is renamed to SPDK_MD_DESCRIPTOR_TYPE_EXTENT_RLE.
Along with any variables throughout the code,
to make it clear which ones are used.
There are no functional changes in this patch.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I7186ccc452d200036188abf1dcea9660dcedee72
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/468230
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This change moves the code related to serializing
extents into serparate function, in order to allow
more clear changes in further patches.
There are no functional changes in this patch.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: If8d7c90a5b01f1608d20fd00c3e4ff6a340ce305
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/466919
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This variable went unused, since logic in
_spdk_blob_persist_write_page_chain() already dealt with
writing metadata from last to first page.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ic70c47df1ea3bb01c8031244339c42e9936f28b0
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/467248
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
SPDK_ERRLOG lists the function name, so remove old references that
assume it doesn't and reprint the function name.
Change-Id: I69da6ca0a25bf0eda07d8dad52bcfadf964ac715
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/469487
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Scan-build has a real issue with reallocs. The original
error from latest version of scan-build is rather complicated,
but it can be greatly simplified with the following change:
> diff --git a/lib/blob/blobstore.c b/lib/blob/blobstore.c
> index 7580c9dd2..6a594edf3 100644
> --- a/lib/blob/blobstore.c
> +++ b/lib/blob/blobstore.c
> @@ -1147,8 +1147,9 @@
> _spdk_blob_persist_clear_clusters_cpl(spdk_bs_sequence_t *seq, void *cb_arg, int
> } else if (blob->active.num_clusters != blob->active.cluster_array_size) {
> tmp = realloc(blob->active.clusters, sizeof(uint64_t) * blob->active.num_clusters);
> assert(tmp != NULL);
> - blob->active.clusters = tmp;
> - blob->active.cluster_array_size = blob->active.num_clusters;
> + ctx->blob->active.clusters = tmp;
> + assert(ctx->blob->active.clusters[0] != 14213);
> + ctx->blob->active.cluster_array_size = ctx->blob->active.num_clusters;
> }
>
> _spdk_blob_persist_complete(seq, ctx, bserrno);
> ```
Scan-build will then complain:
blobstore.c:1151:10: warning: Use of memory after it is freed
assert(ctx->blob->active.clusters[0] != 14213);
Asserting blob == ctx->blob, blob->active.clusters == ctx->...,
or even tmp != blob->active.clusters doesn't work, so use the
last resort scan-build weapon - #ifdef __clang_analyzer__.
The realloc in this case is just down-sizing a buffer to
save some memory. For scan-build, just don't do it. This
finally silences all scan-build false positives.
Change-Id: Ib88ea145370f5035eedd2412e98ee61f96ad1915
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/462868
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
md_start and md_len are values in pages rather than lba.
Those should not be compared against lba of currently
loaded md page.
This patch changes assert to verify if the lba of current
page does not exceed max lba where md is expected to be.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Id445eb9871f82f7fe367bfc396f1b495591511c1
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/460976
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Blob id only is matched to the very first page of md for
that particular blob.
During loading blobstore, we shouldn't verify
further pages in chain against the blobid.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ifc7863ddcb403aedc264c14e6b4c3915bd30dc41
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/460607
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Length of xattr descriptor is equal to length of xattr struct,
xattr name and the len of stored value.
There is no limit to how much can be stored in memory for xattr.
On disk xattr size is limited to single page and within that to
max descriptors that can fit in it.
This size is known at compile time.
Before this patch it was possible to add xattr exceeding
what was possible to be written to disk. This caused issues
when serializing the metadata during spdk_blob_sync_md()
or spdk_blob_close(). Making those fail without specific info
to the user and not actually writting such descriptor.
Since maximum length of xattr descriptor is known at compile time,
this patch compares against this value when setting the xattr.
It will immediately report back to user with error, and will
not store xattr in memory (thus not serialize it).
This patch should not affect any backward compatibility for blobs.
Too large xattrs weren't written to disk before,
API for blobstore stays the same - only reporting ENOMEM
when it should.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I6f4af4d079e47f084e20d7a4969d9a78ec1f8610
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/460450
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Currently we are missing cleanup routine for case when
power failure interrupts creating snapshot. This patch
add such routine.
For the case where we find blob with a parent snapshot ID
matching newly created snapshot we can finish whole process
during recovery by processing forward with setting snpashot
as read only, removing xattr and syncing. We should remove
snapshot only if there is no blob with parent pointing at
snapshot.
Fixes github issue #760
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: I2f0e298164e07a2b4dfa5367e8878facef640702
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455216
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
In _spdk_bs_snapshot_origblob_sync_cpl function on error
path we should not close snapshot as it will be closed during
volume closing when bs_dev is being destroyed.
This issue was found in unit test (see next patch in series).
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: I51c38d1f1f97b134679251b43109b1265e565a17
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455215
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Starting with this patch it is possible to remove a
snapshot if there is only a one clone created from it.
In such case snapshot can be removed without any data
copying. This is achieved with following steps (case
with only one clone):
1. Open snapshot (Snapshot1) that shall be removed
2. Check if the Snapshot1 has no more than 1 clone (Clone1)
3. Remove Clone1 entry from Snapshot1
4. If the Snapshot1 has a parent snapshot (Snapshot2):
4a. Add Clone1 entry to the Snapshot2 clones list
4b. Remove Snapshot1 entry from Snapshot2 clones list
5. Open Clone1 blob
6. Freeze I/O operations on Clone1
7. Temporarily override md_ro flag for Snapshot1 and Clone1
for MD modification
8. Merge Snapshot1 and Clone1 clusters maps into Clone1
clusters map
9a. If Snapshot2 is present switch parent ID and backing
bs_dev on Clone1
9b. If Snapshot2 is not present set parent ID to
SPDK_BLOBID_INVALID and backing bs_dev to zeroes_dev
10. Sync MD on Clone1
11. Sync MD on Snapshot1
12. Restore MD flags for Clone1 and Snapshot1
13. Unfreeze I/O on Clone1
14. Close Clone1 blob
15. Remove Snapshot1
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: I800724b981af894e01e1912d0077c5b34a2ae634
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/445576
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Actually write/writev/write_zeroes/unmap are never be called, and we add
the error code here to keep it same style with snapshot bs_bdev.
Change-Id: I32ad051c1902bd7080b894e36f7c89f1c8d27434
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/456924
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
In some cases user may want to flag blob for removal
then do some operations (before removing it) and while
it happens there might be power failure. In such cases
we should remove this blob on next blobstore load.
Example of such usage is delete snapshot functionality
that will be introduced in upcoming patch.
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: I85f396b73762d2665ba8aec62528bb224acace74
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453835
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This patch moves _spdk_blob_set_thin_provision function
higher in the file as it will be later used during
blobstore load.
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: Ife37ef8c69b88903646b2002b3561101c1eb5135
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455488
Reviewed-by: Piotr Pelpliński <piotr.pelplinski@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Access to field 'tqh_first' results in a dereference of a null pointer
set = TAILQ_FIRST(&channel->reqs).
Add asserts to check if channel got NULL;
Change-Id: Ifd8d131a2432328d683e7fb9357fdd23b2396cf2
Signed-off-by: yidong0635 <dongx.yi@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/454536
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
When new writes come from different threads, cluster allocations can
happen many times at once. The corresponding cluster number for the map
is determined via _spdk_bs_allocate_cluster() and kept in ctx->new_cluster.
The cluster itself is inserted into the map only on md_thread.
When there is conflict of two threads allocating same cluster,
message is returned to the losting thread to release the cluster.
Before this patch, on such failure the cluster to release
was calcualted from the page. This resulted in releasing the
cluster claim for thread that actually won it.
This patch makes it so that cluster allocated and save in ctx
is used instead.
Change-Id: Id10811b887f673f9b89e41e0637d4422f1d7270d
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/452625
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I3858c637d421b58e74fa5573d257e59fed92824a
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/452268
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
This patch add new _spdk_bs_delete_blob_finish function
which will be helpful in future changes.
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: I2d492b6102f33ad35b7b6fe408f709f54b7b2341
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/452251
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This patch adds new function which is used to check
if blob can be removed when requested.
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: Iafa82fba9bf67ffd15cf639f4665087f054b6b7d
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/452242
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This patch creates new function that will be helpful
with further implementation of 'delete snapshot'
feature.
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: I66f138ba217fb4a4186f2703900a2952cdb8e438
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/452240
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
