d/numam-spdk
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

bdev/rbd: make destruct path asynchronous

Browse Source 
When a bdev is being unregistered, after all
channels have been closed, the bdev layer calls
the module's destruct callback for the bdev before
calling the bdev unregister callback.

For the rbd module, the destruct callback is
bdev_rbd_destruct.  This callback unregisters the
rbd io_device which is an asynchronous operation.
We need to return >0 from bdev_rbd_destruct to
inform the bdev layer that this is an asynchronous
operation, so that it does not immediately call
the bdev unregister callback.  Once the rbd io_device
is unregistered, we can call spdk_bdev_destruct_done()
which will trigger the bdev layer to finally call
the bdev unregister callback.

Without this fix, deleting an rbd bdev would
complete before the backing cluster reference
had been released.  This meant that even
if you had deleted all rbd bdevs, there might still
be cluster references in place for a short period of
time.  It's better to wait to complete the delete
operation until the cluster reference has been
released to avoid this issue (which this patch
now does).

Fixes issue #2069.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I8ac156c89d3e235a95ef196308cc349e6078bfd7
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9115
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
This commit is contained in:
Jim Harris 2021-08-06 14:19:27 +00:00 committed by Tomasz Zawadzki
parent a70aae4610
commit abb4baef2d
1 changed files with 20 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

22
module/bdev/rbd/bdev_rbd.c
View File

@ -74,6 +74,7 @@ struct bdev_rbd {
rbd_image_info_t info;
pthread_mutex_t mutex;
struct spdk_thread *main_td;
struct spdk_thread *destruct_td;
uint32_t ch_count;
struct bdev_rbd_group_channel *group_ch;
@ -496,16 +497,30 @@ bdev_rbd_reset(struct bdev_rbd *disk, struct spdk_bdev_io *bdev_io)
}
static void
bdev_rbd_free_cb(void *io_device)
_bdev_rbd_destruct_done(void *io_device)
{
struct bdev_rbd *rbd = io_device;
assert(rbd != NULL);
assert(rbd->ch_count == 0);
spdk_bdev_destruct_done(&rbd->disk, 0);
bdev_rbd_free(rbd);
}
static void
bdev_rbd_free_cb(void *io_device)
{
struct bdev_rbd *rbd = io_device;
/* The io device has been unregistered. Send a message back to the
* original thread that started the destruct operation, so that the
* bdev unregister callback is invoked on the same thread that started
* this whole process.
*/
spdk_thread_send_msg(rbd->destruct_td, _bdev_rbd_destruct_done, rbd);
}
static void
_bdev_rbd_destruct(void *ctx)
{
@ -534,9 +549,12 @@ bdev_rbd_destruct(void *ctx)
* from the main thread, in case there are pending
* channel delete messages in flight to this thread.
*/
assert(rbd->destruct_td == NULL);
rbd->destruct_td = td;
spdk_thread_send_msg(td, _bdev_rbd_destruct, rbd);
return 0;
/* Return 1 to indicate the destruct path is asynchronous. */
return 1;
}
static void