Initial support for bhyve save and restore.
Save and restore (also known as suspend and resume) permits a snapshot
to be taken of a guest's state that can later be resumed. In the
current implementation, bhyve(8) creates a UNIX domain socket that is
used by bhyvectl(8) to send a request to save a snapshot (and
optionally exit after the snapshot has been taken). A snapshot
currently consists of two files: the first holds a copy of guest RAM,
and the second file holds other guest state such as vCPU register
values and device model state.
To resume a guest, bhyve(8) must be started with a matching pair of
command line arguments to instantiate the same set of device models as
well as a pointer to the saved snapshot.
While the current implementation is useful for several uses cases, it
has a few limitations. The file format for saving the guest state is
tied to the ABI of internal bhyve structures and is not
self-describing (in that it does not communicate the set of device
models present in the system). In addition, the state saved for some
device models closely matches the internal data structures which might
prove a challenge for compatibility of snapshot files across a range
of bhyve versions. The file format also does not currently support
versioning of individual chunks of state. As a result, the current
file format is not a fixed binary format and future revisions to save
and restore will break binary compatiblity of snapshot files. The
goal is to move to a more flexible format that adds versioning,
etc. and at that point to commit to providing a reasonable level of
compatibility. As a result, the current implementation is not enabled
by default. It can be enabled via the WITH_BHYVE_SNAPSHOT=yes option
for userland builds, and the kernel option BHYVE_SHAPSHOT.
Submitted by: Mihai Tiganus, Flavius Anton, Darius Mihai
Submitted by: Elena Mihailescu, Mihai Carabas, Sergiu Weisz
Relnotes: yes
Sponsored by: University Politehnica of Bucharest
Sponsored by: Matthew Grooms (student scholarships)
Sponsored by: iXsystems
Differential Revision: https://reviews.freebsd.org/D19495
2020-05-05 00:02:04 +00:00
|
|
|
/*-
|
|
|
|
* SPDX-License-Identifier: BSD-2-Clause-FreeBSD
|
|
|
|
*
|
|
|
|
* Copyright (c) 2016 Flavius Anton
|
|
|
|
* Copyright (c) 2016 Mihai Tiganus
|
|
|
|
* Copyright (c) 2016-2019 Mihai Carabas
|
|
|
|
* Copyright (c) 2017-2019 Darius Mihai
|
|
|
|
* Copyright (c) 2017-2019 Elena Mihailescu
|
|
|
|
* Copyright (c) 2018-2019 Sergiu Weisz
|
|
|
|
* All rights reserved.
|
|
|
|
* The bhyve-snapshot feature was developed under sponsorships
|
|
|
|
* from Matthew Grooms.
|
|
|
|
*
|
|
|
|
* Redistribution and use in source and binary forms, with or without
|
|
|
|
* modification, are permitted provided that the following conditions
|
|
|
|
* are met:
|
|
|
|
* 1. Redistributions of source code must retain the above copyright
|
|
|
|
* notice, this list of conditions and the following disclaimer.
|
|
|
|
* 2. Redistributions in binary form must reproduce the above copyright
|
|
|
|
* notice, this list of conditions and the following disclaimer in the
|
|
|
|
* documentation and/or other materials provided with the distribution.
|
|
|
|
*
|
|
|
|
* THIS SOFTWARE IS PROVIDED BY NETAPP, INC ``AS IS'' AND
|
|
|
|
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
|
|
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
|
|
|
* ARE DISCLAIMED. IN NO EVENT SHALL NETAPP, INC OR CONTRIBUTORS BE LIABLE
|
|
|
|
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
|
|
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
|
|
|
|
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
|
|
|
|
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
|
|
|
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
|
|
|
|
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
|
|
|
|
* SUCH DAMAGE.
|
|
|
|
*
|
|
|
|
* $FreeBSD$
|
|
|
|
*/
|
|
|
|
|
|
|
|
#ifndef _BHYVE_SNAPSHOT_
|
|
|
|
#define _BHYVE_SNAPSHOT_
|
|
|
|
|
|
|
|
#include <machine/vmm_snapshot.h>
|
|
|
|
#include <libxo/xo.h>
|
|
|
|
#include <ucl.h>
|
|
|
|
|
2021-02-19 02:48:40 +00:00
|
|
|
#define BHYVE_RUN_DIR "/var/run/bhyve/"
|
2021-02-27 21:07:35 +00:00
|
|
|
#define MAX_SNAPSHOT_FILENAME PATH_MAX
|
2021-01-29 08:30:31 +00:00
|
|
|
|
Initial support for bhyve save and restore.
Save and restore (also known as suspend and resume) permits a snapshot
to be taken of a guest's state that can later be resumed. In the
current implementation, bhyve(8) creates a UNIX domain socket that is
used by bhyvectl(8) to send a request to save a snapshot (and
optionally exit after the snapshot has been taken). A snapshot
currently consists of two files: the first holds a copy of guest RAM,
and the second file holds other guest state such as vCPU register
values and device model state.
To resume a guest, bhyve(8) must be started with a matching pair of
command line arguments to instantiate the same set of device models as
well as a pointer to the saved snapshot.
While the current implementation is useful for several uses cases, it
has a few limitations. The file format for saving the guest state is
tied to the ABI of internal bhyve structures and is not
self-describing (in that it does not communicate the set of device
models present in the system). In addition, the state saved for some
device models closely matches the internal data structures which might
prove a challenge for compatibility of snapshot files across a range
of bhyve versions. The file format also does not currently support
versioning of individual chunks of state. As a result, the current
file format is not a fixed binary format and future revisions to save
and restore will break binary compatiblity of snapshot files. The
goal is to move to a more flexible format that adds versioning,
etc. and at that point to commit to providing a reasonable level of
compatibility. As a result, the current implementation is not enabled
by default. It can be enabled via the WITH_BHYVE_SNAPSHOT=yes option
for userland builds, and the kernel option BHYVE_SHAPSHOT.
Submitted by: Mihai Tiganus, Flavius Anton, Darius Mihai
Submitted by: Elena Mihailescu, Mihai Carabas, Sergiu Weisz
Relnotes: yes
Sponsored by: University Politehnica of Bucharest
Sponsored by: Matthew Grooms (student scholarships)
Sponsored by: iXsystems
Differential Revision: https://reviews.freebsd.org/D19495
2020-05-05 00:02:04 +00:00
|
|
|
struct vmctx;
|
|
|
|
|
|
|
|
struct restore_state {
|
|
|
|
int kdata_fd;
|
|
|
|
int vmmem_fd;
|
|
|
|
|
|
|
|
void *kdata_map;
|
|
|
|
size_t kdata_len;
|
|
|
|
|
|
|
|
size_t vmmem_len;
|
|
|
|
|
|
|
|
struct ucl_parser *meta_parser;
|
|
|
|
ucl_object_t *meta_root_obj;
|
|
|
|
};
|
|
|
|
|
2021-03-03 06:05:47 +00:00
|
|
|
/* Filename that will be used for save/restore */
|
|
|
|
struct checkpoint_op {
|
|
|
|
char snapshot_filename[MAX_SNAPSHOT_FILENAME];
|
|
|
|
};
|
|
|
|
|
|
|
|
/* Messages that a bhyve process understands. */
|
2021-02-27 21:03:03 +00:00
|
|
|
enum ipc_opcode {
|
|
|
|
START_CHECKPOINT,
|
|
|
|
START_SUSPEND,
|
2021-01-29 08:30:31 +00:00
|
|
|
};
|
|
|
|
|
2021-03-03 06:05:47 +00:00
|
|
|
/*
|
|
|
|
* The type of message and associated data to
|
|
|
|
* send to a bhyve process.
|
|
|
|
*/
|
|
|
|
struct ipc_message {
|
|
|
|
enum ipc_opcode code;
|
|
|
|
union {
|
|
|
|
/*
|
|
|
|
* message specific structures
|
|
|
|
*/
|
|
|
|
struct checkpoint_op op;
|
|
|
|
} data;
|
2021-01-29 08:30:31 +00:00
|
|
|
};
|
|
|
|
|
Initial support for bhyve save and restore.
Save and restore (also known as suspend and resume) permits a snapshot
to be taken of a guest's state that can later be resumed. In the
current implementation, bhyve(8) creates a UNIX domain socket that is
used by bhyvectl(8) to send a request to save a snapshot (and
optionally exit after the snapshot has been taken). A snapshot
currently consists of two files: the first holds a copy of guest RAM,
and the second file holds other guest state such as vCPU register
values and device model state.
To resume a guest, bhyve(8) must be started with a matching pair of
command line arguments to instantiate the same set of device models as
well as a pointer to the saved snapshot.
While the current implementation is useful for several uses cases, it
has a few limitations. The file format for saving the guest state is
tied to the ABI of internal bhyve structures and is not
self-describing (in that it does not communicate the set of device
models present in the system). In addition, the state saved for some
device models closely matches the internal data structures which might
prove a challenge for compatibility of snapshot files across a range
of bhyve versions. The file format also does not currently support
versioning of individual chunks of state. As a result, the current
file format is not a fixed binary format and future revisions to save
and restore will break binary compatiblity of snapshot files. The
goal is to move to a more flexible format that adds versioning,
etc. and at that point to commit to providing a reasonable level of
compatibility. As a result, the current implementation is not enabled
by default. It can be enabled via the WITH_BHYVE_SNAPSHOT=yes option
for userland builds, and the kernel option BHYVE_SHAPSHOT.
Submitted by: Mihai Tiganus, Flavius Anton, Darius Mihai
Submitted by: Elena Mihailescu, Mihai Carabas, Sergiu Weisz
Relnotes: yes
Sponsored by: University Politehnica of Bucharest
Sponsored by: Matthew Grooms (student scholarships)
Sponsored by: iXsystems
Differential Revision: https://reviews.freebsd.org/D19495
2020-05-05 00:02:04 +00:00
|
|
|
struct checkpoint_thread_info {
|
|
|
|
struct vmctx *ctx;
|
|
|
|
int socket_fd;
|
|
|
|
};
|
|
|
|
|
|
|
|
typedef int (*vm_snapshot_dev_cb)(struct vm_snapshot_meta *);
|
|
|
|
typedef int (*vm_pause_dev_cb) (struct vmctx *, const char *);
|
|
|
|
typedef int (*vm_resume_dev_cb) (struct vmctx *, const char *);
|
|
|
|
|
|
|
|
struct vm_snapshot_dev_info {
|
|
|
|
const char *dev_name; /* device name */
|
|
|
|
vm_snapshot_dev_cb snapshot_cb; /* callback for device snapshot */
|
|
|
|
vm_pause_dev_cb pause_cb; /* callback for device pause */
|
|
|
|
vm_resume_dev_cb resume_cb; /* callback for device resume */
|
|
|
|
};
|
|
|
|
|
|
|
|
struct vm_snapshot_kern_info {
|
|
|
|
const char *struct_name; /* kernel structure name*/
|
|
|
|
enum snapshot_req req; /* request type */
|
|
|
|
};
|
|
|
|
|
|
|
|
void destroy_restore_state(struct restore_state *rstate);
|
|
|
|
|
|
|
|
const char *lookup_vmname(struct restore_state *rstate);
|
|
|
|
int lookup_memflags(struct restore_state *rstate);
|
|
|
|
size_t lookup_memsize(struct restore_state *rstate);
|
|
|
|
int lookup_guest_ncpus(struct restore_state *rstate);
|
|
|
|
|
|
|
|
void checkpoint_cpu_add(int vcpu);
|
|
|
|
void checkpoint_cpu_resume(int vcpu);
|
|
|
|
void checkpoint_cpu_suspend(int vcpu);
|
|
|
|
|
|
|
|
int restore_vm_mem(struct vmctx *ctx, struct restore_state *rstate);
|
|
|
|
int vm_restore_kern_structs(struct vmctx *ctx, struct restore_state *rstate);
|
|
|
|
|
|
|
|
int vm_restore_user_devs(struct vmctx *ctx, struct restore_state *rstate);
|
|
|
|
int vm_pause_user_devs(struct vmctx *ctx);
|
|
|
|
int vm_resume_user_devs(struct vmctx *ctx);
|
|
|
|
|
|
|
|
int get_checkpoint_msg(int conn_fd, struct vmctx *ctx);
|
|
|
|
void *checkpoint_thread(void *param);
|
|
|
|
int init_checkpoint_thread(struct vmctx *ctx);
|
2021-05-15 19:58:21 +00:00
|
|
|
void init_snapshot(void);
|
Initial support for bhyve save and restore.
Save and restore (also known as suspend and resume) permits a snapshot
to be taken of a guest's state that can later be resumed. In the
current implementation, bhyve(8) creates a UNIX domain socket that is
used by bhyvectl(8) to send a request to save a snapshot (and
optionally exit after the snapshot has been taken). A snapshot
currently consists of two files: the first holds a copy of guest RAM,
and the second file holds other guest state such as vCPU register
values and device model state.
To resume a guest, bhyve(8) must be started with a matching pair of
command line arguments to instantiate the same set of device models as
well as a pointer to the saved snapshot.
While the current implementation is useful for several uses cases, it
has a few limitations. The file format for saving the guest state is
tied to the ABI of internal bhyve structures and is not
self-describing (in that it does not communicate the set of device
models present in the system). In addition, the state saved for some
device models closely matches the internal data structures which might
prove a challenge for compatibility of snapshot files across a range
of bhyve versions. The file format also does not currently support
versioning of individual chunks of state. As a result, the current
file format is not a fixed binary format and future revisions to save
and restore will break binary compatiblity of snapshot files. The
goal is to move to a more flexible format that adds versioning,
etc. and at that point to commit to providing a reasonable level of
compatibility. As a result, the current implementation is not enabled
by default. It can be enabled via the WITH_BHYVE_SNAPSHOT=yes option
for userland builds, and the kernel option BHYVE_SHAPSHOT.
Submitted by: Mihai Tiganus, Flavius Anton, Darius Mihai
Submitted by: Elena Mihailescu, Mihai Carabas, Sergiu Weisz
Relnotes: yes
Sponsored by: University Politehnica of Bucharest
Sponsored by: Matthew Grooms (student scholarships)
Sponsored by: iXsystems
Differential Revision: https://reviews.freebsd.org/D19495
2020-05-05 00:02:04 +00:00
|
|
|
|
|
|
|
int load_restore_file(const char *filename, struct restore_state *rstate);
|
|
|
|
|
|
|
|
#endif
|