wait for device mounts in zpool and dumpon

If the root file system is composed from multiple devices, wait for
devices to be ready before running zpool and dumpon rc scripts.

An example of this is if the bulk of the root file system exists on a
fast device (e.g. NVMe) but the /var directory comes from a ZFS dataset
on a slower device (e.g. SATA). In this case, it is possible that the
zpool import may run before the slower device has finished being probed,
leaving the system in an intermediate state.

Fix is to add root_hold_wait to the zpool and dumpon (which has a
similar issue) rc scripts.

PR:		242189
Reported by:	osidorkin@gmail.com
Reviewed by:	allanjude
MFC after:	1 month
Differential Revision:	https://reviews.freebsd.org/D29101

libexec/rc/rc.d/dumpon

@@ -42,6 +42,7 @@ dumpon_start()
 	[Nn][Oo] | '')
 		;;
 	[Aa][Uu][Tt][Oo])
+		root_hold_wait
 		dev=$(/bin/kenv -q dumpdev)
 		if [ -n "${dev}" ] ; then
 			dumpon_try "${dev}"
@@ -56,6 +57,7 @@ dumpon_start()
 		return 1
 		;;
 	*)
+		root_hold_wait
 		dumpon_try "${dumpdev}"
 		;;
 	esac

libexec/rc/rc.d/zpool

@@ -22,7 +22,14 @@ zpool_start()
 
 	for cachefile in /etc/zfs/zpool.cache /boot/zfs/zpool.cache; do
 		if [ -r $cachefile ]; then
-			zpool import -c $cachefile -a -N && break
+			zpool import -c $cachefile -a -N
+			if [ $? -ne 0 ]; then
+				echo "Import of zpool cache ${cachefile} failed," \
+				    "will retry after root mount hold release"
+				root_hold_wait
+				zpool import -c $cachefile -a -N
+			fi
+			break
 		fi
 	done
 }