Commit Graph

362 Commits

Author SHA1 Message Date
Pawel Jakub Dawidek
51ea07d722 Now that ioctl(2) is allowed in capability mode and we can limit ioctls for the
given descriptors, use Capsicum sandboxing for hastd in primary and secondary
modes. Allow for DIOCGDELETE and DIOCGFLUSH ioctls on provider descriptor and
for G_GATE_CMD_MODIFY, G_GATE_CMD_START, G_GATE_CMD_DONE and G_GATE_CMD_DESTROY
on GEOM Gate descriptor.

Sponsored by:	The FreeBSD Foundation
2013-03-14 23:14:47 +00:00
Pawel Jakub Dawidek
9cb0633b1c Minor corrections. 2013-03-14 23:11:52 +00:00
Pawel Jakub Dawidek
9bb2b7f535 Delete requests can be larger than MAXPHYS. 2013-03-14 23:03:48 +00:00
Simon J. Gerraty
7cf3a1c6b2 Updated dependencies 2013-03-11 17:21:52 +00:00
Mikolaj Golub
2adbba660d Add i/o error counters to hastd(8) and make hastctl(8) display
them.  This may be useful for detecting problems with HAST disks.

Discussed with and reviewed by:	pjd
MFC after:	1 week
2013-02-25 20:09:07 +00:00
Pawel Jakub Dawidek
d6e636c988 - Add support for 'memsync' mode. This is the fastest replication mode that's
why it will now be the default.
- Bump protocol version to 2 and add backward compatibility for version 1.
- Allow to specify hosts by kern.hostid as well (in addition to hostname and
  kern.hostuuid) in configuration file.

Sponsored by:	Panzura
Tested by:	trociny
2013-02-17 21:12:34 +00:00
Simon J. Gerraty
f5f7c05209 Updated dependencies 2013-02-16 01:23:54 +00:00
David E. O'Brien
d9a447559b Sync with HEAD. 2013-02-08 16:10:16 +00:00
Kevin Lo
a5752d55e0 Fix socket calls on error post-r243965.
Submitted by:	Garrett Cooper
2012-12-21 15:54:13 +00:00
Pawel Jakub Dawidek
771a67330f Revert r228695. We use __func__ here as a format to distinguish between
abort and assert. It would be cleaner to use NULL or "" here, but gcc
complains in both cases.
2012-11-05 00:38:14 +00:00
Simon J. Gerraty
23090366f7 Sync from head 2012-11-04 02:52:03 +00:00
Marcel Moolenaar
7750ad47a9 Sync FreeBSD's bmake branch with Juniper's internal bmake branch.
Requested by: Simon Gerraty <sjg@juniper.net>
2012-08-22 19:25:57 +00:00
Mikolaj Golub
cfbd0d18bb Metaflush on/off values don't need quotes.
Reviewed by:	pjd
MFC after:	3 days
2012-07-16 20:43:28 +00:00
Pawel Jakub Dawidek
c66ee1b31e Make use of GEOM Gate direct reads feature. This allows HAST to serve
reads with native speed of the underlying provider.
There are three situations when direct reads are not used:
1. Data is being synchronized and synchronization source is the secondary
   node, which means secondary node has more recent data and we should read
   from it.
2. Local read failed and we have to try to read from the secondary node.
3. Local component is unavailable and all I/O requests are served from the
   secondary node.

Sponsored by:	Panzura, http://www.panzura.com
MFC after:	1 month
2012-07-04 20:20:48 +00:00
Pawel Jakub Dawidek
4c13f63cba Check if there is cmsg at all.
MFC after:	3 days
2012-07-01 16:26:07 +00:00
Hans Petter Selasky
4f2380c0cf Revert: r236909
Pointyhat: me
2012-06-11 20:27:52 +00:00
Hans Petter Selasky
9ae652f11f Use the correct clock source when computing timeouts.
MFC after:	1 week
2012-06-11 19:20:59 +00:00
Pawel Jakub Dawidek
4c71d26341 Simplify the code by using snprlcat().
MFC after:	3 days
2012-06-03 10:50:46 +00:00
Warren Block
344c81a166 Fixes to man8 groff mandoc style, usage mistakes, or typos.
PR:		168016
Submitted by:	Nobuyuki Koganemaru
Approved by:	gjb
MFC after:	3 days
2012-05-24 02:24:03 +00:00
Baptiste Daroussin
5e2a209a27 Fix world after byacc import:
- old yacc(1) use to magicially append stdlib.h, while new one don't
- new yacc(1) do declare yyparse by itself, fix redundant declaration of
  'yyparse'

Approved by:	des (mentor)
2012-05-22 16:33:10 +00:00
Glen Barber
7b1d17a1bc General mdoc(7) and typo fixes.
PR:		167804
Submitted by:	Nobuyuki Koganemaru (kogane!jp.freebsd.org)
MFC after:	3 days
2012-05-12 15:08:22 +00:00
Mikolaj Golub
7f995f6907 If hastd is invoked with "-P pidfile" option always create pidfile
regardless of whether -F (foreground) option is set or not.

Also, if -P option is specified, ignore pidfile setting from configuration
not only on start but on reload too. This fixes the issue when for hastd
run with -P option reload caused the pidfile change.

Reviewed by:	pjd
MFC after:	1 week
2012-03-29 20:11:16 +00:00
Mikolaj Golub
f9c5a09cfe Fix typo.
MFC after:	3 days
2012-03-23 20:18:48 +00:00
Pawel Jakub Dawidek
86b914642c Nice range comparison.
MFC after:	3 days
2012-02-11 16:41:52 +00:00
Mikolaj Golub
f737157838 If a local write request is from the synchronization thread, when it
is synchronizing data that is out of date on the local component, we
should not send G_GATE_CMD_DONE acknowledge to the kernel.

This fixes the issue, observed in async mode, when on synchronization
from the remote component the worker terminated with "G_GATE_CMD_DONE
failed" error.

Reported by:	Artem Kajalainen <artem kayalaynen ru>
Reviewed by:	pjd
MFC after:	1 week
2012-02-05 15:23:32 +00:00
Mikolaj Golub
2b2cb41812 Fix the regression introduced in r226859: if the local component is
out of date BIO_READ requests got lost instead of being sent to the
remote component.

Reviewed by:	pjd
MFC after:	1 week
2012-02-05 15:21:08 +00:00
Pawel Jakub Dawidek
e0a8ef9d47 Fix typo in comment.
MFC after:	3 days
2012-02-04 07:59:12 +00:00
Pawel Jakub Dawidek
f17b67e14b - Fix documentation to note that /etc/hast.conf is the default configuration
file for hastd(8) and hastctl(8) and not hast.conf.
- In copyright statement correct that this file is documentation, not software.
- Bump date.

MFC after:	3 days
2012-01-24 23:43:13 +00:00
Pawel Jakub Dawidek
4ed472a1cc Free memory that won't be used in child.
MFC after:	1 week
2012-01-22 11:20:42 +00:00
Pawel Jakub Dawidek
2ce9c023fd Fix minor memory leak.
MFC after:	3 days
2012-01-21 20:13:37 +00:00
Pawel Jakub Dawidek
ba1fa0f17e Remove another unused token.
MFC after:	3 days
2012-01-20 21:49:56 +00:00
Pawel Jakub Dawidek
77213228fd Remove unused token 'port'.
MFC after:	3 days
2012-01-20 21:45:24 +00:00
Pawel Jakub Dawidek
3ab1c5a619 Style cleanups.
MFC after:	3 days
2012-01-13 23:25:35 +00:00
Pawel Jakub Dawidek
4bbeefbb07 - Fix a bug where pidfile was removed in SIGHUP when it hasn't changed in
configuration file.
- Log the fact that pidfile has changed.

MFC after:	3 days
2012-01-10 22:41:09 +00:00
Pawel Jakub Dawidek
2b1b224d24 For functions that return -1 on failure check exactly for -1 and not for
any negative number.

MFC after:	3 days
2012-01-10 22:39:07 +00:00
Pawel Jakub Dawidek
45bd093cb0 Don't touch pidfiles when running in foreground. Before that change we
would create an empty pidfile on start and check if it changed on SIGHUP.

MFC after:	3 days
2012-01-10 22:24:57 +00:00
Ulrich Spörlein
4b85a12f71 Spelling fixes for sbin/ 2012-01-07 16:09:33 +00:00
Pawel Jakub Dawidek
dfb1aece41 fork(2) returns -1 on failure, not some random negative number.
MFC after:	3 days
2012-01-06 23:44:26 +00:00
Pawel Jakub Dawidek
f78fe2608a Constify argument.
MFC after:	3 days
2012-01-06 12:27:17 +00:00
Dimitry Andric
6130c10567 Use NO_WCAST_ALIGN for usr.bin/hastctl and usr.bin/hastd; the alignment
warnings in sbin/hastd/lzf.c are only emitted for i386 and amd64, and
there they can be safely ignored.

MFC after:	1 week
2011-12-19 15:46:15 +00:00
Pawel Jakub Dawidek
8a605b3f64 Use lex's standard way of not generating unused function.
Inspired by:	r228555
MFC after:	1 week
2011-12-18 20:41:58 +00:00
Pawel Jakub Dawidek
b6afd24f2b Don't use function name as format string.
Detected by:	clang
MFC after:	1 week
2011-12-18 20:40:19 +00:00
Pawel Jakub Dawidek
f59936d642 Remove redundant assignment.
Found by:	Clang Static Analyzer
MFC after:	1 week
2011-12-15 22:05:23 +00:00
Pawel Jakub Dawidek
8c63ee4ff7 Simplify code by changing functions types from int to avoid, as the functions
always return 0.

Found by:	Clang Static Analyzer
MFC after:	1 week
2011-12-15 22:03:17 +00:00
Pawel Jakub Dawidek
b720f4aad0 Remove redundant setting of the error variable.
Found by:	Clang Static Analyzer
MFC after:	1 week
2011-12-15 22:01:34 +00:00
Pawel Jakub Dawidek
0ebcf9e6cb Remove redundant space.
MFC after:	3 days
2011-10-27 20:36:35 +00:00
Pawel Jakub Dawidek
07ebc3626e Implement 'async' mode for HAST.
MFC after:	3 days
2011-10-27 20:32:57 +00:00
Pawel Jakub Dawidek
3f5bce1822 Minor cleanups.
MFC after:	3 days
2011-10-27 20:15:37 +00:00
Pawel Jakub Dawidek
43b8675beb Reduce indentation.
MFC after:	3 days
2011-10-27 20:13:39 +00:00
Pawel Jakub Dawidek
5a58d22a84 Improve comment so it doesn't suggest race is possible, but that we handle
the race.

MFC after:	3 days
2011-10-27 20:10:21 +00:00
Pawel Jakub Dawidek
949350bb1f - Eliminate the need for hio_nv.
- Introduce hio_clear() function for clearing hio before returning it
  onto free queue.

MFC after:	3 days
2011-10-27 20:01:23 +00:00
Pawel Jakub Dawidek
1212a85c4a Monor cleanups.
MFC after:	3 days
2011-10-27 18:49:16 +00:00
Pawel Jakub Dawidek
8a34134ac2 Delay resuid generation until first connection to secondary, not until first
write. This way on first connection we will synchronize only the extents that
were modified during the lifetime of primary node, not entire GEOM provider.

MFC after:	3 days
2011-10-27 18:45:01 +00:00
Pawel Jakub Dawidek
982369192e Correct comments.
MFC after:	3 days
2011-10-27 16:22:17 +00:00
Pawel Jakub Dawidek
bd738d630c Allow to specify pidfile in HAST configuration file.
MFC after:	1 week
2011-10-17 12:22:09 +00:00
Pawel Jakub Dawidek
89da1a23cd Remove redundant space.
MFC after:	1 week
2011-10-17 09:59:04 +00:00
Pawel Jakub Dawidek
6fea20e297 When path to the configuration file is relative, obtain full path,
so we can always find the file, even after daemonizing and changing
working directory to /.

MFC after:	1 week
2011-10-17 09:54:07 +00:00
Pawel Jakub Dawidek
e3feec94eb Correct typo.
MFC after:	3 days
2011-09-28 13:25:27 +00:00
Pawel Jakub Dawidek
12daf727f6 If the underlying provider doesn't support BIO_FLUSH, log it only once
and don't bother trying in the future.

MFC after:	3 days
2011-09-28 13:19:47 +00:00
Pawel Jakub Dawidek
39852ce89e Break a bit earlier.
MFC after:	3 days
2011-09-28 13:13:43 +00:00
Pawel Jakub Dawidek
518dd4c0d9 After every activemap change flush disk's write cache, so that write
reordering won't make the actual write to be committed before marking
the coresponding extent as dirty.

It can be disabled in configuration file.

If BIO_FLUSH is not supported by the underlying file system we log a warning
and never send BIO_FLUSH again to that GEOM provider.

MFC after:	3 days
2011-09-28 13:08:51 +00:00
Pawel Jakub Dawidek
adf8002bac Use PJDLOG_ASSERT() and PJDLOG_ABORT() everywhere instead of assert().
MFC after:	3 days
2011-09-27 08:50:37 +00:00
Pawel Jakub Dawidek
be1143efb9 No need to wrap pjdlog functions around with KEEP_ERRNO() macro.
MFC after:	3 days
2011-09-27 08:26:09 +00:00
Pawel Jakub Dawidek
7f46e21d82 - Convert some impossible conditions into assertions.
- Add missing 'if' in comment.

MFC after:	3 days
2011-09-27 08:04:01 +00:00
Pawel Jakub Dawidek
09c2e8431a Correct two mistakes when converting asserts to PJDLOG_ASSERT()/PJDLOG_ABORT().
MFC after:	3 days
2011-09-27 07:59:10 +00:00
Pawel Jakub Dawidek
571fdd7e47 Prefer PJDLOG_ASSERT() and PJDLOG_ABORT() over assert() and abort().
pjdlog versions will log problem to syslog when application is running in
background.

MFC after:	3 days
2011-09-27 07:57:15 +00:00
Pawel Jakub Dawidek
1ebc0407fc No need to use KEEP_ERRNO() macro around pjdlog functions, as they don't
modify errno.

MFC after:	3 days
2011-09-27 07:52:39 +00:00
Pawel Jakub Dawidek
80c9ebc27e Ensure that pjdlog functions don't modify errno.
MFC after:	3 days
2011-09-27 06:43:51 +00:00
Mikolaj Golub
adca96f861 Fix indentation.
Approved by:	pjd (mentor)
2011-07-13 05:32:55 +00:00
Mikolaj Golub
0687d71e40 Remove useless initialization.
Approved by:	pjd (mentor)
MFC after:	3 days
2011-07-05 06:12:28 +00:00
Mikolaj Golub
d9f039e0b3 Check the returned value of activemap_write_complete() and update matadata on
disk if needed. This should fix a potential case when extents are cleared in
activemap but metadata is not updated on disk.

Suggested by:	pjd
Approved by:	pjd (mentor)
2011-06-28 21:01:32 +00:00
Mikolaj Golub
aa64b2f494 Make activemap_write_start/complete check the keepdirty list, when
stating if we need to update activemap on disk. This makes keepdirty
serve its purpose -- to reduce number of metadata updates.

Discussed with:	pjd
Approved by:	pjd (mentor)
2011-06-28 20:57:54 +00:00
Pawel Jakub Dawidek
a6de1e5c85 Compile hastd and hastctl with capsicum support.
X-MFC after:	capsicum merge
2011-06-27 09:15:41 +00:00
Pawel Jakub Dawidek
699b26bdce Compile capsicum support only if HAVE_CAPSICUM is defined.
MFC after:	3 days
2011-06-27 09:14:25 +00:00
Pawel Jakub Dawidek
133d75ed18 Log a warning if we cannot sandbox using capsicum, but only under debug level 1.
It would be too noisy to log it as a proper warning as CAPABILITIES are not
compiled into GENERIC by default.

MFC after:	3 days
2011-06-27 09:10:48 +00:00
Mikolaj Golub
ba2a822490 In HAST we use two sockets - one for only sending the data and one for
only receiving the data. In r220271 the unused directions were
disabled using shutdown(2).

Unfortunately, this broke automatic receive buffer sizing, which
currently works only for connections in ETASBLISHED state. It was a
root cause of the issue reported by users, when connection between
primary and secondary could get stuck.

Disable the code introduced in r220271 until the issue with automatic
buffer sizing is not resolved.

Reported by:	Daniel Kalchev <daniel@digsys.bg>, danger, sobomax
Tested by:	Daniel Kalchev <daniel@digsys.bg>, danger
Approved by:	pjd (mentor)
MFC after:	1 week
2011-06-17 07:07:26 +00:00
Maxim Sobolev
e0455434b4 Revert r222688.
Requested by:	Mikolaj Golub
2011-06-16 08:31:06 +00:00
Maxim Sobolev
98453c81af Read from the socket using the same max buffer size as we use while
sending. What happens otherwise is that the sender splits all the
traffic into 32k chunks, while the receiver is waiting for the whole
packet. Then for a certain packet sizes, particularly 66607 bytes in
my case, the communication stucks to secondary is expecting to
read one chunk of 66607 bytes, while primary is sending two chunks
of 32768 bytes and third chunk of 1071. Probably due to TCP windowing
and buffering the final chunk gets stuck somewhere, so neither server
not client can make any progress.

This patch also protect from short reads, as according to the manual
page there are some cases when MSG_WAITALL can give less data than
expected.

MFC after:	3 days
2011-06-04 16:01:30 +00:00
Mikolaj Golub
a01a750f32 If READ from the local node failed we send the request to the remote
node. There is no use in doing this for synchronization requests.

Approved by:	pjd (mentor)
MFC after:	1 week
2011-05-29 21:20:47 +00:00
Pawel Jakub Dawidek
3db86c39ae Keep statistics on number of BIO_READ, BIO_WRITE, BIO_DELETE and BIO_FLUSH
requests as well as number of activemap updates.

Number of BIO_WRITEs and activemap updates are especially interesting, because
if those two are too close to each other, it means that your workload needs
bigger number of dirty extents. Activemap should be updated as rarely as
possible.

MFC after:	1 week
2011-05-23 21:15:19 +00:00
Pawel Jakub Dawidek
1c6689d58d To handle BIO_FLUSH and BIO_DELETE requests in secondary worker we need
to use ioctl(2). This is why we can't use capsicum for now to sandbox
secondary. Capsicum is still used to sandbox hastctl.

MFC after:	1 week
2011-05-23 20:59:50 +00:00
Pawel Jakub Dawidek
aa27d9ef94 Recognize HIO_FLUSH requests.
MFC after:	1 week
2011-05-21 20:21:20 +00:00
Pawel Jakub Dawidek
588e8623d0 Document IPv6 support.
MFC after:	3 weeks
2011-05-20 11:21:39 +00:00
Pawel Jakub Dawidek
89bad89a59 If no listen address is specified, bind by default to:
tcp4://0.0.0.0:8457
	tcp6://[::]:8457

MFC after:	3 weeks
2011-05-20 11:16:25 +00:00
Pawel Jakub Dawidek
a87399ba7f Rename ipv4/ipv6 to tcp4/tcp6.
MFC after:	3 weeks
2011-05-20 11:15:27 +00:00
Pawel Jakub Dawidek
dc18c8ae6c Now that hell is fully frozen it is good time to add IPv6 support to HAST.
MFC after:	3 weeks
2011-05-20 11:14:05 +00:00
Pawel Jakub Dawidek
496a87aa30 Allow [ ] characters in strings. They might be used in IPv6 addresses.
MFC after:	3 weeks
2011-05-20 11:10:39 +00:00
Pawel Jakub Dawidek
bdbd046b35 Rename tcp4 to tcp in preparation for IPv6 support.
MFC after:	3 weeks
2011-05-20 11:09:02 +00:00
Pawel Jakub Dawidek
933728eea2 Rename proto_tcp4.c to proto_tcp.c in preparation for IPv6 support.
MFC after:	2 weeks
2011-05-20 11:06:17 +00:00
Pawel Jakub Dawidek
d4cb6369e6 In preparation for IPv6 support allow to specify multiple addresses to
listen on.

MFC after:	3 weeks
2011-05-19 23:18:42 +00:00
Pawel Jakub Dawidek
0855e42386 - Add support for AF_INET6 sockets for %S format character.
- Use inet_ntop(3) instead of reimplementing it.
- Use %hhu for unsigned char instead of casting it to unsigned int and
  using %u.

MFC after:	1 week
2011-05-18 22:43:56 +00:00
Pawel Jakub Dawidek
0cddb12ffd Currently we are unable to use capsicum for the primary worker process,
because we need to do ioctl(2)s, which are not permitted in the capability
mode. What we do now is to chroot(2) to /var/empty, which restricts access
to file system name space and we drop privileges to hast user and hast
group.

This still allows to access to other name spaces, like list of processes,
network and sysvipc.

To address that, use jail(2) instead of chroot(2). Using jail(2) will restrict
access to process table, network (we use ip-less jails) and sysvipc (if
security.jail.sysvipc_allowed is turned off). This provides much better
separation.

MFC after:	1 week
2011-05-14 17:02:03 +00:00
Pawel Jakub Dawidek
bcc9f32110 When using capsicum to sanbox, still use other methods first, just in case
one of them have some problems.
2011-05-14 16:55:24 +00:00
Pawel Jakub Dawidek
b64a692b64 Allow to specify remote as 'none' again which was broken by r219351, where
'none' was defined as a value for checksum.

Reported by:	trasz
MFC after:	1 week
2011-05-08 11:10:56 +00:00
Mikolaj Golub
0d9d733c57 Fix isitme(), which is used to check if node-specific configuration
belongs to our node, and was returning false positive if the first
part of a node name matches short hostname.

Approved by:	pjd (mentor)
2011-05-08 09:31:17 +00:00
Mikolaj Golub
cfd2b3fb51 Add missing ifdef. This fixes build with NO_OPENSSL.
Reported by:	Pawel Tyll <ptyll@nitronet.pl>
Approved by:	pjd (mentor)
MFC after:	1 week
2011-04-26 19:52:21 +00:00
Mikolaj Golub
7a2b83683d Rename HASTCTL_ defines, which are used for conversion between main
hastd process and workers, remove unused one and set different range
of numbers. This is done in order not to confuse them with HASTCTL_CMD
defines, used for conversation between hastctl and hastd, and to avoid
bugs like the one fixed in in r221075.

Approved by:	pjd (mentor)
MFC after:	1 week
2011-04-26 19:38:30 +00:00
Mikolaj Golub
1768fba5c0 For conversation between hastctl and hastd we should use HASTCTL_CMD
defines.

Approved by:	pjd (mentor)
MFC after:	1 week
2011-04-26 19:22:54 +00:00
Pawel Jakub Dawidek
8e15d1e490 Correct comment.
MFC after:	1 week
2011-04-20 18:49:12 +00:00
Pawel Jakub Dawidek
ac0401e321 When we become primary, we connect to the remote and expect it to be in
secondary role. It is possible that the remote node is primary, but only
because there was a role change and it didn't finish cleaning up (unmounting
file systems, etc.). If we detect such situation, wait for the remote node
to switch the role to secondary before accepting I/Os. If we don't wait for
it in that case, we will most likely cause split-brain.

MFC after:	1 week
2011-04-20 18:43:28 +00:00