Commit Graph

55 Commits

Author SHA1 Message Date
Pawel Jakub Dawidek
9bb2b7f535 Delete requests can be larger than MAXPHYS. 2013-03-14 23:03:48 +00:00
Mikolaj Golub
2adbba660d Add i/o error counters to hastd(8) and make hastctl(8) display
them.  This may be useful for detecting problems with HAST disks.

Discussed with and reviewed by:	pjd
MFC after:	1 week
2013-02-25 20:09:07 +00:00
Pawel Jakub Dawidek
d6e636c988 - Add support for 'memsync' mode. This is the fastest replication mode that's
why it will now be the default.
- Bump protocol version to 2 and add backward compatibility for version 1.
- Allow to specify hosts by kern.hostid as well (in addition to hostname and
  kern.hostuuid) in configuration file.

Sponsored by:	Panzura
Tested by:	trociny
2013-02-17 21:12:34 +00:00
Pawel Jakub Dawidek
3ab1c5a619 Style cleanups.
MFC after:	3 days
2012-01-13 23:25:35 +00:00
Pawel Jakub Dawidek
2b1b224d24 For functions that return -1 on failure check exactly for -1 and not for
any negative number.

MFC after:	3 days
2012-01-10 22:39:07 +00:00
Pawel Jakub Dawidek
dfb1aece41 fork(2) returns -1 on failure, not some random negative number.
MFC after:	3 days
2012-01-06 23:44:26 +00:00
Pawel Jakub Dawidek
f59936d642 Remove redundant assignment.
Found by:	Clang Static Analyzer
MFC after:	1 week
2011-12-15 22:05:23 +00:00
Pawel Jakub Dawidek
0ebcf9e6cb Remove redundant space.
MFC after:	3 days
2011-10-27 20:36:35 +00:00
Pawel Jakub Dawidek
949350bb1f - Eliminate the need for hio_nv.
- Introduce hio_clear() function for clearing hio before returning it
  onto free queue.

MFC after:	3 days
2011-10-27 20:01:23 +00:00
Pawel Jakub Dawidek
982369192e Correct comments.
MFC after:	3 days
2011-10-27 16:22:17 +00:00
Pawel Jakub Dawidek
12daf727f6 If the underlying provider doesn't support BIO_FLUSH, log it only once
and don't bother trying in the future.

MFC after:	3 days
2011-09-28 13:19:47 +00:00
Pawel Jakub Dawidek
39852ce89e Break a bit earlier.
MFC after:	3 days
2011-09-28 13:13:43 +00:00
Pawel Jakub Dawidek
571fdd7e47 Prefer PJDLOG_ASSERT() and PJDLOG_ABORT() over assert() and abort().
pjdlog versions will log problem to syslog when application is running in
background.

MFC after:	3 days
2011-09-27 07:57:15 +00:00
Mikolaj Golub
ba2a822490 In HAST we use two sockets - one for only sending the data and one for
only receiving the data. In r220271 the unused directions were
disabled using shutdown(2).

Unfortunately, this broke automatic receive buffer sizing, which
currently works only for connections in ETASBLISHED state. It was a
root cause of the issue reported by users, when connection between
primary and secondary could get stuck.

Disable the code introduced in r220271 until the issue with automatic
buffer sizing is not resolved.

Reported by:	Daniel Kalchev <daniel@digsys.bg>, danger, sobomax
Tested by:	Daniel Kalchev <daniel@digsys.bg>, danger
Approved by:	pjd (mentor)
MFC after:	1 week
2011-06-17 07:07:26 +00:00
Pawel Jakub Dawidek
3db86c39ae Keep statistics on number of BIO_READ, BIO_WRITE, BIO_DELETE and BIO_FLUSH
requests as well as number of activemap updates.

Number of BIO_WRITEs and activemap updates are especially interesting, because
if those two are too close to each other, it means that your workload needs
bigger number of dirty extents. Activemap should be updated as rarely as
possible.

MFC after:	1 week
2011-05-23 21:15:19 +00:00
Pawel Jakub Dawidek
aa27d9ef94 Recognize HIO_FLUSH requests.
MFC after:	1 week
2011-05-21 20:21:20 +00:00
Pawel Jakub Dawidek
0cddb12ffd Currently we are unable to use capsicum for the primary worker process,
because we need to do ioctl(2)s, which are not permitted in the capability
mode. What we do now is to chroot(2) to /var/empty, which restricts access
to file system name space and we drop privileges to hast user and hast
group.

This still allows to access to other name spaces, like list of processes,
network and sysvipc.

To address that, use jail(2) instead of chroot(2). Using jail(2) will restrict
access to process table, network (we use ip-less jails) and sysvipc (if
security.jail.sysvipc_allowed is turned off). This provides much better
separation.

MFC after:	1 week
2011-05-14 17:02:03 +00:00
Pawel Jakub Dawidek
06cbf54941 Scenario:
- We have two nodes connected and synchronized (local counters on both sides
  are 0).
- We take secondary down and recreate it.
- Primary connects to it and starts synchronization (but local counters are
  still 0).
- We switch the roles.
- Synchronization restarts but data is synchronized now from new primary
  (because local counters are 0) that doesn't have new data yet.

This fix this issue we bump local counter on primary when we discover that
connected secondary was recreated and has no data yet.

Reported by:	trociny
Discussed with:	trociny
Tested by:	trociny
MFC after:	1 week
2011-04-19 19:26:27 +00:00
Pawel Jakub Dawidek
02dfe9724c Declare directions for sockets between primary and secondary.
In HAST we use two sockets - one for only sending the data and one for only
receiving the data.

MFC after:	1 month
2011-04-02 09:25:13 +00:00
Pawel Jakub Dawidek
54987cacfd Add mapsize to the header just before sending the packet.
Before it could change later and we were sending invalid mapsize.
Some time ago I added optimization where when nodes are connected for the
first time and there were no writes to them yet, there is no initial full
synchronization. This bug prevented it from working.

MFC after:	1 week
2011-03-25 20:19:15 +00:00
Pawel Jakub Dawidek
643080b75f Use role2str() when setting process title.
MFC after:	1 week
2011-03-25 20:13:38 +00:00
Pawel Jakub Dawidek
640b7ee623 Don't create socketpair for connection forwarding between parent and secondary.
Secondary doesn't need to connect anywhere.

MFC after:	1 week
2011-03-23 11:09:04 +00:00
Pawel Jakub Dawidek
cd72d521e3 White space cleanups.
MFC after:	1 week
2011-03-22 10:39:34 +00:00
Pawel Jakub Dawidek
4d8dc3b838 When dropping privileges prefer capsicum over chroot+setgid+setuid.
We can use capsicum for secondary worker processes and hastctl.
When working as primary we drop privileges using chroot+setgid+setuid
still as we need to send ioctl(2)s to ggate device, for which capsicum
doesn't allow (yet).

X-MFC after:	capsicum is merged to stable/8
2011-03-21 21:31:50 +00:00
Pawel Jakub Dawidek
756cb15420 Fix typo.
MFC after:	1 week
2011-03-21 21:14:07 +00:00
Pawel Jakub Dawidek
e208a185f0 Be pedantic and free nvout before exiting.
MFC after:	1 week
2011-03-21 14:51:16 +00:00
Pawel Jakub Dawidek
38ea70cadf Detect situation where resource internal identifier differs.
This means that both nodes have separately managed resources that don't
have the same data.

MFC after:	1 week
2011-03-21 14:50:12 +00:00
Pawel Jakub Dawidek
0b626a289e In hast.conf we define the other node's address in 'remote' variable.
This way we know how to connect to secondary node when we are primary.
The same variable is used by the secondary node - it only accepts
connections from the address stored in 'remote' variable.
In cluster configurations it is common that each node has its individual
IP address and there is one addtional shared IP address which is assigned
to primary node. It seems it is possible that if the shared IP address is
from the same network as the individual IP address it might be choosen by
the kernel as a source address for connection with the secondary node.
Such connection will be rejected by secondary, as it doesn't come from
primary node individual IP.

Add 'source' variable that allows to specify source IP address we want to
bind to before connecting to the secondary node.

MFC after:	1 week
2011-03-21 08:54:59 +00:00
Mikolaj Golub
8d7dcf14ff For secondary, set 2 * HAST_KEEPALIVE seconds timeout for incoming
connection so the worker will exit if it does not receive packets from
the primary during this interval.

Reported by:	Christian Vogt <Christian.Vogt@haw-hamburg.de>
Tested by:	Christian Vogt <Christian.Vogt@haw-hamburg.de>
Approved by:	pjd (mentor)
MFC after:	1 week
2011-03-17 21:02:14 +00:00
Mikolaj Golub
bc7a916a25 Make workers inherit debug level from the main process.
Approved by:	pjd (mentor)
MFC after:	1 week
2011-03-11 12:12:35 +00:00
Pawel Jakub Dawidek
32ecf62028 Setup another socketpair between parent and child, so that primary sandboxed
worker can ask the main privileged process to connect in worker's behalf
and then we can migrate descriptor using this socketpair to worker.
This is not really needed now, but will be needed once we start to use
capsicum for sandboxing.

MFC after:	1 week
2011-02-03 11:39:49 +00:00
Pawel Jakub Dawidek
f4c96f944c Let the caller log info about successful privilege drop.
We don't want to log this in hastctl.

MFC after:	1 week
2011-02-03 10:37:44 +00:00
Pawel Jakub Dawidek
2ec483c58e - Use pjdlog for assertions and aborts as this will log assert/abort message
to syslog if we run in background.
- Asserts in proto.c that method we want to call is implemented and remove
  dummy methods from protocols implementation that are only there to abort
  the program with nice message.

MFC after:	1 week
2011-01-31 18:32:17 +00:00
Pawel Jakub Dawidek
6d7967de8a Drop privileges in worker processes.
Accepting connections and handshaking in secondary is still done before
dropping privileges. It should be implemented by only accepting connections in
privileged main process and passing connection descriptors to the worker, but
is not implemented yet.

MFC after:	1 week
2011-01-28 22:35:46 +00:00
Pawel Jakub Dawidek
f463896e5e Use newly added descriptors_assert() function to ensure only expected
descriptors are open.

MFC after:	1 week
2011-01-28 21:57:42 +00:00
Pawel Jakub Dawidek
da1783ea29 Close all unneeded descriptors after fork(2).
MFC after:	1 week
2011-01-28 21:52:37 +00:00
Pawel Jakub Dawidek
ce837469ba Before this change on first connect between primary and secondary we
initialize all the data. This is huge waste of time and resources if
there were no writes yet, as there is no real data to synchronize.

Optimize this by sending "virgin" argument to secondary, which gives it a hint
that synchronization is not needed.

In the common case (where noth nodes are configured at the same time) instead
of synchronizing everything, we don't synchronize at all.

MFC after:	1 week
2010-10-24 17:28:25 +00:00
Pawel Jakub Dawidek
1f39b27946 Simplify code a bit.
MFC after:	3 days
2010-10-24 15:44:23 +00:00
Pawel Jakub Dawidek
d7be7905ae Plug memory leak.
MFC after:	3 days
2010-10-24 15:42:16 +00:00
Pawel Jakub Dawidek
9dd5a6cb0f Switch to sigprocmask(2) API also in the main process and secondary process.
This way the primary process inherits signal mask from the main process,
which fixes a race where signal is delivered to the primary process before
configuring signal mask.

Reported by:	Mikolaj Golub <to.my.trociny@gmail.com>
MFC after:	3 days
2010-09-22 19:08:11 +00:00
Pawel Jakub Dawidek
8b70e6ae9c Fix possible deadlock where worker process sends an event to the main process
while the main process sends control message to the worker process, but worker
process hasn't started control thread yet, because it waits for reply from the
main process.

The fix is to start the control thread before sending any events.

Reported and fix suggested by:	Mikolaj Golub <to.my.trociny@gmail.com>
MFC after:	3 days
2010-09-22 19:03:11 +00:00
Pawel Jakub Dawidek
e43e02f1a4 Add __dead2 to functions that we know they are going to exit.
MFC after:	3 days
2010-09-20 13:23:43 +00:00
Pawel Jakub Dawidek
8ecdeae9d9 Correct error message.
Submitted by:	Mikolaj Golub <to.my.trociny@gmail.com>
MFC after:	2 weeks
2010-08-31 12:03:29 +00:00
Pawel Jakub Dawidek
5bdff860e7 Because it is very hard to make fork(2) from threaded process safe (we are
limited to async-signal safe functions in the child process), move all hooks
execution to the main (non-threaded) process.

Do it by maintaining connection (socketpair) between child and parent
and sending events from the child to parent, so it can execute the hook.

This is step in right direction for others reasons too. For example there is
one less problem to drop privs in worker processes.

MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-30 23:26:10 +00:00
Pawel Jakub Dawidek
5b41e64486 Execute hook when connection between the nodes is established or lost.
MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-30 00:31:30 +00:00
Pawel Jakub Dawidek
2be8fd75ff Execute hook when split-brain is detected.
MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-30 00:12:10 +00:00
Pawel Jakub Dawidek
ecc99c890e Allow to run hooks from the main hastd process.
MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-29 21:41:53 +00:00
Pawel Jakub Dawidek
f7fe83f9f8 Implement keepalive mechanism inside HAST protocol so we can detect secondary
node failures quickly for HAST resources that are rarely modified.

Remove XXX from a comment now that the guard thread never sleeps infinitely.

MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-27 14:26:37 +00:00
Pawel Jakub Dawidek
16bd7026a2 Add QUEUE_INSERT() and QUEUE_TAKE() macros that simplify the code a bit.
MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-27 14:01:28 +00:00
Pawel Jakub Dawidek
a2ef0636b4 Reset signal handlers after fork().
MFC after:	1 month
2010-08-05 18:58:00 +00:00