Commit Graph

61 Commits

Author SHA1 Message Date
Pawel Jakub Dawidek
9446b4536e Initialize localcnt on first write. This fixes assertion when we create
resource, set role to primary, do no writes, then sent it to secondary
and accept connection from primary.

MFC after:	1 week
2011-03-21 21:16:12 +00:00
Pawel Jakub Dawidek
0b626a289e In hast.conf we define the other node's address in 'remote' variable.
This way we know how to connect to secondary node when we are primary.
The same variable is used by the secondary node - it only accepts
connections from the address stored in 'remote' variable.
In cluster configurations it is common that each node has its individual
IP address and there is one addtional shared IP address which is assigned
to primary node. It seems it is possible that if the shared IP address is
from the same network as the individual IP address it might be choosen by
the kernel as a source address for connection with the secondary node.
Such connection will be rejected by secondary, as it doesn't come from
primary node individual IP.

Add 'source' variable that allows to specify source IP address we want to
bind to before connecting to the secondary node.

MFC after:	1 week
2011-03-21 08:54:59 +00:00
Mikolaj Golub
8d7dcf14ff For secondary, set 2 * HAST_KEEPALIVE seconds timeout for incoming
connection so the worker will exit if it does not receive packets from
the primary during this interval.

Reported by:	Christian Vogt <Christian.Vogt@haw-hamburg.de>
Tested by:	Christian Vogt <Christian.Vogt@haw-hamburg.de>
Approved by:	pjd (mentor)
MFC after:	1 week
2011-03-17 21:02:14 +00:00
Mikolaj Golub
bc7a916a25 Make workers inherit debug level from the main process.
Approved by:	pjd (mentor)
MFC after:	1 week
2011-03-11 12:12:35 +00:00
Pawel Jakub Dawidek
fa356f6cfe - Log size of data to synchronize in human readable form (using %N).
- Log synchronization time (using %T).
- Log synchronization speed in human readable form (using %N).

MFC after:	2 weeks
2011-03-07 10:41:12 +00:00
Pawel Jakub Dawidek
8cd3d45ad9 Allow to compress on-the-wire data using two algorithms:
- HOLE - it simply turns all-zero blocks into few bytes header;
	it is extremely fast, so it is turned on by default;
	it is mostly intended to speed up initial synchronization
	where we expect many zeros;
- LZF - very fast algorithm by Marc Alexander Lehmann, which shows
	very decent compression ratio and has BSD license.

MFC after:	2 weeks
2011-03-06 23:09:33 +00:00
Pawel Jakub Dawidek
1fee97b01f Allow to checksum on-the-wire data using either CRC32 or SHA256.
MFC after:	2 weeks
2011-03-06 22:56:14 +00:00
Pawel Jakub Dawidek
32ecf62028 Setup another socketpair between parent and child, so that primary sandboxed
worker can ask the main privileged process to connect in worker's behalf
and then we can migrate descriptor using this socketpair to worker.
This is not really needed now, but will be needed once we start to use
capsicum for sandboxing.

MFC after:	1 week
2011-02-03 11:39:49 +00:00
Pawel Jakub Dawidek
21e7bc5e52 Add missing locking after moving keepalive_send() to remote send thread
in r214692.

MFC after:	1 week
2011-02-03 11:33:32 +00:00
Pawel Jakub Dawidek
f4c96f944c Let the caller log info about successful privilege drop.
We don't want to log this in hastctl.

MFC after:	1 week
2011-02-03 10:37:44 +00:00
Pawel Jakub Dawidek
9d70b24b93 Allow to specify connection timeout by the caller.
MFC after:	1 week
2011-02-02 15:42:00 +00:00
Pawel Jakub Dawidek
2ec483c58e - Use pjdlog for assertions and aborts as this will log assert/abort message
to syslog if we run in background.
- Asserts in proto.c that method we want to call is implemented and remove
  dummy methods from protocols implementation that are only there to abort
  the program with nice message.

MFC after:	1 week
2011-01-31 18:32:17 +00:00
Pawel Jakub Dawidek
6d7967de8a Drop privileges in worker processes.
Accepting connections and handshaking in secondary is still done before
dropping privileges. It should be implemented by only accepting connections in
privileged main process and passing connection descriptors to the worker, but
is not implemented yet.

MFC after:	1 week
2011-01-28 22:35:46 +00:00
Pawel Jakub Dawidek
f463896e5e Use newly added descriptors_assert() function to ensure only expected
descriptors are open.

MFC after:	1 week
2011-01-28 21:57:42 +00:00
Pawel Jakub Dawidek
da1783ea29 Close all unneeded descriptors after fork(2).
MFC after:	1 week
2011-01-28 21:52:37 +00:00
Pawel Jakub Dawidek
d64c0992e4 Add comments to places where we treat errors as ciritical, but it is possible
to handle them more gracefully.

MFC after:	1 week
2011-01-28 21:51:40 +00:00
Pawel Jakub Dawidek
115f4e5c3e Don't open configuration file from worker process. Handle SIGHUP in the
master process only and pass changes to the worker processes over control
socket. This removes access to global namespace in preparation for capsicum
sandboxing.

MFC after:	2 weeks
2011-01-24 15:04:15 +00:00
Pawel Jakub Dawidek
fba1bf5a2c The 'ret' variable is of type ssize_t and we use proper format for it (%zd), so
no (bogus) cast is needed.

MFC after:	3 days
2010-12-16 19:48:03 +00:00
Pawel Jakub Dawidek
cd7b7ee577 Improve problems logging.
MFC after:	3 days
2010-12-16 07:30:47 +00:00
Pawel Jakub Dawidek
7208920499 Don't ignore errors from remote requests.
MFC after:	3 days
2010-12-16 07:29:58 +00:00
Pawel Jakub Dawidek
d448536ceb Move timeout.tv_sec initialization outside the loop - sigtimedwait(2) won't
modify it.

Submitted by:	Mikolaj Golub <to.my.trociny@gmail.com>
MFC after:	3 days
2010-11-15 03:07:42 +00:00
Pawel Jakub Dawidek
1dd5a4bfa2 1. Exit when we cannot create incoming connection.
2. Improve logging to inform which connection can't be created.

Submitted by:	[1] Mikolaj Golub <to.my.trociny@gmail.com>
MFC after:	3 days
2010-11-15 03:05:33 +00:00
Pawel Jakub Dawidek
448efa9421 Send packets to remote node only via the send thread to avoid possible
races - in this case a keepalive packet was send from wrong thread which
lead to connection dropping, because of corrupted packet.

Fix it by sending keepalive packets directly from the send thread.
As a bonus we now send keepalive packets only when connection is idle.

Submitted by:	Mikolaj Golub <to.my.trociny@gmail.com>
MFC after:	3 days
2010-11-02 22:13:08 +00:00
Pawel Jakub Dawidek
ce837469ba Before this change on first connect between primary and secondary we
initialize all the data. This is huge waste of time and resources if
there were no writes yet, as there is no real data to synchronize.

Optimize this by sending "virgin" argument to secondary, which gives it a hint
that synchronization is not needed.

In the common case (where noth nodes are configured at the same time) instead
of synchronizing everything, we don't synchronize at all.

MFC after:	1 week
2010-10-24 17:28:25 +00:00
Pawel Jakub Dawidek
584a9bc3f8 Plug memory leaks.
Found with:	valgrind
MFC after:	3 days
2010-10-24 15:41:23 +00:00
Pawel Jakub Dawidek
51c63dce86 We can't zero out ggio request, as we have some fields in there we initialize
once during start-up.

Reported by:	Mikolaj Golub <to.my.trociny@gmail.com>
MFC after:	3 days
2010-10-08 15:05:39 +00:00
Pawel Jakub Dawidek
4e47b646bb Clear ggate structures before using them. We don't initialize all the field
and there can be some garbage from the stack.

MFC after:	1 week
2010-10-07 18:23:28 +00:00
Pawel Jakub Dawidek
783ee75392 Log error message when we fail to destroy ggate provider.
MFC after:	3 days
2010-10-07 18:20:16 +00:00
Pawel Jakub Dawidek
4a88128b01 Start the guard thread first, so we can handle signals from the very begining.
Reported by:	Mikolaj Golub <to.my.trociny@gmail.com>
MFC after:	1 week
2010-10-07 18:19:02 +00:00
Pawel Jakub Dawidek
b46198a5db Don't close local component on exit as we can hang waiting on g_waitidle.
I'm unable to reproduce the race described in comment anymore and also the
comment is incorrect - localfd represents local component from configuration
file, eg. /dev/da0 and not HAST provider.

Reported by:	Mikolaj Golub <to.my.trociny@gmail.com>
MFC after:	1 week
2010-10-07 18:16:22 +00:00
Pawel Jakub Dawidek
9dd5a6cb0f Switch to sigprocmask(2) API also in the main process and secondary process.
This way the primary process inherits signal mask from the main process,
which fixes a race where signal is delivered to the primary process before
configuring signal mask.

Reported by:	Mikolaj Golub <to.my.trociny@gmail.com>
MFC after:	3 days
2010-09-22 19:08:11 +00:00
Pawel Jakub Dawidek
8b70e6ae9c Fix possible deadlock where worker process sends an event to the main process
while the main process sends control message to the worker process, but worker
process hasn't started control thread yet, because it waits for reply from the
main process.

The fix is to start the control thread before sending any events.

Reported and fix suggested by:	Mikolaj Golub <to.my.trociny@gmail.com>
MFC after:	3 days
2010-09-22 19:03:11 +00:00
Pawel Jakub Dawidek
e43e02f1a4 Add __dead2 to functions that we know they are going to exit.
MFC after:	3 days
2010-09-20 13:23:43 +00:00
Pawel Jakub Dawidek
852ac373cb Mask only those signals that we want to handle.
Suggested by:	jilles
MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-31 06:22:03 +00:00
Pawel Jakub Dawidek
5bdff860e7 Because it is very hard to make fork(2) from threaded process safe (we are
limited to async-signal safe functions in the child process), move all hooks
execution to the main (non-threaded) process.

Do it by maintaining connection (socketpair) between child and parent
and sending events from the child to parent, so it can execute the hook.

This is step in right direction for others reasons too. For example there is
one less problem to drop privs in worker processes.

MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-30 23:26:10 +00:00
Pawel Jakub Dawidek
6be3a25c85 Use pjdlog_exit() before fork().
MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-30 22:28:04 +00:00
Pawel Jakub Dawidek
5b41e64486 Execute hook when connection between the nodes is established or lost.
MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-30 00:31:30 +00:00
Pawel Jakub Dawidek
2be8fd75ff Execute hook when split-brain is detected.
MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-30 00:12:10 +00:00
Pawel Jakub Dawidek
6d0c801ea9 Use sigtimedwait(2) for signals handling in primary process.
This fixes various races and eliminates use of pthread* API in signal handler.

Pointed out by:	kib
With help from:	jilles
MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-30 00:06:05 +00:00
Pawel Jakub Dawidek
ff6bb1f8b3 - Move functionality responsible for checking one connection to separate
function to make code more readable.
- Be sure not to reconnect too often in case of signal delivery, etc.

MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-29 22:55:21 +00:00
Pawel Jakub Dawidek
ee087cdf97 Disconnect after logging errors.
MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-29 22:17:53 +00:00
Pawel Jakub Dawidek
ecc99c890e Allow to run hooks from the main hastd process.
MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-29 21:41:53 +00:00
Pawel Jakub Dawidek
b9cf0cf5fa Correct when we log interrupted synchronization.
MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-27 21:20:32 +00:00
Pawel Jakub Dawidek
eba09893fd Check if no signals were delivered just before going to sleep.
MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-27 20:49:06 +00:00
Pawel Jakub Dawidek
01125a9381 Add hooks execution.
MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-27 20:48:12 +00:00
Pawel Jakub Dawidek
0becad39a7 Allow to execute specified program on various HAST events.
MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-27 15:16:52 +00:00
Pawel Jakub Dawidek
f7fe83f9f8 Implement keepalive mechanism inside HAST protocol so we can detect secondary
node failures quickly for HAST resources that are rarely modified.

Remove XXX from a comment now that the guard thread never sleeps infinitely.

MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-27 14:26:37 +00:00
Pawel Jakub Dawidek
8f8c798c13 - Remove redundant and incorrect 'old' word from debug message.
- Log disconnects as warnings.

MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-27 14:12:53 +00:00
Pawel Jakub Dawidek
e23d2d0187 Don't increase number synchronized bytes in case of an error.
MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-27 14:10:25 +00:00
Pawel Jakub Dawidek
53d9b386eb Log that synchronization was interrupted in a proper place.
MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-27 14:08:10 +00:00