Commit Graph

50 Commits

Author SHA1 Message Date
Pawel Jakub Dawidek
8e15d1e490 Correct comment.
MFC after:	1 week
2011-04-20 18:49:12 +00:00
Pawel Jakub Dawidek
ac0401e321 When we become primary, we connect to the remote and expect it to be in
secondary role. It is possible that the remote node is primary, but only
because there was a role change and it didn't finish cleaning up (unmounting
file systems, etc.). If we detect such situation, wait for the remote node
to switch the role to secondary before accepting I/Os. If we don't wait for
it in that case, we will most likely cause split-brain.

MFC after:	1 week
2011-04-20 18:43:28 +00:00
Pawel Jakub Dawidek
0eb02c031b If we act in different role than requested by the remote node, log it
as a warning and not an error.

MFC after:	1 week
2011-04-20 16:38:05 +00:00
Pawel Jakub Dawidek
640b7ee623 Don't create socketpair for connection forwarding between parent and secondary.
Secondary doesn't need to connect anywhere.

MFC after:	1 week
2011-03-23 11:09:04 +00:00
Pawel Jakub Dawidek
cd72d521e3 White space cleanups.
MFC after:	1 week
2011-03-22 10:39:34 +00:00
Pawel Jakub Dawidek
351758d85b Before handling any events on descriptors check signals so we can update
our info about worker processes if any of them was terminated in the meantime.

This fixes the problem with 'hastctl status' running from a hook called on
split-brain:
1. Secondary calls a hooks and terminates.
2. Hook asks for resource status via 'hastctl status'.
3. The main hastd handles the status request by sending it to the secondary
   worker who is already dead, but because signals weren't checked yet he
   doesn't know that and we get EPIPE.

MFC after:	1 week
2011-03-21 15:29:20 +00:00
Pawel Jakub Dawidek
0b626a289e In hast.conf we define the other node's address in 'remote' variable.
This way we know how to connect to secondary node when we are primary.
The same variable is used by the secondary node - it only accepts
connections from the address stored in 'remote' variable.
In cluster configurations it is common that each node has its individual
IP address and there is one addtional shared IP address which is assigned
to primary node. It seems it is possible that if the shared IP address is
from the same network as the individual IP address it might be choosen by
the kernel as a source address for connection with the secondary node.
Such connection will be rejected by secondary, as it doesn't come from
primary node individual IP.

Add 'source' variable that allows to specify source IP address we want to
bind to before connecting to the secondary node.

MFC after:	1 week
2011-03-21 08:54:59 +00:00
Pawel Jakub Dawidek
4f0ec4797a When creating connection on behalf of primary worker, set pjdlog prefix
to resource name and role, so that any logs related to that can be identified
properly.

MFC after:	1 week
2011-03-21 08:33:58 +00:00
Pawel Jakub Dawidek
c3a8627c9a If there is any traffic on one of out descriptors, we were not checking for
long running hooks. Fix it by not using select(2) timeout to decide if we want
to check hooks or not.

MFC after:	1 week
2011-03-21 08:31:35 +00:00
Pawel Jakub Dawidek
8cd3d45ad9 Allow to compress on-the-wire data using two algorithms:
- HOLE - it simply turns all-zero blocks into few bytes header;
	it is extremely fast, so it is turned on by default;
	it is mostly intended to speed up initial synchronization
	where we expect many zeros;
- LZF - very fast algorithm by Marc Alexander Lehmann, which shows
	very decent compression ratio and has BSD license.

MFC after:	2 weeks
2011-03-06 23:09:33 +00:00
Pawel Jakub Dawidek
1fee97b01f Allow to checksum on-the-wire data using either CRC32 or SHA256.
MFC after:	2 weeks
2011-03-06 22:56:14 +00:00
Pawel Jakub Dawidek
e84a29b629 Now that we break the loop on fstat(2) failure we no longer need to satisfy
gcc's imperfections.

MFC after:	1 week
2011-02-06 14:17:08 +00:00
Pawel Jakub Dawidek
207ee3cdea Add (void) cast before snprintf(3)s for which we are not interested in return
values.

MFC after:	1 week
2011-02-06 14:09:19 +00:00
Pawel Jakub Dawidek
ee3a876c18 Treat fstat(2) failure (different than EBADF) as fatal error.
Reported by:	Mikolaj Golub <to.my.trociny@gmail.com>
MFC after:	1 week
2011-02-06 14:07:58 +00:00
Pawel Jakub Dawidek
18d6e1a5f6 Open syslog when logging sysconf(3) failure.
Reported by:	Mikolaj Golub <to.my.trociny@gmail.com>
MFC after:	1 week
2011-02-06 14:06:37 +00:00
Pawel Jakub Dawidek
5aa85abd1d Close more descriptors that can be open if the worker process for the given
resource is already running.

Submitted by:	Mikolaj Golub <to.my.trociny@gmail.com>
MFC after:	1 week
2011-02-06 12:21:29 +00:00
Pawel Jakub Dawidek
32ecf62028 Setup another socketpair between parent and child, so that primary sandboxed
worker can ask the main privileged process to connect in worker's behalf
and then we can migrate descriptor using this socketpair to worker.
This is not really needed now, but will be needed once we start to use
capsicum for sandboxing.

MFC after:	1 week
2011-02-03 11:39:49 +00:00
Pawel Jakub Dawidek
2ec483c58e - Use pjdlog for assertions and aborts as this will log assert/abort message
to syslog if we run in background.
- Asserts in proto.c that method we want to call is implemented and remove
  dummy methods from protocols implementation that are only there to abort
  the program with nice message.

MFC after:	1 week
2011-01-31 18:32:17 +00:00
Pawel Jakub Dawidek
579fd4b2ff Add function to assert that the only descriptors we have open are the ones
we expect to be open. Also assert that they point at expected type.

Because openlog(3) API is unable to tell us descriptor number it is using, we
have to close syslog socket, remember assert message in local buffer and if we
fail on assertion, reopen syslog socket and log the message.

MFC after:	1 week
2011-01-28 21:56:47 +00:00
Pawel Jakub Dawidek
c3c56f8e41 Add function to close all unneeded descriptors after fork(2).
MFC after:	1 week
2011-01-28 21:48:15 +00:00
Pawel Jakub Dawidek
c0dbce0016 Close the control socket before exiting, so it will be unlinked.
MFC after:	1 week
2011-01-27 19:31:35 +00:00
Pawel Jakub Dawidek
eeb3cd677d Add functions to initialize/finalize pjdlog. This allows to open/close log
file at will.

MFC after:	1 week
2011-01-27 19:24:07 +00:00
Pawel Jakub Dawidek
115f4e5c3e Don't open configuration file from worker process. Handle SIGHUP in the
master process only and pass changes to the worker processes over control
socket. This removes access to global namespace in preparation for capsicum
sandboxing.

MFC after:	2 weeks
2011-01-24 15:04:15 +00:00
Pawel Jakub Dawidek
5ed118d861 - On primary worker reload, update hr_exec field.
- Update comment.

MFC after:	1 week
2011-01-22 22:31:55 +00:00
Pawel Jakub Dawidek
9cc97e5803 Install default signal handlers before masking signals we want to handle.
It is possible that the parent process ignores some of them and sigtimedwait()
will never see them, eventhough they are masked.

The most common situation for this to happen is boot process where init(8)
ignores SIGHUP before starting to execute /etc/rc. This in turn caused
hastd(8) to ignore SIGHUP.

Reported by:	trasz
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
MFC after:	3 days
2011-01-12 14:35:29 +00:00
Pawel Jakub Dawidek
347bde360a Log the fact of launching and include protocol version number.
MFC after:	3 days
2010-12-16 07:28:40 +00:00
Pawel Jakub Dawidek
2964aeb34a Load geom_gate.ko module after parsing arguments.
MFC after:	3 days
2010-10-24 15:38:58 +00:00
Pawel Jakub Dawidek
3f562cce40 Log correct connection when canceling half-open connection.
Submitted by:	Mikolaj Golub <to.my.trociny@gmail.com>
MFC after:	3 days
2010-10-17 15:47:27 +00:00
Pawel Jakub Dawidek
428ad0a9c4 Decrease report interval to 5 seconds, as this also means we will check for
signals every 5 seconds and not every 10 seconds as before.

MFC after:	3 days
2010-10-04 21:44:26 +00:00
Pawel Jakub Dawidek
5f24b330df hook_check() is now only used to report about long-running hooks, so the
argument is redundant, remove it.

MFC after:	3 days
2010-10-04 21:43:06 +00:00
Pawel Jakub Dawidek
41013c0b21 We can't mask ignored signal, so install dummy signal hander for SIGCHLD before
masking it.

This fixes bogus reports about hooks running for too long and other problems
related to garbage-collecting child processes.

Reported by:	Mikolaj Golub <to.my.trociny@gmail.com>
MFC after:	3 days
2010-10-04 21:41:18 +00:00
Pawel Jakub Dawidek
9dd5a6cb0f Switch to sigprocmask(2) API also in the main process and secondary process.
This way the primary process inherits signal mask from the main process,
which fixes a race where signal is delivered to the primary process before
configuring signal mask.

Reported by:	Mikolaj Golub <to.my.trociny@gmail.com>
MFC after:	3 days
2010-09-22 19:08:11 +00:00
Pawel Jakub Dawidek
196abd3518 Assert that descriptor numbers are sane.
MFC after:	3 days
2010-09-22 19:05:54 +00:00
Pawel Jakub Dawidek
0c24d8e2a1 Fix descriptor leaks: when child exits, we have to close control and event
socket pairs. We did that only in one case out of three.

MFC after:	3 days
2010-09-22 18:57:06 +00:00
Pawel Jakub Dawidek
5bdff860e7 Because it is very hard to make fork(2) from threaded process safe (we are
limited to async-signal safe functions in the child process), move all hooks
execution to the main (non-threaded) process.

Do it by maintaining connection (socketpair) between child and parent
and sending events from the child to parent, so it can execute the hook.

This is step in right direction for others reasons too. For example there is
one less problem to drop privs in worker processes.

MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-30 23:26:10 +00:00
Pawel Jakub Dawidek
6b276294af We only want to know if descriptors are ready for reading.
MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-30 23:19:21 +00:00
Pawel Jakub Dawidek
ecc99c890e Allow to run hooks from the main hastd process.
MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-29 21:41:53 +00:00
Pawel Jakub Dawidek
5da2320932 When SIGTERM or SIGINT is received, terminate worker processes.
MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-27 21:28:02 +00:00
Pawel Jakub Dawidek
0becad39a7 Allow to execute specified program on various HAST events.
MFC after:	2 weeks
Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
2010-08-27 15:16:52 +00:00
Pawel Jakub Dawidek
0989854d45 Implement configuration reload on SIGHUP. This includes:
- Load added resources.
- Stop and forget removed resources.
- Update modified resources in least intrusive way, ie. don't touch
  /dev/hast/<name> unless path to local component or provider name were
  modified.

Obtained from:	Wheel Systems Sp. z o.o. http://www.wheelsystems.com
MFC after:	1 month
2010-08-05 19:16:31 +00:00
Pawel Jakub Dawidek
bbbb114cda Prepare configuration parsing code to be called multiple times:
- Don't exit on errors if not requested.
- Don't keep configuration in global variable, but allocate memory for
  configuration.
- Call yyrestart() before yyparse() so that on error in configuration file
  we will start from the begining next time and not from the place we left of.

MFC after:	1 month
2010-08-05 19:08:54 +00:00
Pawel Jakub Dawidek
005f438bf5 - Use pjdlog_exitx() to log errors and exit instead of errx().
- Use 'unable to' (instead of 'cannot') consistently.

MFC after:	1 month
2010-08-05 18:56:24 +00:00
Pawel Jakub Dawidek
f3bd74124a Correct various log messages.
Submitted by:	Mikolaj Golub <to.my.trociny@gmail.com>
MFC after:	3 days
2010-06-14 21:46:48 +00:00
Pawel Jakub Dawidek
9fab3c1b94 Remove macros that are not really needed. The idea was to have them in case
we grow more descriptors, but I'll reconsider readding them once we get there.

Passing (a = b) expression to FD_ISSET() is bad idea, as FD_ISSET() evaluates
its argument twice.

Found by:	Coverity Prevent
CID:		5243
MFC after:	3 days
2010-06-14 21:18:58 +00:00
Pawel Jakub Dawidek
c6ddcbe009 - Check if the worker process was killed by signal and restart it.
- Improve logging.

Pointed out by:	Garrett Cooper <yanefbsd@gmail.com>
MFC after:	3 days
2010-04-29 15:42:24 +00:00
Pawel Jakub Dawidek
5571414ca8 Fix a problem where hastd will stuck in recv(2) after sending request to
secondary, which died between send(2) and recv(2). Do it by adding timeout
to recv(2) for primary incoming and outgoing sockets and secondary outgoing
socket.

Reported by:	Mikolaj Golub <to.my.trociny@gmail.com>
Tested by:	Mikolaj Golub <to.my.trociny@gmail.com>
MFC after:	3 days
2010-04-29 15:36:32 +00:00
Pawel Jakub Dawidek
83a5671405 Restart worker thread only if the problem was temporary.
In case of persistent problem we don't want to loop forever.

MFC after:	3 days
2010-04-28 22:41:06 +00:00
Pawel Jakub Dawidek
06c117d1d1 Use WEXITSTATUS() to obtain real exit code.
MFC after:	3 days
2010-04-28 22:26:30 +00:00
Pawel Jakub Dawidek
09398e9bd4 Fix control socket leak when worker process exits.
Submitted by:	Mikolaj Golub <to.my.trociny@gmail.com>
MFC after:	3 days
2010-04-16 06:47:29 +00:00
Pawel Jakub Dawidek
32115b105a Please welcome HAST - Highly Avalable Storage.
HAST allows to transparently store data on two physically separated machines
connected over the TCP/IP network. HAST works in Primary-Secondary
(Master-Backup, Master-Slave) configuration, which means that only one of the
cluster nodes can be active at any given time. Only Primary node is able to
handle I/O requests to HAST-managed devices. Currently HAST is limited to two
cluster nodes in total.

HAST operates on block level - it provides disk-like devices in /dev/hast/
directory for use by file systems and/or applications. Working on block level
makes it transparent for file systems and applications. There in no difference
between using HAST-provided device and raw disk, partition, etc. All of them
are just regular GEOM providers in FreeBSD.

For more information please consult hastd(8), hastctl(8) and hast.conf(5)
manual pages, as well as http://wiki.FreeBSD.org/HAST.

Sponsored by:	FreeBSD Foundation
Sponsored by:	OMCnet Internet Service GmbH
Sponsored by:	TransIP BV
2010-02-18 23:16:19 +00:00