Commit Graph

105 Commits

Author SHA1 Message Date
Mikolaj Golub
d685f88bee In remote_send_thread, if sending a request fails don't take the
request back from the receive queue -- it might already be processed
by remote_recv_thread, which lead to crashes like below:

  (primary) Unable to receive reply header: Connection reset by peer.
  (primary) Unable to send request (Connection reset by peer):
      WRITE(954662912, 131072).
  (primary) Disconnected from kopusha:7772.
  (primary) Increasing localcnt to 1.
  (primary) Assertion failed: (old > 0), function refcnt_release,
      file refcnt.h, line 62.

Taking the request back was not necessary (it would properly be
processed by the remote_recv_thread) and only complicated things.

MFC after:	2 weeks
2013-12-10 20:05:07 +00:00
Mikolaj Golub
8f04423f25 Add some macros to make the code more readable (no functional chages).
MFC after:	2 weeks
2013-12-10 19:58:10 +00:00
Mikolaj Golub
5d69ed535e For memsync replication, hio_countdown is used not only as an
indication when a request can be moved to done queue, but also for
detecting the current state of memsync request.

This approach has problems, e.g. leaking a request if memsynk ack from
the secondary failed, or racy usage of write_complete, which should be
called only once per write request, but for memsync can be entered by
local_send_thread and ggate_send_thread simultaneously.

So the following approach is implemented instead:

1) Use hio_countdown only for counting components we waiting to
   complete, i.e. initially it is always 2 for any replication mode.

2) To distinguish between "memsync ack" and "memsync fin" responses
   from the secondary, add and use hio_memsyncacked field.

3) write_complete() in component threads is called only before
   releasing hio_countdown (i.e. before the hio may be returned to the
   done queue).

4) Add and use hio_writecount refcounter to detect when
   write_complete() can be called in memsync case.

Reported by:	Pete French petefrench ingresso.co.uk
Tested by:	Pete French petefrench ingresso.co.uk
MFC after:	2 weeks
2013-12-10 19:56:26 +00:00
Mikolaj Golub
6b66c350a7 Make hastctl list command output current queue sizes.
Reviewed by:	pjd
MFC after:	1 month
2013-10-26 08:38:21 +00:00
Mikolaj Golub
d03a08e5e6 Merging local and remote bitmaps must be protected by hr_amp lock.
This is believed to fix hastd crashes, which might occur during
synchronization, triggered by the failed assertion:

 Assertion failed: (amp->am_memtab[ext] > 0),
 function activemap_write_complete, file activemap.c, line 351.

MFC after:	1 week
2013-10-26 08:35:54 +00:00
Mikolaj Golub
a818a4ff09 When updating the map of dirty extents, most recently used extents are
kept dirty to reduce the number of on-disk metadata updates. The
sequence of operations is:

1) acquire the activemap lock;
2) update in-memory map;
3) if the list of keepdirty extents is changed, update on-disk metadata;
4) release the lock.

On-disk updates are not frequent in comparison with in-memory updates,
while require much more time. So situations are possible when one
thread is updating on-disk metadata and another one is waiting for the
activemap lock just to update the in-memory map.

Improve this by introducing additional, on-disk map lock: when
in-memory map is updated and it is detected that the on-disk map needs
update too, the on-disk map lock is acquired and the on-memory lock is
released before flushing the map.

Reported by:	Yamagi Burmeister yamagi.org
Tested by:	Yamagi Burmeister yamagi.org
Reviewed by:	pjd
Approved by:	re (marius)
MFC after:	2 weeks
2013-09-19 20:19:08 +00:00
Mikolaj Golub
1c1310eed7 Use cv_broadcast() instead of cv_signal() when waking up threads
waiting on an empty queue as the queue may have several consumers.

Before the fix the following scenario was possible: 2 threads are
waiting on empty queue, 2 threads are inserting simultaneously. The
first inserting thread detects that the queue is empty and is going to
send the signal, but before it sends the second thread inserts
too. When the first sends the signal only one of the waiting threads
receive it while the other one may wait forever.

The scenario above is is believed to be the cause of the observed
cases, when ggate_recv_thread() was getting stuck on taking free
request, while the free queue was not empty.

Reviewed by:	pjd
Tested by:	Yamagi Burmeister yamagi.org
Approved by:	re (marius)
MFC after:	2 weeks
2013-09-19 20:15:24 +00:00
Ed Schouten
6e87c1514b Use C11 <stdatomic.h> instead of our non-standard <machine/atomic.h>.
Reviewed by:	pjd
2013-04-27 05:01:29 +00:00
Mikolaj Golub
2adbba660d Add i/o error counters to hastd(8) and make hastctl(8) display
them.  This may be useful for detecting problems with HAST disks.

Discussed with and reviewed by:	pjd
MFC after:	1 week
2013-02-25 20:09:07 +00:00
Pawel Jakub Dawidek
d6e636c988 - Add support for 'memsync' mode. This is the fastest replication mode that's
why it will now be the default.
- Bump protocol version to 2 and add backward compatibility for version 1.
- Allow to specify hosts by kern.hostid as well (in addition to hostname and
  kern.hostuuid) in configuration file.

Sponsored by:	Panzura
Tested by:	trociny
2013-02-17 21:12:34 +00:00
Pawel Jakub Dawidek
c66ee1b31e Make use of GEOM Gate direct reads feature. This allows HAST to serve
reads with native speed of the underlying provider.
There are three situations when direct reads are not used:
1. Data is being synchronized and synchronization source is the secondary
   node, which means secondary node has more recent data and we should read
   from it.
2. Local read failed and we have to try to read from the secondary node.
3. Local component is unavailable and all I/O requests are served from the
   secondary node.

Sponsored by:	Panzura, http://www.panzura.com
MFC after:	1 month
2012-07-04 20:20:48 +00:00
Pawel Jakub Dawidek
4c71d26341 Simplify the code by using snprlcat().
MFC after:	3 days
2012-06-03 10:50:46 +00:00
Mikolaj Golub
f737157838 If a local write request is from the synchronization thread, when it
is synchronizing data that is out of date on the local component, we
should not send G_GATE_CMD_DONE acknowledge to the kernel.

This fixes the issue, observed in async mode, when on synchronization
from the remote component the worker terminated with "G_GATE_CMD_DONE
failed" error.

Reported by:	Artem Kajalainen <artem kayalaynen ru>
Reviewed by:	pjd
MFC after:	1 week
2012-02-05 15:23:32 +00:00
Mikolaj Golub
2b2cb41812 Fix the regression introduced in r226859: if the local component is
out of date BIO_READ requests got lost instead of being sent to the
remote component.

Reviewed by:	pjd
MFC after:	1 week
2012-02-05 15:21:08 +00:00
Pawel Jakub Dawidek
2b1b224d24 For functions that return -1 on failure check exactly for -1 and not for
any negative number.

MFC after:	3 days
2012-01-10 22:39:07 +00:00
Ulrich Spörlein
4b85a12f71 Spelling fixes for sbin/ 2012-01-07 16:09:33 +00:00
Pawel Jakub Dawidek
dfb1aece41 fork(2) returns -1 on failure, not some random negative number.
MFC after:	3 days
2012-01-06 23:44:26 +00:00
Pawel Jakub Dawidek
07ebc3626e Implement 'async' mode for HAST.
MFC after:	3 days
2011-10-27 20:32:57 +00:00
Pawel Jakub Dawidek
3f5bce1822 Minor cleanups.
MFC after:	3 days
2011-10-27 20:15:37 +00:00
Pawel Jakub Dawidek
43b8675beb Reduce indentation.
MFC after:	3 days
2011-10-27 20:13:39 +00:00
Pawel Jakub Dawidek
5a58d22a84 Improve comment so it doesn't suggest race is possible, but that we handle
the race.

MFC after:	3 days
2011-10-27 20:10:21 +00:00
Pawel Jakub Dawidek
1212a85c4a Monor cleanups.
MFC after:	3 days
2011-10-27 18:49:16 +00:00
Pawel Jakub Dawidek
8a34134ac2 Delay resuid generation until first connection to secondary, not until first
write. This way on first connection we will synchronize only the extents that
were modified during the lifetime of primary node, not entire GEOM provider.

MFC after:	3 days
2011-10-27 18:45:01 +00:00
Pawel Jakub Dawidek
e3feec94eb Correct typo.
MFC after:	3 days
2011-09-28 13:25:27 +00:00
Pawel Jakub Dawidek
12daf727f6 If the underlying provider doesn't support BIO_FLUSH, log it only once
and don't bother trying in the future.

MFC after:	3 days
2011-09-28 13:19:47 +00:00
Pawel Jakub Dawidek
518dd4c0d9 After every activemap change flush disk's write cache, so that write
reordering won't make the actual write to be committed before marking
the coresponding extent as dirty.

It can be disabled in configuration file.

If BIO_FLUSH is not supported by the underlying file system we log a warning
and never send BIO_FLUSH again to that GEOM provider.

MFC after:	3 days
2011-09-28 13:08:51 +00:00
Pawel Jakub Dawidek
be1143efb9 No need to wrap pjdlog functions around with KEEP_ERRNO() macro.
MFC after:	3 days
2011-09-27 08:26:09 +00:00
Pawel Jakub Dawidek
09c2e8431a Correct two mistakes when converting asserts to PJDLOG_ASSERT()/PJDLOG_ABORT().
MFC after:	3 days
2011-09-27 07:59:10 +00:00
Mikolaj Golub
adca96f861 Fix indentation.
Approved by:	pjd (mentor)
2011-07-13 05:32:55 +00:00
Mikolaj Golub
d9f039e0b3 Check the returned value of activemap_write_complete() and update matadata on
disk if needed. This should fix a potential case when extents are cleared in
activemap but metadata is not updated on disk.

Suggested by:	pjd
Approved by:	pjd (mentor)
2011-06-28 21:01:32 +00:00
Mikolaj Golub
ba2a822490 In HAST we use two sockets - one for only sending the data and one for
only receiving the data. In r220271 the unused directions were
disabled using shutdown(2).

Unfortunately, this broke automatic receive buffer sizing, which
currently works only for connections in ETASBLISHED state. It was a
root cause of the issue reported by users, when connection between
primary and secondary could get stuck.

Disable the code introduced in r220271 until the issue with automatic
buffer sizing is not resolved.

Reported by:	Daniel Kalchev <daniel@digsys.bg>, danger, sobomax
Tested by:	Daniel Kalchev <daniel@digsys.bg>, danger
Approved by:	pjd (mentor)
MFC after:	1 week
2011-06-17 07:07:26 +00:00
Mikolaj Golub
a01a750f32 If READ from the local node failed we send the request to the remote
node. There is no use in doing this for synchronization requests.

Approved by:	pjd (mentor)
MFC after:	1 week
2011-05-29 21:20:47 +00:00
Pawel Jakub Dawidek
3db86c39ae Keep statistics on number of BIO_READ, BIO_WRITE, BIO_DELETE and BIO_FLUSH
requests as well as number of activemap updates.

Number of BIO_WRITEs and activemap updates are especially interesting, because
if those two are too close to each other, it means that your workload needs
bigger number of dirty extents. Activemap should be updated as rarely as
possible.

MFC after:	1 week
2011-05-23 21:15:19 +00:00
Pawel Jakub Dawidek
0cddb12ffd Currently we are unable to use capsicum for the primary worker process,
because we need to do ioctl(2)s, which are not permitted in the capability
mode. What we do now is to chroot(2) to /var/empty, which restricts access
to file system name space and we drop privileges to hast user and hast
group.

This still allows to access to other name spaces, like list of processes,
network and sysvipc.

To address that, use jail(2) instead of chroot(2). Using jail(2) will restrict
access to process table, network (we use ip-less jails) and sysvipc (if
security.jail.sysvipc_allowed is turned off). This provides much better
separation.

MFC after:	1 week
2011-05-14 17:02:03 +00:00
Pawel Jakub Dawidek
ac0401e321 When we become primary, we connect to the remote and expect it to be in
secondary role. It is possible that the remote node is primary, but only
because there was a role change and it didn't finish cleaning up (unmounting
file systems, etc.). If we detect such situation, wait for the remote node
to switch the role to secondary before accepting I/Os. If we don't wait for
it in that case, we will most likely cause split-brain.

MFC after:	1 week
2011-04-20 18:43:28 +00:00
Pawel Jakub Dawidek
06cbf54941 Scenario:
- We have two nodes connected and synchronized (local counters on both sides
  are 0).
- We take secondary down and recreate it.
- Primary connects to it and starts synchronization (but local counters are
  still 0).
- We switch the roles.
- Synchronization restarts but data is synchronized now from new primary
  (because local counters are 0) that doesn't have new data yet.

This fix this issue we bump local counter on primary when we discover that
connected secondary was recreated and has no data yet.

Reported by:	trociny
Discussed with:	trociny
Tested by:	trociny
MFC after:	1 week
2011-04-19 19:26:27 +00:00
Pawel Jakub Dawidek
02dfe9724c Declare directions for sockets between primary and secondary.
In HAST we use two sockets - one for only sending the data and one for only
receiving the data.

MFC after:	1 month
2011-04-02 09:25:13 +00:00
Pawel Jakub Dawidek
2a49afacd1 Handle the problem described in r220264 by using GEOM GATE queue of unlimited
length. This should fix deadlocks reported by HAST users.

MFC after:	1 week
2011-04-02 07:01:09 +00:00
Pawel Jakub Dawidek
7d4df5cd0b Use timeout from configuration file not only when sending and receiving,
but also when establishing connection.

MFC after:	1 week
2011-03-25 20:15:16 +00:00
Pawel Jakub Dawidek
643080b75f Use role2str() when setting process title.
MFC after:	1 week
2011-03-25 20:13:38 +00:00
Mikolaj Golub
9237aa3fa5 After synchronization is complete we should make primary counters be
equal to secondary counters:

  primary_localcnt = secondary_remotecnt
  primary_remotecnt = secondary_localcnt

Previously it was done wrong and split-brain was observed after
primary had synchronized up-to-date data from secondary.

Approved by:	pjd (mentor)
MFC after:	1 week
2011-03-22 20:27:26 +00:00
Mikolaj Golub
b068d5aafb For requests that are sent only to remote component use the
error from remote.
Approved by:	pjd (mentor)
MFC after:	1 week
2011-03-22 19:49:27 +00:00
Pawel Jakub Dawidek
cd72d521e3 White space cleanups.
MFC after:	1 week
2011-03-22 10:39:34 +00:00
Pawel Jakub Dawidek
4d8dc3b838 When dropping privileges prefer capsicum over chroot+setgid+setuid.
We can use capsicum for secondary worker processes and hastctl.
When working as primary we drop privileges using chroot+setgid+setuid
still as we need to send ioctl(2)s to ggate device, for which capsicum
doesn't allow (yet).

X-MFC after:	capsicum is merged to stable/8
2011-03-21 21:31:50 +00:00
Pawel Jakub Dawidek
9446b4536e Initialize localcnt on first write. This fixes assertion when we create
resource, set role to primary, do no writes, then sent it to secondary
and accept connection from primary.

MFC after:	1 week
2011-03-21 21:16:12 +00:00
Pawel Jakub Dawidek
0b626a289e In hast.conf we define the other node's address in 'remote' variable.
This way we know how to connect to secondary node when we are primary.
The same variable is used by the secondary node - it only accepts
connections from the address stored in 'remote' variable.
In cluster configurations it is common that each node has its individual
IP address and there is one addtional shared IP address which is assigned
to primary node. It seems it is possible that if the shared IP address is
from the same network as the individual IP address it might be choosen by
the kernel as a source address for connection with the secondary node.
Such connection will be rejected by secondary, as it doesn't come from
primary node individual IP.

Add 'source' variable that allows to specify source IP address we want to
bind to before connecting to the secondary node.

MFC after:	1 week
2011-03-21 08:54:59 +00:00
Mikolaj Golub
8d7dcf14ff For secondary, set 2 * HAST_KEEPALIVE seconds timeout for incoming
connection so the worker will exit if it does not receive packets from
the primary during this interval.

Reported by:	Christian Vogt <Christian.Vogt@haw-hamburg.de>
Tested by:	Christian Vogt <Christian.Vogt@haw-hamburg.de>
Approved by:	pjd (mentor)
MFC after:	1 week
2011-03-17 21:02:14 +00:00
Mikolaj Golub
bc7a916a25 Make workers inherit debug level from the main process.
Approved by:	pjd (mentor)
MFC after:	1 week
2011-03-11 12:12:35 +00:00
Pawel Jakub Dawidek
fa356f6cfe - Log size of data to synchronize in human readable form (using %N).
- Log synchronization time (using %T).
- Log synchronization speed in human readable form (using %N).

MFC after:	2 weeks
2011-03-07 10:41:12 +00:00
Pawel Jakub Dawidek
8cd3d45ad9 Allow to compress on-the-wire data using two algorithms:
- HOLE - it simply turns all-zero blocks into few bytes header;
	it is extremely fast, so it is turned on by default;
	it is mostly intended to speed up initial synchronization
	where we expect many zeros;
- LZF - very fast algorithm by Marc Alexander Lehmann, which shows
	very decent compression ratio and has BSD license.

MFC after:	2 weeks
2011-03-06 23:09:33 +00:00