Commit Graph

362 Commits

Author SHA1 Message Date
Alan Somers
8907f744ff Improve performance and functionality of the bitstring(3) api
Two new functions are provided, bit_ffs_at() and bit_ffc_at(), which allow
for efficient searching of set or cleared bits starting from any bit offset
within the bit string.

Performance is improved by operating on longs instead of bytes and using
ffsl() for searches within a long. ffsl() is a compiler builtin in both
clang and gcc for most architectures, converting what was a brute force
while loop search into a couple of instructions.

All of the bitstring(3) API continues to be contained in the header file.
Some of the functions are large enough that perhaps they should be uninlined
and moved to a library, but that is beyond the scope of this commit.

sys/sys/bitstring.h:
        Convert the majority of the existing bit string implementation from
        macros to inline functions.

        Properly protect the implementation from inadvertant macro expansion
        when included in a user's program by prefixing all private
        macros/functions and local variables with '_'.

        Add bit_ffs_at() and bit_ffc_at(). Implement bit_ffs() and
        bit_ffc() in terms of their "at" counterparts.

        Provide a kernel implementation of bit_alloc(), making the full API
        usable in the kernel.

        Improve code documenation.

share/man/man3/bitstring.3:
        Add pre-exisiting API bit_ffc() to the synopsis.

        Document new APIs.

        Document the initialization state of the bit strings
        allocated/declared by bit_alloc() and bit_decl().

        Correct documentation for bitstr_size(). The original code comments
        indicate the size is in bytes, not "elements of bitstr_t". The new
        implementation follows this lead. Only hastd assumed "elements"
        rather than bytes and it has been corrected.

etc/mtree/BSD.tests.dist:
tests/sys/Makefile:
tests/sys/sys/Makefile:
tests/sys/sys/bitstring.c:
        Add tests for all existing and new functionality.

include/bitstring.h
	Include all headers needed by sys/bitstring.h

lib/libbluetooth/bluetooth.h:
usr.sbin/bluetooth/hccontrol/le.c:
        Include bitstring.h instead of sys/bitstring.h.

sbin/hastd/activemap.c:
        Correct usage of bitstr_size().

sys/dev/xen/blkback/blkback.c
        Use new bit_alloc.

sys/kern/subr_unit.c:
        Remove hard-coded assumption that sizeof(bitstr_t) is 1.  Get rid of
        unrb.busy, which caches the number of bits set in unrb.map.  When
        INVARIANTS are disabled, nothing needs to know that information.
        callapse_unr can be adapted to use bit_ffs and bit_ffc instead.
        Eliminating unrb.busy saves memory, simplifies the code, and
        provides a slight speedup when INVARIANTS are disabled.

sys/net/flowtable.c:
        Use the new kernel implementation of bit-alloc, instead of hacking
        the old libc-dependent macro.

sys/sys/param.h
        Update __FreeBSD_version to indicate availability of new API

Submitted by:   gibbs, asomers
Reviewed by:    gibbs, ngie
MFC after:      4 weeks
Sponsored by:   Spectra Logic Corp
Differential Revision:  https://reviews.freebsd.org/D6004
2016-05-04 22:34:11 +00:00
Ed Schouten
e33d251e8e Remove useless calls to basename().
There are a couple of places in the source three where we call
basename() on constant strings. This is bad, because the prototype
standardized by POSIX allows the implementation to use its argument as a
storage buffer.

This change eliminates some of these unportable calls to basename() in
cases where it was only added for cosmetical reasons, namely to trim
argv[0]. There's nothing wrong with setting argv[0] to the full path.

Reviewed by:	jilles
Differential Revision:	https://reviews.freebsd.org/D6093
2016-05-01 08:22:11 +00:00
Marcelo Araujo
67210fb3b8 Use MAX() from sys/param.h.
MFC after:	2 weeks.
2016-04-21 06:21:33 +00:00
Marcelo Araujo
46df5db84e Use nitems() from sys/param.h.
MFC after:	2 weeks.
2016-04-20 00:55:35 +00:00
Glen Barber
52259a98ad MFH
Sponsored by:	The FreeBSD Foundation
2016-03-02 16:14:46 +00:00
Bryan Drewery
bd18fd57db DIRDEPS_BUILD: Regenerate without local dependencies.
These are no longer needed after the recent 'beforebuild: depend' changes
and hooking DIRDEPS_BUILD into a subset of FAST_DEPEND which supports
skipping 'make depend'.

Sponsored by:	EMC / Isilon Storage Division
2016-02-24 17:20:11 +00:00
Glen Barber
bcefcb026a Create a package for HAST.
Sponsored by:	The FreeBSD Foundation
2016-02-08 21:15:07 +00:00
Simon J. Gerraty
ccfb965433 Add META_MODE support.
Off by default, build behaves normally.
WITH_META_MODE we get auto objdir creation, the ability to
start build from anywhere in the tree.

Still need to add real targets under targets/ to build packages.

Differential Revision:       D2796
Reviewed by: brooks imp
2015-06-13 19:20:56 +00:00
Simon J. Gerraty
44d314f704 dirdeps.mk now sets DEP_RELDIR 2015-06-08 23:35:17 +00:00
Simon J. Gerraty
98e0ffaefb Merge sync of head 2015-05-27 01:19:58 +00:00
Joel Dahl
f636caf195 mdoc: paragraph improvements. 2015-01-04 12:49:24 +00:00
Baptiste Daroussin
13eb765f2d Convert sbin/ to LIBADD
Reduce overlinking
2014-11-25 11:23:12 +00:00
Simon J. Gerraty
9268022b74 Merge from head@274682 2014-11-19 01:07:58 +00:00
Ed Schouten
974a10854c Add lock annotations to the threading API used by hastd.
Approved by:	pjd@
2014-09-01 18:37:17 +00:00
Enji Cooper
acc1741c58 Garbage collect libl dependency
The application links and runs without libl

Approved by: rpaulo (mentor)
Phabric: D673
X-MFC with: r270117
Submitted by: trociny
2014-08-23 20:45:00 +00:00
Simon J. Gerraty
ee7b0571c2 Merge head from 7/28 2014-08-19 06:50:54 +00:00
Enji Cooper
32529978a7 Add -ll to LDADD to fix "make checkdpadd"
Phabric: D622
MFC after: 2 weeks
Approved by: rpaulo (mentor)
2014-08-18 00:50:09 +00:00
Baptiste Daroussin
bd0891ceb3 use .Mt to mark up email addresses consistently (part1)
PR:		191174
Submitted by:	Franco Fichtner  <franco@lastsummer.de>
2014-06-20 09:40:43 +00:00
Simon J. Gerraty
fae50821ae Updated dependencies 2014-05-16 14:09:51 +00:00
Simon J. Gerraty
cc3f4b9965 Merge from head 2014-05-08 23:54:15 +00:00
Warner Losh
c6063d0da8 Use src.opts.mk in preference to bsd.own.mk except where we need stuff
from the latter.
2014-05-06 04:22:01 +00:00
Simon J. Gerraty
3b8f084595 Merge head 2014-04-28 07:50:45 +00:00
Robert Watson
b881b8be1d Update most userspace consumers of capability.h to use capsicum.h instead.
auditdistd is not updated as I will make the change upstream and then do a
vendor import sometime in the next week or two.

MFC after:	3 weeks
2014-03-16 11:04:44 +00:00
John Baldwin
e432d5f6a7 Drop the 3rd clause from all 3 clause BSD licenses where I am the sole
holder to convert them to 2 clause BSD licenses.

MFC after:	1 week
2014-02-05 18:13:27 +00:00
Mikolaj Golub
f8665529d4 Check remote protocol version only for the first connection (when it
is actually sent by the remote node).

Otherwise it generated confusing "Negotiated protocol version 1" debug
messages when processing the second connection.

MFC after:	2 weeks
2013-12-10 20:09:49 +00:00
Mikolaj Golub
9c53997114 Send wakeup to threads waiting on empty queue before releasing the
lock to decrease spurious wakeups.

Submitted by:	davidxu
MFC after:	2 weeks
2013-12-10 20:06:41 +00:00
Mikolaj Golub
d685f88bee In remote_send_thread, if sending a request fails don't take the
request back from the receive queue -- it might already be processed
by remote_recv_thread, which lead to crashes like below:

  (primary) Unable to receive reply header: Connection reset by peer.
  (primary) Unable to send request (Connection reset by peer):
      WRITE(954662912, 131072).
  (primary) Disconnected from kopusha:7772.
  (primary) Increasing localcnt to 1.
  (primary) Assertion failed: (old > 0), function refcnt_release,
      file refcnt.h, line 62.

Taking the request back was not necessary (it would properly be
processed by the remote_recv_thread) and only complicated things.

MFC after:	2 weeks
2013-12-10 20:05:07 +00:00
Mikolaj Golub
4d70e06ffc Fix compiler warnings.
MFC after:	2 weeks
2013-12-10 20:02:09 +00:00
Mikolaj Golub
8f04423f25 Add some macros to make the code more readable (no functional chages).
MFC after:	2 weeks
2013-12-10 19:58:10 +00:00
Mikolaj Golub
5d69ed535e For memsync replication, hio_countdown is used not only as an
indication when a request can be moved to done queue, but also for
detecting the current state of memsync request.

This approach has problems, e.g. leaking a request if memsynk ack from
the secondary failed, or racy usage of write_complete, which should be
called only once per write request, but for memsync can be entered by
local_send_thread and ggate_send_thread simultaneously.

So the following approach is implemented instead:

1) Use hio_countdown only for counting components we waiting to
   complete, i.e. initially it is always 2 for any replication mode.

2) To distinguish between "memsync ack" and "memsync fin" responses
   from the secondary, add and use hio_memsyncacked field.

3) write_complete() in component threads is called only before
   releasing hio_countdown (i.e. before the hio may be returned to the
   done queue).

4) Add and use hio_writecount refcounter to detect when
   write_complete() can be called in memsync case.

Reported by:	Pete French petefrench ingresso.co.uk
Tested by:	Pete French petefrench ingresso.co.uk
MFC after:	2 weeks
2013-12-10 19:56:26 +00:00
Pawel Jakub Dawidek
cb491a4d63 Correct alignment. 2013-11-03 19:02:18 +00:00
Mikolaj Golub
6b66c350a7 Make hastctl list command output current queue sizes.
Reviewed by:	pjd
MFC after:	1 month
2013-10-26 08:38:21 +00:00
Mikolaj Golub
d03a08e5e6 Merging local and remote bitmaps must be protected by hr_amp lock.
This is believed to fix hastd crashes, which might occur during
synchronization, triggered by the failed assertion:

 Assertion failed: (amp->am_memtab[ext] > 0),
 function activemap_write_complete, file activemap.c, line 351.

MFC after:	1 week
2013-10-26 08:35:54 +00:00
Simon J. Gerraty
3caf0790a8 Merge head@256284 2013-10-13 02:35:19 +00:00
Simon J. Gerraty
34b33809b7 Updated dependencies 2013-10-13 00:24:00 +00:00
Mikolaj Golub
31b81dd333 Fix comments.
Approved by:	re (marius)
MFC after:	3 days
2013-09-19 20:20:59 +00:00
Mikolaj Golub
a818a4ff09 When updating the map of dirty extents, most recently used extents are
kept dirty to reduce the number of on-disk metadata updates. The
sequence of operations is:

1) acquire the activemap lock;
2) update in-memory map;
3) if the list of keepdirty extents is changed, update on-disk metadata;
4) release the lock.

On-disk updates are not frequent in comparison with in-memory updates,
while require much more time. So situations are possible when one
thread is updating on-disk metadata and another one is waiting for the
activemap lock just to update the in-memory map.

Improve this by introducing additional, on-disk map lock: when
in-memory map is updated and it is detected that the on-disk map needs
update too, the on-disk map lock is acquired and the on-memory lock is
released before flushing the map.

Reported by:	Yamagi Burmeister yamagi.org
Tested by:	Yamagi Burmeister yamagi.org
Reviewed by:	pjd
Approved by:	re (marius)
MFC after:	2 weeks
2013-09-19 20:19:08 +00:00
Mikolaj Golub
1c1310eed7 Use cv_broadcast() instead of cv_signal() when waking up threads
waiting on an empty queue as the queue may have several consumers.

Before the fix the following scenario was possible: 2 threads are
waiting on empty queue, 2 threads are inserting simultaneously. The
first inserting thread detects that the queue is empty and is going to
send the signal, but before it sends the second thread inserts
too. When the first sends the signal only one of the waiting threads
receive it while the other one may wait forever.

The scenario above is is believed to be the cause of the observed
cases, when ggate_recv_thread() was getting stuck on taking free
request, while the free queue was not empty.

Reviewed by:	pjd
Tested by:	Yamagi Burmeister yamagi.org
Approved by:	re (marius)
MFC after:	2 weeks
2013-09-19 20:15:24 +00:00
Simon J. Gerraty
d1d0158641 Merge from head 2013-09-05 20:18:59 +00:00
Pawel Jakub Dawidek
7008be5bd7 Change the cap_rights_t type from uint64_t to a structure that we can extend
in the future in a backward compatible (API and ABI) way.

The cap_rights_t represents capability rights. We used to use one bit to
represent one right, but we are running out of spare bits. Currently the new
structure provides place for 114 rights (so 50 more than the previous
cap_rights_t), but it is possible to grow the structure to hold at least 285
rights, although we can make it even larger if 285 rights won't be enough.

The structure definition looks like this:

	struct cap_rights {
		uint64_t	cr_rights[CAP_RIGHTS_VERSION + 2];
	};

The initial CAP_RIGHTS_VERSION is 0.

The top two bits in the first element of the cr_rights[] array contain total
number of elements in the array - 2. This means if those two bits are equal to
0, we have 2 array elements.

The top two bits in all remaining array elements should be 0.
The next five bits in all array elements contain array index. Only one bit is
used and bit position in this five-bits range defines array index. This means
there can be at most five array elements in the future.

To define new right the CAPRIGHT() macro must be used. The macro takes two
arguments - an array index and a bit to set, eg.

	#define	CAP_PDKILL	CAPRIGHT(1, 0x0000000000000800ULL)

We still support aliases that combine few rights, but the rights have to belong
to the same array element, eg:

	#define	CAP_LOOKUP	CAPRIGHT(0, 0x0000000000000400ULL)
	#define	CAP_FCHMOD	CAPRIGHT(0, 0x0000000000002000ULL)

	#define	CAP_FCHMODAT	(CAP_FCHMOD | CAP_LOOKUP)

There is new API to manage the new cap_rights_t structure:

	cap_rights_t *cap_rights_init(cap_rights_t *rights, ...);
	void cap_rights_set(cap_rights_t *rights, ...);
	void cap_rights_clear(cap_rights_t *rights, ...);
	bool cap_rights_is_set(const cap_rights_t *rights, ...);

	bool cap_rights_is_valid(const cap_rights_t *rights);
	void cap_rights_merge(cap_rights_t *dst, const cap_rights_t *src);
	void cap_rights_remove(cap_rights_t *dst, const cap_rights_t *src);
	bool cap_rights_contains(const cap_rights_t *big, const cap_rights_t *little);

Capability rights to the cap_rights_init(), cap_rights_set(),
cap_rights_clear() and cap_rights_is_set() functions are provided by
separating them with commas, eg:

	cap_rights_t rights;

	cap_rights_init(&rights, CAP_READ, CAP_WRITE, CAP_FSTAT);

There is no need to terminate the list of rights, as those functions are
actually macros that take care of the termination, eg:

	#define	cap_rights_set(rights, ...)				\
		__cap_rights_set((rights), __VA_ARGS__, 0ULL)
	void __cap_rights_set(cap_rights_t *rights, ...);

Thanks to using one bit as an array index we can assert in those functions that
there are no two rights belonging to different array elements provided
together. For example this is illegal and will be detected, because CAP_LOOKUP
belongs to element 0 and CAP_PDKILL to element 1:

	cap_rights_init(&rights, CAP_LOOKUP | CAP_PDKILL);

Providing several rights that belongs to the same array's element this way is
correct, but is not advised. It should only be used for aliases definition.

This commit also breaks compatibility with some existing Capsicum system calls,
but I see no other way to do that. This should be fine as Capsicum is still
experimental and this change is not going to 9.x.

Sponsored by:	The FreeBSD Foundation
2013-09-05 00:09:56 +00:00
Mikolaj Golub
14f200d97e Make hastctl(1) ('list' command) output a worker pid.
Reviewed by:	pjd
MFC after:	3 days
2013-07-01 18:41:07 +00:00
Jens Schweikhardt
1862d13b8a Correct some grammar. 2013-06-30 17:59:40 +00:00
Ed Schouten
f758648022 Don't let hastd use C11 atomics.
Due to possible concerns about the stability of C11 atomics, use our
existing atomics API instead.

Requested by:	pjd
2013-06-29 20:13:39 +00:00
Ed Schouten
87f02f6f89 Let hastd use C11 atomics.
C11 atomics now work on all the architectures. Have at least a single
piece of software in our base system that uses C11 atomics. This
somewhat makes it less likely that we break it because of LLVM imports,
etc.
2013-06-15 22:17:59 +00:00
Jung-uk Kim
eb8b73d6ae Improve compatibility with old flex and fix build with GCC. 2013-05-22 17:47:45 +00:00
Mikolaj Golub
1df892f44c Get rid of libl dependency. We needed it only to provide yywrap. But
yywrap is not necessary when parsing a single hast.conf file.

Suggested by:	kib
Reviewed by:	pjd
2013-05-11 09:51:44 +00:00
Ed Schouten
24084cdd91 Partially revert my last change.
I forgot that I still had a locally applied patch to my copy of Clang
that needs to be pushed in before we should use C11 atomics.
2013-04-27 05:06:25 +00:00
Ed Schouten
6e87c1514b Use C11 <stdatomic.h> instead of our non-standard <machine/atomic.h>.
Reviewed by:	pjd
2013-04-27 05:01:29 +00:00
Ed Schouten
7e659f9491 Add the Clang specific -Wmissing-variable-declarations to WARNS=6.
This compiler flag enforces that that people either mark variables
static or use an external declarations for the variable, similar to how
-Wmissing-prototypes works for functions.

Due to the fact that Yacc/Lex generate code that cannot trivially be
changed to not warn because of this (lots of yy* variables), add a
NO_WMISSING_VARIABLE_DECLARATIONS that can be used to turn off this
specific compiler warning.

Announced on:	toolchain@
2013-04-19 19:45:00 +00:00
Simon J. Gerraty
69e6d7b75e sync from head 2013-04-12 20:48:55 +00:00