Commit Graph

12458 Commits

Author SHA1 Message Date
sobomax
0b911c8b6b GC duplicate define. 2016-04-23 07:28:32 +00:00
sobomax
5be3733785 Improve performance in a few key areas:
o Split the compression across several worker threads. By default, "several"
   matches number of CPUs, capped at 24 for sanity when running on a very big
   hardwares. Provide option to set that number manually;

 o Fix bug inherited from the mkulzma (R.I.P) which degraded already slow LZMA
   compression even further by calling function to release compression state
   after processing each block.

   It is neither documented as required nor actually required by the LZMA
   library. This caused spree of system calls to release memory and then map
   it again for every block. LZMA compression is more than 2x faster after this
   change alone;

 o Record time it takes to do compression and report throughput achieved.

 o Add simple first-level 256 entry hash table for de-dup code, so it's not
   becoming a bottleneck at big files.
2016-04-23 07:23:43 +00:00
araujo
c81163a574 Use roundup2() macro from sys/param.h.
MFC after:	2 weeks.
2016-04-22 06:23:23 +00:00
araujo
7f164828e7 Use MIN() macro from sys/param.h.
MFC after:	2 weeks.
2016-04-22 03:46:57 +00:00
araujo
e15ebdf90e Use macro MAX() from sys/param.h.
MFC after:	2 weeks.
2016-04-22 03:43:06 +00:00
araujo
244576d3c3 Use macro MAX() from sys/param.h.
MFC after:	2 weeks.
2016-04-22 03:37:27 +00:00
pfg
3f525b8a34 kernel: use our nitems() macro when it is available through param.h.
No functional change, only trivial cases are done in this sweep,

Discussed in:	freebsd-current
2016-04-21 21:30:51 +00:00
emaste
55c6dd3879 elftoolchain: Use ${SRCTOP} for the top of the FreeBSD tree
It's provided by sys.mk so there's no need to derive it from ${.CURDIR}.

Suggested by:	ngie
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D5998
2016-04-21 12:58:29 +00:00
eadler
242400036c Bring a little more compability with GNU units 2.12
- notionally support a 'history file' flag. This doesn't do much now,
  but is there to prevent scripts written against GNU units from
  breaking
- correctly gracefully quit rather than exit (this will make it easier
  to support a history file in the future)
- remove the "t" flag from fopen which was there to support windows. We
  have not supported windows since at the latest, the introduction of
  capsicum.
2016-04-21 05:24:47 +00:00
bapt
f3bbb3cf24 Plug memory leaks
Reported by:	Coverity
CID=		1338535, 1338536, 1338542, 1338569, 1338570
2016-04-20 21:23:42 +00:00
bapt
d0f2726865 Fix bad checking of the return of realloc(3)
Reported by:	Coverity
CID:		1007335
MFC after:	3 days
2016-04-20 20:56:06 +00:00
bapt
499f728f92 Fix typo: actually test the return of strchr(3)
Reported by:	Coverity
CID:		1007335
MFC after:	3 days
2016-04-20 20:54:47 +00:00
araujo
5329b494a9 Partially revert the change on r298325 where there is an
(-1) casted to a pointer.

Submitted by:	pfg
MFC after:	2 weeks.
2016-04-20 01:38:54 +00:00
araujo
a05218e04b Use NULL instead of 0 for pointers.
Small cosmetic change.

MFC after:	2 weeks.
2016-04-20 01:28:31 +00:00
araujo
b9140f06e3 Use NULL instead of 0 for pointers.
gethostbyname(3) will return NULL for error status.

MFC after:	2 weeks.
2016-04-20 01:26:03 +00:00
delphij
9454d2f01e Fix build breakage introduced by r298253. 2016-04-19 07:28:39 +00:00
eadler
bf4d4b4498 Rename units.lib -> definitions.units
- this matches GNU units 2.12
add ISO country codes from units 2.12
2016-04-19 05:17:59 +00:00
eadler
5ebf5daae9 Remove pathnames.h
- it only holds a single constant
- it doesn't exist in the GNU variant
2016-04-19 05:04:39 +00:00
araujo
30f25483fe Use NULL instead of 0 for pointers.
realloc will return NULL in case it cannot allocate memory.

MFC after:	2 weeks.
2016-04-19 02:05:32 +00:00
araujo
889ed0c7e4 Use NULL instead of 0 for pointers.
strchr(3) will return NULL if the character does not appear in the
string.

MFC after:	2 weeks.
2016-04-18 14:44:01 +00:00
emaste
60b9a17328 Replace ${CURDIR}/../.. with ${CURDIR:H:H} in elftoolchain
This produces a nicer path in debug info and build logs.

MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2016-04-18 13:13:59 +00:00
araujo
cb410646fd User NULL instead of 0 for pointers.
getservent(3) returns NULL on EOF or error.

MFC after:	2 weeks.
2016-04-18 07:14:01 +00:00
araujo
45973d9d15 Use NULL instead of 0 for pointers.
fopen(3) will return NULL in case it cannot open the STREAM.

MFC after:	2 weeks.
2016-04-18 07:09:34 +00:00
araujo
b2f244e71e Use NULL instead of 0 for pointers.
MFC after:	2 weeks.
2016-04-18 07:05:18 +00:00
araujo
a32949ae45 Use NULL instead of 0 for pointers.
realloc will return NULL if it cannot allocate memory.

MFC after:	2 weeks.
2016-04-18 06:12:00 +00:00
araujo
76c4083741 Use NULL for pointers.
strrchr(3) will return NULL if the character does not appears in
the string.

MFC after:	2 weeks.
2016-04-18 06:02:26 +00:00
araujo
d2d25bb0ca Use NULL instead of 0 for pointers.
Also malloc will return NULL if it cannot allocate memory.

MFC after:	2 weeks.
2016-04-18 05:46:18 +00:00
araujo
0bd7ac2f20 Use NULL instead of 0.
strtok(3) will return NULL when no more tokens remain.

MFC after:	2 weeks.
2016-04-18 05:26:32 +00:00
bapt
2f359d8f3e Note that mklocale(1) and colldef(1) are no longer used
mklocale and colldef has been replaced by localedef, but they have to be kept
until 10.2 is EOL for mklocale (it has been added to 10.3 as a bootstrap tools)
and until 10.3 is EOL for colldef (it has never been added to bootstrap tools)
2016-04-17 23:02:49 +00:00
bapt
b519ca02d7 Use the nitems() macro 2016-04-17 22:42:48 +00:00
jilles
8156bf7454 mail: Don't truncate mtime of mailbox to microseconds. 2016-04-17 20:00:24 +00:00
bapt
13b4b9515d Fix a mandoc -Tlint warning 2016-04-16 12:49:26 +00:00
bapt
79d8dcdb25 Directly set the O_CLOEXEC flags via the open(2) attributes
MFC after:	1 week
2016-04-16 12:47:23 +00:00
gjb
6549ef7d12 MFH
Sponsored by:	The FreeBSD Foundation
2016-04-16 02:32:12 +00:00
pfg
26c891f034 Cleanup unnecessary semicolons from utilities we all love. 2016-04-15 22:31:22 +00:00
bdrewery
b40d5fe0a7 META_MODE: Don't rebuild build-tools targets during normal build.
This avoids 'build command changed' due to CFLAGS/CC changes during the
normal build.  Without this the build-tools targets end up rebuilding
for the *target* rather than keeping the native versions built in
build-tools.

Sponsored by:	EMC / Isilon Storage Division
2016-04-14 21:06:10 +00:00
gjb
cad5e8d3bc MFH
Sponsored by:	The FreeBSD Foundation
2016-04-13 02:04:09 +00:00
pfg
5c3b3f706f fmt(1): for pointers use NULL instead of 0
While here clean excessive not lint comment indentation.
2016-04-13 01:57:06 +00:00
pfg
767e032691 fmt(1): reformat with indent(1).
Failed attempt to get nearer to style(9) and the format from the
original OpenBSD code. At least it should be readable now.

No functional change.
2016-04-13 01:46:48 +00:00
cem
f1b3ba4c0d Add a small tool, resizewin(1), to query terminal for window size
Submitted by:	Daniel O'Connor
Reviewed by:	kan, wblock, cem
Sponsored by:	EMC / Isilon Storage Division
Differential Revision:	https://reviews.freebsd.org/D4438
2016-04-13 00:30:42 +00:00
gjb
e0e3598ce1 MFH
Sponsored by:	The FreeBSD Foundation
2016-04-11 15:24:59 +00:00
bofh
1485828b08 Add bofh@ in calendar.freebsd 2016-04-10 16:48:21 +00:00
fanf
3389187e4b Handle whois referrals between RIRs
This is rather ugly, because the RIPE and APNIC whois servers do
not provide referrals for address blocks that they do not manage.
However ARIN is usually the right place or knows the right place
so we try there.

The particular instance which clued me in to this bug is U.Mich.
141.211.0.0/16 for which the referral chain should be IANA ->
RIPE -> ARIN. RIPE's RDAP does provide useful redirects (for
example try `curl -I http://rdap.db.ripe.net/ip/141.211.0.0)
so maybe their whois server can be improved.

AfriNIC's whois server gives more direct referrals, but they are
designed to be human-readable. Ugly, but we can manage.

The issue of referrals between RIRs is likely to become more important
in the future whith the increasing number of cross-region IP address
block transfers increases.
2016-04-08 11:43:22 +00:00
gahr
eb925cdf00 Enhance uuencode with a -r option to produce raw output.
This matches with uudecode's -r option to decode raw data without initial and
final framing lines.

$ echo Test | uuencode -mr - | uudecode -mr
Test

Approved by:	cognet
MFC after:		1 week
2016-04-07 16:12:38 +00:00
trasz
825d80e01c Add four new RCTL resources - readbps, readiops, writebps and writeiops,
for limiting disk (actually filesystem) IO.

Note that in some cases these limits are not quite precise. It's ok,
as long as it's within some reasonable bounds.

Testing - and review of the code, in particular the VFS and VM parts - is
very welcome.

MFC after:	1 month
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D5080
2016-04-07 04:23:25 +00:00
gjb
5fec2ae02d MFH
Sponsored by:	The FreeBSD Foundation
2016-04-06 01:44:21 +00:00
pfg
753486d87a Fix sed functions 'i' and 'a' from discarding leading white space.
This appears to be implementation dependent but convenient and makes
our sed behave more like GNU sed.

Given that it is not the historic behavior, bump FreeBSD_version
should userland/ports somehow depend on it.

Obtained from:	NetBSD (bin/49872)

Reviewed by:	bdrewery
PR:		208554
Merge after:	NEVER
2016-04-06 00:55:39 +00:00
gjb
1dc4c40e3b MFH
Sponsored by:	The FreeBSD Foundation
2016-04-04 23:55:32 +00:00
brooks
f447eee830 Add a cross reference to ktrace(2).
Obtained from:	CheriBSD (9cb420d6b7f04c1b7d2006180b80932e5d3fe50e)
MFC after:	1 week
Sponsored by:	DARPA, AFRL
2016-03-31 23:57:03 +00:00
fanf
b951056e8c Fix whois queries for ARIN AS numbers.
The ARIN whois server likes AS number queries to be in the form
"a 8075" rather than "as7085".
2016-03-31 12:13:01 +00:00
trasz
13af8fa343 Add phttpget(8) .Xr to fetch(1).
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-03-29 11:56:00 +00:00
pfg
a7966dbd97 netstat: avoid returning uninitialized value in p_sockaddr().
In the case the width is less than 0, we are returning an uninitialized
value. For practical purposes the return value is ignored but initialize
it to avoid trouble.

CID:	1341619
2016-03-27 20:02:21 +00:00
ed
dfa2acdc82 Replace the CloudABI system call table by a machine generated version.
The type definitions and constants that were used by COMPAT_CLOUDABI64
are a literal copy of some headers stored inside of CloudABI's C
library, cloudlibc. What is annoying is that we can't make use of
cloudlibc's system call list, as the format is completely different and
doesn't provide enough information. It had to be synced in manually.

We recently decided to solve this (and some other problems) by moving
the ABI definitions into a separate file:

	https://github.com/NuxiNL/cloudabi/blob/master/cloudabi.txt

This file is processed by a pile of Python scripts to generate the
header files like before, documentation (markdown), but in our case more
importantly: a FreeBSD system call table.

This change discards the old files in sys/contrib/cloudabi and replaces
them by the latest copies, which requires some minor changes here and
there. Because cloudabi.txt also enforces consistent names of the system
call arguments, we have to patch up a small number of system call
implementations to use the new argument names.

The new header files can also be included directly in FreeBSD kernel
space without needing any includes/defines, so we can now remove
cloudabi_syscalldefs.h and cloudabi64_syscalldefs.h. Patch up the
sources to include the definitions directly from sys/contrib/cloudabi
instead.
2016-03-24 21:47:15 +00:00
trasz
6fee958d28 Make the autofs(5) -hosts map more robust, primarily to make it correctly
handle NFS shares containing whitespace. This also adds the -E parameter
to showmount(8).

Reviewed by:	emaste@, jhibbits@, wblock@
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D5649
2016-03-23 12:13:53 +00:00
pfg
34220e27d8 localedef(1): minor sorting to match Illumos.
Illumos recently included space in 'print' class. We already had
this but the code had slight sorting differences. Move it some
lines up to reduce diffs with Illumos.

No functional change.

Reference:
https://illumos.org/issues/5227
2016-03-20 03:27:06 +00:00
grembo
9533d74078 Update fetch.1 and fetch.3 to reflect libfetch's actual use of CA bundles
Reviewed by:	wblock
Approved by:	wblock
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D5558
2016-03-19 11:55:21 +00:00
sjg
44636d2ebb Update to bmake-20160315
Archive member handling works again
meta mode, treat missing reads as for writes.

Update dirdeps.mk - much improved startup time.
Update meta.stage.mk - avoid ln when chmod required.
2016-03-18 20:03:09 +00:00
sjg
4ebe624823 xo_config.h no longer in contrib, so -I's needed
PR:		/homes/sjg/commit-logs/freebsd/libxo/xo_config.diff
Reviewed by:	jkim
2016-03-17 04:21:57 +00:00
emaste
9bc25213ea vtfontcvt: support .hex fonts with characters beyond the Unicode BMP
This is already supported by the vt(4) vfnt format mapping from code
points to glyphs. Update the .hex font parser to accept up to six hex
digits.
2016-03-15 21:32:46 +00:00
glebius
c39b2fd5d1 Print running TCP connection counts with TCP statistics. 2016-03-15 00:19:30 +00:00
lidl
33eb7d8e7f Do not truncate office phones in finger's summary listing
When finger is invoked as as "finger username", it produces the
long listing by default, and phones numbers are pretty-printed
by the prphone() function. When invoked as just "finger", the
same pretty-printing happens, but is truncated at 9 characters.
Given the summary listing is already greater than 80 columns,
making it even wider is of no harm.

Approved by:	rpaulo (mentor)
Differential Revision:	https://reviews.freebsd.org/D5638
2016-03-14 22:20:22 +00:00
gjb
086e6f562f MFH
Sponsored by:	The FreeBSD Foundation
2016-03-14 18:54:29 +00:00
sobomax
bd7cbc7f20 In the de-duplication mode, when found matching md5 checksum also read
back block and compare actual content. Just output original block
instead of back reference in the unlikely event of collision.
2016-03-13 21:09:08 +00:00
ian
5d2092c908 Fix a bug in bsdgrep that caused the program to hang in a tight loop for
some combinations of command line options and search patterns.  The code was
examining regexec flags looking for a regcomp flag value.  The fix is to
look in the struct field where the decoded regcomp flag was stored when the
regex was compiled.

With this fix, it's possible to build WITHOUT_GNU_GREP_COMPAT and
WITH_BSDGREP and have a usable GPL-free grep (which of course lacks gnugrep
extensions).  It now passes the kyua tests except for one test that requires
the -z/--null-data gnu extension, and one test involving outputting context
lines across multiple files which appears to sometimes output an extra
delimiter line ("--") between matches (a rather obscure failure of a rather
obscure feature, so bsdgrep should be generally usable now).
2016-03-13 14:53:12 +00:00
kib
c259148074 Fix handling of umtxp resource limit in sh(1)/ulimit(1), limits(1), add
login.conf(5) support.

Reviewed by:	jilles
Sponsored by:	The FreeBSD Foundation
Differential revision:	https://reviews.freebsd.org/D5610
2016-03-12 14:54:34 +00:00
bdrewery
749836195e Fix upgrade of bmake by not setting conflicting MAKE_VERSION.
This may be used in later checks, such as in bsd.dep.mk, to
enable features that rely on the built-in value.

Sponsored by:	EMC / Isilon Storage Division
2016-03-11 04:09:50 +00:00
sjg
218c7825fe Merge bmake-20160307 2016-03-11 01:35:39 +00:00
sobomax
0e0b4ac1f0 When -S is specified dump summary to stdout, not stderr, so it's
easier to capture and process it with external tools via pipe.
2016-03-10 23:19:35 +00:00
sobomax
773a64f003 Add -S option to print out summary after compression has been
completed.

MFC after: 	2 weeks
2016-03-10 21:36:24 +00:00
gjb
1c7e318a9a MFH
Sponsored by:	The FreeBSD Foundation
2016-03-10 21:16:01 +00:00
bdrewery
aab40fdc3d DIRDEPS_BUILD: Connect MK_TESTS.
Sponsored by:	EMC / Isilon Storage Division
2016-03-09 22:46:01 +00:00
jhb
5ca736c477 Use ptrace(2) LWP events to track threads reliably in truss.
- truss can now log the system call invoked by a thread during a
  voluntary process exit.  No return value is logged, but the value passed
  to exit() is included in the trace output.  Arguments passed to thread
  exit system calls such as thr_exit() are not logged as voluntary thread
  exits cannot be distinguished from involuntary thread exits during a
  system call.
- New events are now reported for thread births and exits similar to the
  recently added events for new child processes when following forks.

Reviewed by:	kib
Differential Revision:	https://reviews.freebsd.org/D5561
2016-03-09 18:45:41 +00:00
jhb
e641458f70 Fix reporting of the CloudABI ABI in kdump.
- Advertise the word size for CloudABI ABIs via the SV_LP64 flag.  All of
  the other ABIs include either SV_ILP32 or SV_LP64.
- Fix kdump to not assume a 32-bit ABI if the ABI flags field is non-zero
  but SV_LP64 isn't set.  Instead, only assume a 32-bit ABI if SV_ILP32 is
  set and fallback to the unknown value of "00" if neither SV_LP64 nor
  SV_ILP32 is set.

Reviewed by:	kib, ed
Differential Revision:	https://reviews.freebsd.org/D5560
2016-03-09 18:38:30 +00:00
bdrewery
ca7bcc42dd DIRDEPS_BUILD: Update clang dependencies after r296417.
Sponsored by:	EMC / Isilon Storage Division
2016-03-08 21:26:44 +00:00
bdrewery
1d619dcb69 Record command exit status in the typescript file when running simple commands.
Also capitalize 'command:'.

Relnotes:	yes
MFC after:	2 weeks
Sponsored by:	EMC / Isilon Storage Division
2016-03-08 18:05:23 +00:00
bdrewery
49a860c3b1 Just exit in the child if execve(2) fails.
No functional change.

This is mostly addressing a false-positive from the clang static
analyzer due to it thinking that done() was being called with freed
memory, however the kill(0, SIGTERM) made the done() never reached.
It doesn't make sense to the show the footer from the child anyhow, nor
does it make sense to kill the process group here since the execve(2)
failed in the child.  This code was leftover from many years of refactoring.

MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
2016-03-08 18:05:20 +00:00
bdrewery
f234bc3e54 Filemon: Attach from the child to avoid racing with the parent attach.
This is the same as how the bmake filemon usage works.

This also fixes failed attach not properly flushing the TTY.

MFC after:	1 week
Relnotes:	yes
Sponsored by:	EMC / Isilon Storage Division
2016-03-08 18:05:02 +00:00
gjb
f4bd2481be MFH
Sponsored by:	The FreeBSD Foundation
2016-03-06 04:13:17 +00:00
dim
e9ad9d5ff7 Upgrade our copies of clang, llvm, lldb and compiler-rt to 3.8.0
release.

Please note that from 3.5.0 onwards, clang, llvm and lldb require C++11
support to build; see UPDATING for more information.

Release notes for llvm and clang will soon be available here:
<http://llvm.org/releases/3.8.0/docs/ReleaseNotes.html>
<http://llvm.org/releases/3.8.0/tools/clang/docs/ReleaseNotes.html>

Thanks to Ed Maste, Roman Divacky, Davide Italiano and Antoine Brodin
for their help.

Relnotes:	yes
2016-03-05 20:24:31 +00:00
bapt
338423c265 Fix "files" arguments 2016-03-05 15:53:06 +00:00
dim
9ac7ec8827 Merge ^/head r296007 through r296368. 2016-03-03 23:15:46 +00:00
dim
d50a252468 Regenerate llvm and clang manpages. 2016-03-03 23:09:57 +00:00
gjb
955ce29ea3 MFH
Sponsored by:	The FreeBSD Foundation
2016-03-02 16:14:46 +00:00
kib
e76eb4255b Implement process-shared locks support for libthr.so.3, without
breaking the ABI.  Special value is stored in the lock pointer to
indicate shared lock, and offline page in the shared memory is
allocated to store the actual lock.

Reviewed by:	vangyzen (previous version)
Discussed with:	deischen, emaste, jhb, rwatson,
	Martin Simmons <martin@lispworks.com>
Tested by:	pho
Sponsored by:	The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
bdrewery
eb3588c7b6 Update dependencies.
Sponsored by:	EMC / Isilon Storage Division
2016-02-26 22:14:15 +00:00
pfg
0c969eabba talk(1): Replace select(2) with poll(2)
Hinted by:	OpenBSD and NetBSD
2016-02-25 19:06:44 +00:00
oshogbo
023f14d65b Convert casperd(8) daemon to the libcasper.
After calling the cap_init(3) function Casper will fork from it's original
process, using pdfork(2). Forking from a process has a lot of advantages:
1. We have the same cwd as the original process.
2. The same uid, gid and groups.
3. The same MAC labels.
4. The same descriptor table.
5. The same routing table.
6. The same umask.
7. The same cpuset(1).
From now services are also in form of libraries.
We also removed libcapsicum at all and converts existing program using Casper
to new architecture.

Discussed with:		pjd, jonathan, ed, drysdale@google.com, emaste
Partially reviewed by:	drysdale@google.com, bdrewery
Approved by:		pjd (mentor)
Differential Revision:	https://reviews.freebsd.org/D4277
2016-02-25 18:23:40 +00:00
jhb
85f8ae969b Add simple support for CloudABI processes to kdump(1).
This does not decode arguments to system calls but should properly
decode system call names and error return values.

Reviewed by:	ed
Differential Revision:	https://reviews.freebsd.org/D5412
2016-02-25 17:43:32 +00:00
dim
7ef26dcde6 Merge ^/head r295902 through r296006. 2016-02-24 21:38:51 +00:00
bdrewery
2a891f1feb DIRDEPS_BUILD: Regenerate without local dependencies.
These are no longer needed after the recent 'beforebuild: depend' changes
and hooking DIRDEPS_BUILD into a subset of FAST_DEPEND which supports
skipping 'make depend'.

Sponsored by:	EMC / Isilon Storage Division
2016-02-24 17:20:11 +00:00
bdrewery
b6e8f9c3c9 DIRDEPS_BUILD: Update dependencies.
Sponsored by:	EMC / Isilon Storage Division
2016-02-24 17:18:35 +00:00
bdrewery
56e259a8db Properly fix these builds by adding NO_WMISSING_VARIABLE_DECLARATIONS from r249657.
This reverts r284374.

MFC after:	2 weeks
Sponsored by:	EMC / Isilon Storage Division
2016-02-24 17:18:31 +00:00
gjb
4719e40f5b MFH
Sponsored by:	The FreeBSD Foundation
2016-02-24 03:08:58 +00:00
sobomax
85ce861e46 Obsolete mkulzma(8) and geom_uncompress(4), their functionality
is now provided by mkuzip(8) and geom_uzip(4) respectively.

MFC after:	1 month
2016-02-24 00:39:36 +00:00
sobomax
8abe971b5e Improve mkuzip(8) and geom_uzip(4), merge in LZMA support from mkulzma(8)
and geom_uncompress(4):

1. mkuzip(8):

 - Proper support for eliminating all-zero blocks when compressing an
   image. This feature is already supported by the geom_uzip(4) module
   and CLOOP format in general, so it's just a matter of making mkuzip(8)
   match. It should be noted, however that this feature while it sounds
   great, results in very slight improvement in the overall compression
   ratio, since compressing default 16k all-zero block produces only 39
   bytes compressed output block, which is 99.8% compression ratio. With
   typical average compression ratio of amd64 binaries and data being
   around 60-70% the difference between 99.8% and 100.0% is not that
   great further diluted by the ratio of number of zero blocks in the
   uncompressed image to the overall number of blocks being less than
   0.5 (typically). However, this may be important from performance
   standpoint, so that kernel are not spinning its wheels decompressing
   those empty blocks every time this zero region is read. It could also
   be important when you create huge image mostly filled with zero
   blocks for testing purposes.

 - New feature allowing to de-duplicate output image. It turns out that
   if you twist CLOOP format a bit you can do that as well. And unlike
   zero-blocks elimination, this gives a noticeable improvement in the
   overall compression ratio, reducing output image by something like
   3-4% on my test UFS2 3GB image consisting of full FreeBSD base system
   plus some of the packages (openjdk, apache etc), about 2.3GB worth of
   file data (800+MB compressed). The only caveat is that images created
   with this feature "on" would not work on older versions of FeeBSDxi
   kernel, hence it's turned off by default.

 - provide options to control both features and document them in manual
   page.

 - merge in all relevant LZMA compression support from the mkulzma(8),
   add new option to select between both.

 - switch license from ad-hoc beerware into standard 2-clause BSD.

2. geom_uzip(4):

 - implement support for de-duplicated images;

 - optimize some code paths to handle "all-zero" blocks without reading
   any compressed data;

 - beef up manual page to explain that geom_uzip(4) is not limited only
   to md(4) images. The compressed data can be written to the block
   device and accessed directly via magic of GEOM(4) and devfs(4),
   including to mount root fs from a compressed drive.

 - convert debug log code from being compiled in conditionally into
   being present all the time and provide two sysctls to turn it on or
   off. Due to intended use of the module, it can be used in
   environments where there may not be a luxury to put new kernel with
   debug code enabled. Having those options handy allows debug issues
   without as much problem by just having access to serial console or
   network shell access to a box/appliance. The resulting additional
   CPU cycles are just few int comparisons and branches, and those are
   minuscule when compared to data decompression which is the main
   feature of the module.

 - hopefully improve robustness and resiliency of the geom_uzip(4) by
   performing some of the data validation / range checking on the TOC
   entries and rejecting to attach to an image if those checks fail.

 - merge in all relevant LZMA decompression support from the
   geom_uncompress(4), enable automatically when appropriate format is
   indicated in the header.

 - move compilation work into its own worker thread so that it does not
   clog g_up. This allows multiple instances work in parallel utilizing
   smp cores.

 - document new knobs in the manual page.

Reviewed by:		adrian
MFC after:		1 month
Differential Revision:	https://reviews.freebsd.org/D5333
2016-02-23 23:59:08 +00:00
jhb
ca28498877 Add handling for non-native error values to libsysdecode.
Add two new functions, sysdecode_abi_to_freebsd_errno() and
sysdecode_freebsd_to_abi_errno(), which convert errno values between
the native FreeBSD ABI and other supported ABIs. Note that the
mappings are not necessarily perfect meaning in some cases multiple
errors in one ABI might map to a single error in another ABI. In that
case, the reverse mapping will return one of the errors that maps, but
which error is non-deterministic.

Change truss to always report the raw error value to the user but
use libsysdecode to map it to a native errno value that can be used
with strerror() to generate a description. Previously truss reported
the "converted" error value. Now the user will always see the exact
error value that the application sees.

Change kdump to report the truly raw error value to the user. Previously
kdump would report the absolute value of the raw error value (so for
Linux binaries it didn't output the FreeBSD error value, but the positive
value of the Linux error). Now it reports the real (i.e. negative) error
value for Linux binaries. Also, use libsysdecode to convert the native
FreeBSD error reported in the ktrace record to the raw error used by the
ABI. This means that the Linux ABI can now be handled directly in
ktrsysret() and removes the need for linux_ktrsysret().

Reviewed by:	bdrewery, kib
Helpful notes:	wblock (manpage)
Differential Revision:	https://reviews.freebsd.org/D5314
2016-02-23 20:00:55 +00:00
jhb
dbabd32553 Add support for displaying thread IDs to truss(1).
- Consolidate duplicate code for printing the metadata at the start of
  each line into a shared function.
- Add an -H option which will log the thread ID of the relevant thread
  for each event.

While here, remove some extraneous calls to clock_gettime() in
print_syscall() and print_syscall_ret().  The caller of print_syscall_ret()
always updates the current thread's "after" time before it is called.

Reviewed by:	kib
MFC after:	1 month
Differential Revision:	https://reviews.freebsd.org/D5363
2016-02-23 19:56:29 +00:00
gjb
8bfb527a82 MFH
Sponsored by:	The FreeBSD Foundation
2016-02-22 12:28:23 +00:00
dim
2e1a0cbbd8 Merge ^/head r295601 through r295844. 2016-02-21 13:49:26 +00:00
se
2f3384530c Remove O_SYNC from the options passed to dbmopen().
The output file is created as a temporary file that is moved over the
existing file after completion. Thus there is no need to immediately
flush all created db records to the temporary file.

This speeds up creation of the termcap db by a factor of 40 on my
ZFS based /etc filesytem (from 25 seconds to 0.6 seconds).

I have compared multiple output files created with and without O_SYNC
and they came out identical each time.  Nonetheless it might be best
to MFC this change and the similar one for services_mkdb (r295465) at
the same time when the changes to hash.c in review D5186 are merged.

MFC:	 1 week
2016-02-19 08:42:13 +00:00
bdrewery
72c27c1142 DIRDEPS_BUILD: Add dependencies somehow missed in r295681.
Sponsored by:	EMC / Isilon Storage Division
2016-02-19 00:41:38 +00:00
gjb
adbdbd2fff MFH
Sponsored by:	The FreeBSD Foundation
2016-02-18 00:37:58 +00:00
bdrewery
217f6f397b Fix installation of makewhatis.local(1) since r283777.
The wrapper script has moved to libexec/makewhatis.local since it is not
directlry related to the older makewhatis(1) utility that has been replaced
by the usr.bin/mandoc version.

Reported by:	vangyzen
2016-02-18 00:26:21 +00:00
bdrewery
9d77ef9f08 DIRDEPS_BUILD: Hookup CLANG_EXTRAS.
Sponsored by:	EMC / Isilon Storage Division
2016-02-16 23:11:09 +00:00
jhb
4f1afbf69a Remove an unused variable that snuck into the previous revision. 2016-02-16 22:13:25 +00:00
jhb
36d8f601e5 Fetch the current thread and it's syscall state from the trussinfo object
instead of passing some of that state as arguments to print_syscall() and
print_syscallret().  This just makes the calls of these functions shorter
and easier to read.
2016-02-16 22:00:01 +00:00
pfg
fae984a570 Fix naive use of ftell(3).
Secure coding practices, FIO19-C.
2016-02-16 21:39:19 +00:00
olivier
f3a84e3fe1 Add myself as ports committer (including calendar)
Approved by:	jadawin
Differential Revision:	https://reviews.freebsd.org/D5295
2016-02-16 10:33:45 +00:00
bdrewery
ed99ba5f08 DIRDEPS_BUILD: Update dependencies.
Sponsored by:	EMC / Isilon Storage Division
2016-02-16 02:14:30 +00:00
bdrewery
002a527f3e Test directories can build in parallel fine.
Sponsored by:	EMC / Isilon Storage Division
2016-02-16 02:13:59 +00:00
gjb
dda00fa311 Include correct is*ct*l program in iscsi package, and package the
legacy version separately.

Submitted by:	trasz
Sponsored by:	The FreeBSD Foundation
2016-02-16 01:18:49 +00:00
gjb
64f34d0d71 MFH
Sponsored by:	The FreeBSD Foundation
2016-02-15 21:58:52 +00:00
jhb
18d5c9d9d5 Correct the ABI name for Linux/i386 binaries under FreeBSD/i386.
This allows truss to work for these binaries again after r288424.

MFC after:	3 days
2016-02-15 20:27:16 +00:00
jhb
f921347d5d Sign extend the error value for failing Linux/i386 system calls. This
restores the mapping of Linux errors to native FreeBSD errno values after
the refactoring in r288424.
2016-02-15 20:26:40 +00:00
kevlo
d4f3e58ebf Fix a bug that caused nothing to be skipped when skipping exactly the
number of bytes present in a regular file was requested.

Obtained from:	OpenBSD
2016-02-14 14:23:56 +00:00
dim
f393760a8a Merge ^/head r295544 through r295600. 2016-02-13 16:02:12 +00:00
emaste
5e2392fbf9 elfcopy: enable PE and EFI support
Sponsored by:	The FreeBSD Foundation
2016-02-12 21:15:31 +00:00
dim
afa214d26d Merge ^/head r295351 through r295543. 2016-02-11 20:07:09 +00:00
gjb
e4997c6184 MFH
Sponsored by:	The FreeBSD Foundation
2016-02-10 04:20:39 +00:00
kib
6ea080df6a Rename variable to reflect the condition.
Suggested by:	jhb
Sponsored by:	The FreeBSD Foundation
2016-02-09 18:35:37 +00:00
kib
736e078495 Rename P_KTHREAD struct proc p_flag to P_KPROC.
I left as is an apparent bug in ntoskrnl_var.h:AT_PASSIVE_LEVEL()
definition.

Suggested by:	jhb
Sponsored by:	The FreeBSD Foundation
2016-02-09 16:30:16 +00:00
dim
0abf59f0ae Merge ^/head r294961 through r295350. 2016-02-06 14:07:17 +00:00
gjb
ae6f203603 More 'tests' packaging fixes.
Sponsored by:	The FreeBSD Foundation
2016-02-03 19:08:45 +00:00
gjb
811267757a More 'tests' package fixes.
Sponsored by:	The FreeBSD Foundation
2016-02-03 00:34:23 +00:00
gjb
b795b4ac5f Fix build.
Sponsored by:	The FreeBSD Foundation
2016-02-02 23:23:37 +00:00
gjb
efd5551e55 MFH
Sponsored by:	The FreeBSD Foundation
2016-02-02 22:27:48 +00:00
gjb
a6998ad84f First pass to fix the 'tests' packages.
Sponsored by:	The FreeBSD Foundation
2016-02-02 22:26:49 +00:00
gjb
f7c36465eb Create ee(1) package.
Sponsored by:	The FreeBSD Foundation
2016-02-02 21:04:33 +00:00
gjb
34bfa3fb07 Create a package for vi(1) and related files.
Sponsored by:	The FreeBSD Foundation
2016-02-02 16:47:50 +00:00
alfred
b04e8a547e Increase max allowed backlog for listen sockets
from short to int.

PR: 203922
Submitted by: White Knight <white_knight@2ch.net>
MFC After: 4 weeks
2016-02-02 05:57:59 +00:00
jhb
377cdb0ab1 Add a SYSDECODE_ABI_ prefix to the ABI enums to avoid potential collisions.
Suggested by:	jmallett
Reviewed by:	bdrewery, jmallett
Differential Revision:	https://reviews.freebsd.org/D5123
2016-01-30 01:00:54 +00:00
gjb
5294f47313 Restore the 'beforeinstall' for chpass(1), so files with the
schg flag are handled before being replaced by a symbolic link.

Fix passwd(1)/yppasswd(1) schg issue similarly to chpass(1).

Note:  These are likely temporary fixes while identifying any
remaining files that are not marked as 'flags=schg' via mtree(8).

Sponsored by:	The FreeBSD Foundation
2016-01-28 02:56:30 +00:00
gjb
780ae7713f Ensure mtree(8) recognizes chpass(1) is schg.
Convert hard links to symbolic links, and remove the
'beforeinstall' and 'afterinstall' chflags(8) execution.

Sponsored by:	The FreeBSD Foundation
2016-01-28 01:15:57 +00:00
dim
aef1771e36 Merge ^/head r294777 through r294960. 2016-01-27 22:52:20 +00:00
gjb
589dc73d2e MFH
Sponsored by:	The FreeBSD Foundation
2016-01-27 14:16:13 +00:00
glebius
9487eaf98f Augment struct tcpstat with tcps_states[], which is used for book-keeping
the amount of TCP connections by state.  Provides a cheap way to get
connection count without traversing the whole pcb list.

Sponsored by:	Netflix
2016-01-27 00:45:46 +00:00
dteske
cd54153389 Bump copyright 2016-01-27 00:03:43 +00:00
dteske
d08ad3f736 Add -k' for dpv(3) keep_tite' config option
For scripts using dialog(1) several times, it can be visually distracting
running dpv(1) several times amidst other dialogs. The `-k' option, similar
to dialog(1) `--keep-tite', enables the same functionality to smooth ti/te.
2016-01-26 23:59:30 +00:00
jhb
cd4a0ee266 Add support to libsysdecode for decoding system call names.
A new sysdecode_syscallname() function accepts a system call code and
returns a string of the corresponding name (or NULL if the code is
unknown).  To support different process ABIs, the new function accepts a
value from a new sysdecode_abi enum as its first argument to select the
ABI in use.  Current ABIs supported include FREEBSD (native binaries),
FREEBSD32, LINUX, LINUX32, and CLOUDABI64.  Note that not all ABIs are
supported by all platforms.  In general, a given ABI is only supported
if a platform can execute binaries for that ABI.

To simplify the implementation, libsysdecode's build reuses the
existing pre-generated files from the kernel source tree rather than
duplicating new copies of said files during the build.

kdump(1) and truss(1) now use these functions to map system call
identifiers to names.  For kdump(1), a new 'syscallname()' function
consolidates duplicated code from ktrsyscall() and ktrsyscallret().
The Linux ABI no longer requires custom handling for ktrsyscall() and
linux_ktrsyscall() has been removed as a result.

Reviewed by:	bdrewery
Differential Revision:	https://reviews.freebsd.org/D4823
2016-01-26 19:07:09 +00:00
dim
e94fcba083 Merge ^/head r294599 through r294776. 2016-01-26 07:49:11 +00:00
br
d83e0fe471 Disable gprof and users on RISC-V, they don't build. 2016-01-25 16:49:39 +00:00
gjb
ead3a2f824 MFH
Sponsored by:	The FreeBSD Foundation
2016-01-25 14:13:28 +00:00
emaste
aa1ae99362 elfdump: handle STT_SPARC_REGISTER
STT_SPARC_REGISTER is a SPARC-specific symbol type specified by the
Sparcv9 ABI to provide some information on register use by the object.

Also rework st_info type lookup to avoid out-of-bounds array access.

MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2016-01-25 04:22:01 +00:00
br
6cdde1eb9e We don't support a.out executables on RISC-V.
Reviewed by:	emaste
Sponsored by:	DARPA, AFRL
Sponsored by:	HEIF5
Differential Revision:	https://reviews.freebsd.org/D5048
2016-01-24 15:15:57 +00:00
fanf
7e5d57d7f1 Fix a regression in the .de and .dk whois special cases
Ensure the special cases trigger whether we come via a referral
or via the -c option. Match host names case-insensitively.

Use the default character set supported by .de (UTF-8) since that
is more compatible with the modern world than ISO 8859-1. Persuade
them to give us a useful answer whether an internationalized
domain name is given in UTF-8 or in punycode.
2016-01-23 01:20:58 +00:00
fanf
d92f8a19a9 A lot of the cleverness in whois is no longer needed!
The IANA whois server has the right referral information for domain
names, IP addresses, and AS numbers, so whois does not need to be
able to choose servers itself (except for a few cases where referrals
do not work). We can delete a chunk of code, which is always fun.

This change improves the referral handling to be less sensitive to
all the various formats, and to allow multi-hop referral chains,
such as IANA -> registry -> registrar.

ARIN queries have the "+" flag added if no flags are present, so we
get full details if the query matches multiple objects. The Verisign
anti-spam logic is also now suppressed if the user provided a non-
trivial query string.

Uninformative rubric is now trimmed by default. The -S option
turns off trimming, and disables query fettling.

The -i option is back to its traditional pre-1999 hostname, since
whois.internic.net is more useful than whois.networksolutions.com.
Note that the old fallback/default server whois.crsnic.net is an
alias for whois.internic.net.

The manual is more informative about query syntax.
2016-01-23 00:28:18 +00:00
dim
7cf07624a0 Merge ^/head r294169 through r294598. 2016-01-22 20:41:56 +00:00
fanf
e270f2d00e Update whois synopsis and usage with new options 2016-01-22 19:06:43 +00:00
bjk
1f41c60b30 Bump .Dd after r294575 2016-01-22 17:17:27 +00:00
fanf
8ba35e7674 A few whois usability improvements
Look up AS numbers at ARIN.

Handle more referral formats.

Suppress spammy nameserver objects when querying the .com and .net
whois servers by explicitly querying for domain names by default.
2016-01-22 16:43:49 +00:00