11562 Commits

Author SHA1 Message Date
dteske
bf764fc982 Add new libraries/utilities for data throughput visualization.
dpv(3): dialog progress view library
dpv(1): stream data from stdin or multiple paths with dialog progress view
figpar(3): configuration file parsing library

Reviews:	D714
Reviewed by:	jelischer, shurd
Discussed at:	MeetBSD California 2014 Vendor/Dev Summit
Discussed on:	-current
MFC after:	21 days
X-MFC-to:	stable/10 stable/9
2014-11-04 23:46:01 +00:00
jmg
f431c9f9f8 fix spelling of offset since that is what is used in the body... 2014-10-30 06:54:25 +00:00
jkim
dde27ca539 Replace a magic number with the proper definition. This change actually
fixes broken state field after r273266, i.e., "CPU-1" was displayed in place
of "RUN".
2014-10-29 19:21:19 +00:00
trasz
3af53d3840 Fix iscsictl(8) and ctld(8) to correctly handle Windows newlines
(CRLF) in iscsi.conf and ctl.conf.

MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2014-10-29 12:22:32 +00:00
bapt
99068e4287 Add regression tests for the timeout(1) utility
They are modeled over the regression tests that are provided for the GNU
coreutils timeout(1) utility
2014-10-28 10:39:41 +00:00
bapt
8dd706ab6c Improve compatibility with GNU timeout
According to the coreutils regression testsuite for timeout(1)
It is expect to exit with a status being:
125 in case an invalid duration or signal is passed in arguments
126 in case an invalid command is passed in arguments
127 in case the command passed in arguments does not exists.

While here document this behaviour in the man page
2014-10-28 10:33:31 +00:00
mckusick
12be45777a Replace update from -r271410 accidentally lost in -r273575. 2014-10-26 20:48:26 +00:00
jhb
5dd26e948d The current POSIX semaphore implementation stores the _has_waiters flag
in a separate word from the _count.  This does not permit both items to
be updated atomically in a portable manner.  As a result, sem_post()
must always perform a system call to safely clear _has_waiters.

This change removes the _has_waiters field and instead uses the high bit
of _count as the _has_waiters flag.  A new umtx object type (_usem2) and
two new umtx operations are added (SEM_WAIT2 and SEM_WAKE2) to implement
these semantics.  The older operations are still supported under the
COMPAT_FREEBSD9/10 options.  The POSIX semaphore API in libc has
been updated to use the new implementation.  Note that the new
implementation is not compatible with the previous implementation.
However, this only affects static binaries (which cannot be helped by
symbol versioning).  Binaries using a dynamic libc will continue to work
fine.  SEM_MAGIC has been bumped so that mismatched binaries will error
rather than corrupting a shared semaphore.  In addition, a padding field
has been added to sem_t so that it remains the same size.

Differential Revision:	https://reviews.freebsd.org/D961
Reported by:	adrian
Reviewed by:	kib, jilles (earlier version)
Sponsored by:	Norse
2014-10-24 20:02:44 +00:00
mckusick
bf40fc073d Fix formatting of vmstat (1) so that it looks cleaner on standard 80
column displays. In particular it wraps far less often.

Submitted by: Andy Kosela <akosela@andykosela.com>
PR:           191976
2014-10-24 04:01:14 +00:00
marcel
0172ecaad7 Hook xo(1) to the build -- it's like echo, but uses libxo to
support emitting machine-readable output.

Sponsored by:	Juniper Networks, Inc.
2014-10-23 23:16:21 +00:00
emax
fa3d17f77d Change the code to use the openpty(3) API which uses the pts(4) driver
instead of the pty(4) driver.

PR:		184597
Submitted by:	tobias.rehbein
MFC after:	2 weeks
2014-10-23 15:16:40 +00:00
delphij
7fb2808ce4 Sync with NetBSD.
MFC after:	2 weeks
2014-10-23 01:22:29 +00:00
trasz
853fc00c2a Whitespace fixes.
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2014-10-22 09:17:17 +00:00
jhb
a619f7cffc Retire the unimplemented MAP_RENAME and MAP_NORESERVE flags to mmap(2).
Older binaries are still permitted to use these flags.

PR:		193961 (exp-run in ports)
Differential Revision:	https://reviews.freebsd.org/D848
Reviewed by:	kib
2014-10-18 12:28:51 +00:00
jmg
e9ac97fe26 Fix to col when printing half-line feeds w/ -f option...
Message-Id on openbsd-tech: 20141017195810.GJ132@iris.usta.de
Submitted by:	Ingo Schwarze
Obtained from:	OpenBSD
MFC after:	3 days
2014-10-17 21:09:03 +00:00
jhb
005b66cec8 Decode the arguments passed to _umtx_op(). In particular, decode the
opcode.

MFC after:	1 week
Sponsored by:	Norse
2014-10-13 16:37:06 +00:00
jhb
f805435b74 Fix most of the warnings in kdump(1).
Sponsored by:	Norse
2014-10-13 16:17:42 +00:00
ngie
9de8ee6c0c Integrate usr.bin/gzip/tests from NetBSD into atf/kyua
Sponsored by: EMC / Isilon Storage Division
2014-10-09 02:24:34 +00:00
ngie
9cf2b42641 Integrate usr.bin/grep/tests from NetBSD into atf/kyua
Sponsored by: EMC / Isilon Storage Division
2014-10-09 01:53:23 +00:00
ngie
429a4fc8f2 Integrate usr.bin/cut/tests from NetBSD into atf/kyua
Sponsored by: EMC / Isilon Storage Division
2014-10-09 01:46:31 +00:00
ngie
82aaaa0800 Integrate usr.bin/dirname/tests from NetBSD into atf/kyua
In collaboration with: pho, sjg
Sponsored by: EMC / Isilon Storage Division
2014-10-09 00:55:04 +00:00
ngie
70c01e4074 Integrate usr.bin/cmp/tests from NetBSD into atf/kyua
In collaboration with: sjg
Sponsored by: EMC / Isilon Storage Division
2014-10-09 00:50:33 +00:00
ngie
52c660bb43 Integrate usr.sbin/basename/tests from NetBSD into atf/kyua
In collaboration with: pho
Sponsored by: EMC / Isilon Storage Division
2014-10-09 00:41:42 +00:00
emaste
f9bb385dd3 Correct scale factor for T terabyte suffix
PR:		194250
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2014-10-08 17:40:58 +00:00
eadler
e7ee396a15 don't reinvent the wheel: rely on basename(3)
Reviewed by:	nwhitehorn
2014-10-08 05:04:31 +00:00
marcel
1e5e7f38d1 Update baseline files for EBR, MBR and PC98 now that mkimg fills in
the CHS fields appropriately when -T and -H are given on the command
line.
2014-10-03 20:54:35 +00:00
marcel
ad936aa7f4 Add mkimg_chs() for those schemes that need the LBA broken down into
cylinder, head and track numbers. Return ~0U for these values when
mkimg wasn't given both -T and -H (i.e. no geometry) or the cylinder
would be larger than the provided maximum.

Use mkimgs_chs() for the EBR, MBR and PC98 schemes to fill in the
appropriate fields. Make sure to use a "rounded" size so that the
partition is always a multiple of the track size. We reserved the
room for it in the metadata callback so that's a valid thing to
do.

Bump the mkimg version number.
While doing that again: have mkimg.o depend on the Makefile so that
a version change triggers a rebuild as needed.
2014-10-03 20:48:11 +00:00
bapt
34ba09d82b Make sure to not skip any argument when converting from deprecated
+POS1, -POS2 to -kPOS1,POS2, so that sort +0n gets translated to sort -k1,1n
as it is expected

PR:		193994
Submitted by:	rodrigo
MFC after:	3 days
2014-10-02 06:29:49 +00:00
marcel
41a7dfe8c5 Improve performance of mking(1) by keeping a list of "chunks" in memory,
that keeps track of a particular region of the image. In particular the
image_data() function needs to return to the caller whether a region
contains data or is all zeroes. This required reading the region from
the temporary file and comparing the bytes. When image_data() is used
multiple times for the same region, this will get painful fast.

With a chunk describing a region of the image, we now also have a way
to refer to the image provided on the command line. This means we don't
need to copy the image into a temporary file. We just keep track of the
file descriptor and offset within the source file on a per-chunk basis.

For streams (pipes, sockets, fifos, etc) we now use the temporary file
as a swap file. We read from the input file and create a chunk of type
"zeroes" for each sequence of zeroes that's a multiple of the sector
size. Otherwise, we allocte from the swap file, mmap(2) it, read into
the mmap(2)'d memory and create a chunk representing data.

For regular files, we use SEEK_HOLE and SEEK_DATA to handle sparse files
eficiently and create a chunk of type zeroes for holes and a chunk of
type data for data regions. For data regions, we still compare the bytes
we read to handle differences between a file system's block size and our
sector size.

After reading all files, image_write() is used by schemes to scribble in
the reserved sectors. Since this never amounts to much, keep this data
in memory in chunks of exactly 1 sector.

The output image is created by looking using the chunk list to find the
data and write it out to the output file. For chunks of type "zeroes"
we prefer to seek, but fall back to writing zeroes to handle pipes.
For chunks of type "file" and "memoty" we simply write.

The net effect of this is that for reasonably large images the execution
time drops from 1-2 minutes to 10-20 seconds. A typical speedup is about
5 to 8 times, depending on partition sizes, output format whether in
input files are sparse or not.

Bump version to 20141001.
2014-10-01 21:03:17 +00:00
marcel
a74e992d46 Suffix the cookie constants with ULL to silence warnings from compilers
that try to treat them as 32-bit values.
2014-10-01 20:37:15 +00:00
delphij
4cad8c4b18 Sigh, remove a line that needs to be removed along with previous commit.
Submitted by:	mjg
MFC after:	3 days
X-MFC-with:	272288
2014-09-29 21:54:46 +00:00
delphij
ec42adf015 When setting environment variables in the atrun script, use the
"export foo=bar" form instead of "foo=bar; export foo" since the
former allows the shell to catch variable names that are not valid
shell identifiers.  This will cause /bin/sh to exit with an error
(which gets mailed to the at user) and it will not run the script.

Obtained from:	OpenBSD (r1.63 millert)
MFC after:	3 days
2014-09-29 21:45:57 +00:00
rstone
03edcfbff6 Fix integer truncation in affecting systat -ifstat
The "systat -ifstat" command was using a u_int to store byte counters.
With a 10Gbps or faster interface, this overflows within the default
5 second refresh period.  Switch to using a uint64_t across the board,
which matches the size used for all counters as of r263102.

PR:		182448
MFC after:	1 week
Sponsored by:	Sandvine Inc
2014-09-29 17:38:50 +00:00
marcel
2bf89afb2b Update the usage message and the man page to account for the new long
options. Bump the version number to 20140927.
While here, use explicit fputc() calls to skip a line in the output.
This to avoid having to hunt for extra '\n' characters in the printf
format strings.

MFC after:	1 week
Relnotes:	yes
2014-09-27 17:58:03 +00:00
marcel
bf39d18edb Add 3 long options for getting information about mkimg itself:
--version	print the version of mkimg and also whether it's
		64- or 32-bit.
--formats	list the supported output formats separated by space.
--schemes	list the supported partitioning schemes separated by
		space.

Inspired by a patch from: gjb@

MFC after:	1 week
Relnotes:	yes
2014-09-27 04:53:51 +00:00
allanjude
d7031f8795 Replace the macros used in the previous man(1) commit with literal text,
because the macros seem to be specific to groff, and do not render in
mandoc

Approved by:	bcr (mentor)
Sponsored by:	ScaleEngine Inc.
2014-09-26 11:48:29 +00:00
allanjude
a3b0baba1c Update man(1) to list the different sections of the manual
Add each of the intro sections to the 'Also See' list

PR:		193460
Differential Revision:	D834
Approved by:	bcr (mentor)
Sponsored by:	ScaleEngine Inc.
2014-09-25 21:34:57 +00:00
pfg
be5797835f bsdgrep: Work-around for segmentation fault.
Fix by David Carlier.

Obtained from:	HardenedBSD
PR:		167921
MFC after:	1 month
2014-09-25 19:22:26 +00:00
marcel
be08aa2273 Add baseline files for QCOW2. 2014-09-24 22:58:10 +00:00
marcel
52d98bc2c0 Finish QCOW version 2 and stop making it conditional.
We have a different ordering for the RC block(s) and L2 tables.
This is expected to be a non-issue, because everything is found
through file offsets in the corresponding RC table and L1 table.
Files that grow organically have RC blocks and L2 tables scattered
all over the place anyway.

The reason for the difference is that mkimg needs to be able to
write to a pipe. We can't seek forward and backward to fill in
the bits in non-sequential order.
2014-09-24 22:45:03 +00:00
jch
9ef1acd601 Add myself (jch) to calendar.freebsd and committers-src.dot
Approved by:	jhb (mentor)
2014-09-24 22:31:40 +00:00
marcel
37c5cb3186 Update the baseline for QCOW version 1. A bug was found that rendered
the baseline invalid.
2014-09-24 15:15:14 +00:00
marcel
5d49d67fa7 Fix the creation of the L2 cluster table for version 1. The blkofs
variable was assigned the image offset in bytes and not in blocks
(i.e. sectors). This had image_data() return FALSE, which meant that
we didn't assign a cluster when we needed and also meant that we
didn't write parts of the L2 table when we should have. The result
being that the actual data clusters were written at the wrong offset.

Improve support for QCOW version 2. We're having the right layout
and even know how many refcnt blocks we need. All we need to do is
populate the refcnt blocks for every cluster we write and allocate
a cluster when we need a new refcnt block. The allocation part is
tricky in that it'll interleave with the assignment of clusters to
L2 tables and data. Since version 2 is not quite done, keep it
compiled out for now.
2014-09-24 15:14:01 +00:00
marcel
35a979c5fb Clean the generated baseline files by adding them to CLEANFILES. 2014-09-24 04:17:29 +00:00
mjg
6ca5e81a16 install: re-check failed mkdir for EEXIST
Since the code stats and mkdirs in 2 separate steps, it is possible that
the directory will be created in the meantime by something else (e.g.
concurrent install).[1]

While here alter the code to properly report stat failure, previously it
would always claim it was mkdir which failed.

Noted by: royger [1]
MFC after:	1 week
2014-09-23 11:41:09 +00:00
marcel
24ebed64d0 Rename the tests to something more meaningful. I spent way too long
trying to get the test name right, failed, gave up and used a sequence
number instead. When I realized it wasn't because of the number of
underscores in the name that I really started to think. I didn't have
braces around the variable names ...

Thus: test_1 is now called apm_1x1_512_qcow, which gives you all you
need to run mkimg by hand.

Dumb-ass: marcel
2014-09-22 16:52:09 +00:00
marcel
deb4e9e0ed Don't update the baseline file when the result of the test is identical
to the baseline. Since we don't run gzip with the -n option, the output
of gzip varies for identical result files if and when they are created
at different time. Ouch...

Rather than add -n and commit a 600K+ diff for the changes to all the .uu
files, it's less of a churn to uudecode and gunzip the baseline file and
compare that to the new result file to determine if the baseline file
needs to be updated.

This way, "atf-sh mkimg.sh rebase" can be run as many times as people like
and a subsequent "svn status" will not show unnecessary diffs.
2014-09-22 16:37:37 +00:00
marcel
9abf6bc2c9 Don't echo '# $FreeBSD$' as the first line into the .uu file. Keyword
substitution applies to this file, including the echo command. Avoid
the match (and substitution) by breaking the string up into 3 parts.
2014-09-22 15:27:23 +00:00
marcel
e21e58067d Update the unit tests to include the QCOW (version 1) format.
This is a good safety net for when V2 is added.
2014-09-22 15:08:58 +00:00
marcel
a301a31d64 Add support for QCOW version 1. Version 2 is partially implemented.
And because of that, it's entirely disabled for now. Both versions
are similar enough that a single header definition works for both
of them. The only "diverting" side-effect is that the union of the
two is larger than the official V1 header.

What this means for our V1 support is that we can't put the L1 table
adjacent to the V1 header (i.e. at offset 0x30 in the file), unless
we revert to hackery and klugery. Let's not. Instead, we align the L1
table at the cluster boundary. This is in line with the V2 layout and
perfectly ok for V1 anyway (ok -- as far as I've seen so far).
Due to the alignment, our V1 image seems to be 1 cluster larger than
the V1 image created by qemu-img (on average).

Compression of the clusters is not supported at this time.

MFC after:	2 months
2014-09-22 15:05:28 +00:00