the real build file.
This lessens the need to define DPADD_<lib> and LDADD_<lib> to just very
special cases.
Sponsored by: EMC / Isilon Storage Division
camdd(8) utility.
CCBs may be queued to the driver via the new CAMIOQUEUE ioctl, and
completed CCBs may be retrieved via the CAMIOGET ioctl. User
processes can use poll(2) or kevent(2) to get notification when
I/O has completed.
While the existing CAMIOCOMMAND blocking ioctl interface only
supports user virtual data pointers in a CCB (generally only
one per CCB), the new CAMIOQUEUE ioctl supports user virtual and
physical address pointers, as well as user virtual and physical
scatter/gather lists. This allows user applications to have more
flexibility in their data handling operations.
Kernel memory for data transferred via the queued interface is
allocated from the zone allocator in MAXPHYS sized chunks, and user
data is copied in and out. This is likely faster than the
vmapbuf()/vunmapbuf() method used by the CAMIOCOMMAND ioctl in
configurations with many processors (there are more TLB shootdowns
caused by the mapping/unmapping operation) but may not be as fast
as running with unmapped I/O.
The new memory handling model for user requests also allows
applications to send CCBs with request sizes that are larger than
MAXPHYS. The pass(4) driver now limits queued requests to the I/O
size listed by the SIM driver in the maxio field in the Path
Inquiry (XPT_PATH_INQ) CCB.
There are some things things would be good to add:
1. Come up with a way to do unmapped I/O on multiple buffers.
Currently the unmapped I/O interface operates on a struct bio,
which includes only one address and length. It would be nice
to be able to send an unmapped scatter/gather list down to
busdma. This would allow eliminating the copy we currently do
for data.
2. Add an ioctl to list currently outstanding CCBs in the various
queues.
3. Add an ioctl to cancel a request, or use the XPT_ABORT CCB to do
that.
4. Test physical address support. Virtual pointers and scatter
gather lists have been tested, but I have not yet tested
physical addresses or scatter/gather lists.
5. Investigate multiple queue support. At the moment there is one
queue of commands per pass(4) device. If multiple processes
open the device, they will submit I/O into the same queue and
get events for the same completions. This is probably the right
model for most applications, but it is something that could be
changed later on.
Also, add a new utility, camdd(8) that uses the asynchronous pass(4)
driver interface.
This utility is intended to be a basic data transfer/copy utility,
a simple benchmark utility, and an example of how to use the
asynchronous pass(4) interface.
It can copy data to and from pass(4) devices using any target queue
depth, starting offset and blocksize for the input and ouptut devices.
It currently only supports SCSI devices, but could be easily extended
to support ATA devices.
It can also copy data to and from regular files, block devices, tape
devices, pipes, stdin, and stdout. It does not support queueing
multiple commands to any of those targets, since it uses the standard
read(2)/write(2)/writev(2)/readv(2) system calls.
The I/O is done by two threads, one for the reader and one for the
writer. The reader thread sends completed read requests to the
writer thread in strictly sequential order, even if they complete
out of order. That could be modified later on for random I/O patterns
or slightly out of order I/O.
camdd(8) uses kqueue(2)/kevent(2) to get I/O completion events from
the pass(4) driver and also to send request notifications internally.
For pass(4) devcies, camdd(8) uses a single buffer (CAM_DATA_VADDR)
per CAM CCB on the reading side, and a scatter/gather list
(CAM_DATA_SG) on the writing side. In addition to testing both
interfaces, this makes any potential reblocking of I/O easier. No
data is copied between the reader and the writer, but rather the
reader's buffers are split into multiple I/O requests or combined
into a single I/O request depending on the input and output blocksize.
For the file I/O path, camdd(8) also uses a single buffer (read(2),
write(2), pread(2) or pwrite(2)) on reads, and a scatter/gather list
(readv(2), writev(2), preadv(2), pwritev(2)) on writes.
Things that would be nice to do for camdd(8) eventually:
1. Add support for I/O pattern generation. Patterns like all
zeros, all ones, LBA-based patterns, random patterns, etc. Right
Now you can always use /dev/zero, /dev/random, etc.
2. Add support for a "sink" mode, so we do only reads with no
writes. Right now, you can use /dev/null.
3. Add support for automatic queue depth probing, so that we can
figure out the right queue depth on the input and output side
for maximum throughput. At the moment it defaults to 6.
4. Add support for SATA device passthrough I/O.
5. Add support for random LBAs and/or lengths on the input and
output sides.
6. Track average per-I/O latency and busy time. The busy time
and latency could also feed in to the automatic queue depth
determination.
sys/cam/scsi/scsi_pass.h:
Define two new ioctls, CAMIOQUEUE and CAMIOGET, that queue
and fetch asynchronous CAM CCBs respectively.
Although these ioctls do not have a declared argument, they
both take a union ccb pointer. If we declare a size here,
the ioctl code in sys/kern/sys_generic.c will malloc and free
a buffer for either the CCB or the CCB pointer (depending on
how it is declared). Since we have to keep a copy of the
CCB (which is fairly large) anyway, having the ioctl malloc
and free a CCB for each call is wasteful.
sys/cam/scsi/scsi_pass.c:
Add asynchronous CCB support.
Add two new ioctls, CAMIOQUEUE and CAMIOGET.
CAMIOQUEUE adds a CCB to the incoming queue. The CCB is
executed immediately (and moved to the active queue) if it
is an immediate CCB, but otherwise it will be executed
in passstart() when a CCB is available from the transport layer.
When CCBs are completed (because they are immediate or
passdone() if they are queued), they are put on the done
queue.
If we get the final close on the device before all pending
I/O is complete, all active I/O is moved to the abandoned
queue and we increment the peripheral reference count so
that the peripheral driver instance doesn't go away before
all pending I/O is done.
The new passcreatezone() function is called on the first
call to the CAMIOQUEUE ioctl on a given device to allocate
the UMA zones for I/O requests and S/G list buffers. This
may be good to move off to a taskqueue at some point.
The new passmemsetup() function allocates memory and
scatter/gather lists to hold the user's data, and copies
in any data that needs to be written. For virtual pointers
(CAM_DATA_VADDR), the kernel buffer is malloced from the
new pass(4) driver malloc bucket. For virtual
scatter/gather lists (CAM_DATA_SG), buffers are allocated
from a new per-pass(9) UMA zone in MAXPHYS-sized chunks.
Physical pointers are passed in unchanged. We have support
for up to 16 scatter/gather segments (for the user and
kernel S/G lists) in the default struct pass_io_req, so
requests with longer S/G lists require an extra kernel malloc.
The new passcopysglist() function copies a user scatter/gather
list to a kernel scatter/gather list. The number of elements
in each list may be different, but (obviously) the amount of data
stored has to be identical.
The new passmemdone() function copies data out for the
CAM_DATA_VADDR and CAM_DATA_SG cases.
The new passiocleanup() function restores data pointers in
user CCBs and frees memory.
Add new functions to support kqueue(2)/kevent(2):
passreadfilt() tells kevent whether or not the done
queue is empty.
passkqfilter() adds a knote to our list.
passreadfiltdetach() removes a knote from our list.
Add a new function, passpoll(), for poll(2)/select(2)
to use.
Add devstat(9) support for the queued CCB path.
sys/cam/ata/ata_da.c:
Add support for the BIO_VLIST bio type.
sys/cam/cam_ccb.h:
Add a new enumeration for the xflags field in the CCB header.
(This doesn't change the CCB header, just adds an enumeration to
use.)
sys/cam/cam_xpt.c:
Add a new function, xpt_setup_ccb_flags(), that allows specifying
CCB flags.
sys/cam/cam_xpt.h:
Add a prototype for xpt_setup_ccb_flags().
sys/cam/scsi/scsi_da.c:
Add support for BIO_VLIST.
sys/dev/md/md.c:
Add BIO_VLIST support to md(4).
sys/geom/geom_disk.c:
Add BIO_VLIST support to the GEOM disk class. Re-factor the I/O size
limiting code in g_disk_start() a bit.
sys/kern/subr_bus_dma.c:
Change _bus_dmamap_load_vlist() to take a starting offset and
length.
Add a new function, _bus_dmamap_load_pages(), that will load a list
of physical pages starting at an offset.
Update _bus_dmamap_load_bio() to allow loading BIO_VLIST bios.
Allow unmapped I/O to start at an offset.
sys/kern/subr_uio.c:
Add two new functions, physcopyin_vlist() and physcopyout_vlist().
sys/pc98/include/bus.h:
Guard kernel-only parts of the pc98 machine/bus.h header with
#ifdef _KERNEL.
This allows userland programs to include <machine/bus.h> to get the
definition of bus_addr_t and bus_size_t.
sys/sys/bio.h:
Add a new bio flag, BIO_VLIST.
sys/sys/uio.h:
Add prototypes for physcopyin_vlist() and physcopyout_vlist().
share/man/man4/pass.4:
Document the CAMIOQUEUE and CAMIOGET ioctls.
usr.sbin/Makefile:
Add camdd.
usr.sbin/camdd/Makefile:
Add a makefile for camdd(8).
usr.sbin/camdd/camdd.8:
Man page for camdd(8).
usr.sbin/camdd/camdd.c:
The new camdd(8) utility.
Sponsored by: Spectra Logic
MFC after: 1 week
relax it a bit so gecos can be used to store multibytes data.
This was unseen before FreeBSD 10.2 as this validation function was motly unused
since FreeBSD 10.2 the usage of this function has been generalized to improve
validation.
Reported by: des
MFC after: 1 week
This is not properly respecting WITHOUT or ARCH dependencies in target/.
Doing so requires a massive effort to rework targets/ to do so. A
better approach will be to either include the SUBDIR Makefiles directly
and map to DIRDEPS or just dynamically lookup the SUBDIR. These lose
the benefit of having a userland/lib, userland/libexec, etc, though and
results in a massive package. The current implementation of targets/ is
very unmaintainable.
Currently rescue/rescue and sys/modules are still not connected.
Sponsored by: EMC / Isilon Storage Division
Unfortunately filemon/meta mode tracks all indirect dependencies here
since ld(1) is reading libelf when linking in libkvm. Churn would be
reduced if this was able to be limited to direct dependencies.
Sponsored by: EMC / Isilon Storage Division
When this option is set, the NFS server uses the list of groups
acquired via getgrouplist(3) for the uid instead of the list of
groups in the RPC request. This can be used to avoid the 16 gid
limit for the group list in the RPC request.
Suggested by: jpaetzel
Tested by: jpaetzel
MFC after: 2 weeks
When using syscons, vidfont extracts the font size from the filename
passes it to vidcontrol -f. In vt(4) mode the size argument is not
required, and some of the fonts in /usr/share/vt/fonts do not have the
size in the filename, which caused vidfont to fail. Thus, just omit the
size argument in vt(4) mode.
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Tracking these leads to situations where meta mode will consider the
file to be out of date if /bin/sh or /bin/ln are newer than the source
file. There's no reason for meta mode to do this as make is already
handling the rebuild dependency fine.
Sponsored by: EMC / Isilon Storage Division
This both avoids some dependencies on xinstall.host and allows
bootstrapping on older releases to work due to lack of at least 'install -l'
support.
Sponsored by: EMC / Isilon Storage Division
It included libutil.h for setproctitle(3), which was moved from libutil to libc
in r65353 in 2000.
Reviewed by: gshapiro [sendmail change]
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D4261
bridges. Currently this includes information about what resources a
bridge decodes on the upstream side for use by downstream devices including
bus numbers, I/O port resources, and memory resources. Windows and bus
ranges are enumerated for both PCI-PCI bridges and PCI-CardBus bridges.
To simplify the implementation, all enumeration is done by reading the
appropriate config space registers directly rather than querying the
bridge driver in the kernel via new ioctls. This does result in a few
limitations.
First, an unimplemented window in a PCI-PCI bridge cannot be accurately
detected as accurate detection requires writing to the window base
register. That is not safe for pciconf(8). Instead, this assumes that
any window where both the base and limit read as all zeroes is
unimplemented.
Second, the PCI-PCI bridge driver in a tree has a few quirks for
PCI-PCI bridges that use subtractive decoding but do not indicate that
via the progif config register. The list of quirks is duplicated in
pciconf's source.
Reviewed by: imp
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D4171
is so unusual. Turn off mis-match warnings for building uathload
because the firmware .o file is produced in a way that we
can't get to match exactly. This fixes the build on mips,
so stop excluding it from the build.
pmcstat(8) contains an option to output sampling data in a gmon format
compatible with gprof(1). Currently, it uses the default histcounter,
which is an (unsigned short). With large sets of sampling data, it
is possible to overflow the maximum value provided by an (unsigned
short).
This change adds the -e argument to pmcstat. If -e and -g are both
specified, pmcstat will use a histcounter type of uint64_t.
Differential Revision: https://reviews.freebsd.org/D4151
Reviewed by: jhb, bjk
Approved by: gnn (mentor)
MFC after: 1 month
Sponsored by: Juniper Networks
ypldap -- Intended to be a drop-in replacement for ypserv, gluing in a
LDAP directory and thus providing support for users and groups stored in
LDAP for the get{pw,gr}ent family of functions.
When pmcstat exits after some samples were dropped, give the user an
idea of how many were lost. (Granted, these are global numbers, but
they may still help quantify the scope of the loss.)
Differential Revision: https://reviews.freebsd.org/D4123
Approved by: gnn (mentor)
MFC after: 1 month
Sponsored by: Juniper Networks
- Convert K&R to something a bit less ancient
- Remove an incorrect, duplicate prototype for bootparamprog_1(..)
MFC after: 1 week
PR: 71667
Submitted by: bcran
Sponsored by: EMC / Isilon Storage Division
While CTL has concept of port options, used at least for iSCSI ports now,
before this change it was impossible to set them manually. There still
no user-configurable port options now, but I am planning to change that.
- Don't use contractions (don't -> do not)
- Change "throw away" to "discard" when describing the -o keep-bad-images option
- Revert author e-mail split I brought over from NetBSD, effectively reverting
the change bapt made in r267668
MFC after: 3 days
Submitted by: bjk
Sponsored by: EMC / Isilon Storage Division
indicate service(8) script. While here, add "-l" option for listing the
set of configuration file(s) considered (in order; separated by space).
Also add "-L" for exploring all configuration files and "-E" to omit
files that don't exist from operations.
Differential Revision: https://reviews.freebsd.org/D3551
Reviewed by: allanjude
MFC after: 1 week
X-MFC-to: stable/10
Relnotes: yes
Verify the filesystem type using dumpfs. Add preliminary support
for NetBSD (needs to be validated)
MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division
- Parameterize the mtree keywords as $DEFAULT_MTREE_KEYWORDS
- Test with the extra mtree keywords, `mode,gid,uid`.
- Add a note about mtrees with time support not working with makefs right now
MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division
-- -o isolevel=1 currently fails because of path comparison issues,
so mark it as an expected failure.
-- -o isolevel=3 is not implemented, so expect it to fail as an out
of bounds value [*].
PR: 203645
MFC after: 1 week
X-MFC with: r290264
Sponsored by: EMC / Isilon Storage Division
dereferencing a NULL function pointer
Add some asserts to ensure that isolevel is always either 1 or 2.
PR: 203645
Reported by: Thomas Schmitt <scdbackup@gmx.net>
MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division
Note which options have been implemented and which options haven't
been implemented
Submitted as the following NetBSD PRs: bin/50390 and bin/50392
MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division
When -C was introduced in r114137 the plan was to have -C and -c being used for
"create" due to a typo in FreeBSD <= 4.8 a temporary compatibility hack has been
added to make -c being like -G aka GLOB and a warning was issued for the user to
be aware of the futur change for -c.
12 years later it is more than time to remove that hack and finish the what was
intent in r114137
Submitted by: Alexandre Perrin <alex@kaworu.ch>
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D4000
- Remove -a from the usage message example dealing with specific
interfaces. -a only makes sense when not specifying an interface,
such that it's to be run on all interfaces
- Fix the pidfile option (it's -p, not -P)
- Change `interfaces` to `interface` to match the manpage
MFC after: 3 days
PR: 173744
Sponsored by: EMC / Isilon Storage Division
- Rename -r to -R to avoid the clash with makefs -r in NetBSD
- Note that -R is an FFS-specific option because it's not implemented
in cd9660 today
- Rename the roundup variable to "roundup-size" in the manpage and help
text for consistency with other variables.
- Bump .Dd (missed in r289203)
PR: 203707
MFC after: 1 week
X-MFC with: r289203
Differential Revision: https://reviews.freebsd.org/D3959
Reviewed by: adrian (earlier patch), emaste
Sponsored by: EMC / Isilon Storage Division
because it is also used as an indicator of whether a name or an UID is
being used and we may have undefined results as 'name' may contain
uninitialized stack contents.
MFC after: 2 weeks
The code from mptutil living in the same source tree the day those functions
will be implemented in the mpsutil(8) it can be copy/paste easily
Sponsored by: Gandi.net
device nodes
In particular, use st_rdev (the device type), not st_dev (the device inode),
and fix the comparison to be correct with the st_rdev field
Bug 203648
MFC after: 2 weeks
Submitted by: Thomas Schmitt <scdbackup@gmx.net>
Coverity CID: 1008927
Sponsored by: EMC / Isilon Storage Division
General changes:
- Parameterize out the mount command.
- Use mtree to verify the contents of an image (check_image_contents) instead
of using diff (diff verifies content, but not file metadata).
- Move common logic out to functions (common_cleanup, mount_image,
check_image_contents)
- Add stub testcases for makefs -D (crashes with SIGBUS, similar to bug # 192839)
- Add a note about the ISO-9660 and rockridge specs
- Add testcases that exercise:
-- Creating disk images from an mtree and multiple directories.
-- -F flag use (not really an extensive testcase right now)
cd9660-specific test changes:
- Remove an XXX comment about symlinks; I forgot that non-rockridge images turn
symlinks into hardlinks.
- Add testcases that exercise:
-- -o allow-deep-trees
-- -o allow-max-name stub testcase (doesn't seem to be implemented in makefs)
-- -o preparer (existence in image; not conformance to spec)
-- -o publisher (existence in image; not conformance to spec)
-- -o rockridge (basic)
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division
working.
- Fix a pointer calculation for padding when multiple dnssl[0-9]
parameters are specified [*].
Reported by: http://bugs.dragonflybsd.org/issues/2847 [*]
if vm_activate_cpu(..) fails when called from fbsdrun_addcpu(..)
MFC after: 1 week
PR: 203884
Reviewed by: grehan
Submitted by: William Orr <will@worrbase.com>
these testcases don't need to be nested as much as bin/ls/ls_tests.sh
do when verifying ls -a, ls -A, etc. This allows the tests to make
all paths relative to the top of the temporary directory instead of
always tacking on $ATF_TMPDIR, thus complicating things unnecessarily
Create non-empty files in create_test_inputs as well now, similar to
create_test_inputs2 in bin/ls/ls_tests.sh
Compare the input files to the output file contents using diff where
possible:
- Skip over the fifo comparison for now because it always fails
- Skip over the symlink comparison on cd9660 because it always fails
today
Sponsored by: EMC / Isilon Storage Division
MFC after: 2 weeks
X-MFC with: r289739
buffer_head needs to be freed -- not buffer
Detected by jemalloc, i.e. running makefs failed the arena assert
because my copy of malloc on CURRENT is compiled with the default
!MALLOC_PRODUCTION asserts on
Pointyhat to: ngie
PR: 203647
X-MFC with: r289687
Sponsored by: EMC / Isilon Storage Division
leaking it after returning from the function
MFC after: 1 week
PR: 203647
Submitted by: Thomas Schmitt <scdbackup@gmx.net>
Coverity CID: 978431
Sponsored by: EMC / Isilon Storage Division
Previously, the code prefixes the chroot path to actual file paths to
simulate the effect. This, however, will not work for tzset(3) which
expects the current system have a working set of timezone data files,
and that is not always the case.
This changeset simplifies the handling of paths and use an actual
chroot(2) call to implement the effect.
PR: bin/197313
MFC after: 2 weeks
checked with dumpfs(8). This may lead to other problems, f.e. geom_label kernel
module sanity checks do not like zero fs_old_size value and skips such UFS1
file system while tasting (fs_old_size derives from sblock.fs_providersize).
PR: 203704
Submitted by: eugen@grosbein.net
Reviewed by: marcel
image up to.
From ticket:
While trying to run FreeBSD/mips on some device having very small flash media,
one is forced to compress file system with mkulzma(8) utility. It is desirable
to specify small UFS block/fragment sizes like 4096/512 bytes for makefs(8)
and big compression block size like 65535 bytes to mkulzma at the same time.
Then one obtains very good comression ratios (like 75% and more) but faces
the following problem.
geom_uncompress kernel module reports GEOM provider size rounded up to its
compression block size. Generally, this changes original media size and now
it fails to match the size of embedded UFS file system that leads to other
problems, f.e. geom_label kernel module does not like this and skips the
file system while tasting the GEOM and looking for UFS label.
This makes it impossible to refer to the file system using known UFS label
instead of something like /dev/map/rootfs.uncompress.
The following patch introduces new command line option "-r roundup" for makefs
that makes it round up the image to specified block size. Hence, geom_uncompress
does not change GEOM media size for images rounded that way and geom_label
accepts such GEOMs just fine.
With the patch applied, one can use following commands:
$ makefs -t ffs -r 65536 -o bsize=4096,fsize=512,label=flash optimization=space fs.img fs
$ mkulzma -s 65536 -o fs.img.ulzma fs.img
PR: bin/203707
Submitted by: <eugen@grosbein.net>
netbsd-tests.test.mk (r289151)
- Eliminate explicit OBJTOP/SRCTOP setting
- Convert all ad hoc NetBSD test integration over to netbsd-tests.test.mk
- Remove unnecessary TESTSDIR setting
- Use SRCTOP where possible for clarity
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Divison
This option tells freebsd-update to act as if it is running a specific
release instead of querying the kernel. In particular, this can be
useful when upgrading jails.
Requested by: EuroBSDCon devsummit jails session
Tested by: allanjude
MFC after: 1 week
- Don't advertize trusted-computing capability in the Identify page.
This prevents Windows from issuing a TRUSTED_RECEIVE_DMA command.
- Windows will send down SMART and SECURITY_FREEZE_LOCK
even though smart and security capabilities were not advertized.
Send back a silent abort.
Reviewed by: mav
to the qemu one, and uses the same i/o ports but with different
messaging. Requires the 'bootrom' option to be enabled.
This is used by UEFI (and potentially other BIOSs/firmware) to
request information from bhyve. Currently, only the number of
vCPUs is made available, with more to follow.
A very large thankyou to Ben Perrault who helped out testing
an earlier version of this, and bhyve/Windows in general.
Reviewed by: tychon
Discussed with: neel
Sponsored by: Nahanni Systems
After r288176 kernel debug files have the extension .debug. They also
moved to /usr/lib/debug/boot/kernel by default so in the normal case
kldxref does not encounter them. A src.conf(5) setting may be used to
continue installing them in /boot/kernel though, so have kldxref skip
.debug files in addition to .symbols files.
Reported by: fabient
Sponsored by: The FreeBSD Foundation
always passed a shell by the remote yppasswd. If an NIS client overrides the
shell provided by the ypserv, then yppasswd (pam_unix, actually, afaict)
will pass this new shell to the yppasswdd. If this shell has been set on the
client to a shell which is invalid on the server, a user will never be able
to change their password on the client.
PR: 67142
Submitted by: russell@rucus.ru.ac.za
Approved by: bapt (mentor)
Sponsored by: EuroBSDCon Sweden.
