freebsd-dev/contrib/csup/csup.1

.\" Copyright 1996-2003 John D. Polstra.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions
.\" are met:
.\" 1. Redistributions of source code must retain the above copyright
.\"    notice, this list of conditions and the following disclaimer.
.\" 2. Redistributions in binary form must reproduce the above copyright
.\"    notice, this list of conditions and the following disclaimer in the
.\"    documentation and/or other materials provided with the distribution.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
.\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
.\" OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
.\" IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
.\" INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
.\" NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
.\" THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
.\"
.\" $Id: cvsup.1,v 1.70 2003/03/04 18:23:46 jdp Exp $
.\" $FreeBSD$
.\"
.Dd February 1, 2006
.Os FreeBSD
.Dt CSUP 1
.Sh NAME
.Nm csup
.Nd network distribution package for CVS repositories
.Sh SYNOPSIS
.Nm
.Op Fl 146ksvzZ
.Op Fl A Ar addr
.Op Fl b Ar base
.Op Fl c Ar collDir
.Op Fl d Ar delLimit
.Op Fl h Ar host
.Op Fl i Ar pattern
.Op Fl l Ar lockfile
.Op Fl L Ar verbosity
.Op Fl p Ar port
.Op Fl r Ar maxRetries
.Ar supfile
.Sh DESCRIPTION
.Nm
is a software package for updating collections of files across a network.
It is a rewrite of the
.Nm CVSup
software in C.
This manual page describes the usage of the
.Nm
client program.
.Pp
Unlike more traditional network distribution packages, such as
.Nm rdist
and
.Nm sup ,
.Nm
has specific optimizations for distributing CVS repositories.
.Nm
takes advantage of the properties of CVS repositories and the files they
contain (in particular, RCS files), enabling it to perform updates much
faster than traditional systems.
.Pp
.Nm
is a general-purpose network file updating package.
It is extremely fast,
even for collections of files which have nothing to do with CVS or
RCS.
.Sh OPTIONS
The client program
.Nm
requires at least a single argument,
.Ar supfile .
It names a file describing one or more collections of files to be
transferred and/or updated from the server.
The
.Ar supfile
has a format similar to the corresponding file used by
.Nm sup .
In most cases,
.Nm
can use existing
.Nm sup Ar supfiles .
.Pp
The following options are supported by
.Nm :
.Bl -tag -width Fl
.It Fl 1
Disables automatic retries when transient failures occur.
Without this option, a transient failure such as a dropped network
connection causes
.Nm
to retry repeatedly, using randomized exponential backoff to space the
retries.
This option is equivalent to
.Fl r Cm 0 .
.It Fl 4
Forces
.Nm
to use IPv4 addresses only.
.It Fl 6
Forces
.Nm
to use IPv6 addresses only.
.It Fl A Ar addr
Specifies a local address to bind to when connecting to the server.
The local address might be a hostname or a numeric host address string
consisting of a dotted decimal IPv4 address or an IPv6 address.
This may be useful on hosts which have multiple IP addresses.
.It Fl b Ar base
Specifies the base directory under which
.Nm
will maintain its bookkeeping files, overriding any
.Cm base
specifications in the
.Ar supfile .
.It Fl c Ar collDir
Specifies the subdirectory of
.Ar base
where the information about the collections is maintained.
The default is
.Pa sup .
.It Fl d Ar delLimit
Specifies the maximum number of files that may be deleted in a
single update run.
Any attempt to exceed the limit results in a fatal error.
This can provide some protection against temporary configuration
mistakes on the server.
The default limit is infinity.
.It Fl h Ar host
Specifies the server host to contact, overriding any
.Cm host
specifications in the
.Ar supfile .
.It Fl i Ar pattern
Causes
.Nm
to include only files and directories matching
.Ar pattern
in the update.  If a directory matches the pattern, then the entire
subtree rooted at the directory is included.  If this option is
specified multiple times, the patterns are combined using the
.Ql or
operation.  If no
.Fl i
options are given, the default is to update all files in each
collection.
.Pp
The
.Ar pattern
is a standard file name pattern.
It is interpreted relative to the collection's prefix directory.
Slash characters are matched only by explicit slashes in the pattern.
Leading periods in file name are not treated specially.
.It Fl k
Causes
.Nm
to keep the temporary copies of any incorrectly edited files, in the
event of checksum mismatches.
This option is for debugging, to help determine why the files were
edited incorrectly.
Regardless of whether this option is specified, the permanent versions
of faulty files are replaced with correct versions obtained by
transferring the files in their entirety.
Such transfers are called fixups.
.It Fl l Ar lockfile
Creates and locks the
.Ar lockfile
while the update is in progress.
If
.Ar lockfile
is already locked,
.Nm
fails without performing automatic retries.
This option is useful when
.Nm
is executed periodically from
.Nm cron .
It prevents a job from interfering with an earlier job that is perhaps
taking extra long because of network problems.
.Pp
The process-ID is written to the lock file in text form when the lock
is successfully acquired.
Upon termination of the update, the lock file is removed.
.It Fl L Ar verbosity
Sets the verbosity level for output.
A level of 0 causes
.Nm
to be completely silent unless errors occur.
A level of 1 (the default) causes each updated file to be listed.
A level of 2 provides more detailed information about the updates
performed on each file.
All messages are directed to the standard output.
.It Fl p Ar port
Sets the TCP port to which
.Nm
attempts to connect on the server host.
The default port is 5999.
.It Fl r Ar maxRetries
Limits the number of automatic retries that will be attempted when
transient errors such as lost network connections are encountered.
By default,
.Nm
will retry indefinitely until an update is successfully completed.
The retries are spaced using randomized exponential backoff.
Note that
.Fl r Cm 0
is equivalent to the
.Fl 1
option.
.It Fl s
Suppresses the check of each client file's status against what is
recorded in the list file.  Instead, the list file is assumed to be
accurate.  This option greatly reduces the amount of disk activity and
results in faster updates with less load on the client host.  However
it should only be used if client's files are never modified locally in
any way.  Mirror sites may find this option beneficial to reduce the
disk load on their systems.  For safety, even mirror sites should run
.Nm
occasionally (perhaps once a day) without the
.Fl s
option.
.Pp
Without the
.Fl s
option,
.Nm
performs a
.Xr stat 2
call on each file and verifies that its attributes match those
recorded in the list file.  This ensures that any file changes made
outside of
.Nm
are detected and corrected.
.Pp
If the
.Fl s
option is used when one or more files have been modified locally, the
results are undefined.  Local file damage may remain uncorrected,
updates may be missed, or
.Nm
may abort prematurely.
.It Fl v
Prints the version number and exits, without contacting the server.
.It Fl z
Enables compression for all collections, as if the
.Cm compress
keyword were added to every collection in the
.Ar supfile .
.It Fl Z
Disables compression for all collections, as if the
.Cm compress
keyword were removed from every collection in the
.Ar supfile .
.El
.Pp
The
.Ar supfile
is a text file which specifies the file collections to be updated.
Comments begin with
.Ql #
and extend to the end of the line.  Lines that are empty except for
comments and white space are ignored.  Each remaining line begins
with the name of a server-defined collection of files.  Following the
collection name on the line are zero or more keywords or keyword=value
pairs.
.Pp
Default settings may be specified in lines whose collection name is
.Cm *default .
Such defaults will apply to subsequent lines in the
.Ar supfile .
Multiple
.Cm *default
lines may be present.
New values augment or override any defaults specified earlier in the
.Ar supfile .
Values specified explicitly for a collection override any default
values.
.Pp
The most commonly used keywords are:
.Bl -tag -width Fl
.It Cm release= Ns Ar releaseName
This specifies the release of the files within a collection.
Like collection names, release names are defined by the server
configuration files.  Usually there is only one release in each
collection, but there may be any number.  Collections which come from
a CVS repository often use
.Cm release=cvs
by convention.  Non-CVS collections conventionally use
.Cm release=current .
.It Cm base= Ns Ar base
This specifies a directory under which
.Nm
will maintain its bookkeeping files, describing the state of each
collection on the client machine.
The
.Ar base
directory must already exist;
.Nm
will not create it.
The default
.Ar base
directory is
.Pa /usr/local/etc/csup .
.It Cm prefix= Ns Ar prefix
This is the directory under which updated files will be placed.
By default, it is the same as
.Ar base .
If it is not an absolute pathname, it is interpreted relative to
.Ar base .
The
.Ar prefix
directory must already exist;
.Nm
will not create it.
.Pp
As a special case, if
.Ar prefix
is a symbolic link pointing to a nonexistent file named
.Ql SKIP ,
then
.Nm
will skip the collection.
The parameters associated with the collection are still checked for
validity, but none of its files will be updated.
This feature allows a site to use a standard
.Ar supfile
on several machines, yet control which collections get updated on a
per-machine basis.
.It Cm host= Ns Ar hostname
This specifies the server machine from which all files will be taken.
.Nm
requires that all collections in a single run come from the same host.
If you wish to update collections from several different hosts, you must
run
.Nm
several times.
.It Cm delete
The presence of this keyword gives
.Nm
permission to delete files.
If it is missing, no files will be deleted.
.Pp
The presence of the
.Cm delete
keyword puts
.Nm
into so-called
.Em exact
mode.  In exact mode,
.Nm
does its best to make the client's files correspond to those on the server.
This includes deleting individual deltas and symbolic tags from RCS
files, as well as deleting entire files.
In exact mode,
.Nm
verifies every edited file with a checksum, to ensure that the edits
have produced a file identical to the master copy on the server.
If the checksum test fails for a file, then
.Nm
falls back upon transferring the entire file.
.Pp
In general,
.Nm
deletes only files which are known to the server.
Extra files present in the client's tree are left alone, even in exact
mode.
More precisely,
.Nm
is willing to delete two classes of files:
.Bl -bullet -compact
.It
Files that were previously created or updated by
.Nm
itself.
.It
Checked-out versions of files which are marked as dead on the server.
.El
.It Cm use-rel-suffix
Causes
.Nm
to append a suffix constructed from the release and tag to the name of
each list file that it maintains.
See
.Sx THE LIST FILE
for details.
.It Cm compress
This enables compression of all data sent across the network.
Compression is quite effective, normally eliminating 65% to 75% of the
bytes that would otherwise need to be transferred.
However, it is costly in terms of CPU time on both the client and the
server.
On local area networks, compression is generally counter-productive; it
actually slows down file updates.
On links with speeds of 56K bits/second or less, compression is almost
always beneficial.
For network links with speeds between these two extremes, let
experimentation be your guide.
.Pp
The
.Fl z
command line option enables the
.Cm compress
keyword for all collections, regardless of what is specified in the supfile.
Likewise, the
.Fl Z
command line option disables the
.Cm compress
option for all collections.
.Nm
uses a looser checksum for RCS files, which ignores harmless
differences in white space.  Different versions of CVS and RCS produce
a variety of differences in white space for the same RCS files.  Thus
the strict checksum can report spurious mismatches for files which are
logically identical.  This can lead to numerous unneeded
.Dq fixups ,
and thus to slow updates.
.It Cm umask= Ns Ar n
Causes
.Nm
to use a umask value of
.Ar n
(an octal number) when updating the files in the collection.
This option is ignored if
.Cm preserve
is specified.
.El
.Pp
Some additional, more specialized keywords are described below.
Unrecognized keywords are silently ignored for backward compatibility
with
.Nm sup .
.Sh CVS MODE
.Nm CVSup
supports two primary modes of operation.
They are called
.Em CVS
mode and
.Em checkout
mode.
.Nm
only supports the checkout mode for now.
.Pp
In CVS mode, the client receives copies of the actual RCS files making
up the master CVS repository.  CVS mode is the default mode of operation.
It is appropriate when the user wishes to maintain a full copy of the
CVS repository on the client machine.
.Pp
CVS mode is also appropriate for file collections which are not
based upon a CVS repository.  The files are simply transferred
verbatim, without interpretation.
.Sh CHECKOUT MODE
In checkout mode, the client receives specific revisions of files,
checked out directly from the server's CVS repository.
Checkout mode allows the client to receive any version from the
repository, without requiring any extra disk space on the server for
storing multiple versions in checked-out form.
Checkout mode provides much flexibility beyond that basic functionality,
however.
The client can specify any CVS symbolic tag, or any date, or both, and
.Nm
will provide the corresponding checked-out versions of the files in the
repository.
.Pp
Checkout mode is selected on a per-collection basis, by the presence of
one or both of the following keywords in the
.Ar supfile :
.Bl -tag -width Fl
.It Cm tag= Ns Ar tagname
This specifies a symbolic tag that should be used to select the
revisions that are checked out from the CVS repository.
The tag may refer to either a branch or a specific revision.
It must be symbolic; numeric revision numbers are not supported.
.Pp
For the FreeBSD source repository, the most commonly used tags will be:
.Bl -tag -width RELENG_6
.It Li RELENG_6
The
.Ql stable
branch.
.It Li \&.
The main branch (the
.Ql current
release).
This is the default, if only the
.Cm date
keyword is given.
.El
.Sm off
.It Xo Cm date=
.Op Ar cc
.Ar yy.mm.dd.hh.mm.ss
.Xc
.Sm on
This specifies a date that should be used to select the revisions that
are checked out from the CVS repository.
The client will receive the revisions that were in effect at the
specified date and time.
.Pp
At present, the date format is inflexible.  All 17 or 19 characters must
be specified, exactly as shown.
For the years 2000 and beyond, specify the century
.Ar cc .
For earlier years, specify only the last two digits
.Ar yy .
Dates and times are considered to
be GMT.
The default date is
.Ql \&. ,
which means
.Dq as late as possible .
.El
.Pp
To enable checkout mode, you must specify at least one of these keywords.
If both are missing,
.Nm
defaults to CVS mode.
.Pp
If both a branch tag and a date are specified, then the revisions on the
given branch, as of the given date, will be checked out.  It is
permitted, but not particularly useful, to specify a date with a
specific release tag.
.Pp
In checkout mode, the tag and/or date may be changed between updates.
For example, suppose that a collection has been transferred using the
specification
.Ql tag=. .
The user could later change the specification to
.Ql tag=RELENG_3 .
This would cause
.Nm
to edit the checked-out files in such a way as to transform them from the
.Ql current
versions to the
.Ql stable
versions.
In general,
.Nm
is willing to transform any tag/date combination into any other tag/date
combination, by applying the intervening RCS deltas to the existing files.
.Pp
When transforming a collection of checked-out files from one tag to
another, it is important to specify the
.Cm list
keyword in the
.Ar supfile ,
to ensure that the same list file is used both before and after the
transformation.
The list file is described in
.Sx THE LIST FILE ,
below.
.Sh THE LIST FILE
For efficiency,
.Nm
maintains a bookkeeping file for each collection, called the list file.
The list file contains information about which files and revisions the client
currently possesses.
It also contains information used for verifying that the list file
is consistent with the actual files in the client's tree.
.Pp
The list file is not strictly necessary.  If it is deleted, or becomes
inconsistent with the actual client files,
.Nm
falls back upon a less efficient method of identifying the client's
files and performing its updates.
Depending on
.Nm csup Ns No 's
mode of operation, the fallback method employs time stamps, checksums, or
analysis of RCS files.
.Pp
Because the list file is not essential,
.Nm
is able to
.Dq adopt
an existing file tree acquired by FTP or from a CD-ROM.
.Nm
identifies the client's versions of the files, updates them as
necessary, and creates a list file for future use.
Adopting a foreign file tree is not as fast as performing a normal
update.
It also produces a heavier load on the server.
.Pp
The list file is stored in a collection-specific directory; see
.Sx FILES
for details.
Its name always begins with
.Ql checkouts .
If the keyword
.Cm use-rel-suffix
is specified in the
.Ar supfile ,
a suffix, formed from the release and tag, is appended to the name.
The default suffix can be overridden by specifying an explicit suffix in
the
.Ar supfile :
.Bl -tag -width Fl
.It Cm list= Ns Ar suffix
This specifies a suffix for the name of the list file.  A leading dot is
provided automatically.
For example,
.Ql list=stable
would produce a list file named
.Pa checkouts.stable ,
regardless of the release, tag, or
.Cm use-rel-suffix
keyword.
.El
.Sh REFUSE FILES
The user can specify sets of files that he does not wish to receive.
The files are specified as file name patterns in so-called
.Em refuse
files.
The patterns are separated by whitespace, and multiple patterns are
permitted on each line.
Files and directories matching the patterns are neither updated nor
deleted; they are simply ignored.
.Pp
There is currently no provision for comments in refuse files.
.Pp
The patterns are similar to those of
.Xr sh 1 ,
except that there is no special treatment for slashes or for
filenames that begin with a period.
For example, the pattern
.Ql *.c
will match any file name ending with
.Ql \&.c
including those in subdirectories, such as
.Ql foo/bar/lam.c .
All patterns are interpreted relative to the collection's prefix
directory.
.Pp
If the files are coming from a CVS repository, as is usually
the case, then they will be RCS files. These have a
.Ql \&,v
suffix which must be taken into account in the patterns. For
example, the FreeBSD documentation files are in a sub-directory of
.Ar base
called
.Ql doc .
If
.Ql Makefile
from that directory is not required then the line
.Pp 
.Bl -item -compact -offset indent
.It 
.Pa doc/Makefile
.El
.Pp
will not work because the file on the server is called
.Ql Makefile,v.
A better solution would be
.Pp
.Bl -item -compact -offset indent
.It
.Pa doc/Makefile*
.El 
.Pp 
which will match whether
.Ql Makefile
is an RCS file or not.
.Pp
As another example, to receive the FreeBSD documentation files without
the Japanese, Russian, and Chinese translations, create a refuse file
containing the following lines:
.Pp
.Bl -item -compact -offset indent
.It
.Pa doc/ja*
.It
.Pa doc/ru*
.It
.Pa doc/zh*
.El 
.Pp
As many as three refuse files are examined for each
.Ar supfile
line.
There can be a global refuse file named
.Sm off
.Ar base / Ar collDir Pa /refuse
.Sm on
which applies to all collections and releases.
There can be a per-collection refuse file named
.Sm off
.Xo Ar base / Ar collDir / Ar collection
.Pa /refuse
.Xc
.Sm on
which applies to a specific collection.
Finally, there can be a per-release and tag refuse file which applies only
to a given release/tag combination within a collection.
The name of the latter is formed by suffixing the name of the
per-collection refuse file in the same manner as described above for the
list file.
None of the refuse files are required to exist.
.Pp
.Nm
has a built-in default value of
.Ar /usr/local/etc/cvsup
for
.Ar base
and
.Ar sup
for 
.Ar collDir
but it is possible to override both of these. The value of
.Ar base
can be changed using the
.Fl b
option or a
.Ar base=pathname
entry in the
.Ar supfile .
(If both are used the 
.Fl b
option will override the
.Ar supfile
entry.)  The value of 
.Ar collDir
can only be changed with the
.Fl c
option; there is no
.Ar supfile
command to change it.
.Pp
As an example, suppose that the
.Ar base
and
.Ar collDir
both have their default values, and that the collection and release are
.Ql src-all
and
.Ql cvs ,
respectively.
Assume further that checkout mode is being used with
.Ql tag=RELENG_3 .
The three possible refuse files would then be named:
.Pp
.Bl -item -compact -offset indent
.It
.Pa /usr/local/etc/cvsup/sup/refuse
.It
.Pa /usr/local/etc/cvsup/sup/src-all/refuse
.It
.Pa /usr/local/etc/cvsup/sup/src-all/refuse.cvs:RELENG_3
.El
.Pp
If the
.Ar supfile
includes the command
.Ar base=/foo
the refuse files would be:
.Pp
.Bl -item -compact -offset indent
.It
.Pa /foo/sup/refuse
.It
.Pa /foo/sup/src-all/refuse
.It
.Pa /foo/sup/src-all/refuse.cvs:RELENG_3
.El
.Pp
If
.Fl b
.Ar /bar
is used (even with
.Ar base=/foo
in the
.Ar supfile ) :
.Pp
.Bl -item -compact -offset indent
.It
.Pa /bar/sup/refuse                
.It
.Pa /bar/sup/src-all/refuse                
.It
.Pa /bar/sup/src-all/refuse.cvs:RELENG_3                
.El
.Pp
and with
.Fl c
.Ar stool
as well:
.Pp
.Bl -item -compact -offset indent
.It
.Pa /bar/stool/refuse
.It 
.Pa /bar/stool/src-all/refuse
.It
.Pa /bar/stool/src-all/refuse.cvs:RELENG_3
.El
.Sh csup AND FIREWALLS
In its default mode,
.Nm
will work through any firewall which permits outbound connections to
port 5999 of the server host.
.Sh USING csup WITH SOCKS
.Nm
can be used through a SOCKS proxy server with the standard
.Nm runsocks
command.
Your
.Nm
executable needs to be dynamically-linked with the system
libraries for
.Nm runsocks
to work properly.
.Sh USING ssh PORT FORWARDING
As an alternative to SOCKS, a user behind a firewall can penetrate it
with the TCP port forwarding provided by the Secure Shell package
.Nm ssh .
The user must have a login account on the
.Nm CVSup
server host in order to do this.
The procedure is as follows:
.Bl -enum
.It
Establish a connection to the server host with
.Nm ssh ,
like this:
.Bd -literal
ssh -f -x -L 5999:localhost:5999 serverhost sleep 60
.Ed
.Pp
Replace
.Ar serverhost
with the hostname of the CVSup server, but type
.Ql localhost
literally.
This sets up the required port forwarding.
You must start
.Nm
before the 60-second
.Nm sleep
finishes.
Once the update has begun,
.Nm ssh
will keep the forwarded channels open as long as they are needed.
.It
Run
.Nm
on the local host, including the arguments
.Ql -h localhost
on the command line.
.El
.Sh FILES
.Bl -tag -width base/sup/collection/checkouts*xx -compact
.It Pa /usr/local/etc/cvsup
Default
.Ar base
directory.
.It Pa sup
Default
.Ar collDir
subdirectory.
.Sm off
.It Xo Ar base / Ar collDir / Ar collection
.Pa /checkouts*
.Xc
.Sm on
List files.
.El
.Sh SEE ALSO
.Xr cvs 1 ,
.Xr rcsintro 1 ,
.Xr ssh 1 .
.Pp
.Bd -literal
http://mu.org/~mux/csup.html
.Ed
.Sh AUTHORS
.An -nosplit
.An Maxime Henrion Aq mux@FreeBSD.org
is the author of
.Nm ,
the rewrite of
.Nm CVSup
in C.
.An John Polstra Aq jdp@polstra.com
is the author of
.Nm CVSup .
.Sh LEGALITIES
CVSup is a registered trademark of John D. Polstra.
.Pp
.Nm
is released under a 2-clauses BSD license.
.Sh BUGS
An RCS file is not recognized as such unless its name ends with
.Ql \&,v .
.Pp
Any directory named
.Ql Attic
is assumed to be a CVS Attic, and is treated specially.