Commit Graph

62 Commits

Author SHA1 Message Date
Alfred Perlstein
c13b66c053 Fix spelling of extract.
Pointed out by: gcooper
2011-11-13 06:39:49 +00:00
Alfred Perlstein
d1982439b2 Utilize shell's IFS instead of forking ~6 processes to
handle splitting input files on a '|'.  This greatly
reduces the time taken to process several databases
during the update process.

Additionally add some more debug logging.
2011-11-13 03:18:57 +00:00
Jilles Tjoelker
b77e3cbad1 portsnap: Detect error immediately if we can't fetch the snapshot metadata.
Also add some quotes around command substitution where useful and possible.

Reviewed by:	cperciva
MFC after:	1 week
2011-10-04 22:28:06 +00:00
Simon L. B. Nielsen
52e3f1458b Add --numeric-owner to tar extract operations in portsnap.
This is done to speed up extraction significantly (both for portsnap
extract and update) in the case of slow NSS modules (like nss_ldap) as
it avoids having to look up uid and gid for root / wheel.

The reason this is a bigger problem for portsnap than for many other
system operations, is that portsnap executes tar(1) once for each port
so the internal uid/gid caching in tar(1) only helps a bit, resulting
in many user lookup calls.

Discussed with:	cperciva
2010-12-19 23:09:42 +00:00
Ulrich Spörlein
0d9deed52c mdoc: drop redundant .Pp and .LP calls
They have no effect when coming in pairs, or before .Bl/.Bd
2010-10-08 12:40:16 +00:00
Joel Dahl
c2025a7660 Fix typos, spelling, formatting and mdoc mistakes found by Nobuyuki while
translating these manual pages.  Minor corrections by me.

Submitted by:	Nobuyuki Koganemaru <n-kogane@syd.odn.ne.jp>
2010-08-16 15:18:30 +00:00
Ed Schouten
71ccf09269 The last big commit: let usr.sbin/ use WARNS=6 by default. 2010-01-02 11:07:44 +00:00
Colin Percival
07759af5eb Add undocumented 'portsnap alfred' command, which runs (fetch|cron)
based on whether fd 0 is a terminal, and then runs (extract|update)
based on whether ${PORTSDIR} exists.

Requested by:	alfred
MFC after:	1 month
2009-12-30 06:28:47 +00:00
Colin Percival
04419f3b80 Set SO_NOSIGPIPE on sockets used by phttpget. Without this, if
(1) phttpget is attempting to download enough files that it can't send
all the requests at once, and
(2) the remote server forcibly closes the connection, resulting in RST
packets being sent,
phttpget will receive a SIGPIPE and terminate without downloading all
of the files.

This is probably responsible for a number of hard-to-reproduce errors
with portsnap and freebsd-update.

MFC after:	3 days
2009-04-03 21:13:18 +00:00
Daniel Gerzo
3bea701ec8 - remove superfluous word
PR:		docs/127401
Submitted by:	Mick Charles Beaver <mick@cs.wisc.edu>
MFC after:	1 week
2008-09-15 16:30:06 +00:00
Colin Percival
bdd9aff946 Add support for specifying which INDEX files to build via portsnap.conf.
Requested by:	brooks
Reminded by:	brooks, about halfway through his BSDCan talk
2008-05-17 16:26:27 +00:00
Colin Percival
e81875ba53 Improve conformance to the HTTP specification by using case-insensitive
comparisons for header keywords.  Apparently some proxies use creative
capitalization.

Weird proxy found by:	brooks
MFC after:		3 days
2008-02-13 20:46:23 +00:00
Colin Percival
a4c77fdccb Attempt to reduce the rate of foot-shooting injuries by adding a new
paragraph clarifying that portsnap does not behave the same way as
cvs and cvsup where local modifications are concerned.

Submitted by:	peter
Feet shot:	peter, kris, obrien, + many others
2007-12-28 20:39:18 +00:00
Colin Percival
607fc5a213 When generating ports INDEX files from downloaded "describes" data, build
an index for FreeBSD 7.x too.

MFC After:	1 week
Approved by:	re (hrs)
2007-08-02 02:05:23 +00:00
Colin Percival
220a80611e Add support for HTTP/1.0 Persistent Connections to phttpget. Requests are
be marked as HTTP/1.1 but "Connection: Keep-Alive" is added; this convinces
HTTP/1.0 servers and proxies to hold the TCP connection open despite not
being able to use HTTP pipelining.

This dramatically cuts down on the number of TCP connections (and thus port
numbers) used by portsnap when talking to an HTTP/1.0 proxy (e.g., squid),
and has the side benefit of improving performance in those cases.

Tested by:	simon
Approved by:	re (kensmith)
MFC After:	1 week
2007-06-30 19:48:28 +00:00
Colin Percival
ab5b7534b6 Add support for the HTTP_TIMEOUT environment variable (integer number
of seconds, just like in fetch(1)).

Submitted by:	rdivacky
2006-11-07 09:18:09 +00:00
Colin Percival
f2890dbd24 Set LC_ALL=C in order to avoid problems with character ranges and
sorting.

PR:		bin/104505
MFC after:	3 days
2006-10-21 11:04:34 +00:00
Ruslan Ermilov
bc84aa4ba3 Markup fixes. 2006-09-29 17:57:04 +00:00
Colin Percival
444f80d54b s/MAN8/MAN/
Pointed out by:	ru
2006-09-01 09:49:09 +00:00
Colin Percival
2a569ec900 Use some features of sh(1) which I didn't know about until today ("read"
can read two variables at once; and suffix pattern deletion) to make the
extract command fork fewer processes.

With the portsnap snapshot and the ports tree in swap-backed memory
disks on my 1.4GHz laptop, this reduces 178800 processes and 195/56/126
seconds of real/user/sys time to 44600 processes and 103/34/60 seconds.
2006-05-30 07:08:41 +00:00
Colin Percival
1e1de0e499 Fix typo.
Pointed out by:	ceri
2006-05-13 18:04:48 +00:00
Colin Percival
d600d37002 Add a mechanism for constructing INDEX files which include local ports.
Requested by:	brooks
2006-05-13 15:56:35 +00:00
Colin Percival
343c554c88 Use 'rm -rf foo/bar' to remove a port instead of 'rm -rf foo/bar/'.
When /usr/ports/foo/bar is a symlink pointing outside the tree, this
deletes the symlink instead of the directory it points to.

Requested by:	delphij
2006-05-12 10:42:40 +00:00
Colin Percival
b39855baee Allow the HTTP_PROXY environment variable to be (mis)spelled as
"http_proxy", since some people apparently do this and fetch(3)
allows it.
2006-05-05 04:47:00 +00:00
Colin Percival
a414ca11d5 Fix the test for whether ${HTTP_PROXY} is set -- I got it backwards.
Pointy hat to:	cperciva
Pointed out by:	pjd
2006-05-05 03:44:13 +00:00
Colin Percival
37c8250c90 Remove two lines of debugging which I forgot to remove before the last
commit.

Noticed by:     simon
2006-05-03 22:09:42 +00:00
Colin Percival
4b93a7cae2 The approach portsnap uses of "pick a random HTTP mirror" doesn't
interact very nicely with HTTP proxies: Since proxies do not know
that all the files on portsnap1.freebsd.org are identical to the
files with the same names on portsnap2.freebsd.org, said proxies end
up downloading and storing files in duplicate.

This commit uses the HTTP_PROXY environment variable, if set, to
generate a random number seed for use in selecting a mirror.  This
means that if several systems all have the same HTTP_PROXY value set,
they will ask the proxy to fetch files from the same mirror (unless
that mirror fails, in which case all the systems will use the same
second choice, et cetera).

Portsnap still doesn't interact very well with "transparent" HTTP
proxies, but there's nothing I can do about those.

Requested by:	simon
Sponsored by:   FreeBSD security development fundraiser
2006-05-03 21:29:01 +00:00
Colin Percival
1e87b57202 Instead of selecting a mirror and failing if it is inaccessible, keep
track of which mirrors we have tried and try a different mirror if we
fail when trying to download the SSL public key or the snapshot
signature.

Failures later in the download process will not result in switching to
a different mirror, for two reasons:
1. If is very unlikely that a mirror will fail partway through the
process of downloading updates.
2. If we switched from a more recently updated mirror to a less
recently updated mirror partway through the download process, we would
end up failing anyway because we would be trying to fetch files which
the second mirror didn't have yet.

PR:		bin/96288
Requested by:	lots of people
Sponsored by:	FreeBSD security development fundraiser
2006-05-03 21:19:43 +00:00
Colin Percival
96b468e33b Teach portsnap to parse the output of the host(1) in BIND 8 as well as
the host(1) from BIND 9.  This doesn't matter for HEAD, but will help
people who install portsnap from the ports tree onto older versions of
FreeBSD.

PR:		ports/93901
Sponsored by:	FreeBSD security development fundraiser
2006-05-02 05:27:30 +00:00
Ruslan Ermilov
e1fe3dba5c Reimplementation of world/kernel build options. For details, see:
http://lists.freebsd.org/pipermail/freebsd-current/2006-March/061725.html

The src.conf(5) manpage is to follow in a few days.

Brought to you by:	imp, jhb, kris, phk, ru (all bugs are mine)
2006-03-17 18:54:44 +00:00
Hajimu UMEMOTO
2fd4604aa4 Remove trailing dot from the servername in the serverlist to be able
to use portsnap from behind a http proxy.  Some HTTP proxy don't like
trailing dot in the servername.

Approved by:	cperciva
MFC after:	2 days
2006-03-07 19:13:00 +00:00
Hajimu UMEMOTO
905b98c5f4 Correct handling of HTTP_PROXY_AUTH. A password may have `:'.
Approved by:	cperciva
MFC after:	2 days
2006-03-07 19:04:16 +00:00
Wojciech A. Koszek
6b58c641d6 Reference sha256(1) instead of sha256(8), which doesn't exist.
Approved by:	cognet (mentor)
MFC after:	3 days
2006-02-26 22:59:40 +00:00
Joel Dahl
fbf9b468d5 Expand contractions. 2006-02-01 14:33:14 +00:00
Colin Percival
b258da0fa9 Merge from accidental commit to RELENG_5:
Correctly identify the host and port values on a failed
  getaddrinfo lookup.
2006-01-27 14:42:15 +00:00
Colin Percival
66915bd926 Replace "fetch" with "fetch -r" when downloading the initial ~38MB
snapshot in order to avoid unnecessary re-downloading.

Remove the earlier "rm -f ${SNAPSHOTHASH}.tgz" to make this work.

Suggested by:	Lars Engels
MFC after:	7 days
2006-01-22 23:48:07 +00:00
Colin Percival
afb9481259 Mention that the random time slept by "portsnap cron" is between 1 and
3600 seconds.

Suggested by:	Niki Denev
MFC after:	3 days
2006-01-17 03:02:44 +00:00
Colin Percival
7e6b02f651 Make "portsnap extract" automatically create ${PORTSDIR} if necessary
instead of complaining that "Directory does not exist or is not
writable".

Suggested by:	{tlp, siep} via IRC
MFC after:	1 week
2006-01-06 20:39:11 +00:00
Colin Percival
6382a7be43 Make "missing dependency" errors more useful by indicating which port
is trying to depend upon the non-existent port.

Suggested by:	kris
MFC after:	3 days
2005-12-01 22:14:44 +00:00
Colin Percival
3d19fd3190 An empty file does not have a positive number of lines.
Make sure that the number of lines read is non-zero before in order to
avoid dumping core.

Reported by:	Wojciech A. Koszek
Pointy hat to:	cperciva
2005-11-20 00:50:30 +00:00
Colin Percival
5c67112b9e Correctly handle a TCP connection being shutdown by the server while
we're reading response headers.  (Handle it as a connection-killing
error, rather than entering an infinite loop reading zero bytes.)

Reported by:	simon
Discovered thanks to:	A not-very-transparent transparent HTTP proxy.
MFC after:	3 days
2005-11-17 11:01:32 +00:00
Colin Percival
31c06b3f09 Add HTTP proxy authentication, via the HTTP_PROXY_AUTH environment
variable.

Tested by:	Emil Mikulic
X-MFC-After:	6.0-RELEASE
2005-09-22 07:11:27 +00:00
Colin Percival
3e4858ff09 Style fix: "if (pointer)" -> "if (pointer != NULL)" 2005-09-21 18:42:56 +00:00
Colin Percival
82835af3f1 Add a note pointing out that certain information (IP address, list of
files fetched, FreeBSD release level) is transmitted to the portsnap
server when portsnap is run, but that this information is only used
anonymously and in aggregate.
2005-09-18 12:19:32 +00:00
Colin Percival
6a15313d24 Handle circular dependencies properly (via errx(3)) rather than dumping
core.  This bug was made visible by a recent change to the audio/timidity++
port, which now has itself as a run dependency.

Reported by:	Emil Mikulic, Andreas Klemm
2005-09-17 15:30:16 +00:00
Colin Percival
ac8189712e Portsnap uses host(1) to search for mirrors, but this is only available
when the base system is not compiled with NO_BIND set.  Before we start
searching for mirrors, make sure that host(1) can be found, and if it
doesn't exist then fallback to the A record instead of the SRV records.

Submitted by:	Luca Morettoni
2005-09-15 13:29:10 +00:00
Colin Percival
e2d583aca4 Fix typo: ${PORTDIR} -> ${PORTSDIR}.
Apologies to everyone who has run portsnap in 7.0-CURRENT since
Tuesday; if there is a file "/.portsnap.INDEX" on your system, you can
delete it (or even better, move it to /usr/ports/.portsnap.INDEX).

Big pointy hat to:	cperciva
Reported that things weren't working properly:	Aleksander Fafula
2005-09-11 12:55:07 +00:00
Colin Percival
73b2114b4e Fix typo: Cowarly -> Cowardly.
Reported by:	Joerg Sonnenberger
2005-09-07 19:32:15 +00:00
Colin Percival
ff69e5b71e Teach portsnap how to ignore unwanted parts of the ports tree. A line
of the form "REFUSE foo" in portsnap.conf will result in parts of the
tree matching "^foo" being (a) not extracted by "portsnap extract", (b)
not updated by "portsnap update", and (c) not having any patches or new
ports downloaded by "portsnap fetch" or "portsnap cron". The example
shown in portsnap.conf demonstrates ignoring all the language categories.

As mentioned in portsnap.conf.5, the use of an imcomplete ports tree is
not officially supported; but this is something which many users have
requested, so I'm adding it anyway.

PR:		bin/85619 (but not the patch provided therein)
MFC after:	1 month
2005-09-06 19:28:37 +00:00
Colin Percival
24a3ff142f Spell "cmp > /dev/null" as "cmp -s".
Suggested by:	dougb
2005-08-14 02:45:04 +00:00