handle splitting input files on a '|'. This greatly
reduces the time taken to process several databases
during the update process.
Additionally add some more debug logging.
This is done to speed up extraction significantly (both for portsnap
extract and update) in the case of slow NSS modules (like nss_ldap) as
it avoids having to look up uid and gid for root / wheel.
The reason this is a bigger problem for portsnap than for many other
system operations, is that portsnap executes tar(1) once for each port
so the internal uid/gid caching in tar(1) only helps a bit, resulting
in many user lookup calls.
Discussed with: cperciva
(1) phttpget is attempting to download enough files that it can't send
all the requests at once, and
(2) the remote server forcibly closes the connection, resulting in RST
packets being sent,
phttpget will receive a SIGPIPE and terminate without downloading all
of the files.
This is probably responsible for a number of hard-to-reproduce errors
with portsnap and freebsd-update.
MFC after: 3 days
paragraph clarifying that portsnap does not behave the same way as
cvs and cvsup where local modifications are concerned.
Submitted by: peter
Feet shot: peter, kris, obrien, + many others
be marked as HTTP/1.1 but "Connection: Keep-Alive" is added; this convinces
HTTP/1.0 servers and proxies to hold the TCP connection open despite not
being able to use HTTP pipelining.
This dramatically cuts down on the number of TCP connections (and thus port
numbers) used by portsnap when talking to an HTTP/1.0 proxy (e.g., squid),
and has the side benefit of improving performance in those cases.
Tested by: simon
Approved by: re (kensmith)
MFC After: 1 week
can read two variables at once; and suffix pattern deletion) to make the
extract command fork fewer processes.
With the portsnap snapshot and the ports tree in swap-backed memory
disks on my 1.4GHz laptop, this reduces 178800 processes and 195/56/126
seconds of real/user/sys time to 44600 processes and 103/34/60 seconds.
interact very nicely with HTTP proxies: Since proxies do not know
that all the files on portsnap1.freebsd.org are identical to the
files with the same names on portsnap2.freebsd.org, said proxies end
up downloading and storing files in duplicate.
This commit uses the HTTP_PROXY environment variable, if set, to
generate a random number seed for use in selecting a mirror. This
means that if several systems all have the same HTTP_PROXY value set,
they will ask the proxy to fetch files from the same mirror (unless
that mirror fails, in which case all the systems will use the same
second choice, et cetera).
Portsnap still doesn't interact very well with "transparent" HTTP
proxies, but there's nothing I can do about those.
Requested by: simon
Sponsored by: FreeBSD security development fundraiser
track of which mirrors we have tried and try a different mirror if we
fail when trying to download the SSL public key or the snapshot
signature.
Failures later in the download process will not result in switching to
a different mirror, for two reasons:
1. If is very unlikely that a mirror will fail partway through the
process of downloading updates.
2. If we switched from a more recently updated mirror to a less
recently updated mirror partway through the download process, we would
end up failing anyway because we would be trying to fetch files which
the second mirror didn't have yet.
PR: bin/96288
Requested by: lots of people
Sponsored by: FreeBSD security development fundraiser
the host(1) from BIND 9. This doesn't matter for HEAD, but will help
people who install portsnap from the ports tree onto older versions of
FreeBSD.
PR: ports/93901
Sponsored by: FreeBSD security development fundraiser
snapshot in order to avoid unnecessary re-downloading.
Remove the earlier "rm -f ${SNAPSHOTHASH}.tgz" to make this work.
Suggested by: Lars Engels
MFC after: 7 days
we're reading response headers. (Handle it as a connection-killing
error, rather than entering an infinite loop reading zero bytes.)
Reported by: simon
Discovered thanks to: A not-very-transparent transparent HTTP proxy.
MFC after: 3 days
