be marked as HTTP/1.1 but "Connection: Keep-Alive" is added; this convinces
HTTP/1.0 servers and proxies to hold the TCP connection open despite not
being able to use HTTP pipelining.
This dramatically cuts down on the number of TCP connections (and thus port
numbers) used by portsnap when talking to an HTTP/1.0 proxy (e.g., squid),
and has the side benefit of improving performance in those cases.
Tested by: simon
Approved by: re (kensmith)
MFC After: 1 week
can read two variables at once; and suffix pattern deletion) to make the
extract command fork fewer processes.
With the portsnap snapshot and the ports tree in swap-backed memory
disks on my 1.4GHz laptop, this reduces 178800 processes and 195/56/126
seconds of real/user/sys time to 44600 processes and 103/34/60 seconds.
interact very nicely with HTTP proxies: Since proxies do not know
that all the files on portsnap1.freebsd.org are identical to the
files with the same names on portsnap2.freebsd.org, said proxies end
up downloading and storing files in duplicate.
This commit uses the HTTP_PROXY environment variable, if set, to
generate a random number seed for use in selecting a mirror. This
means that if several systems all have the same HTTP_PROXY value set,
they will ask the proxy to fetch files from the same mirror (unless
that mirror fails, in which case all the systems will use the same
second choice, et cetera).
Portsnap still doesn't interact very well with "transparent" HTTP
proxies, but there's nothing I can do about those.
Requested by: simon
Sponsored by: FreeBSD security development fundraiser
track of which mirrors we have tried and try a different mirror if we
fail when trying to download the SSL public key or the snapshot
signature.
Failures later in the download process will not result in switching to
a different mirror, for two reasons:
1. If is very unlikely that a mirror will fail partway through the
process of downloading updates.
2. If we switched from a more recently updated mirror to a less
recently updated mirror partway through the download process, we would
end up failing anyway because we would be trying to fetch files which
the second mirror didn't have yet.
PR: bin/96288
Requested by: lots of people
Sponsored by: FreeBSD security development fundraiser
the host(1) from BIND 9. This doesn't matter for HEAD, but will help
people who install portsnap from the ports tree onto older versions of
FreeBSD.
PR: ports/93901
Sponsored by: FreeBSD security development fundraiser
snapshot in order to avoid unnecessary re-downloading.
Remove the earlier "rm -f ${SNAPSHOTHASH}.tgz" to make this work.
Suggested by: Lars Engels
MFC after: 7 days
we're reading response headers. (Handle it as a connection-killing
error, rather than entering an infinite loop reading zero bytes.)
Reported by: simon
Discovered thanks to: A not-very-transparent transparent HTTP proxy.
MFC after: 3 days
files fetched, FreeBSD release level) is transmitted to the portsnap
server when portsnap is run, but that this information is only used
anonymously and in aggregate.
core. This bug was made visible by a recent change to the audio/timidity++
port, which now has itself as a run dependency.
Reported by: Emil Mikulic, Andreas Klemm
when the base system is not compiled with NO_BIND set. Before we start
searching for mirrors, make sure that host(1) can be found, and if it
doesn't exist then fallback to the A record instead of the SRV records.
Submitted by: Luca Morettoni
Apologies to everyone who has run portsnap in 7.0-CURRENT since
Tuesday; if there is a file "/.portsnap.INDEX" on your system, you can
delete it (or even better, move it to /usr/ports/.portsnap.INDEX).
Big pointy hat to: cperciva
Reported that things weren't working properly: Aleksander Fafula
of the form "REFUSE foo" in portsnap.conf will result in parts of the
tree matching "^foo" being (a) not extracted by "portsnap extract", (b)
not updated by "portsnap update", and (c) not having any patches or new
ports downloaded by "portsnap fetch" or "portsnap cron". The example
shown in portsnap.conf demonstrates ignoring all the language categories.
As mentioned in portsnap.conf.5, the use of an imcomplete ports tree is
not officially supported; but this is something which many users have
requested, so I'm adding it anyway.
PR: bin/85619 (but not the patch provided therein)
MFC after: 1 month
missing from ${WORKDIR}/files/.
This bug was caused by the astonishing interaction of "return" and
pipelines; in the following code, the "return" does not exit the
function, but instead exits the subshell which was spawned for the last
element of the pipeline; consequently, the output produced is "foo".
foo() {
echo bar | while read baz; do
if [ ${baz} = "bar" ]; then
return 1
fi
done
echo foo
}
Reported by: simon
"portsnap fetch update" or "portsnap -I cron update". They will be
executed in the order that they appear, and duplicates are not removed
(so "portsnap fetch fetch fetch fetch" is meaningful, albeit rather
silly).
Requested by: Roman Divacky
from the port number (if any exists). This unbreaks
env HTTP_PROXY="http://localhost:3128/" portsnap fetch
While I'm here, list both the host and the port in the error message
output if getaddrinfo() fails, since either of them could be responsible
for the failure.
than what we already have" message. [1]
When operating with the undocumented --debug option, replace "xargs
phttpget" with "xargs -t phttpget", since that might conceivably help
debugging.
Noticed by: simon [1]
fast, lightweight, and generally good way for users to keep their
ports trees up to date.
This is version 0.9.4 from the ports tree (sysutils/portsnap) with
the following changes:
1. The experimental pipelined http code is enabled. No seatbelts
in -CURRENT. (^_^)
2. The working directory has moved from /usr/local/portsnap to
/var/db/portsnap (as discussed on -arch two days ago).
3. Portsnap now fetches a list of mirrors (distributed as DNS SRV
records) and selects one randomly. This should help to avoid the
uneven loading which plagues the cvsup mirror network.
4. The license is now 2-clause BSD instead of 3-clause BSD.
5. Various incidental changes to make portsnap fit into the base
system's build mechanics.
X-MFC-After: 6.0-RELEASE
X-MFC-Before: 5.5-RELEASE
X-MFC-To: RELENG_6, RELENG_5, ports
discussed on: -arch and several other places
"yes please" from: simon, remko, flz, Diane Bruce
thinks this is a great idea: bsdimp
Hopes he didn't forget any files: cperciva
