Re-add alarm(2) calls around the calls to fetchStat(3) and fetchXGet(3),
since these calls can still time out on DNS lookups or TCP connect(2).
Remove the alarm(2) calls in the main loop, since all methods properly
handle transfer timeouts (as opposed to connection timeouts).
Set the sigalrm flag if a timeout occurs in the main loop.
Move the signal: label up a little so we still set the atime and mtime
when the transfer times out or is interrupted, so that restarted transfers
will work as expected (as long as the file still exists).
MFC after: 2 weeks
<sys/stat.h> for the declaration of struct timeval.
Removed unused includes (<time.h> doesn't declare anything of interest;
only <sys/time.h> does).
Sorted includes a bit.
to resume a transfer, download the requested document into a temporary file
which we later rename. This avoids leaving half-completed files around in
case of a crash (it'll still leave a half-completed file, but with a hope-
fully non-conflicting name), and should reduce the need for human inter-
vention on ports-building machines.
The temporary file name for "foo/bar" is constructed by invoking mkstemps()
with the pattern "foo/.fetch.XXXXXX.bar"
Requested by: obrien
- if the dates didn't match, fetch would append the received file to the
existing file instead of replacing it.
- if the local file was complete and up-to-date, fetch would miscalculate
the expected size and report a failure instead of a success, because it
had no way of knowing that the server was actually resending the entire
file since the requested offset was invalid.
libfetch features (fetchRestartCalls, fetchXGet()).
Since it doesn't make much sense to have m_flag and r_flag set at the same
time, and it can actually cause trouble in some cases, die if they're both
set.
Set the SA_RESETHAND flag for SIGINT so that when we've caught one, we can
kill ourselves with a second SIGINT (thus notifying our parent of our tragic
fate) instead of just exiting.
These changes fix several problems that would show up when fetching ports,
as well as speeding up HTTP transfers quite a bit (at least for relatively
small files).
Most of these changes were prompted by an interaction problem with an HTTP
server called SWS-1.0, which exhibited two bugs, the first of which prevented
fetch from working around the second (the first was not sending content-type
in reply to HEAD requests, the second was sending garbage after the end of
the requested file).
Always display the completion percentage if stderr is a tty.
Drop the char-by-char transfer mode, it was based on an incorrect assumption
regarding the semantics of fread().
Finally (I hope) straighten out the business of setting the mtime, as well as
when to remove the output file and when not to.
Thanks are owed to the many who have provided nearly instantaneous and
highly constructive feedback and suggestions about these matters.
around the call to fetchStat().
Catch SIGINT, and rework the signal handling so it doesn't skimp on the
cleanup after a timeout or interrupt. Also, don't just bail out after a
timeout; there may be more files to fetch.
Fix a bug where the stats code would print the expected size instead of the
number of bytes received.
Fix the reading code so it'll support partial reads.
of an empty buffer... the output file wasn't readable... also warn that
we can't checksum on stdout and print out the base64 encoded version of the
md5 checksum...
Site to actually return md5 digest: web.golux.com
Verified that fetch was broken: Ken Coar <Ken.Coar@Golux.Com>
Look at the FTP_PASSIVE_MODE environment variable like the man page says.
PR: bin/9464
Submitted by: John A. Shue <John.Shue@symmetron.com>
Add references to RFC's 1790, 959, 850.
PR: doc/6564
Fix usage string. Fix getopt() string. Fix ordering of compatibility options.
fetch.1:
Fix synopsis. Fix ordering of T and t options. Fix minor grammar nit.
- failed to use authorization parameters passed in the environment, if
/dev/tty could not be opened (i.e. if running from cron)
- mixed use of /dev/tty and stdin for prompt and reading of the result
is contrary to RFC1738, which specifies that empty elements between
slashes really mean to send a CWD command with an empty argument.
Most FTP servers consider this an error, so strict RFC compliance
is advisable in this case.
for ftp: URLs as well. This can't possibly be done in the general case,
but since we only claim to support http and ftp, we'll adhere to the
bogus requirement (RFC 2068, s. 14.25) anyway... >sigh<
Submitted by: =?iso-8859-1?Q?=C5ge_R=F8bekk?= <aagero@aage.priv.no>
due to the output not being a regular file. Also split the error message
in any error case here to be on two lines to lessen the likelihood of it
being too long to fit on just one.
been extensively tested, but I now can successfully retrieve
<http://www.apache.org/index.html>, so I guess that's a victory of some
sort.
Also move the initialization of ``autherror'' to hopefully eliminate
the reported loop involving authentication. Still need to implement
MD5 digest authentication.
undocumented FTP_PROXY_USER
Make FTP file errors contian hostname and path.
Pass the FTP port to libftp.
Partially un-HTMLify error messages returned from server
Handle "HTTP NNN" instead of "HTTP/V.vv NNN" response sent by
pre-HTTP/1.0 servers
Reviewed by: wollman
- Fix the bug with URIs of the form ftp://host/filename.
- Fix some more string-termination bugs in util.c.
- Use safe_malloc() rather than testing the return value of
regular malloc() in 15 places.
- Implement HTTP authentication, for both servers and proxies.
Currently only ``basic'' authentication is supported; This Is A Bug
(but less of one tjhan nmot supporting any authentication).
I think there is only one more feature which is required for full
HTTP/1.1 support, which is Transfer-Encoding: chunked; this should
not be toohard, but it isn't very important, either.
1) Implement redirects (or try to, at least).
2) Implement automatic retry after 503 errors when Retry-After is given.
3) Implement a -a flag to enable both of these behaviors.
4) Recognize Transfer-Encoding headers and emit a warning that the file
is likely to be damaged.
5) Bug fix: only write the amount of data we read.
6) Actually document some of these.
7) Fix the usage message to display flags in semi-alphabetical order.
recognize it any more. This makes the following significant changes:
- The main body of the program doesn't know a thing about URIs,
HTTP, or FTP. This makes it possible to easily plug in other
protocols. (The next revision will probably be able to dynamically
add new recognizers.)
- There are no longer arbitrary timeouts for the protocols. If you want
to set one for yourself, use the environment variables.
- FTP proxies are now supported (if I implemented it right).
- The HTTP implementation is much more complete, and can now do restarts,
preserve modtimes, and mrun in mirror mode. It's not yet up to 1.1,
but it's getting there.
- Transaction TCP is now used for sending HTTP requests. The HTTP/1.1 syntax
for requesting that the connection be closed after one request is
implemented.
In all of this, I have doubtless broken somebody. Please test it and tell me
about the bugs.
This will make a number of things easier in the future, as well as (finally!)
avoiding the Id-smashing problem which has plagued developers for so long.
Boy, I'm glad we're not using sup anymore. This update would have been
insane otherwise.
