d/freebsd-skq
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
209,781 Commits 4 Branches 49 Tags 2 GiB
cfe3da09e2
Commit Graph

273 Commits

Author SHA1 Message Date
Dag-Erling Smørgrav
75c3973ab8 Clarify my feelings towards fetch / libfetch. 2002-10-30 04:53:58 +00:00
Dag-Erling Smørgrav
eb87c5a73a Don't complain about not knowing the remote file size when working in 
quiet mode.

PR:		bin/37079
Submitted by:	Nicolas Rachinsky <list@rachinsky.de>
 2002-10-27 17:33:08 +00:00
Dag-Erling Smørgrav
97727a7dc1 Whitespace and indentation cleanup. 2002-10-27 15:32:51 +00:00
Dag-Erling Smørgrav
0c6d34af10 Forgot to disable alarms after fetchXGet() in previous commit. 2002-10-27 15:32:06 +00:00
Dag-Erling Smørgrav
dc161d5582 Add an ETA timer that kicks in after 30 seconds. 
Re-add alarm(2) calls around the calls to fetchStat(3) and fetchXGet(3),
since these calls can still time out on DNS lookups or TCP connect(2).

Remove the alarm(2) calls in the main loop, since all methods properly
handle transfer timeouts (as opposed to connection timeouts).

Set the sigalrm flag if a timeout occurs in the main loop.

Move the signal: label up a little so we still set the atime and mtime
when the transfer times out or is interrupted, so that restarted transfers
will work as expected (as long as the file still exists).

MFC after:	2 weeks
 2002-10-27 15:15:13 +00:00
Dag-Erling Smørgrav
dacff75286 s/optarg/s/ (cut'n'paste braino) 
Spotted by:	kris
 2002-08-27 10:19:24 +00:00
Dag-Erling Smørgrav
35945a74ac fetch(1) is WARNS-5 clean (tested on i386 and Alpha) 2002-07-28 21:09:57 +00:00
Dag-Erling Smørgrav
32411a1b6f Use strtol(3) instead of a home-grown version, and fix a "possibly 
unitialized variable" warning.
 2002-07-28 21:09:25 +00:00
Dag-Erling Smørgrav
6414f0b970 Reorder LDADD to fix static linking. 2002-06-12 12:46:50 +00:00
Ruslan Ermilov
4093807dd9 libfetch now depends on libcrypto and libssl. 2002-06-06 13:45:46 +00:00
Dag-Erling Smørgrav
1bf28f7e94 Figures they had to call it NOCRYPT instead of NOCRYPTO. 2002-06-05 21:28:47 +00:00
Dag-Erling Smørgrav
d9615d7da4 During buildworld, "regular" libraries are built before crypto stuff, so 
libfetch can't depend on lib{crypto,ssl}.  Move the dependency to fetch
until we can figure out how to fix this.
 2002-06-05 21:25:33 +00:00
Dag-Erling Smørgrav
d397408818 Usage style sweep: spell "usage" with a small 'u'. 
Also change one case of blatant __progname abuse (several more remain)
This commit does not touch anything in src/{contrib,crypto,gnu}/.
 2002-04-22 13:44:47 +00:00
Philippe Charnier
3898680cf7 Use `The .Nm utility' 2002-04-19 23:44:58 +00:00
Bruce Evans
05eb035898 Restored the part of 1.45 that was clobbered in rev.1.46: 
#include <sys/time.h> instead of depending on namespace pollution in
<sys/stat.h> for the declaration of struct timeval.
 2002-03-27 02:23:18 +00:00
Philippe Charnier
15baa31aad Add FBSDID. Remove unused include. Getopt returns -1 not EOF. 2002-03-26 12:09:49 +00:00
Bruce Evans
469a45700c #include <sys/time.h> instead of depending on namespace pollution in 
<sys/stat.h> for the declaration of struct timeval.

Removed unused includes (<time.h> doesn't declare anything of interest;
only <sys/time.h> does).

Sorted includes a bit.
 2002-02-25 05:00:39 +00:00
Dag-Erling Smørgrav
8e7cf3debf If downloading to a temporary file, set the timestamp on the temp file, not 
on the file we're replacing.

PR:		bin/34992
Submitted by:	Shunichiro Ariura <syun1rou@blackshell.org>
MFC after:	1 week
 2002-02-16 14:02:57 +00:00
Mark Murray
91bd71d206 Remove leaf node WARNS?=2 (that mainly I added). This should 
help the GCC3 transition and CURRENT in general.
 2002-02-08 22:31:43 +00:00
Ruslan Ermilov
ab47d79913 Document fetch(1) reaction to SIGINFO. 
PR:		docs/33640
Submitted by:	Maxim Konovalov <maxim@macomnet.ru>
Approved by:	des
 2002-01-16 16:16:08 +00:00
Dag-Erling Smørgrav
62f6d0b88e Set fetchDebug if v_level is 3 or more. 
PR:		bin/32615
MFC after:	1 week
 2001-12-09 15:07:26 +00:00
Ian Dowse
650b9e0ed5 Remove an extraneous argument to fprintf that -Wformat noticed. 
Reviewed by:	des
 2001-11-10 18:39:54 +00:00
Dag-Erling Smørgrav
f2a871e75b Remove a debugging warnx() that got committed by mistake. 2001-11-03 01:31:46 +00:00
Dag-Erling Smørgrav
dbcc198301 Don't print running stats unless we're the foreground process. 
PR:		30764
 2001-09-23 22:03:39 +00:00
Mike Barcroft
1c93d61fcc Add a cast to silence a warning (error because of WARNS=2) on Alpha. 
Discussed with:	des
 2001-09-10 17:23:57 +00:00
Dag-Erling Smørgrav
e9dc4f3b74 If the local file does not exist, or is a regular file, and we're not trying 
to resume a transfer, download the requested document into a temporary file
which we later rename.  This avoids leaving half-completed files around in
case of a crash (it'll still leave a half-completed file, but with a  hope-
fully non-conflicting name), and should reduce the need for human inter-
vention on ports-building machines.

The temporary file name for "foo/bar" is constructed by invoking mkstemps()
with the pattern "foo/.fetch.XXXXXX.bar"

Requested by:	obrien
 2001-09-08 15:17:15 +00:00
Dag-Erling Smørgrav
eab5a80417 Remove a blank line that snuck in with the previous commit. 
Set the default timeout to 120 seconds instead of 0 (no timeout).
 2001-08-18 10:48:55 +00:00
Dag-Erling Smørgrav
db695db7f0 Add some comments. 2001-08-17 22:22:45 +00:00
Stefan Eßer
6d64e93980 Include value of command line argument that causes an error message or 
warning in said message, since fetch may be run from a makefile or script
which does not print the command line.

Approved by:	des
 2001-07-28 21:34:56 +00:00
Dag-Erling Smørgrav
04d12c9900 If we're writing to stdout, we don't care about the size. 
Submitted by:	wollman
 2001-07-17 21:23:43 +00:00
Dag-Erling Smørgrav
91404f38f0 WARNS cleanup + remove an unused macro and an unused function argument. 2001-07-17 21:22:32 +00:00
Dima Dorfman
f247324df7 Remove whitespace at EOL. 2001-07-15 08:06:20 +00:00
Dag-Erling Smørgrav
74bd3d7698 Fix an embarassing if-test-reversal bug that broke scheme guessing. 2001-07-08 15:17:26 +00:00
Dag-Erling Smørgrav
1a4a063934 Don't try to print a completion percentage for zero-size files. 
PR:		27780
Submitted by:	Craig Leres <leres@ee.lbl.gov>
 2001-06-01 10:24:58 +00:00
Dag-Erling Smørgrav
5ef824ed2b Damn. Fix typo in previous commit. 2001-05-27 10:46:19 +00:00
Dag-Erling Smørgrav
0d60c7099d When running on a tty, install an authentication callback. 2001-05-26 19:51:05 +00:00
Dag-Erling Smørgrav
dde0dff6dc Bring the usage message in synch with reality. 
PR:		bin/26160
 2001-03-28 21:01:07 +00:00
Dag-Erling Smørgrav
7c480c6c77 Now that libfetch uses the high port range by default, add a -U option to 
make it use the low (default) port range instead.
 2001-03-24 00:32:11 +00:00
Dag-Erling Smørgrav
cd400b67a4 If SIGINFO is received during the transfer, print a status line similar 
to the summary shown at the end of the transfer.

Some days, I really hate Unix...

Suggested by:	cracauer
 2001-03-07 05:33:50 +00:00
Dag-Erling Smørgrav
150fb21bf8 Fix a formatting bug (MFC candidate) 2001-03-07 03:22:11 +00:00
Ruslan Ermilov
d0353b836e mdoc(7) police: split punctuation characters + misc fixes. 2001-02-01 16:38:02 +00:00
Dag-Erling Smørgrav
89a70fbeb3 Warn if the size of the remote file isn't known. 
If the -R option was specified, don't truncate the local file even if its
mtime is incorrect.

PR:		bin/23719
 2000-12-22 18:23:19 +00:00
Dag-Erling Smørgrav
64638f67a7 If the URL did not specify a scheme, try to guess it from the host name. 2000-12-13 11:26:27 +00:00
Dag-Erling Smørgrav
30204f9830 If the transfer timed out, but we don't know how large the file is supposed 
to be, assume it was truncated.
 2000-11-30 10:08:22 +00:00
Kris Kennaway
a6f339d266 Format string auditing 2000-11-27 07:35:53 +00:00
Ruslan Ermilov
f4d874a1db mdoc(7) police: do not split author names in the AUTHORS section. 2000-11-22 09:35:58 +00:00
Ruslan Ermilov
b82f5db0ac Spell the des's name correctly. 2000-11-14 13:19:26 +00:00
Dag-Erling Smørgrav
a0c6ec97c9 Understand the difference between an empty file and a non-existent file. 
This has been sitting in my tree for ages...
 2000-10-19 21:05:59 +00:00
Dag-Erling Smørgrav
74912943bf Don't try to set the mtime of the output file if it's not a regular file. 
Pointed out by:	cwt
 2000-09-02 09:48:34 +00:00
Dag-Erling Smørgrav
e2b41a622b Don't unlink the target file if it's not a regular file. 2000-08-31 11:24:15 +00:00
Dag-Erling Smørgrav
9516ffa7c0 Restore the old semantics of deleting the output file when interrupted. 
Some people liked this and some didn't, so POLA won.
 2000-07-26 07:28:33 +00:00
Dag-Erling Smørgrav
32190ef51f Don't print summary in quiet mode. 2000-07-21 14:21:29 +00:00
Dag-Erling Smørgrav
d939bf77d2 Don't display a running count in quiet mode. 
Don't add the offset to the size; it's libfetch's job to report the correct
size of the requested file.
 2000-07-21 11:08:03 +00:00
Dag-Erling Smørgrav
10e3b1c738 Fix two bugs related to resumed transfers: 
- if the dates didn't match, fetch would append the received file to the
    existing file instead of replacing it.

  - if the local file was complete and up-to-date, fetch would miscalculate
    the expected size and report a failure instead of a success, because it
    had no way of knowing that the server was actually resending the entire
    file since the requested offset was invalid.
 2000-07-19 23:56:45 +00:00
Dag-Erling Smørgrav
a5e3ae21a5 Braino: don't use passive mode unless the user requests it. 
PR:		bin/20025
Submitted by:	HIYAMA Takeshi <th@cis.ibaraki.ac.jp>
 2000-07-19 09:12:36 +00:00
Ben Smithurst
cf575052df * Remove description of obsolete -t option. 
* Insert newline before "and" in ".Fl c and".

Approved by:	des
 2000-07-18 21:05:11 +00:00
Dag-Erling Smørgrav
38a616e49f The description of the semantics of -v and -q has long since been OBE. 2000-07-17 22:44:36 +00:00
Dag-Erling Smørgrav
45b22b2a87 While I'm at it, break a line that was too long, remove a pointless diagnostic 
and adjust the verbosity level of another.
 2000-07-17 22:44:00 +00:00
Dag-Erling Smørgrav
c0b0d54e10 Document the fact that -r and -m are mutually exclusive. 2000-07-17 21:51:48 +00:00
Dag-Erling Smørgrav
e9a039c2cb Rearrange / rewrite large portions of fetch() to take advantage of new 
libfetch features (fetchRestartCalls, fetchXGet()).

Since it doesn't make much sense to have m_flag and r_flag set at the same
time, and it can actually cause trouble in some cases, die if they're both
set.

Set the SA_RESETHAND flag for SIGINT so that when we've caught one, we can
kill ourselves with a second SIGINT (thus notifying our parent of our tragic
fate) instead of just exiting.

These changes fix several problems that would show up when fetching ports,
as well as speeding up HTTP transfers quite a bit (at least for relatively
small files).

Most of these changes were prompted by an interaction problem with an HTTP
server called SWS-1.0, which exhibited two bugs, the first of which prevented
fetch from working around the second (the first was not sending content-type
in reply to HEAD requests, the second was sending garbage after the end of
the requested file).
 2000-07-17 21:49:01 +00:00
Dag-Erling Smørgrav
78394463ee Better signal handling. 
Submitted by:   green
 2000-07-16 00:47:00 +00:00
Dag-Erling Smørgrav
b3c141fd2e Fix a bug in the stats display that showed up when resuming aborted transfers. 
Submitted by:	green
 2000-07-13 08:37:39 +00:00
Dag-Erling Smørgrav
49e62d8f8e Rework the stats code for the nth time. Much cleaner now. 
Always display the completion percentage if stderr is a tty.
Drop the char-by-char transfer mode, it was based on an incorrect assumption
regarding the semantics of fread().
Finally (I hope) straighten out the business of setting the mtime, as well as
when to remove the output file and when not to.
Thanks are owed to the many who have provided nearly instantaneous and
highly constructive feedback and suggestions about these matters.
 2000-07-12 20:53:07 +00:00
Dag-Erling Smørgrav
ef50a72c92 Now that the HTTP code supports timeouts, we don't need to use alarm() 
around the call to fetchStat().
Catch SIGINT, and rework the signal handling so it doesn't skimp on the
cleanup after a timeout or interrupt. Also, don't just bail out after a
timeout; there may be more files to fetch.
 2000-07-12 11:02:01 +00:00
Dag-Erling Smørgrav
aa4b3574a8 Rework the stats code to avoid code duplication. 
Fix a bug where the stats code would print the expected size instead of the
number of bytes received.
Fix the reading code so it'll support partial reads.
 2000-07-12 08:29:52 +00:00
Dag-Erling Smørgrav
ec850e74c0 Don't segfault if given an invalid URL (doh!) 
Submitted by:		green
 2000-07-09 10:14:22 +00:00
Dag-Erling Smørgrav
0b474bc608 Clarify the meaning of -m and -n (thanks to bde). 2000-07-08 09:34:53 +00:00
Dag-Erling Smørgrav
bb11a878ea Straighten out the behvaiour of -m and -n (thanks to bde). 
Check that stderr, not stdout, is a tty (thanks to green).
 2000-07-08 09:34:33 +00:00
Dag-Erling Smørgrav
114c3c1ab8 Add MIT copyright 2000-07-05 12:04:31 +00:00
Dag-Erling Smørgrav
047843dd61 Note that the -h, -c and -f options are deprecated, and remove note that 
incorrectly stated that they were not implemented.
Document the -d option.
 2000-06-29 10:35:24 +00:00
Dag-Erling Smørgrav
a8369cd91b Implement the -c option 2000-06-29 10:32:56 +00:00
Dag-Erling Smørgrav
df6f33d182 Don't forget to delete the output file if the request fails. 
Don't delete the output file if -r was specified.
 2000-06-29 08:39:29 +00:00
Dag-Erling Smørgrav
b39628e763 New libfetch-based fetch. 2000-06-28 16:55:15 +00:00
Jake Burkholder
e39756439c Back out the previous change to the queue(3) interface. 
It was not discussed and should probably not happen.

Requested by:		msmith and others
 2000-05-26 02:09:24 +00:00
Jake Burkholder
740a1973a6 Change the way that the queue(3) structures are declared; don't assume that 
the type argument to *_HEAD and *_ENTRY is a struct.

Suggested by:	phk
Reviewed by:	phk
Approved by:	mdodd
 2000-05-23 20:41:01 +00:00
Martin Cracauer
301cba219f Print a warning and exit with != 0 when at least one downloaded file 
is shorter than previously announced by the server.

Tested by asami.

Approved by:	jkh
 2000-03-08 13:02:11 +00:00
Sheldon Hearn
87faa07bec Remove single-space hard sentence breaks. These degrade the quality 
of the typeset output, tend to make diffs harder to read and provide
bad examples for new-comers to mdoc.
 2000-03-01 12:20:22 +00:00
Brian Feldman
d6a8296aed Add the -F option. This is for forcing restarts with -r by inhibiting 
transmission of the If-Range HTTP header field.
 1999-10-23 15:22:53 +00:00
Peter Wemm
c3aac50f28 $Id$ -> $FreeBSD$ 1999-08-28 01:08:13 +00:00
Brian Feldman
90331af0d2 "transfered" -> "transferred" 
Reviewed by:	dict(1)
 1999-07-02 08:57:06 +00:00
Brian Feldman
378949dbca This fixes the problem of cancelling an FTP transfer resulting in 
a bogus error ("parsing URI".) I also fixed an inconsistency in
another fs_status.
 1999-06-21 02:38:16 +00:00
John-Mark Gurney
85020e14f0 make md5 checksums actually work instead of simply calculating the length 
of an empty buffer...  the output file wasn't readable...  also warn that
we can't checksum on stdout and print out the base64 encoded version of the
md5 checksum...

Site to actually return md5 digest: web.golux.com
Verified that fetch was broken: Ken Coar <Ken.Coar@Golux.Com>
 1999-05-18 19:37:37 +00:00
Garrett Wollman
c7ede78e51 Restore \n at end of usage message. 
Complained-about-by: charnier
 1999-03-08 19:16:14 +00:00
Alexander Langer
c5e72606a7 Improve error message wording when attempting to link to a non-existent 
file on the local host.

PR:		10042
Submitted by:	Chris Costello <phoenix@calldei.com>
 1999-03-02 04:14:33 +00:00
Garrett Wollman
189da04438 Deal with broken Web sites which return 302 responses rather than 404 
and an error document when the requested resource does not exist.  Grrr.

Requested by:	asami
 1999-02-23 18:51:13 +00:00
Bill Fenner
d17bf266c4 Remove the FTP_PASSIVE_MODE "fix"; libftpio handles this. 1999-02-05 01:01:17 +00:00
Bill Fenner
054672c795 Warn about collapsing multiple slashes into 1 in ftp URL's. 
Look at the FTP_PASSIVE_MODE environment variable like the man page says.

PR:		bin/9464
Submitted by:	John A. Shue <John.Shue@symmetron.com>

Add references to RFC's 1790, 959, 850.

PR:		doc/6564
 1999-02-03 20:43:29 +00:00
Bill Fenner
93eb50393d Don't try to parse a colon in a URL as a port 
(e.g. http://www.host.name/foo:bar)

PR:		bin/5072
Submitted by:	Takeshi WATANABE <watanabe@komadori.planet.kobe-u.ac.jp>
 1999-02-03 20:24:53 +00:00
Garrett Wollman
9caa4ddf6b Fix logic error in RFC 850 kluge. 1999-01-15 17:10:31 +00:00
Garrett Wollman
c146843096 For RFC 850 dates received in HTTP responses, implement the century pivot 
described in RFC 2068.  Include a reference to same in the manual page.
 1999-01-15 16:56:22 +00:00
Martin Cracauer
b451f4524e Add -s option, just report size of file that would be fetched. 
Reviewed by:	-current list
 1998-12-08 13:00:49 +00:00
Dag-Erling Smørgrav
9f63f2aa1c main.c: 
Fix usage string. Fix getopt() string. Fix ordering of compatibility options.
fetch.1:
 Fix synopsis. Fix ordering of T and t options. Fix minor grammar nit.
 1998-11-08 23:18:48 +00:00
Bill Fenner
60a53b7d32 If we know the content-length, only read that number of bytes from 
the server.  There exists a broken server which sends a few extra
garbage bytes in response to HTTP/1.1 requests.
 1998-10-26 02:39:21 +00:00
Jordan K. Hubbard
0637c2cccd Add -S flag; this is benign since fetch's behavior is unchanged if you 
don't use it.  That's why I'm bringing it in during our "code slush"
Submitted by:	Stefan Esser <se@mi.uni-koeln.de>
 1998-09-20 00:01:26 +00:00
Dag-Erling Smørgrav
82e3dde754 DTRT for file: URLs when output file is "-". 
PR:		bin/7203
 1998-09-17 00:15:15 +00:00
Stefan Eßer
cc522880b4 Fix proxy authorization code: 
- failed to use authorization parameters passed in the environment, if
  /dev/tty could not be opened (i.e. if running from cron)
- mixed use of /dev/tty and stdin for prompt and reading of the result
 1998-07-12 09:07:36 +00:00
Garrett Wollman
cffe96c903 Fix my last change: the .Fx macro doesn't do anything useful without an 
argument, so use `.Tn FreeBSD' instead.
 1998-05-09 21:39:49 +00:00
Garrett Wollman
eeed58d824 Documentation fix: delete references to T/TCP as they are not relevant 
to the case in question.  (Said case being quite clearly defined by
the basic TCP specification, RFC 792.)
 1998-05-09 20:50:37 +00:00
Dag-Erling Smørgrav
49778f39ac Arrggghhhh... forgot to strip the / after outputting %2f 1998-05-09 08:56:07 +00:00
Dag-Erling Smørgrav
757c6582c1 Map a leading / in -c argument to %2f instead of dropping it silently. 
PR:		bin/6558
 1998-05-09 08:41:23 +00:00
Bill Fenner
5e92200712 Describe absolute paths in FTP URL's. This patch has been sitting 
around uncommitted for 2 months =(
 1998-05-09 07:33:09 +00:00
Philippe Charnier
306005e78c .Sh AUTHOR -> .Sh AUTHORS. Use .An/.Aq. 1998-03-23 07:48:45 +00:00
John Birrell
5362bb64c5 gcc wants to see long long variables passed to %q printf formats, so 
cast to that instead of quad_t.
 1998-02-20 05:11:42 +00:00
John Birrell
2ffc763234 time() needs a pointer to a time_t, but tv_sec in a timeval is a 
long (yuk). So give time() what it wants and let the compiler
promote the variable when it is assigned to tv_sec.
 1998-02-20 05:08:53 +00:00
Bill Fenner
18d1eb6776 Suggest using "-b" or "-t" if the connection is reset (typical symptom 
of broken TCP stack).
 1997-11-22 01:00:42 +00:00
John Polstra
a5bd62c599 Add missing arguments detected by "-Wformat". 1997-11-18 03:27:34 +00:00
David E. O'Brien
e3f8c97942 style-police got me. :-) 
Also fixed some "didn't make the most sense" problems.

Submitted by:	bde
 1997-11-12 04:39:33 +00:00
David E. O'Brien
e4b172e407 If "-l" is given, make sure the target of the link exist. 1997-11-08 22:15:55 +00:00
Andrey A. Chernov
9e8930055e Change diags to see what part of url actually parsed, see 
previous change about #? parms
 1997-11-01 05:47:41 +00:00
Andrey A. Chernov
4ad97ec183 Don't search for ':' port in ?# params part 
Require to correctly parse http://...cgi?http://...
 1997-10-31 23:33:44 +00:00
Bill Fenner
21d466b21a Interpret multiple slashes as a single slash in ftp: URL's. This 
is contrary to RFC1738, which specifies that empty elements between
slashes really mean to send a CWD command with an empty argument.
Most FTP servers consider this an error, so strict RFC compliance
is advisable in this case.
 1997-10-08 18:43:53 +00:00
Bill Fenner
a5cd8a3572 Use the RFC1738 interpretation of ftp: URL's, i.e. CWD to each 
slash-seperated element of the URL and then RETR the last element.

PR:		bin/4670
Reviewed by:	wollman
 1997-10-06 01:09:56 +00:00
Martin Cracauer
3e27c094e4 fetch(3) doesn't get asctime(3) format, wrong length assumed 
PR:		bin/4625
Submitted by:	"Timo J. Rinne" <tri@pooh.tky.hut.fi>
Obtained from:bin/4625
 1997-09-28 11:25:59 +00:00
Satoshi Asami
e5ed1f5994 Be consistent with other utilities in spelling "Kbytes". (Actually, 
there even was one of them about ten lines above in the same file.)

Reminded by:	joerg's commit to usr.bin/ftp/ftp.c rev 1.9
 1997-08-19 01:57:53 +00:00
Andrey A. Chernov
f4e4504f06 Add -t option which turns T/TCP off as workaround for some broken servers 
Submitted by: Marc Slemko <marcs@znep.com>
 1997-08-05 20:18:39 +00:00
Garrett Wollman
12b7829739 Work around a brokenness in the HTTP spec by generating the Host header 
for ftp: URLs as well.  This can't possibly be done in the general case,
but since we only claim to support http and ftp, we'll adhere to the
bogus requirement (RFC 2068, s. 14.25) anyway...  >sigh<

Submitted by: =?iso-8859-1?Q?=C5ge_R=F8bekk?= <aagero@aage.priv.no>
 1997-07-26 20:18:43 +00:00
Garrett Wollman
efd4ad4cd2 Fix the error message for when mirroring and restarting are rejected 
due to the output not being a regular file.  Also split the error message
in any error case here to be on two lines to lessen the likelihood of it
being too long to fit on just one.
 1997-07-26 20:00:05 +00:00
Garrett Wollman
2ec04c7076 Implement HTTP 1.1's ``chunked'' Transfer-Encoding (ick). This hasn't 
been extensively tested, but I now can successfully retrieve
<http://www.apache.org/index.html>, so I guess that's a victory of some
sort.

Also move the initialization of ``autherror'' to hopefully eliminate
the reported loop involving authentication.  Still need to implement
MD5 digest authentication.
 1997-07-26 19:25:56 +00:00
Garrett Wollman
0c8cc99e5b Provide a new `-b' flag to work around some broken HTTP/TCP implementations 
that can't deal with a half-closed connection.
 1997-07-25 19:35:44 +00:00
Philippe Charnier
b28977d169 One newline should be enough. 1997-07-02 06:28:32 +00:00
Philippe Charnier
0bd47384ea Do not use argv[0] in usage(). 1997-07-01 06:37:34 +00:00
Andrey A. Chernov
5e04857ec1 Now understand password in ftp:// URLs 1997-05-31 14:45:41 +00:00
John-Mark Gurney
27e0fa20aa fix a bug in processing of FTP_TIMEOUT environment variable. 
Closes PR#2947

Submitted-by: Peter Childs <pjchilds@imforei.apana.org.au>
 1997-03-11 15:13:28 +00:00
John-Mark Gurney
817eca811d fix restarting http transfers and understanding response from server. 
Slight modification of patch by wollman.

Closes PR#2870

Reviewed by:	wollman
Submitted by:	Marc Slemko <marcs@znep.com>
 1997-03-06 20:01:32 +00:00
John-Mark Gurney
c1599df8b3 fix a couple problems with fetch: 
. don't try to interpet a colon in the pathname as a port number
	. don't report an errno message when one don't exist
 1997-03-06 10:01:54 +00:00
Bill Fenner
8a2f8e395a Fix FTP_PROXY to use user@host[@port] for FTP proxy and eliminate 
undocumented FTP_PROXY_USER
Make FTP file errors contian hostname and path.
Pass the FTP port to libftp.
Partially un-HTMLify error messages returned from server
Handle "HTTP NNN" instead of "HTTP/V.vv NNN" response sent by
	pre-HTTP/1.0 servers

Reviewed by:		wollman
 1997-03-05 18:57:16 +00:00
Wolfram Schneider
bc5d5816e7 These man page will appear in 2.2, not 3.0. 1997-02-22 23:43:32 +00:00
Peter Wemm
c115df18cd Revert $FreeBSD$ to $Id$ 1997-02-22 19:58:13 +00:00
John-Mark Gurney
429100430f Fix fetch so that all status information is sent to stderr, prevents 
corruption of file being recieved to stdout.

Submitted-by: Kent Vander Velden <graphix@iastate.edu>

Closes PR#2739
 1997-02-17 04:11:47 +00:00
John-Mark Gurney
88e8266d80 Modified the display() function to recognize when the output it going to a 
file.  When it is don't do the percentage update, but still display the file
size and rate.

Closes PR#2725

Reviewed by:	wollman
 1997-02-14 19:08:18 +00:00
Garrett Wollman
857292949c Fix remote request for HTTP proxies. Should close PR#2670. 1997-02-11 20:46:06 +00:00
Garrett Wollman
cb0cd746b5 Fix PR#2700: report correct transmission speed, and restart transfers 
from the restart point, not the end of the file (blush!).

Submitted by:	John-Mark Gurney <jmg@nike.efn.org>
 1997-02-10 18:49:42 +00:00
Garrett Wollman
df6c4acc3c Fix off-by-one error in proxy host:port parser. 
Submitted by:	fenner
 1997-02-07 17:55:01 +00:00
Garrett Wollman
76dafb8954 Some bug-fixes, clean-ups, and one new feature: 
- Fix the bug with URIs of the form ftp://host/filename.
- Fix some more string-termination bugs in util.c.
- Use safe_malloc() rather than testing the return value of
  regular malloc() in 15 places.
- Implement HTTP authentication, for both servers and proxies.
  Currently only ``basic'' authentication is supported; This Is A Bug
  (but less of one tjhan nmot supporting any authentication).

I think there is only one more feature which is required for full
HTTP/1.1 support, which is Transfer-Encoding: chunked; this should
not be toohard, but it isn't very important, either.
 1997-02-05 19:59:18 +00:00
Bruce Evans
6525961b64 Null-terminate the string in percent_decode() so that we don't depend on 
malloc() returning zeroed storage or dump core while starting up.

Found-by:	/etc/malloc.conf ->AJ
 1997-02-02 09:16:37 +00:00
Garrett Wollman
2494e810b8 Some fixes for HTTP: 
1) Implement redirects (or try to, at least).
2) Implement automatic retry after 503 errors when Retry-After is given.
3) Implement a -a flag to enable both of these behaviors.
4) Recognize Transfer-Encoding headers and emit a warning that the file
  is likely to be damaged.
5) Bug fix: only write the amount of data we read.
6) Actually document some of these.
7) Fix the usage message to display flags in semi-alphabetical order.
 1997-01-31 19:55:51 +00:00
Garrett Wollman
78be319939 Here is my long-threatened revamping of fetch. Jean-Marc probably won't 
recognize it any more.  This makes the following significant changes:

- The main body of the program doesn't know a thing about URIs,
  HTTP, or FTP.  This makes it possible to easily plug in other
  protocols.  (The next revision will probably be able to dynamically
  add new recognizers.)

- There are no longer arbitrary timeouts for the protocols.  If you want
  to set one for yourself, use the environment variables.

- FTP proxies are now supported (if I implemented it right).

- The HTTP implementation is much more complete, and can now do restarts,
  preserve modtimes, and mrun in mirror mode.  It's not yet up to 1.1,
  but it's getting there.

- Transaction TCP is now used for sending HTTP requests.  The HTTP/1.1 syntax
  for requesting that the connection be closed after one request is
  implemented.

In all of this, I have doubtless broken somebody.  Please test it and tell me
about the bugs.
 1997-01-30 21:43:44 +00:00
Jordan K. Hubbard
ae0fde8e14 Handle timeouts in a slightly less baroque way. 1997-01-17 12:52:12 +00:00
Wolfram Schneider
da1ff3cb8a Sort cross references. 1997-01-15 23:25:55 +00:00
Jordan K. Hubbard
1130b656e5 Make the long-awaited change from $Id$ to $FreeBSD$ 
This will make a number of things easier in the future, as well as (finally!)
avoiding the Id-smashing problem which has plagued developers for so long.

Boy, I'm glad we're not using sup anymore.  This update would have been
insane otherwise.
 1997-01-14 07:20:47 +00:00
Andrey A. Chernov
528baa6dcc Report net connection error via hstrerror(h_errno) now instead of 
simple fact that can't connect
 1996-11-14 07:36:26 +00:00
Andrey A. Chernov
c7a626a567 Change to new ftpLogin interface 
Now clearly say: Not logged in
instead of old: Broken pipe
(note it was original reason for all my libftpio changes)
 1996-11-14 07:06:25 +00:00
Peter Wemm
e721a581cb Cosmetic bugfix. fetch was modifying it's argv[] strings in place, which 
caused ps(1) to show strange things..
 1996-11-10 14:46:50 +00:00
Poul-Henning Kamp
5745420bb0 bin/1933: don't print negative % for huge files (80Mb) 1996-10-31 14:24:35 +00:00
Adam David
bf2226a7db old host:/filename syntax was broken 
implement /filename syntax for completeness
 1996-10-24 00:15:44 +00:00
Jean-Marc Zucconi
b84aaff96e The -h flag implies ftp = 1. 1996-10-06 00:44:24 +00:00
Wolfram Schneider
e0e5145ce6 add missing comma(s) in .Xr macros 1996-09-23 22:24:39 +00:00
Peter Wemm
a43a248c32 Attempt to untangle the timeout code a bit, also make the default ftp 
and http timeouts the same, since when using a http proxy to do ftp
transfers, the http timeout was being used for what is coming in via
ftp.
 1996-09-19 18:07:24 +00:00
Peter Wemm
02fa2cc705 Make fetch 64-bit safe, corresponding to libftpio changes. 
Submitted by: Jason Thorpe <thorpej@nas.nasa.gov>, PR#1641
 1996-09-19 17:31:34 +00:00
Jordan K. Hubbard
f7c176c92b Remove the code which has restart looking at the modtime. That's just 
moronic!  We don't care about the time in restart mode, only the size.
It's *mirror* mode which cares about modtimes.
 1996-09-10 19:49:41 +00:00