d/freebsd-skq
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
258,886 Commits 4 Branches 49 Tags 2 GiB
71776d6719
Commit Graph

426 Commits

Author SHA1 Message Date
Dag-Erling Smørgrav
930105c1e9 style(9): add parentheses to sizeof even when not strictly required. 
MFC after:	3 days
 2003-01-28 08:04:40 +00:00
John W. De Boskey
9015b953d6 Fix signed/unsigned comparison warning/error from 'make release' 2003-01-28 00:33:53 +00:00
Dag-Erling Smørgrav
0ab4a51e4e Document .netrc support. 2003-01-22 18:27:25 +00:00
Dag-Erling Smørgrav
9f808a4dd0 Hook in the .netrc code + don't use pointers as if they were booleans. 2003-01-22 17:54:49 +00:00
Dag-Erling Smørgrav
07350d12cf Experimental support for .netrc. 2003-01-22 17:53:58 +00:00
Dag-Erling Smørgrav
02e6bec10b Initialization paranoia. 2003-01-22 17:53:15 +00:00
Dag-Erling Smørgrav
591c019988 A negative offset means "get it all". 2003-01-22 17:52:27 +00:00
Dag-Erling Smørgrav
f8020ddefe Set auto-retry mode to avoid some spurious errors. 
Submitted by:	Andre Albsmeier <andre.albsmeier@siemens.com>
MFC after:	7 days
 2003-01-03 02:45:10 +00:00
Jens Schweikhardt
57bd0fc6e8 english(4) police. 2002-12-27 12:15:40 +00:00
Jake Burkholder
7b628f43e0 Reduce WARNS level to 3 in the non-crypto case to fix make release on sparc64. 
This may only be turning up now to changes in the cpp predefines for sparc64,
which may be turning on more strict alignment checking.

Approved by:	re
 2002-12-04 04:34:59 +00:00
Dag-Erling Smørgrav
d5216a4fb7 Implement and document support for an HTTP_REFERER environment variable. 
PR:		28171
Submitted by:	Andre Albsmeier <andre.albsmeier@mchp.siemens.de>
Approved by:	re (bmah)
MFC after:	1 week
 2002-11-28 12:07:15 +00:00
Bill Fenner
d5c24aa856 Fix two typos. 2002-11-08 19:10:05 +00:00
Dag-Erling Smørgrav
a49b853d94 Use __func__ and break a long line. 2002-10-30 15:01:29 +00:00
Dag-Erling Smørgrav
7504527ed2 Fix a bug in fenner's _fetch_writev() patch (rev 1.36) 
Submitted by:	fenner
 2002-10-30 14:25:00 +00:00
Dag-Erling Smørgrav
66b8df5fdf Recognize the (incorrect) error code a MediaHawk server sends in reply to 
unrecognized commands such as MDTM.

Requested by:	Stephen Roome <stephen_roome@pepcross.com>
 2002-10-30 06:06:16 +00:00
Dag-Erling Smørgrav
75c3973ab8 Clarify my feelings towards fetch / libfetch. 2002-10-30 04:53:58 +00:00
Dag-Erling Smørgrav
bb13d0af67 Recommit the non-broken parts of 1.34 and 1.37. 
Change the type and name of a variable introduced in 1.33.
 2002-10-30 04:43:00 +00:00
Dag-Erling Smørgrav
cab8495092 Don't forget to set the error flag when _fetch_read() returns -1. 2002-10-30 04:42:01 +00:00
Warner Losh
a4a37038bb Reinstate revs 1.35-36 and 1.38. Revisions 1.34 and 1.37 were specifically 
the root cause of the bus errors I was experiencing.

Submitted by:	fenner
Tested by:	obrien
Prompted by:	peter
 2002-10-30 00:17:16 +00:00
David E. O'Brien
b68fbebd5a Fix `pkg_add -r' by backing out revs 1.34-1.38. 
Revs 1.37-8 produce a bus error in some environments.
Revs 1.34-6 do not bus error, but write corrupted files.
 2002-10-29 12:17:43 +00:00
Dag-Erling Smørgrav
cf2d89d98b libfetch is now WARNS5 clean in the non-SSL case. 2002-10-28 10:37:31 +00:00
Dag-Erling Smørgrav
32a4a82829 Fix an off-by-one error (> where >= should have been used) which caused 
_fetch_writev() to incorrectly report EPIPE in certain cases.

Also fix a number of const warnings by using __DECONST(), plus a signed /
unsigned comparison by casting the rhs to ssize_t.

Submitted by:	fenner, Craig Rodrigues <rodrigc@attbi.com>
 2002-10-28 10:19:03 +00:00
Bill Fenner
8497092d00 Up WARNS to 3 if not building with crypto. 
Approved by:	des
 2002-10-28 01:41:28 +00:00
Dag-Erling Smørgrav
1a5424b137 Slight amendment to rev 1.34: instead of considering any short read an 
error, only report an error if no data was read at all (unless len was
0 to start with).  Otherwise, the final read of practically any transfer
will end in a fatal error.
 2002-10-27 17:20:49 +00:00
Dag-Erling Smørgrav
2761348f78 Introduce _fetch_writev(), which is the conn_t version of writev(2). In 
the SSL case, it is no different from the old _fetch_write(), but in the
non-SSL case it uses writev(2) to send the entire vector as a single
packet (provided it can fit in one packet).  Implement _fetch_write()
and _fetch_putln() in terms of _fetch_writev().

This should improve performance in the non-SSL case (by reducing protocol
overhead) and solve the problem where too-smart-for-their-own-good
firewalls reject FTP packets that do not end in CRLF.

PR:		bin/44123
Submitted by:	fenner
 2002-10-27 16:11:21 +00:00
Dag-Erling Smørgrav
9f788e9c90 Eliminate two cases of undefined behaviour: total in _fetch_write() was 
not initialized before use, and _http_growbuf() did not return a value
on success.

Reported by:	Peter Edwards <pmedwards@eircom.net>
MFC after:	2 weeks
 2002-10-27 15:43:40 +00:00
Dag-Erling Smørgrav
e24f60e74f Back out the previous commit, and fix the bug rather than try to hide its 
symptoms: make timeouts and short transfers fatal, and set errno to an
appropriate value (ETIMEDOUT for a timeout, EPIPE for a short transfer).

MFC after:	2 weeks
 2002-10-27 15:08:21 +00:00
Nate Lawson
13cc1c8394 The FTP connection caching needs a better interface -- connections are 
closed through _fetch_close() which is the only one who knows the connection
REALLY was closed (since ref -> 0).  However, FTP keeps its own local
cached_connection and checks if it is valid by comparing it to NULL.  This
is bogus since it may have been freed elsewhere by _fetch_close().

This change checks if we are closing the cached_connection and the ref is 1
(soon to be 0).  If so, set cached_connection to NULL so we don't
accidentally reuse it.  The REAL fix should be to move connection caching
to the common.c level (_fetch_* functions) and NULL the cache(s) in
_fetch_close().  Then all layers could benefit from caching.
 2002-10-25 01:17:32 +00:00
David E. O'Brien
513829ec5e Use a warns setting we can catch regressions with. 2002-10-20 23:50:28 +00:00
Tim J. Robbins
9d649c1fd0 When recycling a cached connection, increment the reference count so that 
the heap block does not get freed and reused. This should fix the
pkg_add -r crashes that have been happening for months.
 2002-10-20 10:36:27 +00:00
Ruslan Ermilov
721d559c03 HTTP authentication got broken in rev. 1.49. 2002-10-03 10:42:19 +00:00
Peter Wemm
224af215a6 Zap now-unused SHLIB_MINOR 2002-09-28 00:25:32 +00:00
Alfred Perlstein
a6756ecc22 Fix an infinite loop when _fetch_read() can return 0 (if the 
connection is broken), take this into account and return at this
point.
 2002-09-20 21:50:57 +00:00
Bill Fenner
40cfbfd508 Make _fetch_connect() always set the error code. 
Tell ftp that _fetch_connect() always sets the error code (http already knew)
 2002-09-17 05:54:33 +00:00
Dag-Erling Smørgrav
190c185e22 Surround prototypes with __BEGIN_DECLS / __END_DECLS. 
PR:		misc/40399
MFC after:	3 days
 2002-07-22 16:11:39 +00:00
Hajimu UMEMOTO
f8fa093e82 Cope with 2292bis-01 getaddrinfo (no NI_WITHSCOPEID, always attach 
scope identifier).

Approved by:	des
MFC after:	3 weeks
 2002-07-02 11:09:02 +00:00
Dag-Erling Smørgrav
66ffb8a371 Reintroduce debugging code that somehow got lost in a previous revision. 2002-06-24 12:18:41 +00:00
Dag-Erling Smørgrav
a7a4510fd8 Fix incorrect library ordering. I thought I'd committed this already... 2002-06-21 09:56:38 +00:00
Dag-Erling Smørgrav
f8bd33a0ae Don't try to dereference conn when we know it's NULL. 2002-06-19 08:36:00 +00:00
Dag-Erling Smørgrav
f606d589b9 Add a reference count to struct fetchconn so we don't prematurely close and 
free a cached FTP connection.
 2002-06-11 11:27:28 +00:00
Ruslan Ermilov
4093807dd9 libfetch now depends on libcrypto and libssl. 2002-06-06 13:45:46 +00:00
Dag-Erling Smørgrav
3070f6cb06 Make SSL support conditional on NOCRYPT. 2002-06-05 21:35:35 +00:00
Dag-Erling Smørgrav
d9615d7da4 During buildworld, "regular" libraries are built before crypto stuff, so 
libfetch can't depend on lib{crypto,ssl}.  Move the dependency to fetch
until we can figure out how to fix this.
 2002-06-05 21:25:33 +00:00
Dag-Erling Smørgrav
111e251009 Add SSL support + slight cleanup. 
Submitted by:	Henry Whincup <henry@techiebod.com> (in principle)
 2002-06-05 12:46:36 +00:00
Dag-Erling Smørgrav
9601e333a8 Wrap everything in struct connection, and enforce timeouts everywhere 
(except for DNS operations).  Always use funopen() for HTTP, to support
both timeouts and SSL.
 2002-06-05 12:19:08 +00:00
Dag-Erling Smørgrav
4dc0da3f3e Add the necessary dependencies for SSL. 2002-06-05 11:38:19 +00:00
Dag-Erling Smørgrav
ccdd94bdd9 Rename struct cookie to struct httpio to avoid confusion (it's not an HTTP 
cookie) and increase symmetry with equivalent FTP code.
 2002-06-05 10:31:01 +00:00
Dag-Erling Smørgrav
d3b03a9006 Add comments to struct cookie. 2002-06-05 10:27:24 +00:00
Dag-Erling Smørgrav
3f4823c55d Fix a bug I introduced in the chunk decoder in the previous commit.. 2002-06-05 10:23:19 +00:00
Dag-Erling Smørgrav
dea29ca1d5 First step towards SSL support: wrap connections in a 'struct connection' 
which contains the socket descriptor, the input buffer and (yet unused)
SSL state variables.  This has the neat side effect of greatly improving
reentrance (though we're not *quite* there yet) and opening the door to
HTTP connection caching.

This commit is inspired by email conversations with and patches from
Henry Whincup <henry@techiebod.com> last fall.
 2002-06-05 10:05:03 +00:00
Mark Murray
4cd0119367 Do not use __progname directly (except in [gs]etprogname(3)). 
Also, make an internal _getprogname() that is used only inside
libc. For libc, getprogname(3) is a weak symbol in case a
function of the same name is defined in userland.
 2002-03-29 22:43:43 +00:00
Poul-Henning Kamp
06229ad2af Modernize my email address 2002-03-25 13:53:46 +00:00
Dag-Erling Smørgrav
5a51c23be3 Switch to a self-starting allocation scheme. 2002-02-05 22:15:16 +00:00
Dag-Erling Smørgrav
e19e6098b3 Reindent, and add parentheses to return statements. Some functions in 
ftp.c and http.c now have exceedingly long lines due to deep nesting;
this will be corrected by reorganizing the code in a later revision.
 2002-02-05 22:13:51 +00:00
Dag-Erling Smørgrav
551858f0b0 Eliminate a redundant loop. 2002-02-05 21:32:16 +00:00
Dag-Erling Smørgrav
e6f0a33e68 Check the return value from read() when reading the CR/LF at the end of a 
chunk.

PR:		bin/33608
MFC after:	2 weeks
 2002-01-20 19:53:12 +00:00
Dag-Erling Smørgrav
e0583e0c23 Mark uploads as O_WRONLY, not O_RDONLY. 
PR:		misc/34043
MFC after:	2 weeks
 2002-01-20 19:52:25 +00:00
Dag-Erling Smørgrav
d6811c6851 Remove long-obsolete README. 2002-01-01 16:27:26 +00:00
Dag-Erling Smørgrav
a8e9bd8750 In verbose mode, display the full error message from the server, stripping 
it of HTML tags and comments.

PR:		bin/32989
MFC after:	1 week
 2002-01-01 16:25:29 +00:00
Dag-Erling Smørgrav
f67efa37d6 Remove VT100 escapes from debugging messages now that they're enabled by 
default.

PR:		32988
MFC after:	3 days
 2002-01-01 14:48:09 +00:00
Dag-Erling Smørgrav
3c4b4d83f7 Don't build with NDEBUG. 
PR:		bin/32615
MFC after:	1 week
 2001-12-09 15:06:38 +00:00
Dag-Erling Smørgrav
7f807cb867 Conditionalize some debugging code that didn't use the DEBUG macro. 
MFC after:	1 week
 2001-12-09 15:05:58 +00:00
Dag-Erling Smørgrav
7eb2f34d73 Introduce a fetchDebug global. Change the DEBUG macro so it only runs the 
debugging code if fetchDebug is set.

PR:		bin/32615
MFC after:	1 week
 2001-12-09 15:05:19 +00:00
Dag-Erling Smørgrav
6598654546 Properly handle being redirected to an FTP URL. 
Also fix a couple of not-so-minor bugs:
 - missing case for HTTP_SEE_OTHER
 - incorrect definition of HTTP_NEED_PROXY_AUTH

PR:		20259
MFC after:	2 weeks
 2001-12-04 01:12:51 +00:00
Dag-Erling Smørgrav
89db7b8c6d Add prototype for _ftp_request(). 2001-12-04 01:10:34 +00:00
Dag-Erling Smørgrav
102a87c1e1 Reorganize to reduce code duplication. 2001-12-04 01:09:19 +00:00
Dag-Erling Smørgrav
5b31422b36 Add NO_WERROR so the build won't die because of discarded qualifiers 2001-10-19 10:08:38 +00:00
Dag-Erling Smørgrav
93ba13c1bb Back out part of previous commit which was gcc-centric 2001-10-19 10:08:05 +00:00
Dag-Erling Smørgrav
f573a5fc94 Tons of type, style and warning fixes that have been rotting in my tree for 
ages - some of which wouldn't be necessary if gcc wasn't broken or TPTB were
willing to do something (-fno-builtin) about it.
 2001-10-18 08:29:26 +00:00
Ruslan Ermilov
32eef9aeb1 mdoc(7) police: Use the new .In macro for #include statements. 2001-10-01 16:09:29 +00:00
Matthew Dillon
cecb889f1d Add __FBSDID()s to libfetch 2001-09-30 21:36:09 +00:00
Dag-Erling Smørgrav
98b8c4cb10 Use fseeko() instead of fseek() (u->offset is already an off_t), and mark 
some function arguments as unused.
 2001-09-05 12:22:28 +00:00
Brian Somers
778de35906 Handle snprintf() returning < 0 (not just -1) 
MFC after:	2 weeks
 2001-08-20 15:44:17 +00:00
Brian Somers
5f32890558 Handle snrintf overflows. 
Spotted by: bde
 2001-08-20 13:44:13 +00:00
Brian Somers
2449bf28ad Handle snprintf() returning -1. 
MFC after:	2 weeks
 2001-08-20 12:50:21 +00:00
Stefan Eßer
73b3e4df87 The fix for schemeless and hostless URLs (rev. 1.27) broke the schemeless 
proxy specification, which seems to be valid according to the man page.

Change the logic to consider "hostname:port" a hostname and port instead
of a file URL.

Approved by:	des
 2001-07-28 21:28:14 +00:00
Dag-Erling Smørgrav
5b2ad516e9 Unbreak parsing URLs that have a host part but no document part. 
While we're here, fix a snprintf() usage warning.
 2001-07-17 20:22:33 +00:00
Dima Dorfman
7ebcc426ef Remove whitespace at EOL. 2001-07-15 07:53:42 +00:00
Dima Dorfman
70d51341bf mdoc(7) police: remove extraneous .Pp before and/or after .Sh. 2001-07-09 09:54:33 +00:00
Dag-Erling Smørgrav
ab39353eef Handle shemeless, hostless URLs correctly. 2001-07-08 15:59:15 +00:00
Ruslan Ermilov
a7e16a9d0a mdoc(7) police: fixed markup and typo. 2001-07-05 11:24:26 +00:00
Dag-Erling Smørgrav
9e2a792c46 Don't call _ftp_closefn() upon EOF in _ftp_readfn(); just return 0. This 
fixes a bug in pkg_add(1) (which nobody noticed because of another bug).
 2001-06-11 14:12:46 +00:00
Dag-Erling Smørgrav
6a0cf64bce Implement a HTTP_USER_AGENT environment variable. 
PR:		27669
Submitted by:	Eoin Lawless <eoin@maths.tcd.ie>
 2001-05-27 11:00:36 +00:00
Dag-Erling Smørgrav
06b533d32c Document the authentication callback interface. 
Update RFC references (should have done that ages ago...)
 2001-05-26 19:38:34 +00:00
Dag-Erling Smørgrav
7ec6a14ea8 Bump major number. 2001-05-26 19:37:26 +00:00
Dag-Erling Smørgrav
6490b215b3 Add rudimentary support for an authentication callback function. 2001-05-26 19:37:15 +00:00
Dag-Erling Smørgrav
2b26f942d3 MAXPATHLEN -> PATH_MAX 2001-05-26 19:36:49 +00:00
Dag-Erling Smørgrav
b554dea773 Plug memory leak. 
PR:		27506
 2001-05-26 17:23:38 +00:00
Dag-Erling Smørgrav
ea63a39cef Add some error codes that were added in RFC2616. 2001-05-23 18:52:02 +00:00
Archie Cobbs
38c7e4a631 Apply 'const' liberally. 
Fix some other minor glitches.
 2001-04-24 00:06:21 +00:00
Dima Dorfman
5e75e35cca Grammar police: "its", not "it's", is the possessive form of "it". 2001-04-15 19:53:47 +00:00
Dag-Erling Smørgrav
525be862e6 If the server's reply to the SIZE command is unparseable, reset us->size 
to -1 so the caller will know it's invalid.  This is an MFC candidate.
 2001-04-07 15:26:31 +00:00
Ruslan Ermilov
4a558355e5 MAN[1-9] -> MAN. 2001-03-27 17:27:19 +00:00
Dag-Erling Smørgrav
d5f175ce90 Use high port range by default, and replace the 'h' option with an 'l' option 
that forces the ftp code to use the low (default) port range instead.
 2001-03-24 00:28:57 +00:00
Dag-Erling Smørgrav
e238d2a8da Don't remember an EINTR, since the caller may want to restart the call. 2001-03-07 05:12:14 +00:00
Dag-Erling Smørgrav
4cee73c80f Support lower-case versions of the proxy environment variables. 
PR:		bin/25494
 2001-03-07 04:45:55 +00:00
Ruslan Ermilov
9884911506 mdoc(7) police: fixed broken references. 2001-01-16 11:52:00 +00:00
Ruslan Ermilov
32f6256a49 Prepare for mdoc(7)NG. 2001-01-16 09:08:22 +00:00
Dag-Erling Smørgrav
c86c6f748b The user name for anonymous ftp is now "anonymous". 
Remove the period after the last man page reference.
Add a reference to RFC1635, and sort the RFC references by number.
 2001-01-08 13:46:40 +00:00
Dag-Erling Smørgrav
bf4dd407e3 Back out previous (accidental) commit. 2001-01-08 13:17:21 +00:00
Dag-Erling Smørgrav
80ed165e91 Use "anonymous" rather than "ftp" as login name for anonymous ftp. 
Rather than have a separate (misnamed) FTP_ANONYMOUS_PASSWORD constant, use
FTP_ANONYMOUS_USER (i.e. "anonymous") to construct the anonymous ftp password
if getlogin() fails.
 2001-01-08 13:15:14 +00:00
Dag-Erling Smørgrav
659e0d5ef7 Document FTP_LOGIN. 2000-12-22 18:03:21 +00:00
Dag-Erling Smørgrav
e72f0de24e Check the FTP_LOGIN environment variable before falling back on 
FTP_ANONYMOUS_USER.
 2000-12-22 18:01:40 +00:00
Dag-Erling Smørgrav
1ba8497660 Avoid a segfault (due to an unitialized pointer) when parsing URLs that have 
no scheme or host part.
 2000-12-13 11:21:09 +00:00
Dag-Erling Smørgrav
6c81eb523e Somewhere along the line, I misunderstood the whole FTP_PASSIVE_MODE debate 
and had libfetch selecting passive mode even when FTP_PASSIVE_MODE was not
set at all, which is really quite surprising unless you know about it. So
change it to the agreed default behaviour of selecting passive mode if
FTP_PASSIVE_MODE is set, but not "no".
 2000-12-06 09:23:27 +00:00
Dag-Erling Smørgrav
79aad71d3b Remove some obsolete comments. 2000-12-01 11:05:15 +00:00
Dag-Erling Smørgrav
f9c2053bca Clean up the whitespace encoding code. 2000-12-01 11:04:57 +00:00
Dag-Erling Smørgrav
025bb05a82 Add SCHEME_HTTPS. 2000-12-01 11:04:43 +00:00
Dag-Erling Smørgrav
882974d431 Fix old-style proxy specs: default to FTP if FTP_PROXY was set; only default 
to HTTP if HTTP_PROXY was used instead.
 2000-11-27 13:42:56 +00:00
Dag-Erling Smørgrav
b431558526 Don't try to get the proxy port number from /etc/services. 2000-11-27 13:41:55 +00:00
Dag-Erling Smørgrav
d41c0df8f8 Don't go haywire if the server closes the connection in the middle of a 
multiline response (proper fix this time).
 2000-11-22 14:50:46 +00:00
Dag-Erling Smørgrav
04a80993c1 Revert previous commit, it was somewhat hasty. 2000-11-22 14:44:48 +00:00
Dag-Erling Smørgrav
1baad1a183 Don't go haywire if the server closes the connection during a multiline 
response.
 2000-11-22 14:30:28 +00:00
Ruslan Ermilov
725ab6287f log 2000-11-22 09:23:54 +00:00
Ruslan Ermilov
b82f5db0ac Spell the des's name correctly. 2000-11-14 13:19:26 +00:00
Ruslan Ermilov
b5c508fba3 Use Fx macro wherever possible. 2000-11-14 11:20:58 +00:00
Dag-Erling Smørgrav
e828ada709 Use the documented (and historical) defaults. Centralize the decision logic 
in order to avoid this bug in the future.

Submitted by:	se
 2000-11-10 08:43:40 +00:00
Dag-Erling Smørgrav
d74a913b68 Use CHECK_FLAG 2000-10-29 15:56:10 +00:00
Dag-Erling Smørgrav
23109751fd Stricter error checking in the I/O functions. 2000-10-29 15:52:05 +00:00
Dag-Erling Smørgrav
1c2598aa2e Add CHECK_FLAGS, a macro for (safely) checking if a particular flag is set. 2000-10-29 15:45:31 +00:00
Dag-Erling Smørgrav
91c5104600 Document FTP_PASSIVE_MODE and FTP_PASSWORD. 
PR:		docs/20626
Submitted by:	nik
 2000-10-29 12:58:08 +00:00
Dag-Erling Smørgrav
7ca2f6ae1f Fix this my way. David had absolutely no call overriding MAINTAINER without 
even giving me 24 hours to read his mail and find the bug.
 2000-10-28 20:53:02 +00:00
David E. O'Brien
f18255c68b Install per function manpages so one doesn't mistakenly think we don't 
have manpages for libfetch's functions.
 2000-10-28 20:32:39 +00:00
David E. O'Brien
f03bce74fc Repeat after me, "check to see that a pointer isn't NULL before 
dereferenceing it".  This fixes ``pkg_add -r''.
 2000-10-28 20:27:11 +00:00
Dag-Erling Smørgrav
303fd73aa7 MFS: don't expect result code until you close the data connection 2000-10-27 11:37:21 +00:00
Dag-Erling Smørgrav
c7d40ef296 Use funopen() instead of fdopen(). This fixes three problems: 
- ftpTimeout was not honored when reading actual data, as opposed to
   talking protocol

 - connection caching was broken because _ftp_cached_connect() would see
   the result of the transfer instead of the result of the NOOP.

 - if the RETR succeeded, but an error occurred later (as can happen
   when talking to a proxy), the error would not be detected.

There still remains to register an atexit(3) callback to close the cached
connection gracefully instead of just dropping it on the floor.
 2000-10-22 12:07:28 +00:00
Dag-Erling Smørgrav
23fe6d7a4c If the scheme is HTTP or HTTPS, percent-escape whitespace in the document 
part.

Submitted by:	green
 2000-10-21 14:58:18 +00:00
Dag-Erling Smørgrav
8b9ba4668d Argh! Fix passive mode selection (again) 2000-10-17 19:01:18 +00:00
Dag-Erling Smørgrav
cb5b735396 Use the right user name 2000-10-13 09:36:46 +00:00
Dag-Erling Smørgrav
1df2031c15 Document recent changes in URL parsing and proxy handling 2000-10-12 22:11:26 +00:00
Dag-Erling Smørgrav
bbc5af0c5a ftp.c needs _http_request() 2000-10-12 22:10:43 +00:00
Dag-Erling Smørgrav
1a16ed4c9c Rework proxy handling so that proxies can be specified as URLs. 
As a side effect, remove a lot of duplicate and now redundant code.
 2000-10-12 22:10:26 +00:00
Dag-Erling Smørgrav
59769ab192 Relax URL syntax so that schemeless URLs are supported. 2000-10-12 22:07:49 +00:00
Dag-Erling Smørgrav
be9e1d8a68 Add macros for the names of the URL schemes we support. 2000-10-12 21:59:22 +00:00
Dag-Erling Smørgrav
89d755e156 Make the ENVIRONMENT section more legible, and mention that HTTP_AUTH is 
colon-separated.
 2000-10-12 16:55:52 +00:00
Dag-Erling Smørgrav
a898bb8d0d Work around buggy servers such as NCSA httpd which send an incomplete 
HTTP-Version on the Status-Line (see RFC2616 sections 3.1 and 6.1).
 2000-09-24 12:22:12 +00:00
Dag-Erling Smørgrav
2e83f63844 Document the FTP_PROXY and HTTP_PROXY variables better. 2000-08-31 15:13:22 +00:00
Dag-Erling Smørgrav
ea014d857f Grok 125 replies to passive RETR. This fixes an interoperability bug with 
Microsoft FTP Service.

Reported by:	asmodai, eivind
 2000-08-21 07:18:31 +00:00
Dag-Erling Smørgrav
be6aff992d Fix signedness bogon. 2000-08-02 11:25:21 +00:00
Dag-Erling Smørgrav
0f27c7838c Don't fail if _ftp_stat() reports a protocol error, since that most likely 
means that the server doesn't understand SIZE or MDTM, which should not be a
hard failure.

Submitted by:	ume
 2000-07-27 08:48:48 +00:00
Dag-Erling Smørgrav
63428824c9 If the server reports the size as 0, treat it as unknown. This works around 
a bug in some ftp servers (most notably ftp.vmunix.com) which report the
size of a file correctly in ascii mode, but report it as 0 in binary mode.

Reported by:	asmodai

Also remove an unneeded initialization.
 2000-07-25 14:41:02 +00:00
Dag-Erling Smørgrav
10851dc4ad Centralize the default port finding code. 
Work around YA Apache bug: don't send port in Host: header if it's the
default port.
 2000-07-25 11:45:38 +00:00
Dag-Erling Smørgrav
c78f1cc912 Ignore environment variables that are set but empty. 
Sort out the size / length confusion.  Always try to report the *real* file
size in the url_stat structure, no matter how much of it is actually being
sent, and try to detect inconsistencies between sizes.

Rearrange the request loop to avoid having to add meaningless code just to
silence compiler warnings.

Switch to a more sensible and consistent interface for the _http_parse*()
functions.
 2000-07-21 11:02:43 +00:00
Dag-Erling Smørgrav
e0a9ca38aa Document the proxy support better. 2000-07-21 10:57:15 +00:00
Dag-Erling Smørgrav
dfe7c55fcc The decision to use an HTTP proxy for FTP documents belongs in the FTP code 2000-07-21 10:46:22 +00:00
Dag-Erling Smørgrav
2e88106bd1 Don't send port number to proxy unless it's not the standard FTP port 2000-07-21 10:25:33 +00:00
Dag-Erling Smørgrav
5e3f46b51a Fix the case where the file name is preceded by a lone slash. 2000-07-20 02:03:56 +00:00
Dag-Erling Smørgrav
87b41116cc Don't try to skip to the requested offset if the server returns more data 
than requested.  Instead, inform the caller of the real offset by modifying
the offset field in the original struct url, and let him decide how to handle
the situation.
 2000-07-19 23:43:49 +00:00
Dag-Erling Smørgrav
269532d987 Don't take a failure to stat a file to mean that it doesn't exist, as it may 
be a dynamically created file, e.g. a tarball on servers that support sending
tarballs of directories.
 2000-07-18 07:12:26 +00:00
Dag-Erling Smørgrav
1a5faa1061 Introduce fetchXGet*(), which combine the functionalities of fetchGet*() and 
fetchStat*().  In most cases, either fetchGet*() or fetchXGet*() is a wrapper
around the other; in all cases, calling fetchGet*() is identical to calling
fetchXGet*() with the second argument set to NULL.
 2000-07-17 21:25:00 +00:00
Dag-Erling Smørgrav
c789727701 Don't forget to set file descriptor to -1 after closing it, since the code 
outside the loop inspects it to determine whether or not we succeeded in
retrieving the requested document. This fixes a bug where fetchGetHTTP()
would return a FILE with an invalid file descriptor if it hit the redirect
limit without locating the requested document.
 2000-07-17 21:09:11 +00:00
Dag-Erling Smørgrav
5cd33c40d7 Use a symbolic constant instead of hardcoding 999 2000-07-17 21:00:49 +00:00
Dag-Erling Smørgrav
d999dcac2d Don't forget to declare fetchRestartCalls. 2000-07-17 20:52:06 +00:00
Dag-Erling Smørgrav
a1bb3f482a Introduce the (undocumented) variable fetchRestartCalls, which controls whether 
or not interrupted system calls will be restarted. This fixes a bug where
fetch(1) would hang (potentially forever) if a server stopped responding,
because the signal handler would absorb the user's efforts to interrupt the
transfer.
 2000-07-17 20:49:39 +00:00
Dag-Erling Smørgrav
7c80207e33 Remove fetchContentType(), which disappeared in the HTTP rewrite. 2000-07-17 20:42:13 +00:00
Dag-Erling Smørgrav
e66b380285 #ifdef DEBUG -> #ifndef NDEBUG 
Pointed out by:	ache
 2000-07-16 23:18:44 +00:00
Dag-Erling Smørgrav
6290ee735b Reinsert the MIT copyright, which applies to the base64 code and was 
left out by accident during the rewrite.
 2000-07-16 01:04:10 +00:00
Dag-Erling Smørgrav
f4683775d7 Fix a bug (misplaced continue) that caused redirects to fail. Lots of code 
moved around, but the acutal functional changes are small.
Add support for site-internal redirects (where the Location: header gives a
path instead of an absolute URI)

Pointed out by:	kuriyama
 2000-07-13 09:13:58 +00:00
Dag-Erling Smørgrav
98f5573fdd Fully document the HTTP authorization feature. 
Update the BUGS section to reflect the HTTP rewrite.
 2000-07-12 10:41:29 +00:00
Dag-Erling Smørgrav
e4878e39a3 Extensive rewrite of the HTTP code. The new code is significantly cleaner and 
more robust, and somewhat more efficient. It also handles authorization and
redirects properly, and supports timeouts like the FTP code.

Many thanks to Umemoto-san for his assistance with IPv6 support, both here
and in other parts of libfetch.
 2000-07-12 10:39:56 +00:00
Dag-Erling Smørgrav
e137bceb8f Use fetch_putln() 2000-07-11 23:50:22 +00:00
Dag-Erling Smørgrav
a176302737 Add _fetch_putln() 2000-07-11 23:50:08 +00:00
Dag-Erling Smørgrav
bc2a5e683a Document that basic authorization now kinda works. 2000-07-11 18:13:08 +00:00
Dag-Erling Smørgrav
35f723db8a Rework the authorization code. 
Only send absolute URI if connected to a proxy, since Apache doesn't always
understand absolute URIs.
Clean up some of the debugging output.
 2000-07-11 18:12:41 +00:00
Dag-Erling Smørgrav
ec894321e3 Clean up the debugging output 2000-07-11 18:07:09 +00:00
Hajimu UMEMOTO
7487ef6243 Free chunks obtained by getaddrinfo(3). 
Approved by:	des
 2000-07-10 16:28:28 +00:00
Hajimu UMEMOTO
fa5dce6c26 Make EPSV work again. Separate parsing code of 229 replies from 
the code for 227 and 228.

Submitted by:	des
 2000-07-10 08:41:25 +00:00
Dag-Erling Smørgrav
51e3d46ec0 Don't assume the remote address in a 227 reply is enclosed in parentheses. 2000-07-08 09:27:47 +00:00
Dag-Erling Smørgrav
4d029f13b4 Fix basic authentication, and add proxy authentication. 
Submitted by:	se
 2000-07-08 08:08:58 +00:00
Dag-Erling Smørgrav
893980ad29 Make restart work in active mode, too. 
PR:		bin/18688
Submitted by:	Rudolf Cejka <cejkar@dcse.fee.vutbr.cz>
 2000-06-29 10:44:10 +00:00
Dag-Erling Smørgrav
eac7a1e07f Handle multiline replies properly, instead of kinda-right. 2000-06-28 15:48:26 +00:00
Dag-Erling Smørgrav
cf5af79cac Don't incorrectly report a protocol error when we get a 302 and redirects 
are disabled.
 2000-06-21 09:49:51 +00:00
Dag-Erling Smørgrav
def5f54c85 Back out wes' commit with extreme prejudice. 2000-06-20 13:57:00 +00:00
Wes Peters
bda4ef1aa0 Add implementation of fetchListFTP. 
Reviewed by:	silence
 2000-06-20 05:32:41 +00:00
Dag-Erling Smørgrav
6f9febf93c Hackish support for 302 redirect. 2000-05-26 15:34:42 +00:00
Dag-Erling Smørgrav
d02e84a626 Honor FTP_PASSIVE_MODE so caller doesn't need to. 2000-05-26 12:56:53 +00:00
Dag-Erling Smørgrav
a68f8b580f Output newline after debugging message 2000-05-25 16:50:08 +00:00
Dag-Erling Smørgrav
9a964d6a82 Add (and document) fetchMakeURL() 2000-05-25 16:46:09 +00:00
Dag-Erling Smørgrav
ba101983d5 Dump com_err, it's a stinking crock of shit. 2000-05-25 16:24:31 +00:00
Hajimu UMEMOTO
4ab587f4cf Use $FTP_PASSWORD for FTP password. If $FTP_PASSWORD is not 
found, `yourname@yourhost' is used.
 2000-05-22 13:01:13 +00:00
Hajimu UMEMOTO
28c645cf6a IPv6 support. 2000-05-20 18:23:51 +00:00
Dag-Erling Smørgrav
6efb30c8d0 Better handling of some boundary conditions. 
Submitted by:	ume
 2000-05-19 09:45:42 +00:00
Hajimu UMEMOTO
c97925ad4e Make HTTP_PROXY work for FTP. 
Reported by:	Ben Smithurst <ben@scientia.demon.co.uk>
Reviewed by:	des
 2000-05-15 09:05:36 +00:00
Dag-Erling Smørgrav
40ac28454e Document struct url. 
Document the default values for fetchStat*().
 2000-05-15 08:34:42 +00:00
Dag-Erling Smørgrav
7a5b4b1ba6 Initialize the struct url_stat at the beginning of _fetch_stat_file(). 2000-05-15 08:33:58 +00:00
Dag-Erling Smørgrav
0669702c01 Initialize the struct url_stat at the beginning of fetchStatFTP(). 2000-05-15 08:24:58 +00:00
Dag-Erling Smørgrav
f8f4130b77 Remove unused 'verbose'. 
Initialize the struct url_stat at the beginning of fetchStatHTTP().
 2000-05-15 08:24:29 +00:00
Dag-Erling Smørgrav
89474d129e Add workaround for ftpds with the Y2K MDTM bug 2000-05-11 16:01:03 +00:00
Dag-Erling Smørgrav
96d7319afb Bump major number, since struct url has changed. 
Discussed with:	jdp
 2000-05-11 15:29:26 +00:00
Dag-Erling Smørgrav
9364c53225 Fix hard sentence break. 
Submitted by:	sheldonh
 2000-05-11 15:22:52 +00:00
Dag-Erling Smørgrav
60245e42b0 Reorganize some of the http code and split it into more functions. 
Implement fetchStatHTTP().
Unbungle struct url, and add fetchFreeURL().
Document it.
 2000-05-11 13:31:02 +00:00
Sheldon Hearn
0f5fcaa3f1 Supply only one author name per instance of %A, as per mdoc.samples(7). 
PR:		18465
Submitted by:	Kazu TAKAMUNE <takamune@avrl.mei.co.jp>
 2000-05-10 09:49:04 +00:00
Dag-Erling Smørgrav
068cd2851f While we're at it, add a length field too in case we want to fully implement 
partial fetches later.
 2000-05-07 20:52:40 +00:00
Dag-Erling Smørgrav
3d2a847151 Implement restart 2000-05-07 20:51:31 +00:00
Dag-Erling Smørgrav
807c941c90 Use sizeof more consistently. 2000-05-07 20:02:52 +00:00
Dag-Erling Smørgrav
e6182307e8 Use sizeof more consistently. 
Outline sanity checking of server port spec.
 2000-05-07 20:01:55 +00:00
Dag-Erling Smørgrav
32425dafcd Implement restart. 
Use sizeof more consistently.
Outline sanity checking of server port spec.
 2000-05-07 20:00:12 +00:00
Dag-Erling Smørgrav
5ff28c822b Implement restart 2000-05-07 19:56:07 +00:00
Dag-Erling Smørgrav
fd75cc902e Add offset field to struct url 2000-05-07 19:46:33 +00:00
Alexey Zelkin
15efcd39ae Introduce .Lb macro to libfetch manpage 
Sort .Nm values
Change first column width in errors list (table look much better now)
Remove redundant comma
 2000-04-22 16:08:41 +00:00
Sheldon Hearn
c6ff3a1bf7 Remove single-space hard sentence breaks. These degrade the quality 
of the typeset output, tend to make diffs harder to read and provide
bad examples for new-comers to mdoc.
 2000-03-02 09:14:21 +00:00
Dag-Erling Smørgrav
5d32c97ce9 Fix timestamp handling. 
Submitted by:	bde
 2000-01-26 12:50:46 +00:00
Rodney W. Grimes
d05257b0f2 Replace beforeinstall target with new variables used by .mk system. 
Reviewed by:	marcel, and make world
 2000-01-14 07:57:47 +00:00
Dag-Erling Smørgrav
fc6e9e6539 More old uncommitted patches: implement timeouts at the protocol level. 
Currently only supported for ftp connections.
 2000-01-07 12:58:40 +00:00
Dag-Erling Smørgrav
be0d5ff224 Add error codes for protocol errors. 2000-01-07 12:56:27 +00:00
Dag-Erling Smørgrav
f5f109a068 Patches I've had lying around for several months: 
* Add the 'h' ftp flag (allocate local port in high range)
 * Add the 'd' flag (use direct connection even if proxy is defined)
 * Make sure flags != NULL before calling strchr().
 * Minor changes to some comments.
 2000-01-07 10:59:12 +00:00
Peter Wemm
7f3dea244c $Id$ -> $FreeBSD$ 1999-08-28 00:22:10 +00:00
Bruce Evans
e7720fa877 Fixed prototype for fetchParseURL(). 1999-03-05 18:45:32 +00:00
Dag-Erling Smørgrav
5a5375540a Add MAINTAINER. 1999-01-29 12:15:34 +00:00
Dag-Erling Smørgrav
ce71b7364c Implement and document file list retrieval. 1998-12-21 19:41:50 +00:00
Dag-Erling Smørgrav
d941fd2df9 Don't specify a language to compile_et. 
Rename fetchConnect to _fetch_connect since it's internal.
 1998-12-18 14:32:48 +00:00
Dag-Erling Smørgrav
5aea254ff0 Implement and document fetchStatFTP. 
Update description of struct url_stat in the man page.
Clean up error handling in ftp.c.
 1998-12-16 15:29:03 +00:00
Dag-Erling Smørgrav
3b7a67401b Remove unused header files. Fix a few shadowing bugs. Improve error reporting. 1998-12-16 11:44:31 +00:00
Dag-Erling Smørgrav
0fba3a0005 Add verbose flag, and support functions. 
Brucify the Makefile.
Differentiate atime and mtime in fetch*Stat().
Fix a few pointer bugs.
Tweak some error messages.
Don't #include sys/param.h and stdio.h in fetch.h.
Document that sys/param.h and stdio.h must be #included before fetch.h.
 1998-12-16 10:24:55 +00:00
Dag-Erling Smørgrav
2878b30827 Fix the "Unknown HTTP error" message. 
Make compile_et generate prototypes.
 1998-12-15 12:24:26 +00:00
Dag-Erling Smørgrav
08f9fcbfa8 fetch_err.h is a generated file, so don't look for it in ${.CURDIR}. 1998-11-07 08:59:38 +00:00
Dag-Erling Smørgrav
d8acd8dc5c Second of a series of cleanups to libfetch. 
This commit introduces the following features:

 a) the fetchStat*() functions, which return meta-information for a
    document, such as size, modification time, etc.

 b) the use of the com_err(3) facilities to report errors.

It also fixes a bunch of style bugs and a few logic bugs and somewhat
improves the man page.

Changed files, in alphabetical order:

 Makefile:
  Don't generate macros in {ftp,http}err.c.

  Generate category fields for the error message lists.

  Compile the error table.

  Install fetch_err.h along with fetch.h.

 common.c:
  Remove the _netdb_errstring() macro, and add FETCH_ERR_NETDB to the
  error code in the _netdb_seterr() macro.

  Add categories to the _netdb_errlist table.

  Report errors through the Common Error library.

 common.h:
  Add the DEBUG macros.

  Add prototype for fetchConnect().

  Remove the prototype for _fetch_errstring(), which is local to common.c

  Add a categroy field to struct fetcherr, and define constants for
  error categories.

  Define macros for _{url,netdb,ftp,http}_seterr().

 errors.et: (new file)
  List error categories.

 fetch.3:
  Document the fetchStat*() functions.

  Move the "unimplemented functionality" comments from NOTES to BUGS.

  Document that applications which use libfetch must also use
  libcom_err, and list existing error codes.

  Undocument fetchLastErr{Code,String}.

  Remove the (empty) DIAGNOSTICS section.

  Mention Eugene Skepner in the AUTHORS section.

 fetch.c:
  Move the DEBUG macros to common.c

  Add fetchStat() and fetchStatURL().

  Generate error messages for URL parser errors, and fix a minor bug
  in the parser.

  Use 'struct url' instead of 'url_t'.

  Remove fetchLastErr{Code,String}.

 fetch.h:
  Use 'struct url' instead of 'url_t', and remove the typedef.

  Define struct url_stat (used by fetchStat()).

  Add prototypes for fetchStat*().

  Remove the declarations for fetchLastErr{Code,String}.

  Include fetch_err.h.

 fetch_err.et: (new file)
  Error table for libfetch.

 file.c:
  Add fetchStatFile().

  Use 'struct url' instead of 'url_t'.

 ftp.c:
  Add fetchStatFTP().

  Use 'struct url' instead of 'url_t'.

  Don't use fetchLastErrCode.

 ftp.errors:
  Add categories to all error messages.

 http.c:
  Add fetchStatHTTP().

  Use 'struct url' instead of 'url_t'.

  Don't use fetchLastErr{Code,Text}.

 http.errors:
  Add categories to all error messages.

Prompted by: jkh and Eugene Skepner
Numerous sugestions from: Garett Wollman and Eugene Skepner
 1998-11-06 22:14:08 +00:00
Dag-Erling Smørgrav
842a95cc23 First of a series of cleanups to libfetch. Changed files, in 
alphabetical order:

 Makefile:
  Add common.c to SRCS.

  Make debugging easier by making 'CFLAGS += -DNDEBUG' conditional on DEBUG

  Don't declare struct {ftp,http}err in {ftp,http}err.c; use struct fetcherr
  instead.

 README:
  Remove the todo list, which is out of date anyway.

 common.c: (new file)
  Gather utility functions in this file.

  Merge the error reporting functions intp _fetch_errstring(),
  _fetch_seterr() and _fetch_syserr().

  Set fetchLastErrCode and fetchLastErrText appropriately when fetchConnect
  fails.

 common.h: (new file)
  Gather internal prototypes and structures in this files.

 fetch.3:
  Undocument fetchFreeURL().

  Document a few more known bugs.

  Document fetchLastErrCode and fetchLastErrText.

 fetch.c:
  Add descriptive comments to all functions that lacked them.

  Move fetchConnect() to common.c.

  Obviate the need for fetchFreeURL(), and remove it.

 fetch.h:
  Modify struct url_t so the document part is at the end.

 ftp.c:
  Remove code that is duplicated elsewhere.

 http.c:
  Remove code that is duplicated elsewhere.

Prompted by: jkh
 1998-11-05 19:48:17 +00:00
Dag-Erling Smørgrav
db5f8ceb40 Bring the man page more or less up to date with the code. 1998-09-26 20:42:44 +00:00
Bruce Evans
90152d9d21 Made this actually work when there's an object directory: 
- replaced bogus dependencies of distribution sources on generated
  sources by the same ones that bsd.lib.mk would generate if it knew
  all the sources.  We shoot ourself in the foot by not naming the
  generated (included) sources *.h, so we can just put the generated
  sources in SRCS.
- replaced -I${.CURDIR} by `-I.'.  Here `.' is an alias for ${.OBJDIR}.
  -I${.CURDIR} didn't do anything, since ${.CURDIR} is the default in
  all cases here (it would be necessary for ""-style includes made from
  sources in ${.OBJDIR}.

Don't use `+=' for setting CLEANFILES for the first time.

Added $Id$.
 1998-08-17 20:39:09 +00:00
Dag-Erling Smørgrav
ecc9135299 Commit a bunch of patches that have been accumulating: 
- Fix the README to reflect the new status of the ftp code.
 - Change tons of 'if (xxx < 0)' to 'if (xxx == -1)'
 - Add two new interface functions
 - Fix the Makefile so it actually works (yay!)

Now the manpage is lagging even further behind... :( Next on the todo
list is to clean up the http code.
 1998-08-17 09:30:19 +00:00
Dag-Erling Smørgrav
44bcb03a54 Forgot to remove base64.c from SRCS in the Makefile. 1998-07-12 22:37:45 +00:00
Dag-Erling Smørgrav
f62e5228fd Base64 code (and the MIT copyright) moved to http.c 
FTP STORe and APPEnd added.
FTP proxy support added (untested).
 1998-07-12 22:34:40 +00:00
Dag-Erling Smørgrav
346298f0de FTP retrieve works. 1998-07-11 21:29:08 +00:00
Dag-Erling Smørgrav
8e3986ea36 Too many changes to list. Basically, FTP is nearly there and error 
reporting is kinda sorted out. Now HTTP needs to catch up...
 1998-07-11 18:56:01 +00:00
Dag-Erling Smørgrav
4ca1ab9434 Imported libfetch into the tree. It compiles, but there's still some 
work to do. I especially need help with the man page.
 1998-07-09 16:52:44 +00:00