Commit Graph

50 Commits

Author SHA1 Message Date
Mark Murray
4cd0119367 Do not use __progname directly (except in [gs]etprogname(3)).
Also, make an internal _getprogname() that is used only inside
libc. For libc, getprogname(3) is a weak symbol in case a
function of the same name is defined in userland.
2002-03-29 22:43:43 +00:00
Dag-Erling Smørgrav
e19e6098b3 Reindent, and add parentheses to return statements. Some functions in
ftp.c and http.c now have exceedingly long lines due to deep nesting;
this will be corrected by reorganizing the code in a later revision.
2002-02-05 22:13:51 +00:00
Dag-Erling Smørgrav
e6f0a33e68 Check the return value from read() when reading the CR/LF at the end of a
chunk.

PR:		bin/33608
MFC after:	2 weeks
2002-01-20 19:53:12 +00:00
Dag-Erling Smørgrav
a8e9bd8750 In verbose mode, display the full error message from the server, stripping
it of HTML tags and comments.

PR:		bin/32989
MFC after:	1 week
2002-01-01 16:25:29 +00:00
Dag-Erling Smørgrav
f67efa37d6 Remove VT100 escapes from debugging messages now that they're enabled by
default.

PR:		32988
MFC after:	3 days
2002-01-01 14:48:09 +00:00
Dag-Erling Smørgrav
7f807cb867 Conditionalize some debugging code that didn't use the DEBUG macro.
MFC after:	1 week
2001-12-09 15:05:58 +00:00
Dag-Erling Smørgrav
6598654546 Properly handle being redirected to an FTP URL.
Also fix a couple of not-so-minor bugs:
 - missing case for HTTP_SEE_OTHER
 - incorrect definition of HTTP_NEED_PROXY_AUTH

PR:		20259
MFC after:	2 weeks
2001-12-04 01:12:51 +00:00
Dag-Erling Smørgrav
f573a5fc94 Tons of type, style and warning fixes that have been rotting in my tree for
ages - some of which wouldn't be necessary if gcc wasn't broken or TPTB were
willing to do something (-fno-builtin) about it.
2001-10-18 08:29:26 +00:00
Matthew Dillon
cecb889f1d Add __FBSDID()s to libfetch 2001-09-30 21:36:09 +00:00
Dag-Erling Smørgrav
6a0cf64bce Implement a HTTP_USER_AGENT environment variable.
PR:		27669
Submitted by:	Eoin Lawless <eoin@maths.tcd.ie>
2001-05-27 11:00:36 +00:00
Dag-Erling Smørgrav
6490b215b3 Add rudimentary support for an authentication callback function. 2001-05-26 19:37:15 +00:00
Archie Cobbs
38c7e4a631 Apply 'const' liberally.
Fix some other minor glitches.
2001-04-24 00:06:21 +00:00
Dag-Erling Smørgrav
4cee73c80f Support lower-case versions of the proxy environment variables.
PR:		bin/25494
2001-03-07 04:45:55 +00:00
Dag-Erling Smørgrav
e828ada709 Use the documented (and historical) defaults. Centralize the decision logic
in order to avoid this bug in the future.

Submitted by:	se
2000-11-10 08:43:40 +00:00
Dag-Erling Smørgrav
d74a913b68 Use CHECK_FLAG 2000-10-29 15:56:10 +00:00
Dag-Erling Smørgrav
1a16ed4c9c Rework proxy handling so that proxies can be specified as URLs.
As a side effect, remove a lot of duplicate and now redundant code.
2000-10-12 22:10:26 +00:00
Dag-Erling Smørgrav
a898bb8d0d Work around buggy servers such as NCSA httpd which send an incomplete
HTTP-Version on the Status-Line (see RFC2616 sections 3.1 and 6.1).
2000-09-24 12:22:12 +00:00
Dag-Erling Smørgrav
be6aff992d Fix signedness bogon. 2000-08-02 11:25:21 +00:00
Dag-Erling Smørgrav
10851dc4ad Centralize the default port finding code.
Work around YA Apache bug: don't send port in Host: header if it's the
default port.
2000-07-25 11:45:38 +00:00
Dag-Erling Smørgrav
c78f1cc912 Ignore environment variables that are set but empty.
Sort out the size / length confusion.  Always try to report the *real* file
size in the url_stat structure, no matter how much of it is actually being
sent, and try to detect inconsistencies between sizes.

Rearrange the request loop to avoid having to add meaningless code just to
silence compiler warnings.

Switch to a more sensible and consistent interface for the _http_parse*()
functions.
2000-07-21 11:02:43 +00:00
Dag-Erling Smørgrav
87b41116cc Don't try to skip to the requested offset if the server returns more data
than requested.  Instead, inform the caller of the real offset by modifying
the offset field in the original struct url, and let him decide how to handle
the situation.
2000-07-19 23:43:49 +00:00
Dag-Erling Smørgrav
1a5faa1061 Introduce fetchXGet*(), which combine the functionalities of fetchGet*() and
fetchStat*().  In most cases, either fetchGet*() or fetchXGet*() is a wrapper
around the other; in all cases, calling fetchGet*() is identical to calling
fetchXGet*() with the second argument set to NULL.
2000-07-17 21:25:00 +00:00
Dag-Erling Smørgrav
c789727701 Don't forget to set file descriptor to -1 after closing it, since the code
outside the loop inspects it to determine whether or not we succeeded in
retrieving the requested document. This fixes a bug where fetchGetHTTP()
would return a FILE with an invalid file descriptor if it hit the redirect
limit without locating the requested document.
2000-07-17 21:09:11 +00:00
Dag-Erling Smørgrav
e66b380285 #ifdef DEBUG -> #ifndef NDEBUG
Pointed out by:	ache
2000-07-16 23:18:44 +00:00
Dag-Erling Smørgrav
6290ee735b Reinsert the MIT copyright, which applies to the base64 code and was
left out by accident during the rewrite.
2000-07-16 01:04:10 +00:00
Dag-Erling Smørgrav
f4683775d7 Fix a bug (misplaced continue) that caused redirects to fail. Lots of code
moved around, but the acutal functional changes are small.
Add support for site-internal redirects (where the Location: header gives a
path instead of an absolute URI)

Pointed out by:	kuriyama
2000-07-13 09:13:58 +00:00
Dag-Erling Smørgrav
e4878e39a3 Extensive rewrite of the HTTP code. The new code is significantly cleaner and
more robust, and somewhat more efficient. It also handles authorization and
redirects properly, and supports timeouts like the FTP code.

Many thanks to Umemoto-san for his assistance with IPv6 support, both here
and in other parts of libfetch.
2000-07-12 10:39:56 +00:00
Dag-Erling Smørgrav
35f723db8a Rework the authorization code.
Only send absolute URI if connected to a proxy, since Apache doesn't always
understand absolute URIs.
Clean up some of the debugging output.
2000-07-11 18:12:41 +00:00
Dag-Erling Smørgrav
4d029f13b4 Fix basic authentication, and add proxy authentication.
Submitted by:	se
2000-07-08 08:08:58 +00:00
Dag-Erling Smørgrav
cf5af79cac Don't incorrectly report a protocol error when we get a 302 and redirects
are disabled.
2000-06-21 09:49:51 +00:00
Dag-Erling Smørgrav
6f9febf93c Hackish support for 302 redirect. 2000-05-26 15:34:42 +00:00
Hajimu UMEMOTO
28c645cf6a IPv6 support. 2000-05-20 18:23:51 +00:00
Dag-Erling Smørgrav
6efb30c8d0 Better handling of some boundary conditions.
Submitted by:	ume
2000-05-19 09:45:42 +00:00
Hajimu UMEMOTO
c97925ad4e Make HTTP_PROXY work for FTP.
Reported by:	Ben Smithurst <ben@scientia.demon.co.uk>
Reviewed by:	des
2000-05-15 09:05:36 +00:00
Dag-Erling Smørgrav
f8f4130b77 Remove unused 'verbose'.
Initialize the struct url_stat at the beginning of fetchStatHTTP().
2000-05-15 08:24:29 +00:00
Dag-Erling Smørgrav
60245e42b0 Reorganize some of the http code and split it into more functions.
Implement fetchStatHTTP().
Unbungle struct url, and add fetchFreeURL().
Document it.
2000-05-11 13:31:02 +00:00
Dag-Erling Smørgrav
3d2a847151 Implement restart 2000-05-07 20:51:31 +00:00
Dag-Erling Smørgrav
e6182307e8 Use sizeof more consistently.
Outline sanity checking of server port spec.
2000-05-07 20:01:55 +00:00
Dag-Erling Smørgrav
f5f109a068 Patches I've had lying around for several months:
* Add the 'h' ftp flag (allocate local port in high range)
 * Add the 'd' flag (use direct connection even if proxy is defined)
 * Make sure flags != NULL before calling strchr().
 * Minor changes to some comments.
2000-01-07 10:59:12 +00:00
Peter Wemm
7f3dea244c $Id$ -> $FreeBSD$ 1999-08-28 00:22:10 +00:00
Dag-Erling Smørgrav
ce71b7364c Implement and document file list retrieval. 1998-12-21 19:41:50 +00:00
Dag-Erling Smørgrav
d941fd2df9 Don't specify a language to compile_et.
Rename fetchConnect to _fetch_connect since it's internal.
1998-12-18 14:32:48 +00:00
Dag-Erling Smørgrav
3b7a67401b Remove unused header files. Fix a few shadowing bugs. Improve error reporting. 1998-12-16 11:44:31 +00:00
Dag-Erling Smørgrav
0fba3a0005 Add verbose flag, and support functions.
Brucify the Makefile.
Differentiate atime and mtime in fetch*Stat().
Fix a few pointer bugs.
Tweak some error messages.
Don't #include sys/param.h and stdio.h in fetch.h.
Document that sys/param.h and stdio.h must be #included before fetch.h.
1998-12-16 10:24:55 +00:00
Dag-Erling Smørgrav
d8acd8dc5c Second of a series of cleanups to libfetch.
This commit introduces the following features:

 a) the fetchStat*() functions, which return meta-information for a
    document, such as size, modification time, etc.

 b) the use of the com_err(3) facilities to report errors.

It also fixes a bunch of style bugs and a few logic bugs and somewhat
improves the man page.

Changed files, in alphabetical order:

 Makefile:
  Don't generate macros in {ftp,http}err.c.

  Generate category fields for the error message lists.

  Compile the error table.

  Install fetch_err.h along with fetch.h.

 common.c:
  Remove the _netdb_errstring() macro, and add FETCH_ERR_NETDB to the
  error code in the _netdb_seterr() macro.

  Add categories to the _netdb_errlist table.

  Report errors through the Common Error library.

 common.h:
  Add the DEBUG macros.

  Add prototype for fetchConnect().

  Remove the prototype for _fetch_errstring(), which is local to common.c

  Add a categroy field to struct fetcherr, and define constants for
  error categories.

  Define macros for _{url,netdb,ftp,http}_seterr().

 errors.et: (new file)
  List error categories.

 fetch.3:
  Document the fetchStat*() functions.

  Move the "unimplemented functionality" comments from NOTES to BUGS.

  Document that applications which use libfetch must also use
  libcom_err, and list existing error codes.

  Undocument fetchLastErr{Code,String}.

  Remove the (empty) DIAGNOSTICS section.

  Mention Eugene Skepner in the AUTHORS section.

 fetch.c:
  Move the DEBUG macros to common.c

  Add fetchStat() and fetchStatURL().

  Generate error messages for URL parser errors, and fix a minor bug
  in the parser.

  Use 'struct url' instead of 'url_t'.

  Remove fetchLastErr{Code,String}.

 fetch.h:
  Use 'struct url' instead of 'url_t', and remove the typedef.

  Define struct url_stat (used by fetchStat()).

  Add prototypes for fetchStat*().

  Remove the declarations for fetchLastErr{Code,String}.

  Include fetch_err.h.

 fetch_err.et: (new file)
  Error table for libfetch.

 file.c:
  Add fetchStatFile().

  Use 'struct url' instead of 'url_t'.

 ftp.c:
  Add fetchStatFTP().

  Use 'struct url' instead of 'url_t'.

  Don't use fetchLastErrCode.

 ftp.errors:
  Add categories to all error messages.

 http.c:
  Add fetchStatHTTP().

  Use 'struct url' instead of 'url_t'.

  Don't use fetchLastErr{Code,Text}.

 http.errors:
  Add categories to all error messages.

Prompted by: jkh and Eugene Skepner
Numerous sugestions from: Garett Wollman and Eugene Skepner
1998-11-06 22:14:08 +00:00
Dag-Erling Smørgrav
842a95cc23 First of a series of cleanups to libfetch. Changed files, in
alphabetical order:

 Makefile:
  Add common.c to SRCS.

  Make debugging easier by making 'CFLAGS += -DNDEBUG' conditional on DEBUG

  Don't declare struct {ftp,http}err in {ftp,http}err.c; use struct fetcherr
  instead.

 README:
  Remove the todo list, which is out of date anyway.

 common.c: (new file)
  Gather utility functions in this file.

  Merge the error reporting functions intp _fetch_errstring(),
  _fetch_seterr() and _fetch_syserr().

  Set fetchLastErrCode and fetchLastErrText appropriately when fetchConnect
  fails.

 common.h: (new file)
  Gather internal prototypes and structures in this files.

 fetch.3:
  Undocument fetchFreeURL().

  Document a few more known bugs.

  Document fetchLastErrCode and fetchLastErrText.

 fetch.c:
  Add descriptive comments to all functions that lacked them.

  Move fetchConnect() to common.c.

  Obviate the need for fetchFreeURL(), and remove it.

 fetch.h:
  Modify struct url_t so the document part is at the end.

 ftp.c:
  Remove code that is duplicated elsewhere.

 http.c:
  Remove code that is duplicated elsewhere.

Prompted by: jkh
1998-11-05 19:48:17 +00:00
Dag-Erling Smørgrav
ecc9135299 Commit a bunch of patches that have been accumulating:
- Fix the README to reflect the new status of the ftp code.
 - Change tons of 'if (xxx < 0)' to 'if (xxx == -1)'
 - Add two new interface functions
 - Fix the Makefile so it actually works (yay!)

Now the manpage is lagging even further behind... :( Next on the todo
list is to clean up the http code.
1998-08-17 09:30:19 +00:00
Dag-Erling Smørgrav
f62e5228fd Base64 code (and the MIT copyright) moved to http.c
FTP STORe and APPEnd added.
FTP proxy support added (untested).
1998-07-12 22:34:40 +00:00
Dag-Erling Smørgrav
8e3986ea36 Too many changes to list. Basically, FTP is nearly there and error
reporting is kinda sorted out. Now HTTP needs to catch up...
1998-07-11 18:56:01 +00:00
Dag-Erling Smørgrav
4ca1ab9434 Imported libfetch into the tree. It compiles, but there's still some
work to do. I especially need help with the man page.
1998-07-09 16:52:44 +00:00