.\" Copyright (c) 1998 Dag-Erling Coïdan Smørgrav .\" All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions .\" are met: .\" 1. Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" 2. Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" SUCH DAMAGE. .\" .\" $FreeBSD$ .\" .Dd July 1, 1998 .Dt FETCH 3 .Os .Sh NAME .Nm fetchParseURL , .Nm fetchFreeURL , .Nm fetchGetURL , .Nm fetchPutURL , .Nm fetchStatURL , .Nm fetchListURL , .Nm fetchGet , .Nm fetchPut , .Nm fetchStat , .Nm fetchList , .Nm fetchGetFile , .Nm fetchPutFile , .Nm fetchStatFile , .Nm fetchListFile , .Nm fetchGetHTTP , .Nm fetchPutHTTP , .Nm fetchStatHTTP , .Nm fetchListHTTP , .Nm fetchGetFTP , .Nm fetchPutFTP , .Nm fetchStatFTP , .Nm fetchListFTP .Nd file transfer functions .Sh LIBRARY .Lb libfetch .Sh SYNOPSIS .Fd #include .Fd #include .Fd #include .Ft struct url * .Fn fetchParseURL "char *URL" .Ft void .Fn fetchFreeURL "struct url *URL" .Ft FILE * .Fn fetchGetURL "char *URL" "char *flags" .Ft FILE * .Fn fetchPutURL "char *URL" "char *flags" .Ft int .Fn fetchStatURL "char *URL" "struct url_stat *us" "char *flags" .Ft struct url_ent * .Fn fetchListURL "char *URL" "char *flags" .Ft FILE * .Fn fetchGet "struct url *URL" "char *flags" .Ft FILE * .Fn fetchPut "struct url *URL" "char *flags" .Ft int .Fn fetchStat "struct url *URL" "struct url_stat *us" "char *flags" .Ft struct url_ent * .Fn fetchList "struct url *" "char *flags" .Ft FILE * .Fn fetchGetFile "struct url *u" "char *flags" .Ft FILE * .Fn fetchPutFile "struct url *u" "char *flags" .Ft int .Fn fetchStatFile "struct url *URL" "struct url_stat *us" "char *flags" .Ft struct url_ent * .Fn fetchListFile "struct url *" "char *flags" .Ft FILE * .Fn fetchGetHTTP "struct url *u" "char *flags" .Ft FILE * .Fn fetchPutHTTP "struct url *u" "char *flags" .Ft int .Fn fetchStatHTTP "struct url *URL" "struct url_stat *us" "char *flags" .Ft struct url_ent * .Fn fetchListHTTP "struct url *" "char *flags" .Ft FILE * .Fn fetchGetFTP "struct url *u" "char *flags" .Ft FILE * .Fn fetchPutFTP "struct url *u" "char *flags" .Ft int .Fn fetchStatFTP "struct url *URL" "struct url_stat *us" "char *flags" .Ft struct url_ent * .Fn fetchListFTP "struct url *" "char *flags" .Sh DESCRIPTION .Pp These functions implement a high-level library for retrieving and uploading files using Uniform Resource Locators (URLs). .Pp .Fn fetchParseURL takes a URL in the form of a null-terminated string and splits it into its components function according to the Common Internet Scheme Syntax detailed in RFC1738. A regular expression which produces this syntax is: .Bd -literal :(//((:)?@)?(:)?)?/()? .Ed .Pp Note that some components of the URL are not necessarily relevant to all URL schemes. For instance, the file scheme only needs the and components. .Pp .Fn fetchParseURL returns a pointer to a .Fa url structure, which is defined as follows in .Aq Pa fetch.h : .Bd -literal #define URL_SCHEMELEN 16 #define URL_USERLEN 256 #define URL_PWDLEN 256 struct url { char scheme[URL_SCHEMELEN+1]; char user[URL_USERLEN+1]; char pwd[URL_PWDLEN+1]; char host[MAXHOSTNAMELEN+1]; int port; char *doc; off_t offset; size_t length; }; .Ed .Pp The pointer returned by .Fn fetchParseURL should be freed using .Fn fetchFreeURL . .Pp .Fn fetchGetURL and .Fn fetchPutURL constitute the recommended interface to the .Nm fetch library. They examine the URL passed to them to determine the transfer method, and call the appropriate lower-level functions to perform the actual transfer. The .Fa flags argument is a string of characters which specify transfer options. The meaning of the individual flags is scheme-dependent, and is detailed in the appropriate section below. .Pp .Fn fetchStatURL attempts to obtain the requested document's metadata and fill in the structure pointed to by it's second argument. The .Fa url_stat structure is defined as follows in .Aq Pa fetch.h : .Bd -literal struct url_stat { off_t size; time_t atime; time_t mtime; }; .Ed .Pp If the size could not be obtained from the server, the .Fa size field is set to -1. If the modification time could not be obtained from the server, the .Fa mtime field is set to the epoch. If the access time could not be obtained from the server, the .Fa atime field is set to the modification time. .Pp .Fn fetchListURL attempts to list the contents of the directory pointed to by the URL provided. If successful, it returns a malloced array of .Fa url_ent structures. The .Fa url_ent structure is defined as follows in .Aq Pa fetch.h : .Bd -literal struct url_ent { char name[MAXPATHLEN]; struct url_stat stat; }; .Ed .Pp The list is terminated by an entry with an empty name. .Pp The pointer returned by .Fn fetchListURL should be freed using .Fn free . .Pp .Fn fetchGet , .Fn fetchPut and .Fn fetchStat are similar to .Fn fetchGetURL , .Fn fetchPutURL and .Fn fetchStatURL , except that they expect a pre-parsed URL in the form of a pointer to a .Fa struct url rather than a string. .Pp All of the .Fn fetchGetXXX and .Fn fetchPutXXX functions return a pointer to a stream which can be used to read or write data from or to the requested document, respectively. Note that although the implementation details of the individual access methods vary, it can generally be assumed that a stream returned by one of the .Fn fetchGetXXX functions is read-only, and that a stream returned by one of the .Fn fetchPutXXX functions is write-only. .Sh FILE SCHEME .Fn fetchGetFile and .Fn fetchPutFile provide access to documents which are files in a locally mounted file system. Only the component of the URL is used. .Pp .Fn fetchGetFile does not accept any flags. .Pp .Fn fetchPutFile accepts the .Fa a (append to file) flag. If that flag is specified, the data written to the stream returned by .Fn fetchPutFile will be appended to the previous contents of the file, instead of replacing them. .Sh FTP SCHEME .Fn fetchGetFTP and .Fn fetchPutFTP implement the FTP protocol as described in RFC959. .Pp If the .Fa p (passive) flag is specified, a passive (rather than active) connection will be attempted. .Pp If the .Fa h (high) flag is specified, data sockets will be allocated in the high port range (see .Xr ip 4 ). .Pp If the .Fa d (direct) flag is specified, .Fn fetchGetFTP and .Fn fetchPutFTP will use a direct connection even if a proxy server is defined. .Pp If no user name or password is given, the .Nm fetch library will attempt an anonymous login, with user name "ftp" and password "ftp". .Sh HTTP SCHEME The .Fn fetchGetHTTP and .Fn fetchPutHTTP functions implement the HTTP/1.1 protocol. With a little luck, there's even a chance that they comply with RFC2068. .Pp If the .Fa d (direct) flag is specified, .Fn fetchGetHTTP and .Fn fetchPutHTTP will use a direct connection even if a proxy server is defined. .Pp Since there seems to be no good way of implementing the HTTP PUT method in a manner consistent with the rest of the .Nm fetch library, .Fn fetchPutHTTP is currently unimplemented. .Sh RETURN VALUES .Fn fetchParseURL returns a pointer to a .Fa struct url containing the individual components of the URL. If it is unable to allocate memory, or the URL is syntactically incorrect, .Fn fetchParseURL returns a NULL pointer. .Pp The .Fn fetchStat functions return 0 on success and -1 on failure. .Pp All other functions return a stream pointer which may be used to access the requested document, or NULL if an error occurred. .Pp .Nm Libfetch uses the Common Error Library .Nm ( libcom_err ) to report errors. The error code passed to .Fn com_err is one of: .Bl -tag -width 18n .It Bq Er FETCH_ABORT Operation aborted .It Bq Er FETCH_AUTH Authentication failed .It Bq Er FETCH_DOWN Service unavailable .It Bq Er FETCH_EXISTS File exists .It Bq Er FETCH_FULL File system full .It Bq Er FETCH_INFO Informational response .It Bq Er FETCH_MEMORY Insufficient memory .It Bq Er FETCH_MOVED File has moved .It Bq Er FETCH_NETWORK Network error .It Bq Er FETCH_OK No error .It Bq Er FETCH_PROTO Protocol error .It Bq Er FETCH_RESOLV Resolver error .It Bq Er FETCH_SERVER Server error .It Bq Er FETCH_TEMP Temporary error .It Bq Er FETCH_TIMEOUT Operation timed out .It Bq Er FETCH_UNAVAIL File is not available .It Bq Er FETCH_UNKNOWN Unknown error .It Bq Er FETCH_URL Invalid URL .El .Pp The accompanying error message includes a protocol-specific error code and message, e.g. "File is not available (404 Not Found)" .Sh ENVIRONMENT The FTP and HTTP related functions use the .Ev HTTP_PROXY and .Ev FTP_PROXY environment variables, respectively, as the address of a proxy server to use for transferring files. .Sh SEE ALSO .Xr com_err 3 , .Xr fetch 1 , .Xr ftpio 3 , .Xr ip 4 . .Rs .%A T. Berners-Lee .%A L. Masinter .%A M. McCahill .%D December 1994 .%T Uniform Resource Locators (URL) .%O RFC1738 .Re .Rs .%A R. Fielding .%A J. Gettys .%A J. Mogul .%A H. Frystyk .%A T. Berners-Lee .%D Januray 1997 .%B Hypertext Transfer Protocol -- HTTP/1.1 .%O RFC2068 .Re .Rs .%A J. Postel .%A J. K. Reynolds .%D October 1985 .%B File Transfer Protocol .%O RFC959 .Re .Sh NOTES The .Nm fetch library uses the Common Error library, and applications which link with .Nm libfetch must therefore also link with .Nm libcom_err . .Sh HISTORY The .Nm fetch library first appeared in .Fx 3.0 . .Sh AUTHORS The .Nm fetch library was mostly written by .An Dag-Erling Coïdan Smørgrav Aq des@FreeBSD.org with numerous suggestions from .An Jordan K. Hubbard Aq jkh@FreeBSD.org , .An Eugene Skepner Aq eu@qub.com and other FreeBSD developers. It replaces the older .Nm ftpio library written by .An Poul-Henning Kamp Aq pkh@FreeBSD.org and .An Jordan K. Hubbard Aq jkh@FreeBSD.org . .Pp This manual page was written by .An Dag-Erling Coïdan Smørgrav Aq des@FreeBSD.org .Sh BUGS Some parts of the library are not yet implemented. The most notable examples of this are .Fn fetchPutHTTP , .Fn fetchListHTTP , .Fn fetchListFTP and FTP proxy support. .Pp There's no way to select a proxy at run-time other than setting the .Ev HTTP_PROXY or .Ev FTP_PROXY environment variables as appropriate. There is also no way to stop the FTP and HTTP functions from trying to use a proxy if these variables are set. .Pp HTTP authentication doesn't work. I'm not sure that's a bug in my code; as far as I can determine, .Nm libfetch handles HTTP/1.1 basic authentication correctly as outlined in RFC2068, but I haven't been able to find an HTTP server that honors the Authentication: header field. Also, .Nm libfetch does not attempt to interpret and respond to authentication requests from the HTTP server. .Pp No attempt is made to encode spaces etc. within URLs. Spaces in the document part of an URLshould be replaced with "%20" in HTTP URLs and "\\ " in FTP URLs. .Pp Error numbers are unique only within a certain context; the error codes used for FTP and HTTP overlap, as do those used for resolver and system errors. For instance, error code 202 means "Command not implemented, superfluous at this site" in an FTP context and "Accepted" in an HTTP context. .Pp .Fn fetchStatFTP does not check that the result of an MDTM command is a valid date. .Pp The HTTP code needs a complete rewrite, or at least a serious cleanup. .Pp The man page is poorly written and produces badly formatted text. .Pp Tons of other stuff.