This change adds SOCKS5 support to the library fetch(3) and updates the man
page.
Details: Within the fetch_connect() function, fetch(3) checks if the
SOCKS5_PROXY environment variable is set. If so, it connects to this host
rather than the end-host. It then initializes the SOCKS5 connection in
accordance with RFC 1928 and returns the resulting conn_t (file descriptor)
for usage by the regular FTP/HTTP handlers.
Design Decision: This change defaults all DNS resolutions through the proxy
by sending all IPs as hostnames. Going forward, another feature might be to
create another environmental variable to toggle resolutions through the
proxy or not..
One may set the SOCKS5_PROXY environment variable in any of the formats:
SOCKS5_PROXY=proxy.example.com
SOCKS5_PROXY=proxy.example.com:1080
SOCKS5_PROXY=192.0.2.0
SOCKS5_PROXY=198.51.100.0:1080
SOCKS5_PROXY=[2001:db8::1]
SOCKS5_PROXY=[2001:db8::2]:1080
Then perform a request with fetch(1).
(note by kevans)
I've since been informed that Void Linux/xbps has a fork of libfetch that
also implements SOCKS5. I may compare/contrast the two in the mid-to-near
future.
Submitted by: Farhan Khan <farhan farhan codes>
Differential Revision: https://reviews.freebsd.org/D18908
Mainly focus on files that use BSD 2-Clause license, however the tool I
was using mis-identified many licenses so this was mostly a manual - error
prone - task.
The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.
simply not trying to return exactly what the caller asked for - just
return whatever we got and let the caller be the judge of whether it
was enough. If an error occurs or the connection times out after we
already received some data, return a short read, under the assumption
that the next call will fail or time out before we read anything.
As it turns out, none of the code that calls fetch_read() assumes an
all-or-nothing result anyway, except for a couple of lines where we
read the CR LF at the end of a hunk in HTTP hunked encoding, so the
changes outside of fetch_read() and http_readfn() are minimal.
While there, replace select(2) with poll(2).
MFC after: 3 days
progress information. The first is that fetch_read() (used in the HTTP
code but not the FTP code) can enter an infinite loop if it has previously
been interrupted by a signal. The second is that when it is interrupted,
fetch_read() will discard any data it may have read up to that point.
Luckily, both bugs are extremely timing-sensitive and therefore difficult
to trigger.
PR: bin/153240
Submitted by: Mark <markjdb@gmail.com>
MFC after: 3 weeks
lynx, curl etc. Note that this patch differs significantly from that
in the PR, as the submitter refined it after submitting the PR.
PR: 110388
Submitted by: Alexander Pohoyda <alexander.pohoyda@gmx.net>
MFC after: 3 weeks
the SSL case, it is no different from the old _fetch_write(), but in the
non-SSL case it uses writev(2) to send the entire vector as a single
packet (provided it can fit in one packet). Implement _fetch_write()
and _fetch_putln() in terms of _fetch_writev().
This should improve performance in the non-SSL case (by reducing protocol
overhead) and solve the problem where too-smart-for-their-own-good
firewalls reject FTP packets that do not end in CRLF.
PR: bin/44123
Submitted by: fenner
which contains the socket descriptor, the input buffer and (yet unused)
SSL state variables. This has the neat side effect of greatly improving
reentrance (though we're not *quite* there yet) and opening the door to
HTTP connection caching.
This commit is inspired by email conversations with and patches from
Henry Whincup <henry@techiebod.com> last fall.
Brucify the Makefile.
Differentiate atime and mtime in fetch*Stat().
Fix a few pointer bugs.
Tweak some error messages.
Don't #include sys/param.h and stdio.h in fetch.h.
Document that sys/param.h and stdio.h must be #included before fetch.h.
This commit introduces the following features:
a) the fetchStat*() functions, which return meta-information for a
document, such as size, modification time, etc.
b) the use of the com_err(3) facilities to report errors.
It also fixes a bunch of style bugs and a few logic bugs and somewhat
improves the man page.
Changed files, in alphabetical order:
Makefile:
Don't generate macros in {ftp,http}err.c.
Generate category fields for the error message lists.
Compile the error table.
Install fetch_err.h along with fetch.h.
common.c:
Remove the _netdb_errstring() macro, and add FETCH_ERR_NETDB to the
error code in the _netdb_seterr() macro.
Add categories to the _netdb_errlist table.
Report errors through the Common Error library.
common.h:
Add the DEBUG macros.
Add prototype for fetchConnect().
Remove the prototype for _fetch_errstring(), which is local to common.c
Add a categroy field to struct fetcherr, and define constants for
error categories.
Define macros for _{url,netdb,ftp,http}_seterr().
errors.et: (new file)
List error categories.
fetch.3:
Document the fetchStat*() functions.
Move the "unimplemented functionality" comments from NOTES to BUGS.
Document that applications which use libfetch must also use
libcom_err, and list existing error codes.
Undocument fetchLastErr{Code,String}.
Remove the (empty) DIAGNOSTICS section.
Mention Eugene Skepner in the AUTHORS section.
fetch.c:
Move the DEBUG macros to common.c
Add fetchStat() and fetchStatURL().
Generate error messages for URL parser errors, and fix a minor bug
in the parser.
Use 'struct url' instead of 'url_t'.
Remove fetchLastErr{Code,String}.
fetch.h:
Use 'struct url' instead of 'url_t', and remove the typedef.
Define struct url_stat (used by fetchStat()).
Add prototypes for fetchStat*().
Remove the declarations for fetchLastErr{Code,String}.
Include fetch_err.h.
fetch_err.et: (new file)
Error table for libfetch.
file.c:
Add fetchStatFile().
Use 'struct url' instead of 'url_t'.
ftp.c:
Add fetchStatFTP().
Use 'struct url' instead of 'url_t'.
Don't use fetchLastErrCode.
ftp.errors:
Add categories to all error messages.
http.c:
Add fetchStatHTTP().
Use 'struct url' instead of 'url_t'.
Don't use fetchLastErr{Code,Text}.
http.errors:
Add categories to all error messages.
Prompted by: jkh and Eugene Skepner
Numerous sugestions from: Garett Wollman and Eugene Skepner
alphabetical order:
Makefile:
Add common.c to SRCS.
Make debugging easier by making 'CFLAGS += -DNDEBUG' conditional on DEBUG
Don't declare struct {ftp,http}err in {ftp,http}err.c; use struct fetcherr
instead.
README:
Remove the todo list, which is out of date anyway.
common.c: (new file)
Gather utility functions in this file.
Merge the error reporting functions intp _fetch_errstring(),
_fetch_seterr() and _fetch_syserr().
Set fetchLastErrCode and fetchLastErrText appropriately when fetchConnect
fails.
common.h: (new file)
Gather internal prototypes and structures in this files.
fetch.3:
Undocument fetchFreeURL().
Document a few more known bugs.
Document fetchLastErrCode and fetchLastErrText.
fetch.c:
Add descriptive comments to all functions that lacked them.
Move fetchConnect() to common.c.
Obviate the need for fetchFreeURL(), and remove it.
fetch.h:
Modify struct url_t so the document part is at the end.
ftp.c:
Remove code that is duplicated elsewhere.
http.c:
Remove code that is duplicated elsewhere.
Prompted by: jkh
