Commit Graph

422 Commits

Author SHA1 Message Date
Kyle Evans
5ac6a2c94d fetch(3): plug some leaks
In the successful case, sockshost is not freed prior to return.

The failure case can now be hit after fetch_reopen(), which was not true
before. Thus, we need to make sure to clean up all of the conn resources
which will also close sd. For all of the points prior to fetch_reopen(), we
continue to just close sd.

CID:		1419598, 1419616
2020-02-21 18:21:57 +00:00
Kyle Evans
86fd2105dc fetch(3): don't leak sockshost on failure
fetch_socks5_getenv will allocate memory for the host (or set it to NULL) in
all cases through the function; the caller is responsible for freeing it if
we end up allocating.

While I'm here, I've eliminated a label that just jumps to the next line...
2020-02-15 19:47:49 +00:00
Kyle Evans
3dc455e897 fetch(3): fix regression in IPv6:port spec from r357977
In case the port was specified, we never actually populated *host. Do so
now.

Pointy hat:	kevans
2020-02-15 19:39:50 +00:00
Kyle Evans
0f3fa96016 fetch(3): move bits of fetch_socks5_getenv around
This commit separates out port parsing and validation from grabbing the host
from the env var. The only related bit really is that we need to be more
specific with the delimiter in the IPv6 case.
2020-02-15 19:31:40 +00:00
Kyle Evans
c44be5aa0a fetch(3): Add SOCKS5 support
This change adds SOCKS5 support to the library fetch(3) and updates the man
page.

Details: Within the fetch_connect() function, fetch(3) checks if the
SOCKS5_PROXY environment variable is set. If so, it connects to this host
rather than the end-host. It then initializes the SOCKS5 connection in
accordance with RFC 1928 and returns the resulting conn_t (file descriptor)
for usage by the regular FTP/HTTP handlers.

Design Decision: This change defaults all DNS resolutions through the proxy
by sending all IPs as hostnames. Going forward, another feature might be to
create another environmental variable to toggle resolutions through the
proxy or not..

One may set the SOCKS5_PROXY environment variable in any of the formats:

SOCKS5_PROXY=proxy.example.com
SOCKS5_PROXY=proxy.example.com:1080
SOCKS5_PROXY=192.0.2.0
SOCKS5_PROXY=198.51.100.0:1080
SOCKS5_PROXY=[2001:db8::1]
SOCKS5_PROXY=[2001:db8::2]:1080

Then perform a request with fetch(1).

(note by kevans)
I've since been informed that Void Linux/xbps has a fork of libfetch that
also implements SOCKS5. I may compare/contrast the two in the mid-to-near
future.

Submitted by:	Farhan Khan <farhan farhan codes>
Differential Revision:	https://reviews.freebsd.org/D18908
2020-02-15 18:03:16 +00:00
Ed Maste
83372bda16 libfetch: disallow invalid escape sequences
Per RFC1738 escape is "% hex hex"; other sequences do not form a valid URL.

Suggested by:	Matthew Dillon
Reviewed by:	Matthew Dillon
MFC after:	1 week
2020-02-05 16:55:00 +00:00
Gordon Tetlow
6fb3f9944f Fix urldecode buffer overrun.
Reported by:	Duncan Overbruck
Security:	CVE-2020-7450
2020-01-28 18:37:18 +00:00
Simon J. Gerraty
2c9a9dfc18 Update Makefile.depend files
Update a bunch of Makefile.depend files as
a result of adding Makefile.depend.options files

Reviewed by:	 bdrewery
MFC after:	1 week
Sponsored by:   Juniper Networks
Differential Revision:  https://reviews.freebsd.org/D22494
2019-12-11 17:37:53 +00:00
Simon J. Gerraty
5ab1c5846f Add Makefile.depend.options
Leaf directories that have dependencies impacted
by options need a Makefile.depend.options file
to avoid churn in Makefile.depend

DIRDEPS for cases such as OPENSSL, TCP_WRAPPERS etc
can be set in local.dirdeps-options.mk
which can add to those set in Makefile.depend.options

See share/mk/dirdeps-options.mk

Reviewed by:	 bdrewery
MFC after:	1 week
Sponsored by:   Juniper Networks
Differential Revision:  https://reviews.freebsd.org/D22469
2019-12-11 17:37:37 +00:00
Emmanuel Vadot
4c1a82cea5 pkgbase: Create a FreeBSD-utilities package and make it the default one
The default package use to be FreeBSD-runtime but it should only contain
binaries and libs enough to boot to single user and repair the system, it
is also very handy to have a package that can be tranform to a small mfsroot.
So create a new package named FreeBSD-utilities and make it the default one.
Also move a few binaries and lib into this package when it make sense.
Reviewed by:	bapt, gjb
Differential Revision:	https://reviews.freebsd.org/D21506
2019-09-05 14:15:47 +00:00
Mark Johnston
f4e05cc55d Document fetchReqHTTP().
Submitted by:	Farhan Khan <khanzf@gmail.com>
Reviewed by:	0mp
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D18788
2019-08-28 17:01:28 +00:00
Adrian Chadd
819082e103 [libfetch] Fix compilation with WITHOUT_CRYPT. 2019-05-03 06:06:39 +00:00
Dag-Erling Smørgrav
a768df3e91 When deciding whether to send the complete URL or just the document part,
we were looking at the original URL rather than the one we were currently
processing.  This meant that if we were trying to retrieve an HTTP URL but
were redirected to an HTTPS URL, and HTTPS proxying was enabled, we would
send an invalid request and most likely get garbage back.

MFC after:	3 days
2018-11-27 16:23:17 +00:00
Dag-Erling Smørgrav
ceedec4bce A few more cases where strcasecmp() is no longer required.
MFC after:	1 week
2018-11-27 11:22:19 +00:00
Dag-Erling Smørgrav
8d9de5b10a Improve URL parsing. In particular, convert scheme and host to lowercase.
MFC after:	1 week
2018-11-27 10:45:14 +00:00
Dag-Erling Smørgrav
b68815f088 Support proxying FTP over HTTPS, not just HTTP.
There is probably a PR for this, but I can't find this, or remember who
submitted it.  The patch got lost in the noise of another that wasn't
ready to commit.

MFC after:	3 days
2018-11-27 10:06:41 +00:00
Jung-uk Kim
3fd49fe2d4 Make libfetch buildable. 2018-09-19 07:04:15 +00:00
Dag-Erling Smørgrav
5f04ebd4d3 Fix an inverted conditional in the netrc code, which would ignore the
value of $HOME and always use the home directory from the passwd
database, unless $HOME was unset, in which case it would use (null).

While there, clean up handling of netrcfd and add debugging aids.

MFC after:	3 weeks
2018-05-29 13:07:36 +00:00
Dag-Erling Smørgrav
f2eac20246 Fix a few (but far from all) style issues.
MFC after:	3 weeks
2018-05-29 10:29:43 +00:00
Dag-Erling Smørgrav
c5712d6da1 Use __VA_ARGS__ to simplify the DEBUG macro.
MFC after:	3 weeks
2018-05-29 10:28:20 +00:00
Dag-Erling Smørgrav
b847b083a8 Preserve if-modified-since timestamps across redirects.
PR:		224426
MFC after:	1 week
2018-05-12 17:02:27 +00:00
Pedro F. Giffuni
64de3fdd58 SPDX: use the Beerware identifier. 2017-11-30 20:33:45 +00:00
Pedro F. Giffuni
5e53a4f90f lib: further adoption of SPDX licensing ID tags.
Mainly focus on files that use BSD 2-Clause license, however the tool I
was using mis-identified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.
2017-11-26 02:00:33 +00:00
Bryan Drewery
ea825d0274 DIRDEPS_BUILD: Update dependencies.
Sponsored by:	Dell EMC Isilon
2017-10-31 00:07:04 +00:00
Dag-Erling Smørgrav
3954b52f28 In fetch_resolve(), if the port number or service name is included in
the host argument (e.g. "www.freebsd.org:443"), the service pointer,
which is supposed to point to the port or service part, instead points
to the separator, causing getaddrinfo() to fail.

Note that I have not been able to trigger this bug with fetch(1), nor
do I believe it is possible, as libfetch always parses the host:port
specification itself.  I discovered it when I copied fetch_resolve()
into an unrelated project.

MFC after:	3 days
2017-08-18 18:20:36 +00:00
Dag-Erling Smørgrav
08a49957b3 r308996 broke IP literals by assuming that a colon could only occur as
a separator between host and port, and using strchr() to search for it.
Rewrite fetch_resolve() so it handles bracketed literals correctly, and
remove similar code elsewhere to avoid passing unbracketed literals to
fetch_resolve().  Remove #ifdef INET6 so we still parse IP literals
correctly even if we do not have the ability to connect to them.

While there, fix an off-by-one error which caused HTTP 400 errors to be
misinterpreted as redirects.

PR:		217723
MFC after:	1 week
Reported by:	bapt, bz, cem, ngie
2017-03-17 14:18:52 +00:00
Dag-Erling Smørgrav
64c422355d Add a __printflike() that would have caught the bug fixed in r314396. 2017-03-12 11:54:35 +00:00
Pedro F. Giffuni
0012b66baf libfetch: extra bounds checking through reallocarray(3).
Reviewed by:	des
MFC after:	1 week
2017-03-06 15:38:03 +00:00
Dag-Erling Smørgrav
c8453e5bf4 Fix partial requests (used by fetch -r) when the requested file is
already complete.

Since 416 is an error code, any Content-Range header in the response
would refer to the error message, not the requested document, so
relying on the value of size when we know we got a 416 is wrong.
Instead, just verify that offset == 0 and assume that we've reached
the end of the document (if offset > 0, we did not request a range,
and the server is screwing with us).  Note that we cannot distinguish
between reaching the end and going past it, but that is a flaw in the
protocol, not in the code, so we just have to assume that the caller
knows what it's doing.  A smart caller would request an offset
slightly before what it believes is the end and compare the result to
what is already in the file.

PR:		212065
Reported by:	mandree
MFC after:	3 weeks
2017-03-05 12:06:45 +00:00
Baptiste Daroussin
5d1ce10bee Properly initialize netrcfd in fetchParseURL
This fixes ftp with fetch(1) which was broken after r313974

Submitted by:	dim
Reported by:	olivier
Pointyhat to:	bapt
2017-03-03 12:51:16 +00:00
Bjoern A. Zeeb
6d91604093 Properly indent a default: label and avoid crashing when running
under -v but cannot connect due to trying to print an int as %s [1].

Reported by:	andrew [1]
MFC after:	3 days
2017-02-28 18:10:03 +00:00
Baptiste Daroussin
d8713bf361 Add a file descriptor in struct url for netrc
When using libfetch in an application that drops privileges when fetching
like pkg(8) then user complain because the application does not read anymore
${HOME}/.netrc. Now a caller can prepare a fd to the said file and manually
assign it to the structure.

It is also a first step to allow to capsicumize libfetch applications

Reviewed by:	allanjude, des
Approved by:	des
Differential Revision:	https://reviews.freebsd.org/D9678
2017-02-20 00:14:31 +00:00
Dag-Erling Smørgrav
21ca0912c6 Fix inverted loop condition which broke multi-line responses to CONNECT.
PR:		194483
Submitted by:	Miłosz Kaniewski <milosz.kaniewski@gmail.com>
MFC after:	1 week
2016-12-30 14:54:54 +00:00
Dag-Erling Smørgrav
3af3efd115 More debugging code I missed in r309051.
Reported by:	jbeich, jkim
2016-12-09 16:14:55 +00:00
Dag-Erling Smørgrav
1da400983c Remove debugging code. 2016-11-23 10:52:19 +00:00
Dag-Erling Smørgrav
792ef1ae7b Refactor fetch_connect() and fetch_bind() to improve readability and avoid
repeating the same DNS lookups.

MFC after:	3 weeks
2016-11-22 13:30:07 +00:00
Dag-Erling Smørgrav
a5fc9a29bb r169386 (PR 112515) was incomplete: it treated 307 as an error except
in verbose mode, and did not handle 308 at all.

r241840 (PR 172451) added support for 308, but with the same bug.

Correctly handle both by recognizing them as redirects in all places
where we check the HTTP result code.

PR:		112515 173451 209546
Submitted by:	novel@
MFC after:	1 week
2016-05-31 08:27:39 +00:00
Don Lewis
65380b5f2f Call closedir() before returning from fetchListFile() to avoid a leak.
Reported by:	Coverity
CID:		1016697
MFC after:	1 week
2016-05-25 07:43:32 +00:00
Don Lewis
429bf952ae Don't leak addrinfo in fetch_bind()
Reported by:	Coverity
CID:		1225038
MFC after:	1 week
2016-05-25 07:39:48 +00:00
Don Lewis
77b822dbc0 Use strlcpy() instead of strncpy() to copy the string returned by
setlocale() so that static analyzers know that the string is NUL
terminated.  This was causing a false positive in Coverity even
though the longest string returned by setlocale() is ENCODING_LEN
(31) and we are copying into a 64 byte buffer.  This change is also
a bit of an optimization since we don't need the strncpy() feature
of padding the rest of the destination buffer with NUL characters.

Reported by:	Coverity
CID:		974654
2016-05-12 06:39:13 +00:00
Pedro F. Giffuni
75f46cf6c8 lib: minor spelling fixes in comments.
No functional change.
2016-05-01 19:37:33 +00:00
Glen Barber
876d357fa7 MFH
Sponsored by:	The FreeBSD Foundation
2016-04-11 15:24:59 +00:00
Pedro F. Giffuni
3cd7f4295f libfetch: replace 0 with NULL for pointers.
Found with devel/coccinelle.

Reviewed by:	des
2016-04-09 18:08:21 +00:00
Glen Barber
d60840138f MFH
Sponsored by:	The FreeBSD Foundation
2016-04-04 23:55:32 +00:00
Edward Tomasz Napierala
225636dccb Fix bunch of .Xrs.
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-03-28 16:48:28 +00:00
Michael Gmelin
1ec923fd6d Update fetch.1 and fetch.3 to reflect libfetch's actual use of CA bundles
Reviewed by:	wblock
Approved by:	wblock
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D5558
2016-03-19 11:55:21 +00:00
Glen Barber
52259a98ad MFH
Sponsored by:	The FreeBSD Foundation
2016-03-02 16:14:46 +00:00
Bryan Drewery
bd18fd57db DIRDEPS_BUILD: Regenerate without local dependencies.
These are no longer needed after the recent 'beforebuild: depend' changes
and hooking DIRDEPS_BUILD into a subset of FAST_DEPEND which supports
skipping 'make depend'.

Sponsored by:	EMC / Isilon Storage Division
2016-02-24 17:20:11 +00:00
Glen Barber
0fe0fe112f MFH
Sponsored by:	The FreeBSD Foundation
2016-02-15 21:58:52 +00:00
Dag-Erling Smørgrav
a982c4c7f5 Fix double-free error: r289419 moved all error handling in http_connect()
to the end of the function, but did not remove a fetch_close() call which
was made redundant by the one in the shared error-handling code.

PR:		206774
Submitted by:	Christian Heckendorf <heckendorfc@gmail.com>
MFC after:	3 days
2016-02-11 17:48:15 +00:00