Commit Graph

373 Commits

Author SHA1 Message Date
bdrewery
2a891f1feb DIRDEPS_BUILD: Regenerate without local dependencies.
These are no longer needed after the recent 'beforebuild: depend' changes
and hooking DIRDEPS_BUILD into a subset of FAST_DEPEND which supports
skipping 'make depend'.

Sponsored by:	EMC / Isilon Storage Division
2016-02-24 17:20:11 +00:00
des
4f0a0f6221 Fix double-free error: r289419 moved all error handling in http_connect()
to the end of the function, but did not remove a fetch_close() call which
was made redundant by the one in the shared error-handling code.

PR:		206774
Submitted by:	Christian Heckendorf <heckendorfc@gmail.com>
MFC after:	3 days
2016-02-11 17:48:15 +00:00
bapt
7eac7d8684 Test for /etc/ssl/cert.pem existence to avoid masking SSL_CA_CERT_PATH
Prior to this patch, unless SSL_CA_CERT_FILE is set in the environment,
libfetch will set the CA file to "/usr/local/etc/cert.pem" if it exists,
and to "/etc/ssl/cert.pem" otherwise. This has the consequence of
masking SSL_CA_CERT_PATH, because OpenSSL will ignore the CA path if a CA
file is set but fails to load (see X509_STORE_load_locations()).

While here, fall back to OpenSSL defaults if neither SSL_CA_CERT_FILE nor
SSL_CA_CERT_PATH are set in the environment, and if neither of the
libfetch default CA files exists.

PR:		193871
Submitted by:	John W. O'Brien <john@saltant.com>
Approved by:	des
MFC after:	1 week
2016-01-19 15:02:37 +00:00
des
9f3bc9ef2e As a followup to r292330, standardize on size_t and add a few comments. 2015-12-16 09:20:45 +00:00
des
f487c94c08 Reset bufpos to 0 immediately after refilling the buffer. Otherwise, we
risk leaving the connection in an indeterminate state if the server fails
to send a chunk delimiter.  Depending on the application and on the sizes
of the preceding chunks, the result can be anything from missing data to a
segfault.  With this patch, it will be reported as a protocol error.

PR:		204771
MFC after:	1 week
2015-12-16 09:17:07 +00:00
dim
fbecea9b07 Fix buildworld after r291453, similar to r284346: url->user and url->pwd
are arrays, so they can never be NULL.

Reported by:	many
Pointy hat to:	des
2015-11-29 22:37:48 +00:00
des
d93cecbf5b Use .netrc for HTTP sites and proxies, not just FTP.
PR:		193740
Submitted by:	TEUBEL György <tgyurci@gmail.com>
MFC after:	1 week
2015-11-29 14:26:59 +00:00
bdrewery
e13d6f8b3f META MODE: Prefer INSTALL=tools/install.sh to lessen the need for xinstall.host.
This both avoids some dependencies on xinstall.host and allows
bootstrapping on older releases to work due to lack of at least 'install -l'
support.

Sponsored by:	EMC / Isilon Storage Division
2015-11-25 19:10:28 +00:00
des
095b2710ed Use fopen()'s newfangled "e" flag instead of explicit fcntl() calls.
PR:		199801
Submitted by:	Jukka Ukkonen <jau@iki.fi>
MFC after:	1 week
2015-10-16 12:53:22 +00:00
des
7cd58b9488 Fix two bugs in HTTPS tunnelling:
- If the proxy returns a non-200 result, set the error code accordingly
   so the caller / user gets a somewhat meaningful error message.
 - Consume and discard any HTTP response header following the result line.

PR:		194483
Tested by:	Fabian Keil <fk@fabiankeil.de>
MFC after:	1 week
2015-10-16 12:21:44 +00:00
grembo
adca4f81cb Fix non-POSIX-compliant use of getaddrinfo in libfetch
Submitted by:	Boris Kolpackov <boris@codesynthesis.com>
Reviewed by:	bapt
Approved by:	bapt
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D3724
2015-09-25 14:24:23 +00:00
araujo
886fc0a8be Remove unused variable to silence clang warning.
Differential Revision:	D2683
Reviewed by:		rodrigc, bapt
2015-07-04 17:22:07 +00:00
sjg
852129abd1 new depends 2015-06-16 23:37:19 +00:00
dim
8cd240ced1 Fix the following clang 3.7.0 warnings in lib/libfetch/http.c:
lib/libfetch/http.c:1628:26: error: address of array 'purl->user'
    will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
                                    aparams.user = purl->user ?
                                                   ~~~~~~^~~~ ~
    lib/libfetch/http.c:1630:30: error: address of array 'purl->pwd'
    will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
                                    aparams.password = purl->pwd?
                                                       ~~~~~~^~~~
    lib/libfetch/http.c:1657:25: error: address of array 'url->user'
    will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
                                    aparams.user = url->user ?
                                                   ~~~~~^~~~ ~
    lib/libfetch/http.c:1659:29: error: address of array 'url->pwd'
    will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
                                    aparams.password = url->pwd ?
                                                       ~~~~~^~~ ~
    lib/libfetch/http.c:1669:25: error: address of array 'url->user'
    will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
                                    aparams.user = url->user ?
                                                   ~~~~~^~~~ ~
    lib/libfetch/http.c:1671:29: error: address of array 'url->pwd'
    will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
                                    aparams.password = url->pwd ?
                                                       ~~~~~^~~ ~

Since url->user and url->pwd are arrays, they can never be NULL, so the
checks can be removed.

Reviewed by:	bapt
MFC after:	3 days
Differential Revision: https://reviews.freebsd.org/D2673
2015-06-13 19:26:48 +00:00
sjg
008d7c831f Add META_MODE support.
Off by default, build behaves normally.
WITH_META_MODE we get auto objdir creation, the ability to
start build from anywhere in the tree.

Still need to add real targets under targets/ to build packages.

Differential Revision:       D2796
Reviewed by: brooks imp
2015-06-13 19:20:56 +00:00
sjg
75a137820d dirdeps.mk now sets DEP_RELDIR 2015-06-08 23:35:17 +00:00
sjg
65145fa4c8 Merge sync of head 2015-05-27 01:19:58 +00:00
jkim
3cb01b490f Remove defunct SSLv2 support from fetch(1) and fetch(3). 2015-03-25 18:56:36 +00:00
bapt
6adce30d28 Convert libraries to use LIBADD
While here reduce a bit overlinking
2014-11-25 11:07:26 +00:00
sjg
b137080f19 Merge from head@274682 2014-11-19 01:07:58 +00:00
des
9be332cf60 As pointed out by several people, r273114 was incorrect: it unconditionally
disabled everything except TLS 1.0.  Replace it with a more carefully
wrought patch:

 - Switch the default for SSLv3 from on to off
 - Add environment variables to control TLS 1.1 and 1.2
 - In verbose mode, report which version is used
 - Update the man page to reflect these changes.

MFC after:	1 week
2014-10-15 07:35:50 +00:00
des
95a6b7a20f Drop support for SSLv3.
MFC after:	1 week
2014-10-15 02:34:26 +00:00
sjg
d7cd1d425c Merge head from 7/28 2014-08-19 06:50:54 +00:00
bapt
7eba29e60e Add a comment to explain the EAGAIN is only there for POSIX compliance
Resquested by:	kib
Reviewed by:	des
2014-07-15 15:29:43 +00:00
bapt
a25f9f5e21 Support EAGAIN in fetch_writev
Reviewed by:	des
Approved by:	des
2014-07-10 13:04:52 +00:00
bapt
091c55ab4e Use Mt macro to properly format mailto links (patch from Franco Fichtner <franco@lastsummer.de> for Dragonfly)
Reviewed by:	des
Approved by:	des
Obtained from:	Dragonfly
MFC after:	1 week
2014-06-11 20:20:14 +00:00
bapt
1b8c8ebb57 Add support for arbitrary http requests
Submitted by:	Alex Hornung <alex@alexhornung.com>
Reviewed by:	des
Obtained from:	Dragonfly
MFC after:	3 week
2014-06-05 22:16:26 +00:00
bapt
c84894dbf3 Remove unnecessary semicolons
Patch by Sascha Wildner <saw@online.de> for Dragonfly

Reviewed by:	des
Obtained from:	Dragonfly
MFC after:	1 week
2014-06-05 22:13:30 +00:00
bapt
e9021f7004 Use NULL instead of 0
Patch by Sascha Wildner <saw@online.de> for Dragonfly

Reviewed by:	des
Obtained from:	Dragonfly
MFC after:	1 week
2014-06-05 22:10:25 +00:00
des
055be14e5c If HTTP_USER_AGENT is defined but empty, don't send User-Agent at all.
PR:		184507
Submitted by:	jbeich@tormail.org (with modifications)
MFC after:	1 week
2014-06-05 20:27:16 +00:00
des
fd274d1e17 Look for root certificates in /usr/local/etc/ssl before /etc/ssl.
MFH:	1 week
2014-05-17 03:39:56 +00:00
sjg
5860f0d106 Updated dependencies 2014-05-16 14:09:51 +00:00
sjg
ed3fc70bf5 Merge from head 2014-05-08 23:54:15 +00:00
imp
2118f42afd Use src.opts.mk in preference to bsd.own.mk except where we need stuff
from the latter.
2014-05-06 04:22:01 +00:00
sjg
0c7e03a54c Merge head 2014-04-27 08:13:43 +00:00
bdrewery
ef589c91da Support Last-Modified behind proxies which return UTC instead of GMT.
The standard states that GMT must be used, but that UTC is equivalent. Still
parse UTC as otherwise this causes problems for pkg(8). It will refetch
the repository every time 'pkg update' or other remote operations
are used behind these proxies.

RFC2616: "All HTTP date/time stamps MUST be represented in Greenwich Mean
Time (GMT), without exception. For the purposes of HTTP, GMT is exactly equal
to UTC (Coordinated Universal Time).""

Approved by:	bapt (mentor)
Reviewed by:	des, peter
Sponsored by:	EMC / Isilon Storage Division
MFC after:	1 week
2014-03-11 13:47:11 +00:00
des
8ca08329ca Bump copyright dates 2014-01-30 08:37:23 +00:00
des
3dc25be505 r261230 broke the cases where the amount of data to be read is not
known in advance, or where the caller doesn't care and just keeps
reading until it hits EOF.

In fetch_read(): the socket is non-blocking, so read() will return 0
on EOF, and -1 (errno == EAGAIN) when the connection is still open but
there is no data waiting.  In the first case, we should immediately
return 0.  The EINTR case was also broken, although not in a way that
matters.

In fetch_writev(): use timersub() and timercmp() as in fetch_read().

In http_fillbuf(): set errno to a sensible value when an invalid chunk
header is encountered.

In http_readfn(): as in fetch_read(), a zero return from down the
stack indicates EOF, not an error.  Furthermore, when io->error is
EINTR, clear it (but no errno) before returning so the caller can
retry after dealing with the interrupt.

MFC after:	3 days
2014-01-29 12:48:19 +00:00
des
2010baa9c1 Solve http buffering issues and hangs once and for all (hopefully!) by
simply not trying to return exactly what the caller asked for - just
return whatever we got and let the caller be the judge of whether it
was enough.  If an error occurs or the connection times out after we
already received some data, return a short read, under the assumption
that the next call will fail or time out before we read anything.

As it turns out, none of the code that calls fetch_read() assumes an
all-or-nothing result anyway, except for a couple of lines where we
read the CR LF at the end of a hunk in HTTP hunked encoding, so the
changes outside of fetch_read() and http_readfn() are minimal.

While there, replace select(2) with poll(2).

MFC after:	3 days
2014-01-28 12:48:17 +00:00
des
61d98c4b31 Fix format string.
Submitted by:	Jörg Sonnenberger <joerg@NetBSD.org>
MFC after:	1 week
2014-01-20 11:13:05 +00:00
bdrewery
6f5000b05b Fix build with GCC
SSL_set_tlsext_host_name(3) internally does not modify the host buffer
pased to it. So it is safe to DECONST the struct url* here.

Reported by:	gjb
Approved by:	bapt (implicit)
MFC after:	1 week
X-MFC-With:	r258347
2013-11-19 16:11:03 +00:00
bdrewery
71362d24e8 Support SNI in libfetch
SNI is Server Name Indentification which is a protocol for TLS that
indicates the host that is being connected to at the start of the
handshake. It allows to use Virtual Hosts on HTTPS.

Submitted by:	sbz
Submitted by:	Michael Gmelin <freebsd@grem.de> [1]
PR:		kern/183583 [1]
Reviewed by:	des
Approved by:	bapt
MFC after:	1 week
2013-11-19 15:35:26 +00:00
sjg
62bb106222 Merge from head 2013-09-05 20:18:59 +00:00
des
0b46e38a09 Even though it doesn't really make sense in the context of a CONNECT
request, RFC 2616 14.23 mandates the presence of the Host: header in
all HTTP 1.1 requests.

PR:		kern/181445
Submitted by:	Kimo <kimor79@yahoo.com>
MFC after:	3 days
2013-08-22 07:43:36 +00:00
des
0b1a1b2338 Include an Accept header in requests.
PR:		kern/180917
MFC after:	1 week
2013-07-30 13:07:55 +00:00
des
29d3efe0a8 Implement certificate verification, and many other SSL-related
imrovements; complete details in the PR.

PR:		kern/175514
Submitted by:	Michael Gmelin <freebsd@grem.de>
MFC after:	1 week
2013-07-26 15:53:43 +00:00
des
50344d2c6f Use the correct request syntax for proxied (tunneled) HTTPS requests.
PR:		bin/180666
MFC after:	3 days
2013-07-21 06:59:56 +00:00
kientzle
7a1b8ef999 Fix -Wunsequenced warning.
Submitted by:	dt71@gmx.com
2013-06-29 15:51:27 +00:00
des
b2e53cc2ea Use the CONNECT method to proxy HTTPS connections through HTTP proxies.
PR:		bin/80176
Submitted by:	Yuichiro NAITO <naito.yuichiro@gmail.com>
2013-04-12 22:05:15 +00:00
sjg
6d37b86f2b Updated dependencies 2013-03-11 17:21:52 +00:00