Commit Graph

344 Commits

Author SHA1 Message Date
Jung-uk Kim
01ed3ca3de Remove defunct SSLv2 support from fetch(1) and fetch(3). 2015-03-25 18:56:36 +00:00
Baptiste Daroussin
6b129086dc Convert libraries to use LIBADD
While here reduce a bit overlinking
2014-11-25 11:07:26 +00:00
Dag-Erling Smørgrav
767207650c As pointed out by several people, r273114 was incorrect: it unconditionally
disabled everything except TLS 1.0.  Replace it with a more carefully
wrought patch:

 - Switch the default for SSLv3 from on to off
 - Add environment variables to control TLS 1.1 and 1.2
 - In verbose mode, report which version is used
 - Update the man page to reflect these changes.

MFC after:	1 week
2014-10-15 07:35:50 +00:00
Dag-Erling Smørgrav
d75ca0c1fd Drop support for SSLv3.
MFC after:	1 week
2014-10-15 02:34:26 +00:00
Baptiste Daroussin
3674911c29 Add a comment to explain the EAGAIN is only there for POSIX compliance
Resquested by:	kib
Reviewed by:	des
2014-07-15 15:29:43 +00:00
Baptiste Daroussin
4472d6e1df Support EAGAIN in fetch_writev
Reviewed by:	des
Approved by:	des
2014-07-10 13:04:52 +00:00
Baptiste Daroussin
6cfac37b03 Use Mt macro to properly format mailto links (patch from Franco Fichtner <franco@lastsummer.de> for Dragonfly)
Reviewed by:	des
Approved by:	des
Obtained from:	Dragonfly
MFC after:	1 week
2014-06-11 20:20:14 +00:00
Baptiste Daroussin
c41991303c Add support for arbitrary http requests
Submitted by:	Alex Hornung <alex@alexhornung.com>
Reviewed by:	des
Obtained from:	Dragonfly
MFC after:	3 week
2014-06-05 22:16:26 +00:00
Baptiste Daroussin
4bd8c06c3a Remove unnecessary semicolons
Patch by Sascha Wildner <saw@online.de> for Dragonfly

Reviewed by:	des
Obtained from:	Dragonfly
MFC after:	1 week
2014-06-05 22:13:30 +00:00
Baptiste Daroussin
6064928d01 Use NULL instead of 0
Patch by Sascha Wildner <saw@online.de> for Dragonfly

Reviewed by:	des
Obtained from:	Dragonfly
MFC after:	1 week
2014-06-05 22:10:25 +00:00
Dag-Erling Smørgrav
c257f99e9b If HTTP_USER_AGENT is defined but empty, don't send User-Agent at all.
PR:		184507
Submitted by:	jbeich@tormail.org (with modifications)
MFC after:	1 week
2014-06-05 20:27:16 +00:00
Dag-Erling Smørgrav
e9de92ad83 Look for root certificates in /usr/local/etc/ssl before /etc/ssl.
MFH:	1 week
2014-05-17 03:39:56 +00:00
Warner Losh
c6063d0da8 Use src.opts.mk in preference to bsd.own.mk except where we need stuff
from the latter.
2014-05-06 04:22:01 +00:00
Bryan Drewery
b36853caf1 Support Last-Modified behind proxies which return UTC instead of GMT.
The standard states that GMT must be used, but that UTC is equivalent. Still
parse UTC as otherwise this causes problems for pkg(8). It will refetch
the repository every time 'pkg update' or other remote operations
are used behind these proxies.

RFC2616: "All HTTP date/time stamps MUST be represented in Greenwich Mean
Time (GMT), without exception. For the purposes of HTTP, GMT is exactly equal
to UTC (Coordinated Universal Time).""

Approved by:	bapt (mentor)
Reviewed by:	des, peter
Sponsored by:	EMC / Isilon Storage Division
MFC after:	1 week
2014-03-11 13:47:11 +00:00
Dag-Erling Smørgrav
4524013cd3 Bump copyright dates 2014-01-30 08:37:23 +00:00
Dag-Erling Smørgrav
9c1ca3a1dd r261230 broke the cases where the amount of data to be read is not
known in advance, or where the caller doesn't care and just keeps
reading until it hits EOF.

In fetch_read(): the socket is non-blocking, so read() will return 0
on EOF, and -1 (errno == EAGAIN) when the connection is still open but
there is no data waiting.  In the first case, we should immediately
return 0.  The EINTR case was also broken, although not in a way that
matters.

In fetch_writev(): use timersub() and timercmp() as in fetch_read().

In http_fillbuf(): set errno to a sensible value when an invalid chunk
header is encountered.

In http_readfn(): as in fetch_read(), a zero return from down the
stack indicates EOF, not an error.  Furthermore, when io->error is
EINTR, clear it (but no errno) before returning so the caller can
retry after dealing with the interrupt.

MFC after:	3 days
2014-01-29 12:48:19 +00:00
Dag-Erling Smørgrav
215a27f1a4 Solve http buffering issues and hangs once and for all (hopefully!) by
simply not trying to return exactly what the caller asked for - just
return whatever we got and let the caller be the judge of whether it
was enough.  If an error occurs or the connection times out after we
already received some data, return a short read, under the assumption
that the next call will fail or time out before we read anything.

As it turns out, none of the code that calls fetch_read() assumes an
all-or-nothing result anyway, except for a couple of lines where we
read the CR LF at the end of a hunk in HTTP hunked encoding, so the
changes outside of fetch_read() and http_readfn() are minimal.

While there, replace select(2) with poll(2).

MFC after:	3 days
2014-01-28 12:48:17 +00:00
Dag-Erling Smørgrav
3443b2bc31 Fix format string.
Submitted by:	Jörg Sonnenberger <joerg@NetBSD.org>
MFC after:	1 week
2014-01-20 11:13:05 +00:00
Bryan Drewery
5313966ea6 Fix build with GCC
SSL_set_tlsext_host_name(3) internally does not modify the host buffer
pased to it. So it is safe to DECONST the struct url* here.

Reported by:	gjb
Approved by:	bapt (implicit)
MFC after:	1 week
X-MFC-With:	r258347
2013-11-19 16:11:03 +00:00
Bryan Drewery
4cb272a079 Support SNI in libfetch
SNI is Server Name Indentification which is a protocol for TLS that
indicates the host that is being connected to at the start of the
handshake. It allows to use Virtual Hosts on HTTPS.

Submitted by:	sbz
Submitted by:	Michael Gmelin <freebsd@grem.de> [1]
PR:		kern/183583 [1]
Reviewed by:	des
Approved by:	bapt
MFC after:	1 week
2013-11-19 15:35:26 +00:00
Dag-Erling Smørgrav
615c5740ef Even though it doesn't really make sense in the context of a CONNECT
request, RFC 2616 14.23 mandates the presence of the Host: header in
all HTTP 1.1 requests.

PR:		kern/181445
Submitted by:	Kimo <kimor79@yahoo.com>
MFC after:	3 days
2013-08-22 07:43:36 +00:00
Dag-Erling Smørgrav
1453595f49 Include an Accept header in requests.
PR:		kern/180917
MFC after:	1 week
2013-07-30 13:07:55 +00:00
Dag-Erling Smørgrav
dcd47379ff Implement certificate verification, and many other SSL-related
imrovements; complete details in the PR.

PR:		kern/175514
Submitted by:	Michael Gmelin <freebsd@grem.de>
MFC after:	1 week
2013-07-26 15:53:43 +00:00
Dag-Erling Smørgrav
ba7c6aec97 Use the correct request syntax for proxied (tunneled) HTTPS requests.
PR:		bin/180666
MFC after:	3 days
2013-07-21 06:59:56 +00:00
Tim Kientzle
9bc22394d8 Fix -Wunsequenced warning.
Submitted by:	dt71@gmx.com
2013-06-29 15:51:27 +00:00
Dag-Erling Smørgrav
4056bae982 Use the CONNECT method to proxy HTTPS connections through HTTP proxies.
PR:		bin/80176
Submitted by:	Yuichiro NAITO <naito.yuichiro@gmail.com>
2013-04-12 22:05:15 +00:00
Dag-Erling Smørgrav
eab7a548ba Fix weird indentation. 2012-11-16 12:31:43 +00:00
Eitan Adler
8d049fb235 Implement HTTP 305 redirect handling.
PR:		172452
Submitted by:	gcooper
Reviewed by:	des
Approved by:	cperciva
MFC after:	1 week
2012-10-22 03:00:15 +00:00
Eitan Adler
c4fa1489ec Don't deny non-temporary redirects if the -A option is set (per
the man page) [0]

While here add support for draft-reschke-http-status-308-07

PR:		172451 [0]
Submitted by:	gcooper [0]
Reviewed by:	des
Approved by:	cperciva
MFC after:	1 week
2012-10-22 03:00:10 +00:00
Eitan Adler
e6c0e200f4 Be a bit more lenient in the maximum number of redirects allowed.
Chrome and Firefox have a limit of 20. IE has a limit of 8.

Reviewed by:	des
Approved by:	cperciva
MFC after:	3 days
2012-10-22 03:00:04 +00:00
Dag-Erling Smørgrav
0e50a83330 Use libmd if and only if OpenSSL is not available.
PR:		bin/171402
MFC after:	3 days
2012-09-14 13:00:43 +00:00
Eitan Adler
42415fcb52 Adding missing return statements during error conditions.
PR:		kern/171187
Submitted by:	Mark Johnston <markjdb@gmail.com>
Reviewed by:	des
Approved by:	cperciva
MFC after:	2 weeks
2012-09-14 12:15:13 +00:00
Jung-uk Kim
1f13597d10 Merge OpenSSL 1.0.1c.
Approved by:	benl (maintainer)
2012-07-12 19:30:53 +00:00
Jilles Tjoelker
28fd93073a libfetch: Avoid SIGPIPE on network connections.
To avoid unexpected process termination from SIGPIPE when writing to a
closed network connection, enable SO_NOSIGPIPE on all network connections.

The POSIX standard MSG_NOSIGNAL is not used since it requires modifying all
send calls to add this flag. This is particularly nasty for SSL connections.

Reviewed by:	des
Tested by:	bapt
MFC after:	5 days
2012-05-28 19:22:23 +00:00
Dag-Erling Smørgrav
f51b84bcc4 Don't reuse credentials if redirected to a different host.
Submitted by:	Niels Heinen <heinenn@google.com>
MFC after:	3 weeks
2012-04-30 12:12:48 +00:00
Dag-Erling Smørgrav
30787285b5 Since the socket is non-blocking, it is necessary to use select(2) even
when there is no timeout, because read(2) will return immediately if there
is no data waiting in the TCP buffer, causing fetch_read() to busy-loop on
slow connections.

MFC after:	3 weeks
Noticed by:	Yanhui Shen <shen.elf@gmail.com>
2012-04-30 12:11:45 +00:00
Ed Maste
0fa391997a Support percent-encoded user and password
RFC 1738 specifies that any ":", "@", or "/" within a user name or
password in a URL is percent-encoded, to avoid ambiguity with the use
of those characters as URL component separators.

Reviewed by:    rstone@
MFC after:	1 month
2012-04-11 15:42:02 +00:00
Eitan Adler
50d675f7a9 Remove trailing whitespace per mdoc lint warning
Disussed with:	gavin
No objection from:	doc
Approved by:	joel
MFC after:	3 days
2012-03-29 05:02:12 +00:00
Dag-Erling Smørgrav
e6e3bcd376 Fix two nits in previous commit pointed out by pjd@.
MFC after:	3 weeks
2012-01-23 09:23:07 +00:00
Dag-Erling Smørgrav
2a7daafe67 Fix two issues related to the use of SIGINFO in fetch(1) to display
progress information.  The first is that fetch_read() (used in the HTTP
code but not the FTP code) can enter an infinite loop if it has previously
been interrupted by a signal.  The second is that when it is interrupted,
fetch_read() will discard any data it may have read up to that point.
Luckily, both bugs are extremely timing-sensitive and therefore difficult
to trigger.

PR:		bin/153240
Submitted by:	Mark <markjdb@gmail.com>
MFC after:	3 weeks
2012-01-18 15:13:21 +00:00
Dag-Erling Smørgrav
578153f1ba latin1 -> utf8 2011-10-19 11:43:51 +00:00
Dag-Erling Smørgrav
6337341d81 Update copyright dates and strip my middle name. 2011-09-27 18:57:26 +00:00
Dag-Erling Smørgrav
76b94eb6e9 Bump date. 2011-09-27 18:53:35 +00:00
Dag-Erling Smørgrav
ecd18c961b Think first, commit second.
1. Allow the caller to select active mode.
2. Fix the envar logic so it *always* overrides the caller's flags.
3. Document the change from active to passive.
2011-09-27 18:42:09 +00:00
Dag-Erling Smørgrav
4f411f8601 Long overdue: make passive mode the default for ftp. 2011-09-27 18:23:58 +00:00
Dag-Erling Smørgrav
15b68c63bb Mark all socket and file descriptors close-on-exec.
PR:		bin/151866
MFC after:	3 weeks
2011-05-13 07:21:41 +00:00
Dag-Erling Smørgrav
ee3cb0e502 Builds cleanly at the default WARNS level (WARNS=6). 2011-05-12 21:30:46 +00:00
Dag-Erling Smørgrav
eb9b80c30d Increase WARNS to 4. 2011-05-12 21:26:42 +00:00
Dag-Erling Smørgrav
c12c6e3cda Mechanical whitespace cleanup. 2011-05-12 21:18:55 +00:00
Dag-Erling Smørgrav
a42eecded0 Increase WARNS to 3. 2011-05-12 21:12:24 +00:00