Make shutdown() return ENOTCONN as required by POSIX, part deux.

Summary:
Back in 2005, maxim@ attempted to fix shutdown() to return ENOTCONN in case the socket was not connected (r150152). This had to be rolled back (r150155), as it broke some of the existing programs that depend on this behavior. I reapplied this change on my system and indeed, syslogd failed to start up. I fixed this back in February (279016) and MFC'ed it to the supported stable branches. Apart from that, things seem to work out all right.

Since at least Linux and Mac OS X do the right thing, I'd like to go ahead and give this another try. To keep old copies of syslogd working, only start returning ENOTCONN for recent binaries.

I took a look at the XNU sources and they seem to test against both SS_ISCONNECTED, SS_ISCONNECTING and SS_ISDISCONNECTING, instead of just SS_ISCONNECTED. That seams reasonable, so let's do the same.

Test Plan:
This issue was uncovered while writing tests for shutdown() in CloudABI:

https://github.com/NuxiNL/cloudlibc/blob/master/src/libc/sys/socket/shutdown_test.c#L26

Reviewers: glebius, rwatson, #manpages, gnn, #network

Reviewed By: gnn, #network

Subscribers: bms, mjg, imp

Differential Revision: https://reviews.freebsd.org/D3039

lib/libc/sys/shutdown.2

@@ -29,7 +29,7 @@
 .\"     @(#)shutdown.2	8.1 (Berkeley) 6/4/93
 .\" $FreeBSD$
 .\"
-.Dd March 5, 2007
+.Dd July 27, 2015
 .Dt SHUTDOWN 2
 .Os
 .Sh NAME
@@ -79,40 +79,26 @@ The following protocol specific actions apply to the use of
 based on the properties of the socket associated with the file descriptor
 .Fa s .
 .Bl -column ".Dv PF_INET6" ".Dv SOCK_STREAM" ".Dv IPPROTO_SCTP"
-.It Sy Domain Ta Sy Type Ta Sy Protocol Ta Sy Return value and action
+.It Sy Domain Ta Sy Type Ta Sy Protocol Ta Sy Action
 .It Dv PF_INET Ta Dv SOCK_DGRAM Ta Dv IPPROTO_SCTP Ta
-Return \-1.
-The global variable
-.Va errno
-will be set to
-.Er EOPNOTSUPP .
+Failure,
+as socket is not connected.
 .It Dv PF_INET Ta Dv SOCK_DGRAM Ta Dv IPPROTO_UDP Ta
-Return 0.
-ICMP messages will
-.Em not
-be generated.
+Failure,
+as socket is not connected.
 .It Dv PF_INET Ta Dv SOCK_STREAM Ta Dv IPPROTO_SCTP Ta
-Return 0.
 Send queued data and tear down association.
 .It Dv PF_INET Ta Dv SOCK_STREAM Ta Dv IPPROTO_TCP Ta
-Return 0.
 Send queued data, wait for ACK, then send FIN.
 .It Dv PF_INET6 Ta Dv SOCK_DGRAM Ta Dv IPPROTO_SCTP Ta
-Return \-1.
-The global variable
-.Va errno
-will be set to
-.Er EOPNOTSUPP .
+Failure,
+as socket is not connected.
 .It Dv PF_INET6 Ta Dv SOCK_DGRAM Ta Dv IPPROTO_UDP Ta
-Return 0.
-ICMP messages will
-.Em not
-be generated.
+Failure,
+as socket is not connected.
 .It Dv PF_INET6 Ta Dv SOCK_STREAM Ta Dv IPPROTO_SCTP Ta
-Return 0.
 Send queued data and tear down association.
 .It Dv PF_INET6 Ta Dv SOCK_STREAM Ta Dv IPPROTO_TCP Ta
-Return 0.
 Send queued data, wait for ACK, then send FIN.
 .El
 .\"
@@ -131,16 +117,10 @@ argument is not a valid file descriptor.
 The
 .Fa how
 argument is invalid.
-.It Bq Er EOPNOTSUPP
-The socket associated with the file descriptor
-.Fa s
-does not support this operation.
 .It Bq Er ENOTCONN
 The
 .Fa s
-argument specifies a
-.Dv SOCK_STREAM
-socket which is not connected.
+argument specifies a socket which is not connected.
 .It Bq Er ENOTSOCK
 The
 .Fa s

sys/kern/uipc_socket.c

@@ -2334,6 +2334,9 @@ soshutdown(struct socket *so, int how)
 
 	if (!(how == SHUT_RD || how == SHUT_WR || how == SHUT_RDWR))
 		return (EINVAL);
+	if ((so->so_state &
+	    (SS_ISCONNECTED | SS_ISCONNECTING | SS_ISDISCONNECTING)) == 0)
+		return (ENOTCONN);
 
 	CURVNET_SET(so->so_vnet);
 	if (pr->pr_usrreqs->pru_flush != NULL)

sys/kern/uipc_syscalls.c

@@ -1383,6 +1383,15 @@ sys_shutdown(td, uap)
 	if (error == 0) {
 		so = fp->f_data;
 		error = soshutdown(so, uap->how);
+		/*
+		 * Previous versions did not return ENOTCONN, but 0 in
+		 * case the socket was not connected. Some important
+		 * programs like syslogd up to r279016, 2015-02-19,
+		 * still depend on this behavior.
+		 */
+		if (error == ENOTCONN &&
+		    td->td_proc->p_osrel < P_OSREL_SHUTDOWN_ENOTCONN)
+			error = 0;
 		fdrop(fp, td);
 	}
 	return (error);

sys/sys/param.h

@@ -81,6 +81,7 @@
 #define	P_OSREL_SIGSEGV			700004
 #define	P_OSREL_MAP_ANON		800104
 #define	P_OSREL_MAP_FSTRICT		1100036
+#define	P_OSREL_SHUTDOWN_ENOTCONN	1100077
 
 #define	P_OSREL_MAJOR(x)		((x) / 100000)
 #endif