Change the way the advertized TCP window scaling is computed. Instead of

upper-bounding it to the size of the initial socket buffer lower-bound it
to the smallest MSS we accept.  Ideally we'd use the actual MSS information
here but it is not available yet.

For socket buffer auto sizing to be effective we need room to grow the
receive window.  The window scale shift is determined at connection setup
and can't be changed afterwards.  The previous, original, method effectively
just did a power of two roundup of the socket buffer size at connection
setup severely limiting the headroom for larger socket buffers.

Tested by:	many (as part of the socket buffer auto sizing patch)
MFC after:	1 month

sys/netinet/tcp_syncache.c

@@ -1014,9 +1014,15 @@ syncache_add(struct in_conninfo *inc, struct tcpopt *to, struct tcphdr *th,
 		if (to->to_flags & TOF_SCALE) {
 			int wscale = 0;
 
-			/* Compute proper scaling value from buffer space */
+			/*
+			 * Compute proper scaling value from buffer space.
+			 * Leave enough room for the socket buffer to grow
+			 * with auto sizing.  This allows us to scale the
+			 * receive buffer over a wide range while not losing
+			 * any efficiency or fine granularity.
+			 */
 			while (wscale < TCP_MAX_WINSHIFT &&
-			    (TCP_MAXWIN << wscale) < sb_hiwat)
+			    (0x1 << wscale) < tcp_minmss)
 				wscale++;
 			sc->sc_requested_r_scale = wscale;
 			sc->sc_requested_s_scale = to->to_requested_s_scale;

sys/netinet/tcp_usrreq.c

@@ -1131,9 +1131,14 @@ tcp_connect(tp, nam, td)
 	inp->inp_laddr = laddr;
 	in_pcbrehash(inp);
 
-	/* Compute window scaling to request.  */
+	/*
+	 * Compute window scaling to request:
+	 * Scale to fit into sweet spot.  See tcp_syncache.c.
+	 * XXX: This should move to tcp_output().
+	 * XXX: This should be based on the actual MSS.
+	 */
 	while (tp->request_r_scale < TCP_MAX_WINSHIFT &&
-	    (TCP_MAXWIN << tp->request_r_scale) < so->so_rcv.sb_hiwat)
+	    (0x1 << tp->request_r_scale) < tcp_minmss)
 		tp->request_r_scale++;
 
 	soisconnecting(so);