The previous incarnation of this would call wcrtomb() on the destination
buffer, and only check for overflow *after* it's happened.
Additionally, the conversion error / VIS_NOLOCALE path also didn't check
for overflow, and the overflow check at the end didn't account for the
fact that we still need to write a NUL terminator afterward.
Start by only doing the multibyte conversion into mbdst directly if we
have enough buffer space to guarantee it'll fit. An additional
MB_CUR_MAX buffer has been stashed on the stack to write into if we're
cutting it close at the end of the buffer, since we don't really have a
good way to determine the length of the wchar_t without just doing the
conversion. We'll do the conversion into the buffer that's guaranteed
to fit, then copy it over if the copy won't overflow.
The byte-for-byte overflow is a little bit easier, as we simply check
for overflow with each byte written and make sure we can still NUL
terminate after.
Tests added to exercise these edge cases.
Reviewed by: des
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D41328
2f489a509e