According to POSIX, The mbtowc() function shall fail if:
[EILSEQ] An invalid character sequence is detected.
Reviewed by: bapt
Differential Revision: https://reviews.freebsd.org/D5496
Obtained from: OpenBSD (Ingo Schwarze)
MFC after: 1 month
The NONE:US-ASCII case isn't necessary. The "NONE:" case will handle
US-ASCII, so let's remove the redundant handling.
Submitted by: marino
Obtained from: DragonflyBSD
The US-ASCII format was getting treated identically to POSIX. It is
supposed to throw an ILSEQ errno if a value of 0x80 or greater is
encountered, so let's bring back the "ASCII" handling.
While here, change nl_codeset to return US-ASCII only when the encoding
really is "US-ASCII". Before "C" and "POSIX" encoding returned this
string, so now they return "POSIX".
Discussed with: ache
Submitted by: marino
Obtained from: DragonflyBSD
Sorting eucJP text with "sort" resulted in an illegal sequence while
"gsort" worked. This was traced back to mbrtowc handling which was
broken for eucJP (probably eucCN, eucKR, and eucTW as well). This
small fix took hours to figure out. The OR operation to build the
wide character requires an unsigned character to work correctly. The
euc wcrtowc conversion is probably broken upstream in Illumos as well.
Triggered by: misc/freebsd-doc-ja in ports (encoded in eucJP)
Submitted by: marino
Obtained from: DragonflyBSD
The output of "locale charmap" is identical to the result of
nl_langinfo (CODESET) for any given locale. The logic for returning the
codeset was very simplistic. It just returned portion of the locale name
after the period (e.g. en_FR.ISO8859-1 returned "ISO8859-1").
When softlinks were added to locales, this broke. e.g.:
en_US returned ""
en_FR.UTF8 returned "UTF8"
en_FR.UTF-8 returned "UTF-8"
zh_Hant_HK.Big5HKSCS returned "Big5HKSCS"
zh_Hant_TW.Big5 returned "Big5"
es_ES@euro returned ""
In order to fix this properly, the named locale cannot be used to
determine the encoding. This information was almost available in the
rune data. Unfortunately, all the single byte encodings were listed
as "NONE" encoding.
So I adjusted localedef tool to provide more information about the
encoding. For example, instead of "NONE", the LC_CTYPE used by
fr_FR.ISO8859-15 is now encoded as "NONE:ISO8859-15". The locale
handlers now check if the first four characters of the encoding is
"NONE" and if so, treats it as a single-byte encoding.
The nl_langinfo handling of CODESET was adjusting accordingly. Now the
following is returned:
en_US returns "ISO8859-1"
fr_FR.UTF8 returns "UTF-8"
fr_FR.UTF-8 returns "UTF-8"
zh_Hant_HK.Big5HKSCS returns "Big5"
zh_Hant_TW.Big5 returns "Big5"
es_ES@euro returns "ISO8859-15"
as before, "C" and "POSIX" locales return "US-ASCII". This is a big
improvement. The result of nl_langinfo can never be a zero-length
string and it will always exclusively one of the values of the
character maps of /usr/src/tools/tools/locale/etc/final-maps.
Submitted by: marino
Obtained from: DragonflyBSD
Through testing, the user noted that some Cyrillic characters were not
sorting correctly, and this was confirmed.
After extensive testing and review, the localedef tool was eliminated
as the culprit. The sustitutions were encoded correctly in LC_COLLATE.
The error was mainly in wcscoll where character expansions were
mishandled. The main directive pass routines had to be written to
go back for a new collation value when the "state" variable was set.
Before pointers were being advanced, the second lookup was gettting
applied to the wrong character, etc.
The "eat expansion codes" section on collate.c also had a bug. Later
own, the "state" variable logic was changed to only set if next
code was greater than zero (rather than >= 0).
Some additional cleanups got captured from previous work:
1) The previous commit moved the binary search comment from the
correct location to a wrong location because it's wrong upstream
in Illumos. The comment has little value so I just removed it.
2) Don't check if pointers are null before freeing, this is
redundant as free() handles null pointers.
3) The two binary search trees were standardized wrt initialization
4) On the binary search trees, a negative "high" exits rather than
checking the table count again.
Submitted by: marino
Obtained from: DragonflyBSD
The main "fix" here is properly setting a collate loading error for each
early return. Tweaks include removing unnecessary null checks, adding
assertions (from Illumos) and a couple of variables to reduces code
differences and improve readability. For normal use, there are no
functional changes here.
Obtained from: DragonflyBSD, Illumos
FreeBSD extended ctypes to include numbers (e.g. isnumber()) but never
actually implemented it. The isnumber() function was equivalent to the
isdigit() function in every case.
Now that DragonFly's ctype source files have number definitions, the
number ctype can finally be implemented. It's given a new flag _CTYPE_N.
The isalnum() and iswalnum() functions have been changed to use this
flag rather than the _CTYPE_D digit flag.
While isalnum(), isnumber(), and their wide equivalents now return
different values in locale cases, the ishexnumber() and iswhexnumber()
functions are unchanged. They are still aliases for isxdigit() and
iswxdigit().
Also change ctype.h for isdigit and isxdigit to use sbistype like the
other functions.
Obtained from: dragonfly
- in mbrtowc() we need to disallow codepoints above 0x10ffff.
- In wcrtomb() we need to disallow codepoints between 0xd800 and 0xdfff.
Reviewed by: bapt
Differential Revision: https://reviews.freebsd.org/D3399
packed LC_COLLATE binary formats. These were generated with the colldef
tool, but the new LC_COLLATE files are going to be generated by the new
localedef tool using CLDR POSIX files as input. The BSD-flavored
version of localedef identifies the format as "BSD 1.0". Any
LC_COLLATE file with a different version will simply not be loaded, and
all LC* categories will get set to "C" (aka "POSIX") locale.
This work is based off of Nexenta's contribution to Illumos.
The integration with xlocale is John Marino's work for Dragonfly.
The following commits will enable localedef tool, disable the colldef
tool, add generated colldef directory, and finally remove colldef from
base.
The only difference with Dragonfly are:
- a few fixes to build with clang
- And identification of the flavor as "BSD 1.0" instead of "Dragonfly 4.4"
Obtained from: Dragonfly
Fix a missing .h and change the recommended include for the POSIX2008 functions from xlocale.h to locale.h. Including xlocale.h is for legacy / Darwin compatibility so should not be encouraged.
causing mb* functions (and similar) to be called with the wrong data
(possibly a null pointer, causing a crash).
PR: standards/188036
MFC after: 1 week
if not already defined. This allows building libc from outside of
lib/libc using a reach-over makefile.
A typical use-case is to build a standard ILP32 version and a COMPAT32
version in a single iteration by building the COMPAT32 version using a
reach-over makefile.
Obtained from: Juniper Networks, Inc.
a macro with parameters. Remove a __DECONST hack and add consts instead
for gnu libiconv API compatability. This makes it work with things like
devel/boost-libs that expects to use "iconv" as though it were a pointer.
I initially thought wchar_t was locale independent, but this seems to be
only the case on Linux. This means that we cannot depend on the *wc*()
routines to implement *c16*() and *c32*(). Instead, use the Citrus
libiconv that is part of libc.
I'll see if there is anything I can do to make the existing functions
somewhat useful in case the system is built without libiconv in the
nearby future. If not, I'll simply remove the broken implementations.
Reviewed by: jilles, gabor
The <uchar.h> header, part of C11, adds a small number of utility
functions for 16/32-bit "universal" characters, which may or may not be
UTF-16/32. As our wchar_t is already ISO 10646, simply add light-weight
wrappers around wcrtomb() and mbrtowc().
While there, also add (non-yet-standard) _l functions, similar to the
ones we already have for the other locale-dependent functions.
Reviewed by: theraven
