Provide a way to ask for an opaque version string for a locale_t, so
that potential changes in sort order can be detected. Similar to
ICU's ucol_getVersion() and Windows' GetNLSVersionEx(), this API is
intended to allow databases to detect when text order-based indexes
might need to be rebuilt.
The CLDR version is extracted from CLDR source data by the Makefile
under tools/tools/locale, written into the machine-generated Makefile
under shared/colldef, passed to localedef -V, and then written into
LC_COLLATE file headers. The initial version is 34.0.
tools/tools/locale was recently updated to pull down 35.0, but the
output hasn't been committed under share/colldef yet, so that will
provide the first observable change when it happens. Other versioning
schemes are possible in future, because the format is unspecified.
Reviewed by: bapt, 0mp, kib, yuripv (albeit a long time ago)
Differential Revision: https://reviews.freebsd.org/D17166
newlocale() optionally takes a "base" locale, from which components not
specified in the mask are inherited. POSIX says that newlocale() may
modify "base" and return it, or free "base" and return a newly allocated
locale. We were not doing either, so applications which use newlocale()
to modify an existing base locale end up leaking memory on FreeBSD.
This diff fixes the leak by releasing a reference to the base locale
before returning. This is less efficient than modifying "base"
directly, but is simpler for an initial bug fix. Also, update the man
page to clarify behaviour with respect to "base".
PR: 249416
MFC after: 3 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D26522
Posix says that the interpretation of the locale string is
"implementation-defined", so we ought to document what is
actually recognized.
Also add a cross reference to locale(1).
PR: 247553
MFC after: 1 week
In a single-threaded program pthread_getspecific() always returns NULL,
so the old locale would not end up being freed.
PR: 239520
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
- Fix checks for mmap() failures. [1]
- Set the "map" and "maplen" fields of struct xlocale_collate so that
the table destructor actually does something.
- Free an already-mapped collation file before loading a new one into
the global table.
- Harmonize the prototype and definition of __collate_load_tables_l() by
adding the "static" qualifier to the latter.
PR: 243195
Reported by: cem [1]
Reviewed by: cem, yuripv
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D23109
- Fix the statement that big5 is a de facto standard of Traditional Chinese
text [1]
- Add a BUGS section describes the problem of big5 and suggests use utf8
PR: 189095
Submitted by: Brennan Vincent <brennan@umanwizard.com> [1]
Reviewed by: Ting-Wei Lan <lantw44@gmail.com>
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D21622
This was shown to be a problem by side effect of now-enabled test case,
which was going through C, en_US.UTF-8, ja_JP.SJIS, and ja_JP.eucJP,
and failing eventually as data in mbrtowc's mbstate, that was
perfectly correct for en_US.UTF-8 was treated as incorrect for
ja_JP.SJIS, failing the entire test case.
This makes the persistent mbstates to be per ctype-component,
and not per-locale so we could easily reset the mbstates when
only LC_CTYPE is changed.
Reviewed by: bapt, pfg
Approved by: kib (mentor, implicit)
Differential Revision: https://reviews.freebsd.org/D17796
that it uses the same ctype maps and functions as other UTF-8 locales.
Reviewed by: bapt, cem, eadler
Approved by: kib (mentor, implicit)
Differential Revision: https://reviews.freebsd.org/D17833
it appropriately when building share/ctypedef and share/colldef.
This makes the resulting locale data in EL->EB (amd64->powerpc64) cross
build and in the native EB build match. Revert the changes done to libc
in r308170 as they are no longer needed.
PR: 231965
Reviewed by: bapt, emaste, sbruno, 0mp
Approved by: kib (mentor)
Differential Revision: https://reviews.freebsd.org/D17603
Mainly focus on files that use BSD 2-Clause license, however the tool I
was using mis-identified many licenses so this was mostly a manual - error
prone - task.
The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.
Mainly focus on files that use BSD 3-Clause license.
The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.
Special thanks to Wind River for providing access to "The Duke of
Highlander" tool: an older (2014) run over FreeBSD tree was useful as a
starting point.
The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.
Special thanks to Wind River for providing access to "The Duke of
Highlander" tool: an older (2014) run over FreeBSD tree was useful as a
starting point.
Initially, only tag files that use BSD 4-Clause "Original" license.
RelNotes: yes
Differential Revision: https://reviews.freebsd.org/D13133
Also fix the style of the asprintf(3) call in __collate_load_tables_l().
Both of these lines were modified away from snprintf(3) during the
import from DragonFly/Illumos.
Reviewed by: jilles (briefly over shoulder)
MFC after: 2 weeks
Sponsored by: Dell EMC Isilon
Illumos and Schillix is adopting some of the locale code and our style(9)
sometimes matches the Solaris cstyle, so the changes are also useful as a
way to reduce diffs.
No functional change.
Discussed with: Joerg Schilling
MFC after: 1 week
Renumber cluase 4 to 3, per what everybody else did when BSD granted
them permission to remove clause 3. My insistance on keeping the same
numbering for legal reasons is too pedantic, so give up on that point.
Submitted by: Jan Schaumann <jschauma@stevens.edu>
Pull Request: https://github.com/freebsd/freebsd/pull/96
Apple had them at the start but moving them to the end is better for
faster reading and fits better what is done in other FreeBSD headers.
MFC after: 5 days
When wcstof() skipped initial space and then parsing failed, it set
endptr to the first non-space character. Fix it to correctly report
failure by setting endptr to the beginning of the input string.
The fix is from theraven@, who fixed this bug in wcstod() and
wcstold() in r227753.
While I'm here:
Move assignments out of declarations in wcstod() and wcstold().
This is against my personal preference, but it is our agreed style(9).
Set endptr correctly on malloc() failure in all three functions.
Remove an incorrect comment: This is pointer arithmetic,
so the code was not actually making that assumption.
wcstold() advanced the wcp pointer beyond leading whitespace
and then reset it back to the beginning of the string.
Do not reset it. This seems to have no functional effect,
since strtold_l() also skips leading whitespace. I'm making
the change to keep this function consistent with wcstof() and
wcstod(), and because the C11 spec prescribes the use of iswspace()
to skip leading space.
Reported by: libc++ unit test for std::stof(std::wstring)
MFC after: 8 days
Sponsored by: Dell EMC
We have locale files generated on EL machines (e.g. during cross-build
on amd64 host), but then we are using them on EB machines (e.g. MIPS64EB),
so proceed byte-swap if necessary.
All the libc tests passed successfully, including Russian collation.
Tested by: br@, Hongyan Xia <hx242@cam.ac.uk>
Sponsored by: DARPA, AFRL
Sponsored by: HEIF5
Differential Revision: https://reviews.freebsd.org/D8281
Our version of this function currently returns an integer indicating
failure or success, whereas POSIX specifies that this function has no
return value. It returns void. Patch up the header, sources and man page
to use the right type. While there, use the opportunity to simplify the
body of this function.
Theoretically speaking, this change breaks the ABI of this function.
That said, I have yet to find any code that makes use of freelocale()'s
return value. I couldn't find any of it in the base system, nor did an
exp-run reveal any breakage caused by this change.
PR: 211394 (exp-run)
locale (which cause core dump) by removing whole 'table' argument
by which it passed.
2) Restore __collate_range_cmp() in __sccl().
3) Collating [a-z] range in regcomp() only for single bytes locales
(we can't do it now for other ones). In previous state only first 256
wchars are considered and all others are just silently dropped from the
range.
Instead of changing whole course to another POSIX-permitted way
for consistency and uniformity I decide to completely ignore missing
regex fucntionality and concentrace on fixing bugs in what we have now,
too many small obstacles instead, counting ports.
Only first 256 wide chars are considered currently, all other are just
dropped from the range. Proper implementation require reverse tables
database lookup, since objects are really big as max UTF-8 (1114112
code points), so just the same scanning as it was for 256 chars will
slow things down.
POSIX does not require collation for [a-z] type ranges and does not
prohibit it for non-POSIX locales. POSIX require collation for ranges
only for POSIX (or C) locale which is equal to ASCII and binary for
other chars, so we already have it.
No other *BSD implements collation for [a-z] type ranges.
Restore ABI compatibility with unused now __collate_range_cmp() which
is visible from outside (will be removed later).
When collation support was brought in, the second and third
arguments in __collate_range_cmp() were changed from int to
wchar_t, breaking the ABI. Change them to a "char" type which
makes more sense and keeps the ABI compatible.
Also introduce __wcollate_range_cmp() which does work with wide
characters. This function is used only internally in libc so
we don't export it. Use the new function in glob(3), fnmatch(3),
and regexec(3).
PR: 179721
Suggested by: ache. jilles
MFC after: 3 weeks (perhaps partial only)
While here also cleanup some surrounding code; particularly
drop some malloc() casts.
Found with devel/coccinelle.
Reviewed by: bde (previous version - all new bugs are mine)
According to POSIX, The mbtowc() function shall fail if:
[EILSEQ] An invalid character sequence is detected.
Reviewed by: bapt
Differential Revision: https://reviews.freebsd.org/D5496
Obtained from: OpenBSD (Ingo Schwarze)
MFC after: 1 month
The NONE:US-ASCII case isn't necessary. The "NONE:" case will handle
US-ASCII, so let's remove the redundant handling.
Submitted by: marino
Obtained from: DragonflyBSD
The US-ASCII format was getting treated identically to POSIX. It is
supposed to throw an ILSEQ errno if a value of 0x80 or greater is
encountered, so let's bring back the "ASCII" handling.
While here, change nl_codeset to return US-ASCII only when the encoding
really is "US-ASCII". Before "C" and "POSIX" encoding returned this
string, so now they return "POSIX".
Discussed with: ache
Submitted by: marino
Obtained from: DragonflyBSD
