Commit Graph

19 Commits

Author SHA1 Message Date
Yuri Pankov
2d1cfed1b1 localedef: define characters in "space" class also as "print", except
for the known conflicts ("control" characters can't be "print"able).
POSIX doesn't explicitly forbid this, and actually includes <space>
character in "print".

PR:		225692
Reviewed by:	bapt, cem (previous version), pfg (previous version)
Approved by:	kib (mentor)
Differential Revision:	https://reviews.freebsd.org/D17467
2018-10-27 23:31:42 +00:00
Yuri Pankov
4644f9bef6 Add -b/-l options to localedef(1) to specify output endianness and use
it appropriately when building share/ctypedef and share/colldef.

This makes the resulting locale data in EL->EB (amd64->powerpc64) cross
build and in the native EB build match.  Revert the changes done to libc
in r308170 as they are no longer needed.

PR:		231965
Reviewed by:	bapt, emaste, sbruno, 0mp
Approved by:	kib (mentor)
Differential Revision:	https://reviews.freebsd.org/D17603
2018-10-20 20:51:05 +00:00
Eitan Adler
dae3a64fb9 userland: Fix several typos and minor errors
- duplicate words
- typos
- references to old versions of FreeBSD

Reviewed by:	imp, benno
2017-12-27 03:23:01 +00:00
Pedro F. Giffuni
e12a957f8d localedef(1): Add comment markings for license. 2017-03-10 16:12:16 +00:00
Pedro F. Giffuni
56b1edd680 localedef(1): Fix mismatch.
Obtained from:	illumos
X-MFC with:	r314974
2017-03-10 16:06:14 +00:00
Pedro F. Giffuni
1bb0ddf99d localedef(1): Fix small coverity issues.
- Operands don't affect result (CONSTANT_EXPRESSION_RESULT)
- Buffer not null terminated (BUFFER_SIZE_WARNING)

CID:	1338557, 1338565

Obtained from:	illumos
MFC after:	5 days
2017-03-09 21:49:11 +00:00
Pedro F. Giffuni
c48c87b790 Revert r314969, r314961:
The localdef(1) changes are breaking world:

00:18:40.750 /usr/src/share/colldef/af_ZA.UTF-8.src: 2421: error: Bad file
descriptor

I will fix them offline.

Reported by:	lwshu and many others
2017-03-09 19:02:36 +00:00
Pedro F. Giffuni
830784ef0f localedef(1): Fix for memory leaks reported by coverity.
Also some small cleanups to match better current illumos.

CID: 1338540, 1338541, 1338557, 1338566

Obtained from:	illumos
Discussed with:	Yuri Pankov (@Nexenta)
MFC after:	5 days
2017-03-09 15:21:03 +00:00
Baptiste Daroussin
c7edf4fd0b localedef: Fix ctype dump (fixed wide spread errors)
This commit is from John Marino in dragonfly with the following commit log:

====
This was a CTYPE encoding error involving consecutive points of the same
ctype.  It was reported by myself to Illumos over a year ago but I was
unsure if it was only happening on BSD.  Given the cause, the bug is also
present on Illumos.

Basically, if consecutive points were of the exact same ctype, they would
be defined as a range regardless.  For example, all of these would be
considered equivalent:

  <A> ... <C>, <H>  (converts to <A> .. <H>)
  <A>, <B>, <H>     (converts to <A> .. <H>)
  <A>, <J> ... <H>  (converts to <A> .. <H>)

So all the points that shouldn't have been defined got "bridged" by the
extreme points.

The effects were recently reported to FreeBSD on PR 213013.  There are
countless places were the ctype flags are misdefined, so this is a major
fix that has to be MFC'd.
====

This reveals a bad change I did on the testsuite: while 0x07FF is a valid
unicode it is not used yet (reserved for future use)

PR:		213013
Submitted by:	marino@
Reported by:	Kurtis Rader <krader@skepticism.us>
Obtained from:	Dragonfly
MFC after:	1 month
2016-10-06 19:46:43 +00:00
Pedro F. Giffuni
046c3cda83 localedef(1): minor sorting to match Illumos.
Illumos recently included space in 'print' class. We already had
this but the code had slight sorting differences. Move it some
lines up to reduce diffs with Illumos.

No functional change.

Reference:
https://illumos.org/issues/5227
2016-03-20 03:27:06 +00:00
Baptiste Daroussin
8c859b074e Fix build of localedef(1) on arm where wchar_t is an unsigned int 2015-11-07 22:57:00 +00:00
Baptiste Daroussin
71e8badedc Actually only T_ISDIGIT should be flagged as _E4 2015-10-19 14:48:31 +00:00
Baptiste Daroussin
227d35dac0 With regard to ctype, digits (e.g. 0 to 9) and xdigits (the 0 to 9 portion
of hexidecimal numbers) are all considered "numbers".  (Note that while
all digits are numbers, not all numbers are digits).

Enhance localedef to automatically set the "number" characteristic when
it encounters a digit or xdigit definition. This fixes malfunctionning
isalnum(3)

Obtained from:	DragonflyBSD
2015-10-19 14:30:28 +00:00
Baptiste Daroussin
8833f5e9c2 eliminate need for "print" definition
By having space automatically classified as "print" type, we can
eliminate the print section from ctype src files completely (they
are just "graph" plus "<space>".

Obtained from:	Dragonfly
2015-10-13 20:45:29 +00:00
Baptiste Daroussin
f5dde0166d Commit log from Dragonfly:
FreeBSD extended ctypes to include numbers (e.g. isnumber()) but never
actually implemented it.  The isnumber() function was equivalent to the
isdigit() function in every case.

Now that DragonFly's ctype source files have number definitions, the
number ctype can finally be implemented.  It's given a new flag _CTYPE_N.
The isalnum() and iswalnum() functions have been changed to use this
flag rather than the _CTYPE_D digit flag.

While isalnum(), isnumber(), and their wide equivalents now return
different values in locale cases, the ishexnumber() and iswhexnumber()
functions are unchanged.  They are still aliases for isxdigit() and
iswxdigit().

Also change ctype.h for isdigit and isxdigit to use sbistype like the
other functions.

Obtained from:	dragonfly
2015-10-13 20:43:49 +00:00
Baptiste Daroussin
3a444dbdbf Convert localedef(1) from avl to RB trees 2015-08-08 22:57:17 +00:00
Baptiste Daroussin
e30c75b1f5 Prefer static generation of functions 2015-08-08 22:01:54 +00:00
Baptiste Daroussin
6131dc6a88 Convert ctype generation to Red Black tree 2015-08-08 21:53:02 +00:00
Baptiste Daroussin
057ca2d437 Add localedef(1), a locale definition generator tool
The localedef tool can read entire (and unmodified) CLDR posix definition
files, and generate all 6 LC categories: LC_COLLATE, LC_CTYPE, LC_TIME,
LC_NUMERIC, LC_MONETARY and LC_MESSAGES.

This tool has a long history with Solaris.  The Nexenta developers
modified it to read CLDR files and created the much richer collation
formats.  The libc collation functions have to be modified to read the
new format (called "BSD-1.0") and to handle the new data structures.

The result will be that locale-sensitive tools and functions will now
properly sort multibyte and unicode strings.

Obtained from:	Dragonfly
2015-08-07 23:53:31 +00:00