Commit Graph

35 Commits

Author SHA1 Message Date
Baptiste Daroussin
bbf9a45630 localedef: Improve cc_list parsing
original commit log:
=====
I had originally suspected the parsing of ctype definition files as being
the source of the ctype flag mis-definitions, but it wasn't.  In the
process, I simplified the cc_list parsing so I'm committing the no-impact
improvement separately.  It removes some parsing redundancies and
won't parse partial range definitions anymore.
====

Submitted by:	marino
Obtained from:	Dragonfly
MFC after:	1 month
2016-10-06 19:51:30 +00:00
Baptiste Daroussin
c7edf4fd0b localedef: Fix ctype dump (fixed wide spread errors)
This commit is from John Marino in dragonfly with the following commit log:

====
This was a CTYPE encoding error involving consecutive points of the same
ctype.  It was reported by myself to Illumos over a year ago but I was
unsure if it was only happening on BSD.  Given the cause, the bug is also
present on Illumos.

Basically, if consecutive points were of the exact same ctype, they would
be defined as a range regardless.  For example, all of these would be
considered equivalent:

  <A> ... <C>, <H>  (converts to <A> .. <H>)
  <A>, <B>, <H>     (converts to <A> .. <H>)
  <A>, <J> ... <H>  (converts to <A> .. <H>)

So all the points that shouldn't have been defined got "bridged" by the
extreme points.

The effects were recently reported to FreeBSD on PR 213013.  There are
countless places were the ctype flags are misdefined, so this is a major
fix that has to be MFC'd.
====

This reveals a bad change I did on the testsuite: while 0x07FF is a valid
unicode it is not used yet (reserved for future use)

PR:		213013
Submitted by:	marino@
Reported by:	Kurtis Rader <krader@skepticism.us>
Obtained from:	Dragonfly
MFC after:	1 month
2016-10-06 19:46:43 +00:00
Pedro F. Giffuni
be4391a2d5 localedef(1): make better use of calloc(3) arguments.
The first argument of calloc(3) should be an ordinal type, and the
second a size: split a multiplication to make better use of calloc(3)
and detect overflows.

Do some other re-ordering and style fixes while here.

MFC after:	3 weeks
2016-09-14 16:47:17 +00:00
Marcelo Araujo
4c22fda976 - Invert calloc(3) argument order.
MFC after:	4 weeks
2016-09-01 15:23:33 +00:00
Pedro F. Giffuni
fcc7baa1ae localedef(1): minor spelling fixes on comments.
No functional change.
2016-05-01 16:10:56 +00:00
Pedro F. Giffuni
0b33b55b01 Small typo. 2016-04-28 15:20:08 +00:00
Baptiste Daroussin
e6d8c0e2dd Plug memory leaks
Reported by:	Coverity
CID=		1338535, 1338536, 1338542, 1338569, 1338570
2016-04-20 21:23:42 +00:00
Pedro F. Giffuni
046c3cda83 localedef(1): minor sorting to match Illumos.
Illumos recently included space in 'print' class. We already had
this but the code had slight sorting differences. Move it some
lines up to reduce diffs with Illumos.

No functional change.

Reference:
https://illumos.org/issues/5227
2016-03-20 03:27:06 +00:00
Bryan Drewery
bd18fd57db DIRDEPS_BUILD: Regenerate without local dependencies.
These are no longer needed after the recent 'beforebuild: depend' changes
and hooking DIRDEPS_BUILD into a subset of FAST_DEPEND which supports
skipping 'make depend'.

Sponsored by:	EMC / Isilon Storage Division
2016-02-24 17:20:11 +00:00
Bryan Drewery
393608176b META MODE: Fix 'make the-lot' with recent locale changes
Sponsored by:	EMC / Isilon Storage Division
2015-11-25 19:13:28 +00:00
Baptiste Daroussin
c5aac62ae4 lower again the warnings and remove the pragmas unsupported by gcc 4.2.1 2015-11-08 22:23:21 +00:00
Baptiste Daroussin
55b270e68c Eliminate some gcc pragmas 2015-11-08 21:22:24 +00:00
Baptiste Daroussin
8c859b074e Fix build of localedef(1) on arm where wchar_t is an unsigned int 2015-11-07 22:57:00 +00:00
Baptiste Daroussin
00d10c2c70 Rewrite the histoty part
Fix information about "Dragonfly-style" format which on freebsd is named
BSD-style

Noted by:	bdrewery
2015-11-07 21:07:40 +00:00
Baptiste Daroussin
5b3b54e06c Improve localedef(1) manpage
Obtained from:	DragonflyBSD
2015-11-07 20:36:54 +00:00
Baptiste Daroussin
29660f86e2 Bump warning level 2015-11-07 20:31:23 +00:00
Baptiste Daroussin
a0e395a47f Use const where needed instead of using pragmas to work around the warnings 2015-11-07 20:29:23 +00:00
Baptiste Daroussin
557a07f08a Make bsd declaration static 2015-11-07 20:27:31 +00:00
Baptiste Daroussin
5d21db0905 Fix an off by one due to bad conversion from avl(3) to tree(3)
Readd calloc as it was not the issue just the messenger

Submitted by:	dim
Found by:	Address Sanitizer
2015-11-07 19:54:40 +00:00
Baptiste Daroussin
e12838d367 Run memset only after having checked the return of malloc
Submitted by:	pluknet
2015-11-07 16:45:51 +00:00
Baptiste Daroussin
6cdc211add Workaround an issue on i386 to unbreak the build until the real issue is tracked
down
2015-11-07 16:22:29 +00:00
Baptiste Daroussin
78be8e6732 Fix build on arm64 2015-11-07 15:03:45 +00:00
Baptiste Daroussin
99b72f8fa4 Add missing header 2015-11-07 12:11:17 +00:00
Baptiste Daroussin
9f3e8dc233 Fix typo 2015-11-07 11:08:19 +00:00
Baptiste Daroussin
d79cdd21de libc: Fix (and improve) nl_langinfo (CODESET)
The output of "locale charmap" is identical to the result of
nl_langinfo (CODESET) for any given locale.  The logic for returning the
codeset was very simplistic.  It just returned portion of the locale name
after the period (e.g. en_FR.ISO8859-1 returned "ISO8859-1").

When softlinks were added to locales, this broke.  e.g.:
   en_US returned ""
   en_FR.UTF8 returned "UTF8"
   en_FR.UTF-8 returned "UTF-8"
   zh_Hant_HK.Big5HKSCS returned "Big5HKSCS"
   zh_Hant_TW.Big5 returned "Big5"
   es_ES@euro returned ""

In order to fix this properly, the named locale cannot be used to
determine the encoding.  This information was almost available in the
rune data.  Unfortunately, all the single byte encodings were listed
as "NONE" encoding.

So I adjusted localedef tool to provide more information about the
encoding.  For example, instead of "NONE", the LC_CTYPE used by
fr_FR.ISO8859-15 is now encoded as "NONE:ISO8859-15".  The locale
handlers now check if the first four characters of the encoding is
"NONE" and if so, treats it as a single-byte encoding.

The nl_langinfo handling of CODESET was adjusting accordingly.  Now the
following is returned:
   en_US returns "ISO8859-1"
   fr_FR.UTF8 returns "UTF-8"
   fr_FR.UTF-8 returns "UTF-8"
   zh_Hant_HK.Big5HKSCS returns "Big5"
   zh_Hant_TW.Big5 returns "Big5"
   es_ES@euro returns "ISO8859-15"

as before, "C" and "POSIX" locales return "US-ASCII".  This is a big
improvement.  The result of nl_langinfo can never be a zero-length
string and it will always exclusively one of the values of the
character maps of /usr/src/tools/tools/locale/etc/final-maps.

Submitted by:	marino
Obtained from:	DragonflyBSD
2015-11-01 12:00:55 +00:00
Baptiste Daroussin
71e8badedc Actually only T_ISDIGIT should be flagged as _E4 2015-10-19 14:48:31 +00:00
Baptiste Daroussin
227d35dac0 With regard to ctype, digits (e.g. 0 to 9) and xdigits (the 0 to 9 portion
of hexidecimal numbers) are all considered "numbers".  (Note that while
all digits are numbers, not all numbers are digits).

Enhance localedef to automatically set the "number" characteristic when
it encounters a digit or xdigit definition. This fixes malfunctionning
isalnum(3)

Obtained from:	DragonflyBSD
2015-10-19 14:30:28 +00:00
Baptiste Daroussin
8833f5e9c2 eliminate need for "print" definition
By having space automatically classified as "print" type, we can
eliminate the print section from ctype src files completely (they
are just "graph" plus "<space>".

Obtained from:	Dragonfly
2015-10-13 20:45:29 +00:00
Baptiste Daroussin
f5dde0166d Commit log from Dragonfly:
FreeBSD extended ctypes to include numbers (e.g. isnumber()) but never
actually implemented it.  The isnumber() function was equivalent to the
isdigit() function in every case.

Now that DragonFly's ctype source files have number definitions, the
number ctype can finally be implemented.  It's given a new flag _CTYPE_N.
The isalnum() and iswalnum() functions have been changed to use this
flag rather than the _CTYPE_D digit flag.

While isalnum(), isnumber(), and their wide equivalents now return
different values in locale cases, the ishexnumber() and iswhexnumber()
functions are unchanged.  They are still aliases for isxdigit() and
iswxdigit().

Also change ctype.h for isdigit and isxdigit to use sbistype like the
other functions.

Obtained from:	dragonfly
2015-10-13 20:43:49 +00:00
Baptiste Daroussin
23a32822d2 Merge from HEAD 2015-08-25 20:14:50 +00:00
Baptiste Daroussin
0c648b8384 Pet mandoc -Tlint 2015-08-09 13:20:53 +00:00
Baptiste Daroussin
3a444dbdbf Convert localedef(1) from avl to RB trees 2015-08-08 22:57:17 +00:00
Baptiste Daroussin
e30c75b1f5 Prefer static generation of functions 2015-08-08 22:01:54 +00:00
Baptiste Daroussin
6131dc6a88 Convert ctype generation to Red Black tree 2015-08-08 21:53:02 +00:00
Baptiste Daroussin
057ca2d437 Add localedef(1), a locale definition generator tool
The localedef tool can read entire (and unmodified) CLDR posix definition
files, and generate all 6 LC categories: LC_COLLATE, LC_CTYPE, LC_TIME,
LC_NUMERIC, LC_MONETARY and LC_MESSAGES.

This tool has a long history with Solaris.  The Nexenta developers
modified it to read CLDR files and created the much richer collation
formats.  The libc collation functions have to be modified to read the
new format (called "BSD-1.0") and to handle the new data structures.

The result will be that locale-sensitive tools and functions will now
properly sort multibyte and unicode strings.

Obtained from:	Dragonfly
2015-08-07 23:53:31 +00:00