Commit Graph

23 Commits

Author SHA1 Message Date
yuripv
b78b1e2544 Mark "private use area" characters as printable.
At least some of the characters in E000-F8FF range are used by Powerline
fonts, and having no attributes for these ranges in UnicodeData.txt
other than "Other, Private Use" it should be safe to mark all of them as
printable.  Some actually were before r340491, so this fixes the
regression introduced there as well.

PR:		240911
Reviewed by:	bapt
Tested by:	Daniel Ponte <amigan@gmail.com>
Differential Revision:	https://reviews.freebsd.org/D21850
2019-10-05 22:17:54 +00:00
yuripv
8e370b212f Add ga_IE.UTF-8 locale.
PR:		228587
Submitted by:	Micil <micheal.maclochlainn@oegaillimh.ie> (LC_TIME)
Reviewed by:	bapt
Approved by:	kib (mentor, implicit)
MFC after:	3 days
Differential revision:	https://reviews.freebsd.org/D17997
2018-11-26 19:39:49 +00:00
yuripv
c6e4d24106 Use UnicodeData.txt to create UTF-8 ctype map.
This should provide more complete coverage of currently defined Unicode
characters as compared to manually assembled one we use currently.

Comparison of original and new UTF-8 ctype maps by character class:

TYPE    ORIG    NEW
alnum   94229   126029
alpha   93557   125419
blank   4       2
cntrl   73      137685
digit   469     622
graph   109615  137203
lower   1478    2145
print   109641  137222
punct   3428    797
rune    110481  274907
space   33      24
upper   983     1781
xdigit  469     622

Large number of added cntrl definitions is due to the fact that private-use
planes are currently defined as such, this can change in the future.

Discussed with:	bapt
Approved by:	kib (mentor, implicit)
MFC after:	1 month
Differential revision:	https://reviews.freebsd.org/D17842
2018-11-17 10:36:00 +00:00
yuripv
b6fca3ee80 Add hybrid C.UTF-8 locale being identical to default C locale except
that it uses the same ctype maps and functions as other UTF-8 locales.

Reviewed by:	bapt, cem, eadler
Approved by:	kib (mentor, implicit)
Differential Revision:	https://reviews.freebsd.org/D17833
2018-11-04 22:13:22 +00:00
yuripv
85d52a2654 Update to CLDR 34 and UNICODE 11.
Discussed with:	bapt
Approved by:	kib (mentor)
MFC after:	1 month
2018-11-03 22:02:10 +00:00
yuripv
d641b0caff Add -b/-l options to localedef(1) to specify output endianness and use
it appropriately when building share/ctypedef and share/colldef.

This makes the resulting locale data in EL->EB (amd64->powerpc64) cross
build and in the native EB build match.  Revert the changes done to libc
in r308170 as they are no longer needed.

PR:		231965
Reviewed by:	bapt, emaste, sbruno, 0mp
Approved by:	kib (mentor)
Differential Revision:	https://reviews.freebsd.org/D17603
2018-10-20 20:51:05 +00:00
yuripv
a7a80f58ca Restore some of the ctype definitions reported in the PR from pre-CLDR
data, namely 0xE000-0xF8FF private use area, and 0xFF00-0xFFF half- and
fullwidth punctuation.

While here, update tools/tools/locale/README based on my experience
rebuilding the locale data.

PR:		225692
Reviewed by:	bapt, cem (previous version)
Approved by:	re (gjb), kib (mentor)
Differential Revision:	https://reviews.freebsd.org/D17471
2018-10-11 18:30:12 +00:00
bapt
fa6a6b77d9 Update to CLDR 32 and Unicode 10
MFC after:	2 weeks
Relnotes:	yes
2017-11-03 13:52:34 +00:00
bapt
3287651efa Readd Big5: some large databases setup are still requiring it.
Reported by:	"張君天(Chun-Tien Chang)" <tcs@kitty.2y.idv.tw>
2017-04-20 18:21:50 +00:00
bapt
f18b447322 Remove zh_TW.Big5 locale
After discussion with many Taiwanese, in IT or not. Big5 is not used anymore.
It is not able to represent lots of the characters used in the language.
2017-03-19 17:52:46 +00:00
ngie
f3b9b77d64 share: normalize paths using SRCTOP-relative paths or :H when possible
This simplifies make logic/output

MFC after:	1 month
Sponsored by:	Dell EMC Isilon
2017-03-04 11:26:40 +00:00
bapt
411ea23c1c Update locales cldr to v30.0.3 and unicode to 9.0.0
MFC after:	1 month
Relnotes:	yes
2016-12-18 04:17:13 +00:00
bapt
9889527f5a Posixify the locales name for variants
For all locales with variants:
- if no ambiguity on the locale (only one variant) just use the regular name
- if ambiguity, pick one as default and append @<variant> to the others
  respecting POSIX

As a result:
- All the 3 components locales added recently are renamed to the usual 2
  components version for all but sr_RS.UTF-8
- Set sr_RS.UTF-8 to the cyrillic variant
- Add sr_RS.UTF-8@latin
- Remove the symlinks aliases they were created to represent the 2 components
  version as aliasas and are now useless
- Update the OptionalObsoleteFiles.inc and ObsoleteFiles.inc to reflect those
  changes

Discussed with:	ache@
Approved by:	re@ (gjb)
2016-07-03 18:21:11 +00:00
bapt
2d1226d5e3 Regenerates locales Makefiles and locales 2016-04-16 17:55:11 +00:00
bapt
1cc4eb0cac Use :E instead of regex in ctypedef Makefiles 2015-12-19 12:26:01 +00:00
bapt
429291909c Simplify the locale generated Makefiles by using bmake multi variables for loops 2015-12-18 21:34:28 +00:00
bdrewery
a23213fe40 META MODE: Fix 'make the-lot' with recent locale changes
Sponsored by:	EMC / Isilon Storage Division
2015-11-25 19:13:28 +00:00
bapt
3f00265ee4 Bring back a couple of ISO8859-15 locales:
- af_ZA.ISO8859-15
- en_AU.ISO8859-15
- en_CA.ISO8859-15
- en_NZ.ISO8859-15
- en_US.ISO8859-15
- fr_CA.ISO8859-15
2015-11-16 12:58:47 +00:00
bapt
ded369f5b9 Regenerate locales after readding ISO8859-1 for locales that have ISO8859-15
Requested by:	arche
2015-11-15 14:51:10 +00:00
bdrewery
bf614d3ddb Use explicit filename when creating locale symlinks to avoid creating a
directory symlink when the target directory does not exist.  This will
cause an error instead of a broken setup.

Sponsored by:	EMC / Isilon Storage Division
2015-11-11 18:45:48 +00:00
bapt
1477bec6db Merge from head 2015-11-01 21:17:38 +00:00
bapt
9e21b4c174 Regenerate locales 2015-10-13 20:24:57 +00:00
bapt
671598262f Add newly generated locales from CLDR 27.0.1
only skip timedef until I figure out how to workaround svn painful mime-type handling
2015-08-08 01:45:53 +00:00