Commit Graph

567 Commits

Author SHA1 Message Date
Baptiste Daroussin
764a768e16 Merge from HEAD 2015-08-09 00:15:17 +00:00
Baptiste Daroussin
8bb93485fb Remove 5 and 6 bytes sequences which are illegal in UTF-8 space. (part2)
Per rfc3629 value greater than 0x10ffff should be rejected

Suggested by:	jilles
2015-08-09 00:06:56 +00:00
Baptiste Daroussin
c9d24bcfd5 Remove 5 and 6 bytes sequences which are illegal in UTF-8 space.
Per rfc3629 value greater than 0x10ffff should be rejected

Suggested by:	jilles
2015-08-08 23:59:15 +00:00
Baptiste Daroussin
d0a68f8d38 Fix typo 2015-08-08 23:17:10 +00:00
Baptiste Daroussin
7b2473410f Revamp CTYPE support (from Illumos & Dragonfly)
Obtained from:	Dragonfly
2015-08-08 18:22:14 +00:00
Baptiste Daroussin
2a6abeebef The collate functions within libc have been using version 1 and 1.2 of the
packed LC_COLLATE binary formats. These were generated with the colldef
tool, but the new LC_COLLATE files are going to be generated by the new
localedef tool using CLDR POSIX files as input.  The BSD-flavored
version of localedef identifies the format as "BSD 1.0".  Any
LC_COLLATE file with a different version will simply not be loaded, and
all LC* categories will get set to "C" (aka "POSIX") locale.

This work is based off of Nexenta's contribution to Illumos.
The integration with xlocale is John Marino's work for Dragonfly.

The following commits will enable localedef tool, disable the colldef
tool, add generated colldef directory, and finally remove colldef from
base.

The only difference with Dragonfly are:
- a few fixes to build with clang
- And identification of the flavor as "BSD 1.0" instead of "Dragonfly 4.4"

Obtained from:	Dragonfly
2015-08-07 23:41:26 +00:00
David Chisnall
710ec77e3a __xlocale_C_ctype should not be const. It contains a reference count that is modified by newlocale / duplocale / freelocale.
MFC after:	1 week
2015-04-24 10:21:20 +00:00
David Chisnall
58912ae7c3 Small changes to locale-related man pages.
Fix a missing .h and change the recommended include for the POSIX2008 functions from xlocale.h to locale.h.  Including xlocale.h is for legacy / Darwin compatibility so should not be encouraged.
2015-04-24 10:17:55 +00:00
Tijl Coosemans
1243a98e38 Remove the const qualifier from iconv(3) to comply with POSIX:
http://pubs.opengroup.org/onlinepubs/9699919799/functions/iconv.html

Adjust all code that calls iconv.

PR:		199099
Exp-run by:	antoine
MFC after:	2 weeks
2015-04-15 09:09:20 +00:00
Joel Dahl
4990a1c050 mdoc: improvements to SEE ALSO. 2014-12-27 08:31:52 +00:00
Pedro F. Giffuni
9908eab82e libc/locale: Remove a wrong comma.
This only had some effect when debugging.

Obtained from:	DragonflyBSD
MFC after:	3 days
2014-09-04 17:36:21 +00:00
Pedro F. Giffuni
0716c0ff7a minor perf enhancement for UTF-8
Reduce some duplicate code.

Reference:
https://www.illumos.org/issues/628

Obtained from:	Illumos
MFC after:	1 week
2014-07-04 22:39:39 +00:00
Pedro F. Giffuni
0f5132cd25 citrus: Avoid invalid code points.
From the OpenBSD log:
The UTF-8 decoder should not accept byte sequences which decode to unicode
code positions U+D800 to U+DFFF (UTF-16 surrogates), U+FFFE, and U+FFFF.

http://www.cl.cam.ac.uk/~mgk25/unicode.html#utf-8
http://unicode.org/faq/utf_bom.html#utf8-4

Reported by:	Stefan Sperling
Obtained from:	OpenBSD
MFC after:	5 days
2014-05-01 01:42:48 +00:00
Pedro F. Giffuni
97ecaa8907 citrus: Avoid invalid code points.
From the OpenBSD log:
The UTF-8 decoder should not accept byte sequences which decode to unicode
code positions U+D800 to U+DFFF (UTF-16 surrogates), U+FFFE, and U+FFFF.

http://www.cl.cam.ac.uk/~mgk25/unicode.html#utf-8
http://unicode.org/faq/utf_bom.html#utf8-4

Reported by:	Stefan Sperling
Obtained from:	OpenBSD
MFC after:	5 days
2014-04-29 15:25:57 +00:00
David Chisnall
8d07b7deff Fix an issue where the locale and rune locale could become out of sync,
causing mb* functions (and similar) to be called with the wrong data
(possibly a null pointer, causing a crash).

PR:		standards/188036
MFC after:	1 week
2014-04-02 11:10:46 +00:00
Marcel Moolenaar
8876613dc5 Replace use of ${.CURDIR} by ${LIBC_SRCTOP} and define ${LIBC_SRCTOP}
if not already defined. This allows building libc from outside of
lib/libc using a reach-over makefile.

A typical use-case is to build a standard ILP32 version and a COMPAT32
version in a single iteration by building the COMPAT32 version using a
reach-over makefile.

Obtained from:	Juniper Networks, Inc.
2014-03-04 02:19:39 +00:00
Peter Wemm
19dfd82d81 Replace the #define for "iconv" so it is for the function name instead of
a macro with parameters.  Remove a __DECONST hack and add consts instead
for gnu libiconv API compatability.  This makes it work with things like
devel/boost-libs that expects to use "iconv" as though it were a pointer.
2013-07-03 07:03:19 +00:00
Ed Schouten
49111f0092 Add libiconv based versions of *c16*() and *c32*().
I initially thought wchar_t was locale independent, but this seems to be
only the case on Linux. This means that we cannot depend on the *wc*()
routines to implement *c16*() and *c32*(). Instead, use the Citrus
libiconv that is part of libc.

I'll see if there is anything I can do to make the existing functions
somewhat useful in case the system is built without libiconv in the
nearby future. If not, I'll simply remove the broken implementations.

Reviewed by:	jilles, gabor
2013-06-03 17:17:56 +00:00
Ed Schouten
50c77c6e8b Add <uchar.h>.
The <uchar.h> header, part of C11, adds a small number of utility
functions for 16/32-bit "universal" characters, which may or may not be
UTF-16/32. As our wchar_t is already ISO 10646, simply add light-weight
wrappers around wcrtomb() and mbrtowc().

While there, also add (non-yet-standard) _l functions, similar to the
ones we already have for the other locale-dependent functions.

Reviewed by:	theraven
2013-05-21 19:59:37 +00:00
Sergey Kandaurov
424b842a5d Document that the return type is different from 1003.1-2008.
MFC after:	1 week
2013-05-04 17:21:44 +00:00
Sergey Kandaurov
4f79ce7b4c mdoc: missing comma in .Dd macro. 2013-05-04 17:06:47 +00:00
Sergey Kandaurov
67ff590b9e Also, add a missing period. 2013-05-03 13:27:13 +00:00
Sergey Kandaurov
cc7a693fa7 Remove an extra comma. 2013-05-03 12:45:45 +00:00
Sergey Kandaurov
88bbbe88d7 Remove the STANDARDS section.
querylocale is not part of IEEE Std 1003.1-2008.

MFC after:	3 days
2013-05-03 12:42:43 +00:00
Jilles Tjoelker
86bca8fb51 btowc(3), isblank(3): Correct prototypes for _l variants.
MFC after:	1 week
2013-03-27 21:31:40 +00:00
Jilles Tjoelker
07588bf421 libc: Make various internal file descriptors close-on-exec.
These are obtained via fopen().
2012-12-11 22:52:56 +00:00
Brooks Davis
1652751915 Improve style(9) compliance of function declarations. 2012-12-10 17:34:33 +00:00
Jilles Tjoelker
05eb11cbc4 libc: Use O_CLOEXEC for various internal file descriptors.
This fixes a race condition where another thread may fork() before CLOEXEC
is set, unintentionally passing the descriptor to the child process.

This commit only adds O_CLOEXEC flags to open() or openat() calls where no
fcntl(fd, F_SETFD, FD_CLOEXEC) follows. The separate fcntl() call still
leaves a race window so it should be fixed later.
2012-09-29 11:54:34 +00:00
Joel Dahl
76c9beff1b Remove trailing whitespace. 2012-07-30 21:02:44 +00:00
Isabell Long
ea91541917 Add more locale-specific functions to the relevant man pages and Makefile:
- lib/libc/locale/islower.3
  - lib/libc/locale/ispunct.3
  - lib/libc/locale/nl_langinfo.3
  - lib/libc/locale/isgraph.3
  - lib/libc/locale/isspace.3

Reviewed by:	bz
Approved by:	theraven
MFC after:	5 days
2012-07-30 20:56:19 +00:00
Joel Dahl
a18f885152 Start manpage with Dd macro and also remove a trailing whitespace
while here.
2012-07-26 14:46:19 +00:00
Isabell Long
b3f18e536a Add a new man page containing details of new locale-specific functions for
wctype.h, iswalnum_l(3).  Add it and its functions to the Makefile.

Reviewed by:	gavin, jilles
Approved by:	theraven
MFC after:	5 days
2012-07-25 22:17:44 +00:00
David Chisnall
bb4317bf3c Restore the __collate_load_error global that was accidentally removed in the
xlocale refactoring.

MFC after:	1 week
2012-07-06 20:16:22 +00:00
Jilles Tjoelker
57ca132fc8 ctype_l(3): Note that not all these functions are specified by POSIX.1-2008.
The function isascii_l() is not in POSIX even though isascii() is,
probably because isascii() is marked as obsolete.  The other functions,
like digittoint_l() and ishexnumber_l(), are FreeBSD-specific just like
their non-_l versions.
2012-07-01 22:18:20 +00:00
David Chisnall
1967332863 Fix a leak when setting the global character locale to "C" from something else.
Reported by:	mm
2012-06-11 14:02:02 +00:00
Glen Barber
a3fb6da9ba General mdoc(7) and typo fixes.
PR:		167734
Submitted by:	Nobuyuki Koganemaru (kogane!jp.freebsd.org)
MFC after:	3 days
2012-05-11 20:06:46 +00:00
Dimitry Andric
83c476cd09 Fix copy/paste error in lib/libc/locale/toupper.c.
Submitted by:	Kohji Okuno <okuno.kohji@jp.panasonic.com>
2012-05-10 20:03:34 +00:00
David Chisnall
6ed2e496fd Fix some incorrect symbol versions.
Reported by:	das
2012-04-22 18:51:38 +00:00
Joel Dahl
26c9eb4a77 mdoc: fix function type. 2012-04-07 09:26:21 +00:00
Joel Dahl
748611c9c0 mdoc: fix column names, indentation, column separation within each row, and
quotation. Also make sure we have the same amount of columns in each row as
the number of columns we specify in the head arguments.

Reviewed by:	brueffer
2012-04-07 09:05:30 +00:00
Eitan Adler
50d675f7a9 Remove trailing whitespace per mdoc lint warning
Disussed with:	gavin
No objection from:	doc
Approved by:	joel
MFC after:	3 days
2012-03-29 05:02:12 +00:00
Joel Dahl
6ae6870fe8 mdoc: remove unknown macro. 2012-03-26 19:06:59 +00:00
Joel Dahl
5b40a50e9a mdoc: remove empty lines. 2012-03-26 15:30:28 +00:00
Joel Dahl
edf5bf9919 mdoc: document title should be all caps. 2012-03-25 19:34:05 +00:00
Joel Dahl
12afe06c06 Make sure sections are sorted into conventional order. 2012-03-25 16:00:56 +00:00
Joel Dahl
41949a1ed5 Remove superfluous paragraph macro. 2012-03-25 12:13:24 +00:00
David Chisnall
5a674eeb8f Make __get_locale() static inline, not just inline, so when compiling libc in
debug mode (without optimisations) it doesn't generate a linker failure.

Approved by:	dim (mentor)
2012-03-19 11:47:37 +00:00
David Chisnall
c889dd01b9 First set of xlocale man pages. More to follow...
Approved by:	dim (mentor)
2012-03-13 20:02:41 +00:00
David Chisnall
4d7815ecc4 More xlocale cleanups.
Approved by:	dim (mentor)
2012-03-13 14:14:13 +00:00
Dimitry Andric
cc09dfac61 Fix build of libc.so after r232620. This caused a duplicate definition
of __getCurrentRuneLocale().

Pointy hat to:	me
2012-03-06 21:56:30 +00:00