freebsd-dev/lib/libc/locale/multibyte.3

.\" Copyright (c) 2002-2004 Tim J. Robbins. All rights reserved.
.\" Copyright (c) 1993
.\"	The Regents of the University of California.  All rights reserved.
.\"
.\" This code is derived from software contributed to Berkeley by
.\" Donn Seeley of BSDI.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions
.\" are met:
.\" 1. Redistributions of source code must retain the above copyright
.\"    notice, this list of conditions and the following disclaimer.
.\" 2. Redistributions in binary form must reproduce the above copyright
.\"    notice, this list of conditions and the following disclaimer in the
.\"    documentation and/or other materials provided with the distribution.
.\" 3. Neither the name of the University nor the names of its contributors
.\"    may be used to endorse or promote products derived from this software
.\"    without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
.\" ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
.\" SUCH DAMAGE.
.\"
.\"	@(#)multibyte.3	8.1 (Berkeley) 6/4/93
.\" $FreeBSD$
.\"
.Dd September 9, 2019
.Dt MULTIBYTE 3
.Os
.Sh NAME
.Nm multibyte
.Nd multibyte and wide character manipulation functions
.Sh LIBRARY
.Lb libc
.Sh SYNOPSIS
.In limits.h
.In stdlib.h
.In wchar.h
.Sh DESCRIPTION
The basic elements of some written natural languages, such as Chinese,
cannot be represented uniquely with single C
.Vt char Ns s .
The C standard supports two different ways of dealing with
extended natural language encodings:
wide characters and
multibyte characters.
Wide characters are an internal representation
which allows each basic element to map
to a single object of type
.Vt wchar_t .
Multibyte characters are used for input and output
and code each basic element as a sequence of C
.Vt char Ns s .
Individual basic elements may map into one or more
(up to
.Dv MB_LEN_MAX )
bytes in a multibyte character.
.Pp
The current locale
.Pq Xr setlocale 3
governs the interpretation of wide and multibyte characters.
The locale category
.Dv LC_CTYPE
specifically controls this interpretation.
The
.Vt wchar_t
type is wide enough to hold the largest value
in the wide character representations for all locales.
.Pp
Multibyte strings may contain
.Sq shift
indicators to switch to and from
particular modes within the given representation.
If explicit bytes are used to signal shifting,
these are not recognized as separate characters
but are lumped with a neighboring character.
There is always a distinguished
.Sq initial
shift state.
Some functions (e.g.,
.Xr mblen 3 ,
.Xr mbtowc 3
and
.Xr wctomb 3 )
maintain static shift state internally, whereas
others store it in an
.Vt mbstate_t
object passed by the caller.
Shift states are undefined after a call to
.Xr setlocale 3
with the
.Dv LC_CTYPE
or
.Dv LC_ALL
categories.
.Pp
For convenience in processing,
the wide character with value 0
(the null wide character)
is recognized as the wide character string terminator,
and the character with value 0
(the null byte)
is recognized as the multibyte character string terminator.
Null bytes are not permitted within multibyte characters.
.Pp
The C library provides the following functions for dealing with
multibyte characters:
.Bl -column "Description"
.It Sy "Function	Description"
.It Xr mblen 3 Ta "get number of bytes in a character"
.It Xr mbrlen 3 Ta "get number of bytes in a character (restartable)"
.It Xr mbrtowc 3 Ta "convert a character to a wide-character code (restartable)"
.It Xr mbsrtowcs 3 Ta "convert a character string to a wide-character string (restartable)"
.It Xr mbstowcs 3 Ta "convert a character string to a wide-character string"
.It Xr mbtowc 3 Ta "convert a character to a wide-character code"
.It Xr wcrtomb 3 Ta "convert a wide-character code to a character (restartable)"
.It Xr wcstombs 3 Ta "convert a wide-character string to a character string"
.It Xr wcsrtombs 3 Ta "convert a wide-character string to a character string (restartable)"
.It Xr wctomb 3 Ta "convert a wide-character code to a character"
.El
.Sh SEE ALSO
.Xr localedef 1 ,
.Xr setlocale 3 ,
.Xr stdio 3 ,
.Xr big5 5 ,
.Xr euc 5 ,
.Xr gb18030 5 ,
.Xr gb2312 5 ,
.Xr gbk 5 ,
.Xr mskanji 5 ,
.Xr utf8 5
.Sh STANDARDS
These functions conform to
.St -isoC-99 .
Update manual pages for change to C99 mbrtowc() semantics. 2004-04-08 09:59:02 +00:00			`.\" Copyright (c) 2002-2004 Tim J. Robbins. All rights reserved.`
BSD 4.4 Lite Lib Sources 1994-05-27 05:00:24 +00:00			`.\" Copyright (c) 1993`
			`.\" The Regents of the University of California. All rights reserved.`
			`.\"`
			`.\" This code is derived from software contributed to Berkeley by`
			`.\" Donn Seeley of BSDI.`
			`.\"`
			`.\" Redistribution and use in source and binary forms, with or without`
			`.\" modification, are permitted provided that the following conditions`
			`.\" are met:`
			`.\" 1. Redistributions of source code must retain the above copyright`
			`.\" notice, this list of conditions and the following disclaimer.`
			`.\" 2. Redistributions in binary form must reproduce the above copyright`
			`.\" notice, this list of conditions and the following disclaimer in the`
			`.\" documentation and/or other materials provided with the distribution.`
Renumber copyright clause 4 Renumber cluase 4 to 3, per what everybody else did when BSD granted them permission to remove clause 3. My insistance on keeping the same numbering for legal reasons is too pedantic, so give up on that point. Submitted by: Jan Schaumann <jschauma@stevens.edu> Pull Request: https://github.com/freebsd/freebsd/pull/96 2017-02-28 23:42:47 +00:00			`.\" 3. Neither the name of the University nor the names of its contributors`
BSD 4.4 Lite Lib Sources 1994-05-27 05:00:24 +00:00			`.\" may be used to endorse or promote products derived from this software`
			`.\" without specific prior written permission.`
			`.\"`
			.\" THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
			`.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE`
			`.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE`
			`.\" ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE`
			`.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL`
			`.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS`
			`.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)`
			`.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT`
			`.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY`
			`.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF`
			`.\" SUCH DAMAGE.`
			`.\"`
			`.\" @(#)multibyte.3 8.1 (Berkeley) 6/4/93`
$Id$ -> $FreeBSD$ 1999-08-28 00:22:10 +00:00			`.\" $FreeBSD$`
BSD 4.4 Lite Lib Sources 1994-05-27 05:00:24 +00:00			`.\"`
In FreeBSD 11 localedef(1) has replaced the mklocale(1) and colldef(1) tools to generate the locales data. state it in the libc manpages. MFC after: 3 days 2019-09-10 07:47:52 +00:00			`.Dd September 9, 2019`
BSD 4.4 Lite Lib Sources 1994-05-27 05:00:24 +00:00			`.Dt MULTIBYTE 3`
			`.Os`
			`.Sh NAME`
Split multibyte(3) into separate manual pages for each function. Instead of just deleting it, turn the original page into a general overview of the multibyte character conversion functions, somewhat similar to stdio(3). 2003-12-07 06:33:52 +00:00			`.Nm multibyte`
			`.Nd multibyte and wide character manipulation functions`
Introduce ".Lb" macro to libc manpages. More libraries manpages updates following. 2000-04-21 09:42:15 +00:00			`.Sh LIBRARY`
			`.Lb libc`
BSD 4.4 Lite Lib Sources 1994-05-27 05:00:24 +00:00			`.Sh SYNOPSIS`
Split multibyte(3) into separate manual pages for each function. Instead of just deleting it, turn the original page into a general overview of the multibyte character conversion functions, somewhat similar to stdio(3). 2003-12-07 06:33:52 +00:00			`.In limits.h`
mdoc(7) police: Use the new .In macro for #include statements. 2001-10-01 16:09:29 +00:00			`.In stdlib.h`
Split multibyte(3) into separate manual pages for each function. Instead of just deleting it, turn the original page into a general overview of the multibyte character conversion functions, somewhat similar to stdio(3). 2003-12-07 06:33:52 +00:00			`.In wchar.h`
BSD 4.4 Lite Lib Sources 1994-05-27 05:00:24 +00:00			`.Sh DESCRIPTION`
Split multibyte(3) into separate manual pages for each function. Instead of just deleting it, turn the original page into a general overview of the multibyte character conversion functions, somewhat similar to stdio(3). 2003-12-07 06:33:52 +00:00			`The basic elements of some written natural languages, such as Chinese,`
BSD 4.4 Lite Lib Sources 1994-05-27 05:00:24 +00:00			`cannot be represented uniquely with single C`
Markup, grammar, and spelling fixes. 2004-06-30 20:09:10 +00:00			`.Vt char Ns s .`
BSD 4.4 Lite Lib Sources 1994-05-27 05:00:24 +00:00			`The C standard supports two different ways of dealing with`
Split multibyte(3) into separate manual pages for each function. Instead of just deleting it, turn the original page into a general overview of the multibyte character conversion functions, somewhat similar to stdio(3). 2003-12-07 06:33:52 +00:00			`extended natural language encodings:`
			`wide characters and`
			`multibyte characters.`
BSD 4.4 Lite Lib Sources 1994-05-27 05:00:24 +00:00			`Wide characters are an internal representation`
			`which allows each basic element to map`
			`to a single object of type`
Markup, grammar, and spelling fixes. 2004-06-30 20:09:10 +00:00			`.Vt wchar_t .`
BSD 4.4 Lite Lib Sources 1994-05-27 05:00:24 +00:00			`Multibyte characters are used for input and output`
			`and code each basic element as a sequence of C`
Markup, grammar, and spelling fixes. 2004-06-30 20:09:10 +00:00			`.Vt char Ns s .`
BSD 4.4 Lite Lib Sources 1994-05-27 05:00:24 +00:00			`Individual basic elements may map into one or more`
mdoc(7) police: Avoid using parenthesis enclosure macros (.Pq and .Po/.Pc) with plain text. Not only this slows down the mdoc(7) processing significantly, but it also has an undesired (in this case) effect of disabling hyphenation within the entire enclosed block. 2001-08-07 15:48:51 +00:00			`(up to`
Typo: refer to MB_LEN_MAX instead of MB_CHAR_MAX (which does not exist). 2002-09-01 07:21:58 +00:00			`.Dv MB_LEN_MAX )`
BSD 4.4 Lite Lib Sources 1994-05-27 05:00:24 +00:00			`bytes in a multibyte character.`
			`.Pp`
			`The current locale`
			`.Pq Xr setlocale 3`
			`governs the interpretation of wide and multibyte characters.`
			`The locale category`
			`.Dv LC_CTYPE`
			`specifically controls this interpretation.`
			`The`
Markup, grammar, and spelling fixes. 2004-06-30 20:09:10 +00:00			`.Vt wchar_t`
BSD 4.4 Lite Lib Sources 1994-05-27 05:00:24 +00:00			`type is wide enough to hold the largest value`
			`in the wide character representations for all locales.`
			`.Pp`
			`Multibyte strings may contain`
			`.Sq shift`
			`indicators to switch to and from`
			`particular modes within the given representation.`
			`If explicit bytes are used to signal shifting,`
			`these are not recognized as separate characters`
			`but are lumped with a neighboring character.`
			`There is always a distinguished`
			`.Sq initial`
			`shift state.`
Markup, grammar, and spelling fixes. 2004-06-30 20:09:10 +00:00			`Some functions (e.g.,`
			`.Xr mblen 3 ,`
			`.Xr mbtowc 3`
BSD 4.4 Lite Lib Sources 1994-05-27 05:00:24 +00:00			`and`
Markup, grammar, and spelling fixes. 2004-06-30 20:09:10 +00:00			`.Xr wctomb 3 )`
Split multibyte(3) into separate manual pages for each function. Instead of just deleting it, turn the original page into a general overview of the multibyte character conversion functions, somewhat similar to stdio(3). 2003-12-07 06:33:52 +00:00			`maintain static shift state internally, whereas`
Markup, grammar, and spelling fixes. 2004-06-30 20:09:10 +00:00			`others store it in an`
Split multibyte(3) into separate manual pages for each function. Instead of just deleting it, turn the original page into a general overview of the multibyte character conversion functions, somewhat similar to stdio(3). 2003-12-07 06:33:52 +00:00			`.Vt mbstate_t`
			`object passed by the caller.`
			`Shift states are undefined after a call to`
Markup, grammar, and spelling fixes. 2004-06-30 20:09:10 +00:00			`.Xr setlocale 3`
BSD 4.4 Lite Lib Sources 1994-05-27 05:00:24 +00:00			`with the`
			`.Dv LC_CTYPE`
			`or`
			`.Dv LC_ALL`
			`categories.`
			`.Pp`
			`For convenience in processing,`
			`the wide character with value 0`
mdoc(7) police: Avoid using parenthesis enclosure macros (.Pq and .Po/.Pc) with plain text. Not only this slows down the mdoc(7) processing significantly, but it also has an undesired (in this case) effect of disabling hyphenation within the entire enclosed block. 2001-08-07 15:48:51 +00:00			`(the null wide character)`
BSD 4.4 Lite Lib Sources 1994-05-27 05:00:24 +00:00			`is recognized as the wide character string terminator,`
			`and the character with value 0`
mdoc(7) police: Avoid using parenthesis enclosure macros (.Pq and .Po/.Pc) with plain text. Not only this slows down the mdoc(7) processing significantly, but it also has an undesired (in this case) effect of disabling hyphenation within the entire enclosed block. 2001-08-07 15:48:51 +00:00			`(the null byte)`
BSD 4.4 Lite Lib Sources 1994-05-27 05:00:24 +00:00			`is recognized as the multibyte character string terminator.`
			`Null bytes are not permitted within multibyte characters.`
			`.Pp`
Split multibyte(3) into separate manual pages for each function. Instead of just deleting it, turn the original page into a general overview of the multibyte character conversion functions, somewhat similar to stdio(3). 2003-12-07 06:33:52 +00:00			`The C library provides the following functions for dealing with`
			`multibyte characters:`
			`.Bl -column "Description"`
Markup, grammar, and spelling fixes. 2004-06-30 20:09:10 +00:00			`.It Sy "Function Description"`
			`.It Xr mblen 3 Ta "get number of bytes in a character"`
			`.It Xr mbrlen 3 Ta "get number of bytes in a character (restartable)"`
			`.It Xr mbrtowc 3 Ta "convert a character to a wide-character code (restartable)"`
			`.It Xr mbsrtowcs 3 Ta "convert a character string to a wide-character string (restartable)"`
			`.It Xr mbstowcs 3 Ta "convert a character string to a wide-character string"`
			`.It Xr mbtowc 3 Ta "convert a character to a wide-character code"`
			`.It Xr wcrtomb 3 Ta "convert a wide-character code to a character (restartable)"`
			`.It Xr wcstombs 3 Ta "convert a wide-character string to a character string"`
			`.It Xr wcsrtombs 3 Ta "convert a wide-character string to a character string (restartable)"`
			`.It Xr wctomb 3 Ta "convert a wide-character code to a character"`
Split multibyte(3) into separate manual pages for each function. Instead of just deleting it, turn the original page into a general overview of the multibyte character conversion functions, somewhat similar to stdio(3). 2003-12-07 06:33:52 +00:00			`.El`
The .Fn function 2003-02-06 11:04:47 +00:00			`.Sh SEE ALSO`
In FreeBSD 11 localedef(1) has replaced the mklocale(1) and colldef(1) tools to generate the locales data. state it in the libc manpages. MFC after: 3 days 2019-09-10 07:47:52 +00:00			`.Xr localedef 1 ,`
BSD 4.4 Lite Lib Sources 1994-05-27 05:00:24 +00:00			`.Xr setlocale 3 ,`
Markup, grammar, and spelling fixes. 2004-06-30 20:09:10 +00:00			`.Xr stdio 3 ,`
Add cross references to the new character encoding manual pages, and to mbsinit(3) while I'm at it. 2003-08-10 09:25:52 +00:00			`.Xr big5 5 ,`
Update cross references after utf2/euc move. 2003-11-15 02:26:04 +00:00			`.Xr euc 5 ,`
Add cross references to the new character encoding manual pages, and to mbsinit(3) while I'm at it. 2003-08-10 09:25:52 +00:00			`.Xr gb18030 5 ,`
Add one more cross-reference to gb2312(5). 2003-11-08 03:23:11 +00:00			`.Xr gb2312 5 ,`
Cross-reference gbk(5). 2003-08-10 11:38:28 +00:00			`.Xr gbk 5 ,`
Add cross references to the new character encoding manual pages, and to mbsinit(3) while I'm at it. 2003-08-10 09:25:52 +00:00			`.Xr mskanji 5 ,`
Add a UTF-8 encoding method, which will eventually replace the antique "UTF2" method. Although UTF-8 and the old UTF2 encoding are compatible for 16-bit characters, the new UTF-8 implementation is much more strict about rejecting malformed input and also handles the full 31 bit range of characters. 2002-10-10 22:56:18 +00:00			`.Xr utf8 5`
BSD 4.4 Lite Lib Sources 1994-05-27 05:00:24 +00:00			`.Sh STANDARDS`
Split multibyte(3) into separate manual pages for each function. Instead of just deleting it, turn the original page into a general overview of the multibyte character conversion functions, somewhat similar to stdio(3). 2003-12-07 06:33:52 +00:00			`These functions conform to`
Update manual pages for change to C99 mbrtowc() semantics. 2004-04-08 09:59:02 +00:00			`.St -isoC-99 .`