d/freebsd-nq
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
b1ba0846f1
freebsd-nq/lib/libc/locale/utf8.5

103 lines
3.7 KiB
Groff
Raw Normal View History

Add a UTF-8 encoding method, which will eventually replace the antique "UTF2" method. Although UTF-8 and the old UTF2 encoding are compatible for 16-bit characters, the new UTF-8 implementation is much more strict about rejecting malformed input and also handles the full 31 bit range of characters.
2002-10-10 22:56:18 +00:00
.\" Copyright (c) 1993
.\" The Regents of the University of California. All rights reserved.
.\"
.\" This code is derived from software contributed to Berkeley by
.\" Paul Borman at Krystal Technologies.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions
.\" are met:
.\" 1. Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" 2. Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
Renumber copyright clause 4 Renumber cluase 4 to 3, per what everybody else did when BSD granted them permission to remove clause 3. My insistance on keeping the same numbering for legal reasons is too pedantic, so give up on that point. Submitted by: Jan Schaumann <jschauma@stevens.edu> Pull Request: https://github.com/freebsd/freebsd/pull/96
2017-02-28 23:42:47 +00:00
.\" 3. Neither the name of the University nor the names of its contributors
Add a UTF-8 encoding method, which will eventually replace the antique "UTF2" method. Although UTF-8 and the old UTF2 encoding are compatible for 16-bit characters, the new UTF-8 implementation is much more strict about rejecting malformed input and also handles the full 31 bit range of characters.
2002-10-10 22:56:18 +00:00
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
.\" ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
.\" SUCH DAMAGE.
.\"
.\" @(#)utf2.4 8.1 (Berkeley) 6/4/93
.\" $FreeBSD$
.\"
Remove a nonsensical remark about byte order markers in UTF-8 streams.
2004-04-12 12:58:41 +00:00
.Dd April 7, 2004
Add a UTF-8 encoding method, which will eventually replace the antique "UTF2" method. Although UTF-8 and the old UTF2 encoding are compatible for 16-bit characters, the new UTF-8 implementation is much more strict about rejecting malformed input and also handles the full 31 bit range of characters.
2002-10-10 22:56:18 +00:00
.Dt UTF8 5
.Os
.Sh NAME
.Nm utf8
.Nd "UTF-8, a transformation format of ISO 10646"
.Sh SYNOPSIS
.Nm ENCODING
.Qq UTF-8
.Sh DESCRIPTION
The
.Nm UTF-8
encoding represents UCS-4 characters as a sequence of octets, using
between 1 and 6 for each character.
It is backwards compatible with
.Tn ASCII ,
so 0x00-0x7f refer to the
.Tn ASCII
character set.
mdoc(7) police: sweep.
2002-11-29 17:35:09 +00:00
The multibyte encoding of
.No non- Ns Tn ASCII
Add a UTF-8 encoding method, which will eventually replace the antique "UTF2" method. Although UTF-8 and the old UTF2 encoding are compatible for 16-bit characters, the new UTF-8 implementation is much more strict about rejecting malformed input and also handles the full 31 bit range of characters.
2002-10-10 22:56:18 +00:00
characters
consist entirely of bytes whose high order bit is set.
The actual
encoding is represented by the following table:
.Bd -literal
[0x00000000 - 0x0000007f] [00000000.0bbbbbbb] -> 0bbbbbbb
[0x00000080 - 0x000007ff] [00000bbb.bbbbbbbb] -> 110bbbbb, 10bbbbbb
[0x00000800 - 0x0000ffff] [bbbbbbbb.bbbbbbbb] ->
1110bbbb, 10bbbbbb, 10bbbbbb
[0x00010000 - 0x001fffff] [00000000.000bbbbb.bbbbbbbb.bbbbbbbb] ->
11110bbb, 10bbbbbb, 10bbbbbb, 10bbbbbb
[0x00200000 - 0x03ffffff] [000000bb.bbbbbbbb.bbbbbbbb.bbbbbbbb] ->
111110bb, 10bbbbbb, 10bbbbbb, 10bbbbbb, 10bbbbbb
[0x04000000 - 0x7fffffff] [0bbbbbbb.bbbbbbbb.bbbbbbbb.bbbbbbbb] ->
1111110b, 10bbbbbb, 10bbbbbb, 10bbbbbb, 10bbbbbb, 10bbbbbb
.Ed
.Pp
If more than a single representation of a value exists (for example,
0x00; 0xC0 0x80; 0xE0 0x80 0x80) the shortest representation is always
used.
Longer ones are detected as an error as they pose a potential
security risk, and destroy the 1:1 character:octet sequence mapping.
.Sh SEE ALSO
Remove support for the obsolete UTF2 encoding.
2004-10-17 02:29:15 +00:00
.Xr euc 5
Add a UTF-8 encoding method, which will eventually replace the antique "UTF2" method. Although UTF-8 and the old UTF2 encoding are compatible for 16-bit characters, the new UTF-8 implementation is much more strict about rejecting malformed input and also handles the full 31 bit range of characters.
2002-10-10 22:56:18 +00:00
.Rs
Add two additional references to the See Also section, which contain much better descriptions of UTF-8 and related issues.
2002-10-30 11:49:05 +00:00
.%A "Rob Pike"
.%A "Ken Thompson"
.%T "Hello World"
.%J "Proceedings of the Winter 1993 USENIX Technical Conference"
.%Q "USENIX Association"
.%D "January 1993"
.Re
.Rs
Add a UTF-8 encoding method, which will eventually replace the antique "UTF2" method. Although UTF-8 and the old UTF2 encoding are compatible for 16-bit characters, the new UTF-8 implementation is much more strict about rejecting malformed input and also handles the full 31 bit range of characters.
2002-10-10 22:56:18 +00:00
.%A "F. Yergeau"
.%T "UTF-8, a transformation format of ISO 10646"
.%O "RFC 2279"
.%D "January 1998"
.Re
Add two additional references to the See Also section, which contain much better descriptions of UTF-8 and related issues.
2002-10-30 11:49:05 +00:00
.Rs
.%Q "The Unicode Consortium"
.%T "The Unicode Standard, Version 3.0"
.%D "2000"
.%O "as amended by the Unicode Standard Annex #27: Unicode 3.1 and by the Unicode Standard Annex #28: Unicode 3.2"
.Re
Add a UTF-8 encoding method, which will eventually replace the antique "UTF2" method. Although UTF-8 and the old UTF2 encoding are compatible for 16-bit characters, the new UTF-8 implementation is much more strict about rejecting malformed input and also handles the full 31 bit range of characters.
2002-10-10 22:56:18 +00:00
.Sh STANDARDS
The
.Nm
Add two additional references to the See Also section, which contain much better descriptions of UTF-8 and related issues.
2002-10-30 11:49:05 +00:00
encoding is compatible with RFC 2279 and Unicode 3.2.
Reference in New Issue Copy Permalink