9f7e5bdad1
Sort(1)'s radixsort implementation was broken for multibyte LC_CTYPEs in at
least two ways:
* In actual radix sort, it would only bucket the least significant
byte from each wchar, ignoring the 24 most-significant bits of each
unicode character.
* In degenerate cases / "fast paths," it would fall back to another
sorting algorithm (default: mergesort) with a bogus comparator
offset. The string comparison functions in sort(1) take an offset
in units of the operating character size. However, radixsort was
passing an offset in units of bytes. The byte offset must be
divided by sizeof(wchar_t).
This revision addresses both discovered issues.
Some example testcases:
$ (echo 耳 ; echo 脳 ; echo 耳) | \
LC_CTYPE=ja_JP.UTF-8 LC_COLLATE=C LANG=C sort --radixsort --debug
$ (echo 耳 ; echo 脳 ; echo 耳) | \
LC_CTYPE=C LC_COLLATE=C LANG=C sort --radixsort --debug
$ (for i in $(jot 34); do echo 耳耳耳耳耳; echo 耳耳耳耳脳; echo 耳耳耳耳脴; done) | \
LC_CTYPE=ja_JP.UTF-8 LC_COLLATE=C LANG=C sort --radixsort --debug
PR: 247494
Reported by: knu
MFC after: I do not intend to, but parties interested in stable might want to
FreeBSD Source:
This is the top level of the FreeBSD source directory. This file
was last revised on:
FreeBSD
FreeBSD is an operating system used to power modern servers, desktops, and embedded platforms. A large community has continually developed it for more than thirty years. Its advanced networking, security, and storage features have made FreeBSD the platform of choice for many of the busiest web sites and most pervasive embedded networking and storage devices.
For copyright information, please see the file COPYRIGHT in this directory. Additional copyright information also exists for some sources in this tree - please see the specific source directories for more information.
The Makefile in this directory supports a number of targets for building components (or all) of the FreeBSD source tree. See build(7), config(8), https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/makeworld.html, and https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig.html for more information, including setting make(1) variables.
Source Roadmap:
bin System/user commands.
cddl Various commands and libraries under the Common Development
and Distribution License.
contrib Packages contributed by 3rd parties.
crypto Cryptography stuff (see crypto/README).
etc Template files for /etc.
gnu Various commands and libraries under the GNU Public License.
Please see gnu/COPYING* for more information.
include System include files.
kerberos5 Kerberos5 (Heimdal) package.
lib System libraries.
libexec System daemons.
release Release building Makefile & associated tools.
rescue Build system for statically linked /rescue utilities.
sbin System commands.
secure Cryptographic libraries and commands.
share Shared resources.
stand Boot loader sources.
sys Kernel sources.
sys/<arch>/conf Kernel configuration files. GENERIC is the configuration
used in release builds. NOTES contains documentation of
all possible entries.
tests Regression tests which can be run by Kyua. See tests/README
for additional information.
tools Utilities for regression testing and miscellaneous tasks.
usr.bin User commands.
usr.sbin System administration commands.
For information on synchronizing your source tree with one or more of the FreeBSD Project's development branches, please see:
https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/current-stable.html