regcomp.c uses the "start + count < end" idiom to check that there are
"count" bytes available in an array of char "start" and "end" both point to.
This is fine, unless "start + count" goes beyond the last element of the
array. In this case, pedantic interpretation of the C standard makes the
comparison of such a pointer against "end" undefined, and optimizers from
hell will happily remove as much code as possible because of this.
An example of this occurs in regcomp.c's bothcases(), which defines
bracket[3], sets "next" to "bracket" and "end" to "bracket + 2". Then it
invokes p_bracket(), which starts with "if (p->next + 5 < p->end)"...
Because bothcases() and p_bracket() are static functions in regcomp.c, there
is a real risk of miscompilation if aggressive inlining happens.
The following diff rewrites the "start + count < end" constructs into "end -
start > count". Assuming "end" and "start" are always pointing in the array
(such as "bracket[3]" above), "end - start" is well-defined and can be
compared without trouble.
As a bonus, MORE2() implies MORE() therefore SEETWO() can be simplified a
bit.
PR: 252403
It was realized just a little too late that this was a hack that belonged in
individual regex(3)-using applications. It was surrounded in NOTYET and not
implemented in the engine, so remove it.
This is the last of the needed GNU expressions before we can unleash bsdgrep
by default. \b is effectively an agnostic equivalent of \< and \>, while
\B will match every space that isn't making a transition from
nonchar -> char or char -> nonchar.
These are GNU extensions, generally equivalent to ^ and $ except that the
new syntax will not match beginning of line after the first in a multi-line
expression or the end of line before absolute last in a multi-line
expression.
The entire patch-set is not yet mature enough for commit, but this usable
subset is generally enough for googletest to be happy with and mostly map to
some existing concepts, so they're not as invasive.
The specific changes included here are:
- Branching in BREs with \|
- \w and \W for [[:alnum:]] and [^[:alnum:]] respectively
- \s and \S for [[:space:]] and [^[:space:]] respectively
- Additional quantifiers in BREs, \? and \+ (self-explanatory)
There's some #ifdef'd out work for allowing empty branches as a match-all.
This is a feature that's under assessment... future work will determine
how standard this behavior is and act accordingly.
In IEEE 1003.1-2008 [1] and earlier revisions, BRE/ERE grammar allows for
any character to be escaped, but "ORD_CHAR preceded by an unescaped
<backslash> character [gives undefined results]".
Historically, we've interpreted an escaped ordinary character as the
ordinary character itself. This becomes problematic when some extensions
give special meanings to an otherwise ordinary character
(e.g. GNU's \b, \s, \w), meaning we may have two different valid
interpretations of the same sequence.
To make this easier to deal with and given that the standard calls this
undefined, we should throw an error (EESCAPE) if we run into this scenario
to ease transition into a state where some escaped ordinaries are blessed
with a special meaning -- it will either error out or have extended
behavior, rather than have two entirely different versions of undefined
behavior that leave the consumer of regex(3) guessing as to what behavior
will be used or leaving them with false impressions.
This change bumps the symbol version of regcomp to FBSD_1.6 and provides the
old escape semantics for legacy applications, just in case one has an older
application that would immediately turn into a pumpkin because of an
extraneous escape that's embedded or otherwise critical to its operation.
This is the final piece needed before enhancing libregex with GNU extensions
and flipping the switch on bsdgrep.
[1] http://pubs.opengroup.org/onlinepubs/9699919799.2016edition/
PR: 229925 (exp-run, courtesy of antoine)
Differential Revision: https://reviews.freebsd.org/D10510
This fixes the obscure endless loop seen with case-insensitive
patterns containing characters in 128-255 range; originally
found running GNU grep test suite.
Our regex implementation being kludgy translates the characters
in case-insensitive pattern to bracket expression containing both
cases for the character and doesn't correctly handle the case when
original character is in bitmap and the other case is not, falling
into the endless loop going through in p_bracket(), ordinary(),
and bothcases().
Reducing the bitmap to 0-127 range for multibyte locales solves this
as none of these characters have other case mapping outside of bitmap.
We are also safe in the case when the original character outside of
bitmap has other case mapping in the bitmap (there are several of those
in our current ctype maps having unidirectional mapping into bitmap).
Reviewed by: bapt, kevans, pfg
Differential revision: https://reviews.freebsd.org/D18302
Matcher function incorrectly assumed that moffset that we get from
findmust is in bytes. Fix this by introducing a stepback function,
taking short path if MB_CUR_MAX is 1, and going back byte-by-byte,
checking if we have a legal character sequence otherwise.
PR: 153502
Reviewed by: pfg, kevans
Approved by: kib (mentor, implicit)
Differential revision: https://reviews.freebsd.org/D18297
The issue found with gcc6 build (originally on illumos, confirmed on FreeBSD).
Mark it __unused.
Differential Revision: https://reviews.freebsd.org/D13109
regcomp uses some libc internal collation bits that are not available in the
libregex context. It's easy enough to bring in the needed parts that can
work in a libregex world, so do so.
Pointy hat to: me
libregex is a regex(3) implementation intended to feature GNU extensions and
any other non-POSIX compliant extensions that are deemed worthy.
These extensions are separated out into a separate library for the sake of
not cluttering up libc further with them as well as not deteriorating the
speed (or lack thereof) of the libc implementation.
libregex is implemented as a build of the libc implementation with LIBREGEX
defined to distinguish this from a libc build. The reasons for
implementation like this are two-fold:
1.) Maintenance- This reduces the overhead induced by adding yet another
regex implementation to base.
2.) Ease of use- Flipping on GNU extensions will be as simple as linking
against libregex, and POSIX-compliant compilations can be guaranteed with a
REG_POSIX cflag that should be ignored by libc/regex and disables extensions
in libregex. It is also easier to keep REG_POSIX sane and POSIX pure when
implemented in this fashion.
Tests are added for future functionality, but left disconnected for the time
being while other testing is done.
Reviewed by: cem (previous version)
Differential Revision: https://reviews.freebsd.org/D12934
libc is set for WARNS=2, but the incoming libregex will use WARNS=6.
Sprinkle some casts and (void)bc's to alleviate the warnings that come along
with the higher WARNS level.
These 'bc' parameters could be outright removed, but as of right now they
will be used in some parts of libregex land. Silence the warnings instead
rather than flip-flopping.
Mainly focus on files that use BSD 3-Clause license.
The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.
Special thanks to Wind River for providing access to "The Duke of
Highlander" tool: an older (2014) run over FreeBSD tree was useful as a
starting point.
The bug is an out-of-bounds read detected with address sanitizer that
happens when 'sp' in p_b_coll_elems() includes NUL byte[s], e.g. if it's
equal to "GS\x00". In that case len will be equal to 4, and the
strncmp(cp->name, sp, len) call will succeed when cp->name is "GS" but the
cp->name[len] == '\0' comparison will cause the read to go out-of-bounds.
Checking the length using strlen() instead eliminates the issue.
The bug was found in LLVM with oss-fuzz:
https://reviews.llvm.org/D39380
MFC after: 1 week
Obtained from: Vlad Tsyrklevich through posting on openbsd-tech
Adding features for matching is fairly straightforward, but this requires
some duplication because of this fast/slow setup. They can be fairly
trivially combined into a single walk(), so do it to make future additions
less error prone.
Reviewed by: cem (earlier version), emaste, pfg
Approved by: emaste (mentor)
Differential Revision: https://reviews.freebsd.org/D11233
Currently, regex(3) exhibits the following wrong behavior as demonstrated
with sed:
- echo "a{1,2,3}b" | sed -r "s/{/_/" (1)
- echo "a{1,2,3}b" | sed "s/\}/_/" (2)
- echo "a{1,2,3}b" | sed -r "s/{}/_/" (3)
Cases (1) and (3) should throw errors but they actually succeed, and (2)
throws an error when it should match the literal '}'. The correct behavior
was decided by comparing to the behavior with the equivalent BRE (1)(3) or
ERE (2) and consulting POSIX, along with some reasonable evaluation.
Tests were also adjusted/added accordingly.
PR: 166861
Reviewed by: emaste, ngie, pfg
Approved by: emaste (mentor)
MFC after: never
Differential Revision: https://reviews.freebsd.org/D10315
An oversight in r320742 caused BREs to become sensitive to the branching operator prematurely, which caused
breakage in some limited situations -- namely, those that tried to use branching in a BRE. Most of these scenarios
had already been corrected beforehand to properly use gsed or grep for GNU extensions, so the damage is
slightly mitigated.
Reported by: antoine
Reported by: antoine
Approved by: emaste (mentor)
Differential Revision: https://reviews.freebsd.org/D11522
p_branch_empty was declared but never used due to an oversight. Use it as
designed, further comment on its return value.
Reported by: Jenkins (head-sparc64)
Reviewed by: emaste
Approved by: emaste (mentor)
MFC with: r320742
Differential Revision: https://reviews.freebsd.org/D11506
EREs closer together. Prepare for this and reduce the diff of libregex changes by
refactoring and combining the top-level parsers for EREs/BREs ahead of time.
Branching functionality has been split out to make it easier to follow the combined
version of the top-level parser. It may also be enabled in the parsing context to make
it easier when libregex enables branching for BREs.
A branching context was also added for the various branching functions and so that
BREs, for instance, can determine if they're the first expression in a chain of expressions
within the current branch and treat '*' as ordinary if so.
This should have no functional impact and negligible performance impact.
Reviewed by: cem, emaste, pfg
Approved by: emaste (mentor)
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D10920
When passed the invalid regular expression "a**", the error is
eventually detected and seterr() is called. It sets p->error
appropriatly and p->next and p->end to nuls which is a never used char
nuls[10] which is zeros due to .bss initialization. Unfortunatly,
p_ere_exp() and p_simp_re() both have fall through cases where they set
the error, decrement p->next and access it which means a read from what
ever .bss variable comes before nuls.
Found with regex_test:repet_multi and CHERI bounds checking.
Reviewed by: ngie, pfg, emaste
Obtained from: CheriBSD
Sponsored by: DARPA, AFRL
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D10541
Taking some hints from the regex variant in nvi(1) and higher-level
compiler warnings, update some types in our regex(3) implementation.
Joint work with: Kyle Evans
MFC after: 2 weeks
reallocarray(3) is a non portable extension that originated in OpenBSD.
Given that it is already in FreeBSD's libc it is useful for the cases
where reallocation involves a multiplication.
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D9955
Renumber cluase 4 to 3, per what everybody else did when BSD granted
them permission to remove clause 3. My insistance on keeping the same
numbering for legal reasons is too pedantic, so give up on that point.
Submitted by: Jan Schaumann <jschauma@stevens.edu>
Pull Request: https://github.com/freebsd/freebsd/pull/96
This reduces build output, need for recalculating paths, and makes it clearer
which paths are relative to what areas in the source tree. The change in
performance over a locally mounted UFS filesystem was negligible in my testing,
but this may more positively impact other filesystems like NFS.
LIBC_SRCTOP was left alone so Juniper (and other users) can continue to
manipulate lib/libc/Makefile (and other Makefile.inc's under lib/libc) as
include Makefiles with custom options.
Discussed with: marcel, sjg
MFC after: 1 week
Reviewed by: emaste
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D9207
locale (which cause core dump) by removing whole 'table' argument
by which it passed.
2) Restore __collate_range_cmp() in __sccl().
3) Collating [a-z] range in regcomp() only for single bytes locales
(we can't do it now for other ones). In previous state only first 256
wchars are considered and all others are just silently dropped from the
range.
Instead of changing whole course to another POSIX-permitted way
for consistency and uniformity I decide to completely ignore missing
regex fucntionality and concentrace on fixing bugs in what we have now,
too many small obstacles instead, counting ports.
Only first 256 wide chars are considered currently, all other are just
dropped from the range. Proper implementation require reverse tables
database lookup, since objects are really big as max UTF-8 (1114112
code points), so just the same scanning as it was for 256 chars will
slow things down.
POSIX does not require collation for [a-z] type ranges and does not
prohibit it for non-POSIX locales. POSIX require collation for ranges
only for POSIX (or C) locale which is equal to ASCII and binary for
other chars, so we already have it.
No other *BSD implements collation for [a-z] type ranges.
Restore ABI compatibility with unused now __collate_range_cmp() which
is visible from outside (will be removed later).
When collation support was brought in, the second and third
arguments in __collate_range_cmp() were changed from int to
wchar_t, breaking the ABI. Change them to a "char" type which
makes more sense and keeps the ABI compatible.
Also introduce __wcollate_range_cmp() which does work with wide
characters. This function is used only internally in libc so
we don't export it. Use the new function in glob(3), fnmatch(3),
and regexec(3).
PR: 179721
Suggested by: ache. jilles
MFC after: 3 weeks (perhaps partial only)
Change the behavior of when REG_STARTEND is combined with REG_NOTBOL.
From the original posting[1]:
"Enable the assumption that pmatch[0].rm_so is a continuation offset
to a string and allows us to do a proper assessment of the character
in regards to it's word position ('^' or '\<'), without risking going
into unallocated memory."
This change makes us similar to how glibc handles REG_STARTEND |
REG_NOTBOL, and is closely related to a soon-to-land fix to sed.
Special thanks to Martijn van Duren and Ingo Schwarze for working
out some consistent behaviour.
Differential Revision: https://reviews.freebsd.org/D6257
Taken from: openbsd-tech 2016-05-24 [1] (Martijn van Duren)
Relnotes: yes
MFC after: 1 month
Fix some rather complex regex issues found on OpenBSD as part of some
ongoing work to fix a sed(1) bug.
Curiously the OpenBSD tests don't trigger segfaults on FreeBSD but the
bugs were confirmed by running a port of FreeBSD's regex under OpenBSD's
malloc. Huge thanks to Ingo for confirming the behavior.
Taken from: Ingo Schwarze (through openbsd-tech 2016-05-15)
MFC after: 1 week
In matcher() we used an integer to index nsub of type size_t.
In print() we used an integer to index nstates of type sopno,
typedef'd long.
In both cases the indexes never take negative values.
Match the types to avoid any error.
MFC after: 5 days
- Create automatically generated include header for split.c
main.c:
- Use function definitions from debug.ih and split.ih instead of externs
Sponsored by: EMC / Isilon Storage Division
