d/freebsd-dev
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
a41b48ea12
freebsd-dev/lib/libc/regex/engine.c

1136 lines
29 KiB
C
Raw Normal View History

BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
/*-
General further adoption of SPDX licensing ID tags. Mainly focus on files that use BSD 3-Clause license. The Software Package Data Exchange (SPDX) group provides a specification to make it easier for automated tools to detect and summarize well known opensource licenses. We are gradually adopting the specification, noting that the tags are considered only advisory and do not, in any way, superceed or replace the license texts. Special thanks to Wind River for providing access to "The Duke of Highlander" tool: an older (2014) run over FreeBSD tree was useful as a starting point.
2017-11-20 19:49:47 +00:00
* SPDX-License-Identifier: BSD-3-Clause
*
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
* Copyright (c) 1992, 1993, 1994 Henry Spencer.
* Copyright (c) 1992, 1993, 1994
* The Regents of the University of California. All rights reserved.
*
* This code is derived from software contributed to Berkeley by
* Henry Spencer.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
Renumber copyright clause 4 Renumber cluase 4 to 3, per what everybody else did when BSD granted them permission to remove clause 3. My insistance on keeping the same numbering for legal reasons is too pedantic, so give up on that point. Submitted by: Jan Schaumann <jschauma@stevens.edu> Pull Request: https://github.com/freebsd/freebsd/pull/96
2017-02-28 23:42:47 +00:00
* 3. Neither the name of the University nor the names of its contributors
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*
* @(#)engine.c 8.5 (Berkeley) 3/20/94
*/
Fix the style of the SCM ID's. I believe have made all of libc .c's as consistent as possible.
2002-03-22 21:53:29 +00:00
#include <sys/cdefs.h>
__FBSDID("$FreeBSD$");
regex(3): Refactor fast/slow stepping bits in the matching engine Adding features for matching is fairly straightforward, but this requires some duplication because of this fast/slow setup. They can be fairly trivially combined into a single walk(), so do it to make future additions less error prone. Reviewed by: cem (earlier version), emaste, pfg Approved by: emaste (mentor) Differential Revision: https://reviews.freebsd.org/D11233
2017-08-09 01:04:36 +00:00
#include <stdbool.h>
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
/*
* The matching engine and friends. This file is #included by regexec.c
* after suitable #defines of a variety of macros used herein, so that
* different state representations can be used without duplicating masses
* of code.
*/
#ifdef SNAMES
regexec: fix processing multibyte strings. Matcher function incorrectly assumed that moffset that we get from findmust is in bytes. Fix this by introducing a stepback function, taking short path if MB_CUR_MAX is 1, and going back byte-by-byte, checking if we have a legal character sequence otherwise. PR: 153502 Reviewed by: pfg, kevans Approved by: kib (mentor, implicit) Differential revision: https://reviews.freebsd.org/D18297
2018-11-23 15:49:18 +00:00
#define stepback sstepback
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
#define matcher smatcher
regex(3): Refactor fast/slow stepping bits in the matching engine Adding features for matching is fairly straightforward, but this requires some duplication because of this fast/slow setup. They can be fairly trivially combined into a single walk(), so do it to make future additions less error prone. Reviewed by: cem (earlier version), emaste, pfg Approved by: emaste (mentor) Differential Revision: https://reviews.freebsd.org/D11233
2017-08-09 01:04:36 +00:00
#define walk swalk
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
#define dissect sdissect
#define backref sbackref
#define step sstep
#define print sprint
#define at sat
#define match smat
#endif
#ifdef LNAMES
regexec: fix processing multibyte strings. Matcher function incorrectly assumed that moffset that we get from findmust is in bytes. Fix this by introducing a stepback function, taking short path if MB_CUR_MAX is 1, and going back byte-by-byte, checking if we have a legal character sequence otherwise. PR: 153502 Reviewed by: pfg, kevans Approved by: kib (mentor, implicit) Differential revision: https://reviews.freebsd.org/D18297
2018-11-23 15:49:18 +00:00
#define stepback lstepback
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
#define matcher lmatcher
regex(3): Refactor fast/slow stepping bits in the matching engine Adding features for matching is fairly straightforward, but this requires some duplication because of this fast/slow setup. They can be fairly trivially combined into a single walk(), so do it to make future additions less error prone. Reviewed by: cem (earlier version), emaste, pfg Approved by: emaste (mentor) Differential Revision: https://reviews.freebsd.org/D11233
2017-08-09 01:04:36 +00:00
#define walk lwalk
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
#define dissect ldissect
#define backref lbackref
#define step lstep
#define print lprint
#define at lat
#define match lmat
#endif
Make regular expression matching aware of multibyte characters. The general idea is that we perform multibyte->wide character conversion while parsing and compiling, then convert byte sequences to wide characters when they're needed for comparison and stepping through the string during execution. As with tr(1), the main complication is to efficiently represent sets of characters in bracket expressions. The old bitmap representation is replaced by a bitmap for the first 256 characters combined with a vector of individual wide characters, a vector of character ranges (for [A-Z] etc.), and a vector of character classes (for [[:alpha:]] etc.). One other point of interest is that although the Boyer-Moore algorithm had to be disabled in the general multibyte case, it is still enabled for UTF-8 because of its self-synchronizing nature. This greatly speeds up matching by reducing the number of multibyte conversions that need to be done.
2004-07-12 07:35:59 +00:00
#ifdef MNAMES
regexec: fix processing multibyte strings. Matcher function incorrectly assumed that moffset that we get from findmust is in bytes. Fix this by introducing a stepback function, taking short path if MB_CUR_MAX is 1, and going back byte-by-byte, checking if we have a legal character sequence otherwise. PR: 153502 Reviewed by: pfg, kevans Approved by: kib (mentor, implicit) Differential revision: https://reviews.freebsd.org/D18297
2018-11-23 15:49:18 +00:00
#define stepback mstepback
Make regular expression matching aware of multibyte characters. The general idea is that we perform multibyte->wide character conversion while parsing and compiling, then convert byte sequences to wide characters when they're needed for comparison and stepping through the string during execution. As with tr(1), the main complication is to efficiently represent sets of characters in bracket expressions. The old bitmap representation is replaced by a bitmap for the first 256 characters combined with a vector of individual wide characters, a vector of character ranges (for [A-Z] etc.), and a vector of character classes (for [[:alpha:]] etc.). One other point of interest is that although the Boyer-Moore algorithm had to be disabled in the general multibyte case, it is still enabled for UTF-8 because of its self-synchronizing nature. This greatly speeds up matching by reducing the number of multibyte conversions that need to be done.
2004-07-12 07:35:59 +00:00
#define matcher mmatcher
regex(3): Refactor fast/slow stepping bits in the matching engine Adding features for matching is fairly straightforward, but this requires some duplication because of this fast/slow setup. They can be fairly trivially combined into a single walk(), so do it to make future additions less error prone. Reviewed by: cem (earlier version), emaste, pfg Approved by: emaste (mentor) Differential Revision: https://reviews.freebsd.org/D11233
2017-08-09 01:04:36 +00:00
#define walk mwalk
Make regular expression matching aware of multibyte characters. The general idea is that we perform multibyte->wide character conversion while parsing and compiling, then convert byte sequences to wide characters when they're needed for comparison and stepping through the string during execution. As with tr(1), the main complication is to efficiently represent sets of characters in bracket expressions. The old bitmap representation is replaced by a bitmap for the first 256 characters combined with a vector of individual wide characters, a vector of character ranges (for [A-Z] etc.), and a vector of character classes (for [[:alpha:]] etc.). One other point of interest is that although the Boyer-Moore algorithm had to be disabled in the general multibyte case, it is still enabled for UTF-8 because of its self-synchronizing nature. This greatly speeds up matching by reducing the number of multibyte conversions that need to be done.
2004-07-12 07:35:59 +00:00
#define dissect mdissect
#define backref mbackref
#define step mstep
#define print mprint
#define at mat
#define match mmat
#endif
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
/* another structure passed up and down to avoid zillions of parameters */
struct match {
struct re_guts *g;
int eflags;
regmatch_t *pmatch; /* [nsub+1] (0 element unused) */
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
const char *offp; /* offsets work from here */
const char *beginp; /* start of string -- virtual NUL precedes */
const char *endp; /* end of string -- virtual NUL here */
const char *coldp; /* can be no match starting before here */
const char **lastpos; /* [nplus+1] */
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
STATEVARS;
states st; /* current states */
states fresh; /* states for a fresh start */
states tmp; /* temporary */
states empty; /* empty set of states */
Make regular expression matching aware of multibyte characters. The general idea is that we perform multibyte->wide character conversion while parsing and compiling, then convert byte sequences to wide characters when they're needed for comparison and stepping through the string during execution. As with tr(1), the main complication is to efficiently represent sets of characters in bracket expressions. The old bitmap representation is replaced by a bitmap for the first 256 characters combined with a vector of individual wide characters, a vector of character ranges (for [A-Z] etc.), and a vector of character classes (for [[:alpha:]] etc.). One other point of interest is that although the Boyer-Moore algorithm had to be disabled in the general multibyte case, it is still enabled for UTF-8 because of its self-synchronizing nature. This greatly speeds up matching by reducing the number of multibyte conversions that need to be done.
2004-07-12 07:35:59 +00:00
mbstate_t mbs; /* multibyte conversion state */
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
};
/* ========= begin header generated by ./mkh ========= */
#ifdef __cplusplus
extern "C" {
#endif
/* === engine.c === */
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
static int matcher(struct re_guts *g, const char *string, size_t nmatch, regmatch_t pmatch[], int eflags);
static const char *dissect(struct match *m, const char *start, const char *stop, sopno startst, sopno stopst);
static const char *backref(struct match *m, const char *start, const char *stop, sopno startst, sopno stopst, sopno lev, int);
regex(3): Refactor fast/slow stepping bits in the matching engine Adding features for matching is fairly straightforward, but this requires some duplication because of this fast/slow setup. They can be fairly trivially combined into a single walk(), so do it to make future additions less error prone. Reviewed by: cem (earlier version), emaste, pfg Approved by: emaste (mentor) Differential Revision: https://reviews.freebsd.org/D11233
2017-08-09 01:04:36 +00:00
static const char *walk(struct match *m, const char *start, const char *stop, sopno startst, sopno stopst, bool fast);
Make regular expression matching aware of multibyte characters. The general idea is that we perform multibyte->wide character conversion while parsing and compiling, then convert byte sequences to wide characters when they're needed for comparison and stepping through the string during execution. As with tr(1), the main complication is to efficiently represent sets of characters in bracket expressions. The old bitmap representation is replaced by a bitmap for the first 256 characters combined with a vector of individual wide characters, a vector of character ranges (for [A-Z] etc.), and a vector of character classes (for [[:alpha:]] etc.). One other point of interest is that although the Boyer-Moore algorithm had to be disabled in the general multibyte case, it is still enabled for UTF-8 because of its self-synchronizing nature. This greatly speeds up matching by reducing the number of multibyte conversions that need to be done.
2004-07-12 07:35:59 +00:00
static states step(struct re_guts *g, sopno start, sopno stop, states bef, wint_t ch, states aft);
Only stop evaluation of a back reference if the match length is zero and the recursion level is too deep. Obtained from: OpenBSD
2007-03-05 09:43:55 +00:00
#define MAX_RECURSION 100
Make regular expression matching aware of multibyte characters. The general idea is that we perform multibyte->wide character conversion while parsing and compiling, then convert byte sequences to wide characters when they're needed for comparison and stepping through the string during execution. As with tr(1), the main complication is to efficiently represent sets of characters in bracket expressions. The old bitmap representation is replaced by a bitmap for the first 256 characters combined with a vector of individual wide characters, a vector of character ranges (for [A-Z] etc.), and a vector of character classes (for [[:alpha:]] etc.). One other point of interest is that although the Boyer-Moore algorithm had to be disabled in the general multibyte case, it is still enabled for UTF-8 because of its self-synchronizing nature. This greatly speeds up matching by reducing the number of multibyte conversions that need to be done.
2004-07-12 07:35:59 +00:00
#define BOL (OUT-1)
#define EOL (BOL-1)
#define BOLEOL (BOL-2)
#define NOTHING (BOL-3)
#define BOW (BOL-4)
#define EOW (BOL-5)
#define BADCHAR (BOL-6)
#define NONCHAR(c) ((c) <= OUT)
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
#ifdef REDEBUG
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
static void print(struct match *m, const char *caption, states st, int ch, FILE *d);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
#endif
#ifdef REDEBUG
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
static void at(struct match *m, const char *title, const char *start, const char *stop, sopno startst, sopno stopst);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
#endif
#ifdef REDEBUG
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
static const char *pchar(int ch);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
#endif
#ifdef __cplusplus
}
#endif
/* ========= end header generated by ./mkh ========= */
#ifdef REDEBUG
#define SP(t, s, c) print(m, t, s, c, stdout)
#define AT(t, p1, p2, s1, s2) at(m, t, p1, p2, s1, s2)
#define NOTE(str) { if (m->eflags&REG_TRACE) printf("=%s\n", (str)); }
#else
#define SP(t, s, c) /* nothing */
#define AT(t, p1, p2, s1, s2) /* nothing */
#define NOTE(s) /* nothing */
#endif
regexec: fix processing multibyte strings. Matcher function incorrectly assumed that moffset that we get from findmust is in bytes. Fix this by introducing a stepback function, taking short path if MB_CUR_MAX is 1, and going back byte-by-byte, checking if we have a legal character sequence otherwise. PR: 153502 Reviewed by: pfg, kevans Approved by: kib (mentor, implicit) Differential revision: https://reviews.freebsd.org/D18297
2018-11-23 15:49:18 +00:00
/*
* Given a multibyte string pointed to by start, step back nchar characters
* from current position pointed to by cur.
*/
static const char *
stepback(const char *start, const char *cur, int nchar)
{
const char *ret;
int wc, mbc;
mbstate_t mbs;
size_t clen;
if (MB_CUR_MAX == 1)
return ((cur - nchar) > start ? cur - nchar : NULL);
ret = cur;
for (wc = nchar; wc > 0; wc--) {
for (mbc = 1; mbc <= MB_CUR_MAX; mbc++) {
if ((ret - mbc) < start)
return (NULL);
memset(&mbs, 0, sizeof(mbs));
clen = mbrtowc(NULL, ret - mbc, mbc, &mbs);
if (clen != (size_t)-1 && clen != (size_t)-2)
break;
}
if (mbc > MB_CUR_MAX)
return (NULL);
ret -= mbc;
}
return (ret);
}
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
/*
- matcher - the actual matching engine
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
== static int matcher(struct re_guts *g, const char *string, \
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
== size_t nmatch, regmatch_t pmatch[], int eflags);
*/
static int /* 0 success, REG_NOMATCH failure */
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
matcher(struct re_guts *g,
const char *string,
size_t nmatch,
regmatch_t pmatch[],
int eflags)
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
{
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
const char *endp;
regex: prevent two improbable signed integer overflows. In matcher() we used an integer to index nsub of type size_t. In print() we used an integer to index nstates of type sopno, typedef'd long. In both cases the indexes never take negative values. Match the types to avoid any error. MFC after: 5 days
2016-04-23 20:45:09 +00:00
size_t i;
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
struct match mv;
Remove 'register' keyword.
2002-03-21 18:49:23 +00:00
struct match *m = &mv;
regex(3): Fix uninitialized pointer values. CID: 405582 (also clang static checker) CID: 1018724
2015-02-20 21:21:38 +00:00
const char *dp = NULL;
Remove 'register' keyword.
2002-03-21 18:49:23 +00:00
const sopno gf = g->firststate+1; /* +1 for OEND */
const sopno gl = g->laststate;
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
const char *start;
const char *stop;
Add Boyler-Moore algorithm to pre-matching test. The BM algorithm works by scanning the pattern from right to left, and jumping as many characters as viable based on the text's mismatched character and the pattern's already matched suffix. This typically enable us to test only a fraction of the text's characters, but has a worse performance than the straight-forward method for small patterns. Because of this, the BM algorithm will only be used if the pattern size is at least 4 characters. Notice that this pre-matching is done on the largest substring of the regular expression that _must_ be present on the text for a succesful match to be possible at all. For instance, "(xyzzy|grues)" will yield a null "must" substring, and, therefore, not benefit from the BM algorithm at all. Because of the lack of intelligence of the algorithm that finds the "must" string, things like "charjump|matchjump" will also yield a null string. To optimize that, "(char|match)jump" should be used. The setup time (at regcomp()) for the BM algorithm will most likely outweight any benefits for one-time matches. Given the slow regex(3) we have, this is unlikely to be even perceptible, though. The size of a regex_t structure is increased by 2*sizeof(char*) + 256*sizeof(int) + strlen(must)*sizeof(int). This is all inside the regex_t's "guts", which is allocated dynamically by regcomp(). If allocation of either of the two tables fail, the other one is freed. In this case, the straight-forward algorithm is used for pre-matching. Tests exercising the code path affected have shown a speed increase of 50% for "must" strings of length four or five. API and ABI remain unchanged by this commit. The patch submitted on the PR was not used, as it was non-functional. PR: 14342
2000-06-29 04:48:34 +00:00
/* Boyer-Moore algorithms variables */
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
const char *pp;
Add Boyler-Moore algorithm to pre-matching test. The BM algorithm works by scanning the pattern from right to left, and jumping as many characters as viable based on the text's mismatched character and the pattern's already matched suffix. This typically enable us to test only a fraction of the text's characters, but has a worse performance than the straight-forward method for small patterns. Because of this, the BM algorithm will only be used if the pattern size is at least 4 characters. Notice that this pre-matching is done on the largest substring of the regular expression that _must_ be present on the text for a succesful match to be possible at all. For instance, "(xyzzy|grues)" will yield a null "must" substring, and, therefore, not benefit from the BM algorithm at all. Because of the lack of intelligence of the algorithm that finds the "must" string, things like "charjump|matchjump" will also yield a null string. To optimize that, "(char|match)jump" should be used. The setup time (at regcomp()) for the BM algorithm will most likely outweight any benefits for one-time matches. Given the slow regex(3) we have, this is unlikely to be even perceptible, though. The size of a regex_t structure is increased by 2*sizeof(char*) + 256*sizeof(int) + strlen(must)*sizeof(int). This is all inside the regex_t's "guts", which is allocated dynamically by regcomp(). If allocation of either of the two tables fail, the other one is freed. In this case, the straight-forward algorithm is used for pre-matching. Tests exercising the code path affected have shown a speed increase of 50% for "must" strings of length four or five. API and ABI remain unchanged by this commit. The patch submitted on the PR was not used, as it was non-functional. PR: 14342
2000-06-29 04:48:34 +00:00
int cj, mj;
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
const char *mustfirst;
const char *mustlast;
Remove 'register' keyword.
2002-03-21 18:49:23 +00:00
int *matchjump;
int *charjump;
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
/* simplify the situation where possible */
if (g->cflags&REG_NOSUB)
nmatch = 0;
if (eflags&REG_STARTEND) {
start = string + pmatch[0].rm_so;
stop = string + pmatch[0].rm_eo;
} else {
start = string;
stop = start + strlen(start);
}
if (stop < start)
return(REG_INVARG);
/* prescreening; this does wonders for this rather slow code */
if (g->must != NULL) {
Enhance the optimization provided by pre-matching. Fix style bugs with previous commits. At the time we search the pattern for the "must" string, we now compute the longest offset from the beginning of the pattern at which the must string might be found. If that offset is found to be infinite (through use of "+" or "*"), we set it to -1 to disable the heuristics applied later. After we are done with pre-matching, we use that offset and the point in the text at which the must string was found to compute the earliest point at which the pattern might be found. Special care should be taken here. The variable "start" is passed to the automata-processing functions fast() and slow() to indicate the point in the text at which they should start working from. The real beginning of the text is passed in a struct match variable m, which is used to check for anchors. That variable, though, is initialized with "start", so we must not adjust "start" before "m" is properly initialized. Simple tests showed a speed increase from 100% to 400%, but they were biased in that regexec() was called for the whole file instead of line by line, and parenthized subexpressions were not searched for. This change adds a single integer to the size of the "guts" structure, and does not change the ABI. Further improvements possible: Since the speed increase observed here is so huge, one intuitive optimization would be to introduce a bias in the function that computes the "must" string so as to prefer a smaller string with a finite offset over a larger one with an infinite offset. Tests have shown this to be a bad idea, though, as the cost of false pre-matches far outweights the benefits of a must offset, even in biased situations. A number of other improvements suggest themselves, though: * identify the cases where the pattern is identical to the must string, and avoid entering fast() and slow() in these cases. * compute the maximum offset from the must string to the end of the pattern, and use that to set the point at which fast() and slow() should give up trying to find a match, and return then return to pre-matching. * return all the way to pre-matching if a "match" was found and later invalidated by back reference processing. Since back references are evil and should be avoided anyway, this is of little use.
2000-07-02 10:58:07 +00:00
if (g->charjump != NULL && g->matchjump != NULL) {
Add Boyler-Moore algorithm to pre-matching test. The BM algorithm works by scanning the pattern from right to left, and jumping as many characters as viable based on the text's mismatched character and the pattern's already matched suffix. This typically enable us to test only a fraction of the text's characters, but has a worse performance than the straight-forward method for small patterns. Because of this, the BM algorithm will only be used if the pattern size is at least 4 characters. Notice that this pre-matching is done on the largest substring of the regular expression that _must_ be present on the text for a succesful match to be possible at all. For instance, "(xyzzy|grues)" will yield a null "must" substring, and, therefore, not benefit from the BM algorithm at all. Because of the lack of intelligence of the algorithm that finds the "must" string, things like "charjump|matchjump" will also yield a null string. To optimize that, "(char|match)jump" should be used. The setup time (at regcomp()) for the BM algorithm will most likely outweight any benefits for one-time matches. Given the slow regex(3) we have, this is unlikely to be even perceptible, though. The size of a regex_t structure is increased by 2*sizeof(char*) + 256*sizeof(int) + strlen(must)*sizeof(int). This is all inside the regex_t's "guts", which is allocated dynamically by regcomp(). If allocation of either of the two tables fail, the other one is freed. In this case, the straight-forward algorithm is used for pre-matching. Tests exercising the code path affected have shown a speed increase of 50% for "must" strings of length four or five. API and ABI remain unchanged by this commit. The patch submitted on the PR was not used, as it was non-functional. PR: 14342
2000-06-29 04:48:34 +00:00
mustfirst = g->must;
mustlast = g->must + g->mlen - 1;
charjump = g->charjump;
matchjump = g->matchjump;
pp = mustlast;
Deal with the signed/unsigned chars issue in a more proper manner. We use a CHAR_MIN-based array, like elsewhere in the code. Remove a number of unused variables (some due to the above change, one that was left after a number of optimizing steps through the source). Brucified by: bde
2000-07-07 07:46:36 +00:00
for (dp = start+g->mlen-1; dp < stop;) {
Add Boyler-Moore algorithm to pre-matching test. The BM algorithm works by scanning the pattern from right to left, and jumping as many characters as viable based on the text's mismatched character and the pattern's already matched suffix. This typically enable us to test only a fraction of the text's characters, but has a worse performance than the straight-forward method for small patterns. Because of this, the BM algorithm will only be used if the pattern size is at least 4 characters. Notice that this pre-matching is done on the largest substring of the regular expression that _must_ be present on the text for a succesful match to be possible at all. For instance, "(xyzzy|grues)" will yield a null "must" substring, and, therefore, not benefit from the BM algorithm at all. Because of the lack of intelligence of the algorithm that finds the "must" string, things like "charjump|matchjump" will also yield a null string. To optimize that, "(char|match)jump" should be used. The setup time (at regcomp()) for the BM algorithm will most likely outweight any benefits for one-time matches. Given the slow regex(3) we have, this is unlikely to be even perceptible, though. The size of a regex_t structure is increased by 2*sizeof(char*) + 256*sizeof(int) + strlen(must)*sizeof(int). This is all inside the regex_t's "guts", which is allocated dynamically by regcomp(). If allocation of either of the two tables fail, the other one is freed. In this case, the straight-forward algorithm is used for pre-matching. Tests exercising the code path affected have shown a speed increase of 50% for "must" strings of length four or five. API and ABI remain unchanged by this commit. The patch submitted on the PR was not used, as it was non-functional. PR: 14342
2000-06-29 04:48:34 +00:00
/* Fast skip non-matches */
Eliminate 61 warnings emitted at WARNS=2 (leaving 53 to go). Only warnings that could be fixed without changing the generated object code and without restructuring the source code have been handled. Reviewed by: /sbin/md5
2003-02-16 17:29:11 +00:00
while (dp < stop && charjump[(int)*dp])
dp += charjump[(int)*dp];
Add Boyler-Moore algorithm to pre-matching test. The BM algorithm works by scanning the pattern from right to left, and jumping as many characters as viable based on the text's mismatched character and the pattern's already matched suffix. This typically enable us to test only a fraction of the text's characters, but has a worse performance than the straight-forward method for small patterns. Because of this, the BM algorithm will only be used if the pattern size is at least 4 characters. Notice that this pre-matching is done on the largest substring of the regular expression that _must_ be present on the text for a succesful match to be possible at all. For instance, "(xyzzy|grues)" will yield a null "must" substring, and, therefore, not benefit from the BM algorithm at all. Because of the lack of intelligence of the algorithm that finds the "must" string, things like "charjump|matchjump" will also yield a null string. To optimize that, "(char|match)jump" should be used. The setup time (at regcomp()) for the BM algorithm will most likely outweight any benefits for one-time matches. Given the slow regex(3) we have, this is unlikely to be even perceptible, though. The size of a regex_t structure is increased by 2*sizeof(char*) + 256*sizeof(int) + strlen(must)*sizeof(int). This is all inside the regex_t's "guts", which is allocated dynamically by regcomp(). If allocation of either of the two tables fail, the other one is freed. In this case, the straight-forward algorithm is used for pre-matching. Tests exercising the code path affected have shown a speed increase of 50% for "must" strings of length four or five. API and ABI remain unchanged by this commit. The patch submitted on the PR was not used, as it was non-functional. PR: 14342
2000-06-29 04:48:34 +00:00
Deal with the signed/unsigned chars issue in a more proper manner. We use a CHAR_MIN-based array, like elsewhere in the code. Remove a number of unused variables (some due to the above change, one that was left after a number of optimizing steps through the source). Brucified by: bde
2000-07-07 07:46:36 +00:00
if (dp >= stop)
Add Boyler-Moore algorithm to pre-matching test. The BM algorithm works by scanning the pattern from right to left, and jumping as many characters as viable based on the text's mismatched character and the pattern's already matched suffix. This typically enable us to test only a fraction of the text's characters, but has a worse performance than the straight-forward method for small patterns. Because of this, the BM algorithm will only be used if the pattern size is at least 4 characters. Notice that this pre-matching is done on the largest substring of the regular expression that _must_ be present on the text for a succesful match to be possible at all. For instance, "(xyzzy|grues)" will yield a null "must" substring, and, therefore, not benefit from the BM algorithm at all. Because of the lack of intelligence of the algorithm that finds the "must" string, things like "charjump|matchjump" will also yield a null string. To optimize that, "(char|match)jump" should be used. The setup time (at regcomp()) for the BM algorithm will most likely outweight any benefits for one-time matches. Given the slow regex(3) we have, this is unlikely to be even perceptible, though. The size of a regex_t structure is increased by 2*sizeof(char*) + 256*sizeof(int) + strlen(must)*sizeof(int). This is all inside the regex_t's "guts", which is allocated dynamically by regcomp(). If allocation of either of the two tables fail, the other one is freed. In this case, the straight-forward algorithm is used for pre-matching. Tests exercising the code path affected have shown a speed increase of 50% for "must" strings of length four or five. API and ABI remain unchanged by this commit. The patch submitted on the PR was not used, as it was non-functional. PR: 14342
2000-06-29 04:48:34 +00:00
break;
/* Greedy matcher */
/* We depend on not being used for
* for strings of length 1
*/
Deal with the signed/unsigned chars issue in a more proper manner. We use a CHAR_MIN-based array, like elsewhere in the code. Remove a number of unused variables (some due to the above change, one that was left after a number of optimizing steps through the source). Brucified by: bde
2000-07-07 07:46:36 +00:00
while (*--dp == *--pp && pp != mustfirst);
Add Boyler-Moore algorithm to pre-matching test. The BM algorithm works by scanning the pattern from right to left, and jumping as many characters as viable based on the text's mismatched character and the pattern's already matched suffix. This typically enable us to test only a fraction of the text's characters, but has a worse performance than the straight-forward method for small patterns. Because of this, the BM algorithm will only be used if the pattern size is at least 4 characters. Notice that this pre-matching is done on the largest substring of the regular expression that _must_ be present on the text for a succesful match to be possible at all. For instance, "(xyzzy|grues)" will yield a null "must" substring, and, therefore, not benefit from the BM algorithm at all. Because of the lack of intelligence of the algorithm that finds the "must" string, things like "charjump|matchjump" will also yield a null string. To optimize that, "(char|match)jump" should be used. The setup time (at regcomp()) for the BM algorithm will most likely outweight any benefits for one-time matches. Given the slow regex(3) we have, this is unlikely to be even perceptible, though. The size of a regex_t structure is increased by 2*sizeof(char*) + 256*sizeof(int) + strlen(must)*sizeof(int). This is all inside the regex_t's "guts", which is allocated dynamically by regcomp(). If allocation of either of the two tables fail, the other one is freed. In this case, the straight-forward algorithm is used for pre-matching. Tests exercising the code path affected have shown a speed increase of 50% for "must" strings of length four or five. API and ABI remain unchanged by this commit. The patch submitted on the PR was not used, as it was non-functional. PR: 14342
2000-06-29 04:48:34 +00:00
Deal with the signed/unsigned chars issue in a more proper manner. We use a CHAR_MIN-based array, like elsewhere in the code. Remove a number of unused variables (some due to the above change, one that was left after a number of optimizing steps through the source). Brucified by: bde
2000-07-07 07:46:36 +00:00
if (*dp == *pp)
Add Boyler-Moore algorithm to pre-matching test. The BM algorithm works by scanning the pattern from right to left, and jumping as many characters as viable based on the text's mismatched character and the pattern's already matched suffix. This typically enable us to test only a fraction of the text's characters, but has a worse performance than the straight-forward method for small patterns. Because of this, the BM algorithm will only be used if the pattern size is at least 4 characters. Notice that this pre-matching is done on the largest substring of the regular expression that _must_ be present on the text for a succesful match to be possible at all. For instance, "(xyzzy|grues)" will yield a null "must" substring, and, therefore, not benefit from the BM algorithm at all. Because of the lack of intelligence of the algorithm that finds the "must" string, things like "charjump|matchjump" will also yield a null string. To optimize that, "(char|match)jump" should be used. The setup time (at regcomp()) for the BM algorithm will most likely outweight any benefits for one-time matches. Given the slow regex(3) we have, this is unlikely to be even perceptible, though. The size of a regex_t structure is increased by 2*sizeof(char*) + 256*sizeof(int) + strlen(must)*sizeof(int). This is all inside the regex_t's "guts", which is allocated dynamically by regcomp(). If allocation of either of the two tables fail, the other one is freed. In this case, the straight-forward algorithm is used for pre-matching. Tests exercising the code path affected have shown a speed increase of 50% for "must" strings of length four or five. API and ABI remain unchanged by this commit. The patch submitted on the PR was not used, as it was non-functional. PR: 14342
2000-06-29 04:48:34 +00:00
break;
/* Jump to next possible match */
mj = matchjump[pp - mustfirst];
Eliminate 61 warnings emitted at WARNS=2 (leaving 53 to go). Only warnings that could be fixed without changing the generated object code and without restructuring the source code have been handled. Reviewed by: /sbin/md5
2003-02-16 17:29:11 +00:00
cj = charjump[(int)*dp];
Deal with the signed/unsigned chars issue in a more proper manner. We use a CHAR_MIN-based array, like elsewhere in the code. Remove a number of unused variables (some due to the above change, one that was left after a number of optimizing steps through the source). Brucified by: bde
2000-07-07 07:46:36 +00:00
dp += (cj < mj ? mj : cj);
pp = mustlast;
Add Boyler-Moore algorithm to pre-matching test. The BM algorithm works by scanning the pattern from right to left, and jumping as many characters as viable based on the text's mismatched character and the pattern's already matched suffix. This typically enable us to test only a fraction of the text's characters, but has a worse performance than the straight-forward method for small patterns. Because of this, the BM algorithm will only be used if the pattern size is at least 4 characters. Notice that this pre-matching is done on the largest substring of the regular expression that _must_ be present on the text for a succesful match to be possible at all. For instance, "(xyzzy|grues)" will yield a null "must" substring, and, therefore, not benefit from the BM algorithm at all. Because of the lack of intelligence of the algorithm that finds the "must" string, things like "charjump|matchjump" will also yield a null string. To optimize that, "(char|match)jump" should be used. The setup time (at regcomp()) for the BM algorithm will most likely outweight any benefits for one-time matches. Given the slow regex(3) we have, this is unlikely to be even perceptible, though. The size of a regex_t structure is increased by 2*sizeof(char*) + 256*sizeof(int) + strlen(must)*sizeof(int). This is all inside the regex_t's "guts", which is allocated dynamically by regcomp(). If allocation of either of the two tables fail, the other one is freed. In this case, the straight-forward algorithm is used for pre-matching. Tests exercising the code path affected have shown a speed increase of 50% for "must" strings of length four or five. API and ABI remain unchanged by this commit. The patch submitted on the PR was not used, as it was non-functional. PR: 14342
2000-06-29 04:48:34 +00:00
}
Enhance the optimization provided by pre-matching. Fix style bugs with previous commits. At the time we search the pattern for the "must" string, we now compute the longest offset from the beginning of the pattern at which the must string might be found. If that offset is found to be infinite (through use of "+" or "*"), we set it to -1 to disable the heuristics applied later. After we are done with pre-matching, we use that offset and the point in the text at which the must string was found to compute the earliest point at which the pattern might be found. Special care should be taken here. The variable "start" is passed to the automata-processing functions fast() and slow() to indicate the point in the text at which they should start working from. The real beginning of the text is passed in a struct match variable m, which is used to check for anchors. That variable, though, is initialized with "start", so we must not adjust "start" before "m" is properly initialized. Simple tests showed a speed increase from 100% to 400%, but they were biased in that regexec() was called for the whole file instead of line by line, and parenthized subexpressions were not searched for. This change adds a single integer to the size of the "guts" structure, and does not change the ABI. Further improvements possible: Since the speed increase observed here is so huge, one intuitive optimization would be to introduce a bias in the function that computes the "must" string so as to prefer a smaller string with a finite offset over a larger one with an infinite offset. Tests have shown this to be a bad idea, though, as the cost of false pre-matches far outweights the benefits of a must offset, even in biased situations. A number of other improvements suggest themselves, though: * identify the cases where the pattern is identical to the must string, and avoid entering fast() and slow() in these cases. * compute the maximum offset from the must string to the end of the pattern, and use that to set the point at which fast() and slow() should give up trying to find a match, and return then return to pre-matching. * return all the way to pre-matching if a "match" was found and later invalidated by back reference processing. Since back references are evil and should be avoided anyway, this is of little use.
2000-07-02 10:58:07 +00:00
if (pp != mustfirst)
Add Boyler-Moore algorithm to pre-matching test. The BM algorithm works by scanning the pattern from right to left, and jumping as many characters as viable based on the text's mismatched character and the pattern's already matched suffix. This typically enable us to test only a fraction of the text's characters, but has a worse performance than the straight-forward method for small patterns. Because of this, the BM algorithm will only be used if the pattern size is at least 4 characters. Notice that this pre-matching is done on the largest substring of the regular expression that _must_ be present on the text for a succesful match to be possible at all. For instance, "(xyzzy|grues)" will yield a null "must" substring, and, therefore, not benefit from the BM algorithm at all. Because of the lack of intelligence of the algorithm that finds the "must" string, things like "charjump|matchjump" will also yield a null string. To optimize that, "(char|match)jump" should be used. The setup time (at regcomp()) for the BM algorithm will most likely outweight any benefits for one-time matches. Given the slow regex(3) we have, this is unlikely to be even perceptible, though. The size of a regex_t structure is increased by 2*sizeof(char*) + 256*sizeof(int) + strlen(must)*sizeof(int). This is all inside the regex_t's "guts", which is allocated dynamically by regcomp(). If allocation of either of the two tables fail, the other one is freed. In this case, the straight-forward algorithm is used for pre-matching. Tests exercising the code path affected have shown a speed increase of 50% for "must" strings of length four or five. API and ABI remain unchanged by this commit. The patch submitted on the PR was not used, as it was non-functional. PR: 14342
2000-06-29 04:48:34 +00:00
return(REG_NOMATCH);
} else {
for (dp = start; dp < stop; dp++)
if (*dp == g->must[0] &&
stop - dp >= g->mlen &&
memcmp(dp, g->must, (size_t)g->mlen) == 0)
break;
if (dp == stop) /* we didn't find g->must */
return(REG_NOMATCH);
}
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
}
/* match struct setup */
m->g = g;
m->eflags = eflags;
m->pmatch = NULL;
m->lastpos = NULL;
m->offp = string;
m->beginp = start;
m->endp = stop;
STATESETUP(m, 4);
SETUP(m->st);
SETUP(m->fresh);
SETUP(m->tmp);
SETUP(m->empty);
CLEAR(m->empty);
Make regular expression matching aware of multibyte characters. The general idea is that we perform multibyte->wide character conversion while parsing and compiling, then convert byte sequences to wide characters when they're needed for comparison and stepping through the string during execution. As with tr(1), the main complication is to efficiently represent sets of characters in bracket expressions. The old bitmap representation is replaced by a bitmap for the first 256 characters combined with a vector of individual wide characters, a vector of character ranges (for [A-Z] etc.), and a vector of character classes (for [[:alpha:]] etc.). One other point of interest is that although the Boyer-Moore algorithm had to be disabled in the general multibyte case, it is still enabled for UTF-8 because of its self-synchronizing nature. This greatly speeds up matching by reducing the number of multibyte conversions that need to be done.
2004-07-12 07:35:59 +00:00
ZAPSTATE(&m->mbs);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
Enhance the optimization provided by pre-matching. Fix style bugs with previous commits. At the time we search the pattern for the "must" string, we now compute the longest offset from the beginning of the pattern at which the must string might be found. If that offset is found to be infinite (through use of "+" or "*"), we set it to -1 to disable the heuristics applied later. After we are done with pre-matching, we use that offset and the point in the text at which the must string was found to compute the earliest point at which the pattern might be found. Special care should be taken here. The variable "start" is passed to the automata-processing functions fast() and slow() to indicate the point in the text at which they should start working from. The real beginning of the text is passed in a struct match variable m, which is used to check for anchors. That variable, though, is initialized with "start", so we must not adjust "start" before "m" is properly initialized. Simple tests showed a speed increase from 100% to 400%, but they were biased in that regexec() was called for the whole file instead of line by line, and parenthized subexpressions were not searched for. This change adds a single integer to the size of the "guts" structure, and does not change the ABI. Further improvements possible: Since the speed increase observed here is so huge, one intuitive optimization would be to introduce a bias in the function that computes the "must" string so as to prefer a smaller string with a finite offset over a larger one with an infinite offset. Tests have shown this to be a bad idea, though, as the cost of false pre-matches far outweights the benefits of a must offset, even in biased situations. A number of other improvements suggest themselves, though: * identify the cases where the pattern is identical to the must string, and avoid entering fast() and slow() in these cases. * compute the maximum offset from the must string to the end of the pattern, and use that to set the point at which fast() and slow() should give up trying to find a match, and return then return to pre-matching. * return all the way to pre-matching if a "match" was found and later invalidated by back reference processing. Since back references are evil and should be avoided anyway, this is of little use.
2000-07-02 10:58:07 +00:00
/* Adjust start according to moffset, to speed things up */
regexec: fix processing multibyte strings. Matcher function incorrectly assumed that moffset that we get from findmust is in bytes. Fix this by introducing a stepback function, taking short path if MB_CUR_MAX is 1, and going back byte-by-byte, checking if we have a legal character sequence otherwise. PR: 153502 Reviewed by: pfg, kevans Approved by: kib (mentor, implicit) Differential revision: https://reviews.freebsd.org/D18297
2018-11-23 15:49:18 +00:00
if (dp != NULL && g->moffset > -1) {
const char *nstart;
nstart = stepback(start, dp, g->moffset);
if (nstart != NULL)
start = nstart;
}
Enhance the optimization provided by pre-matching. Fix style bugs with previous commits. At the time we search the pattern for the "must" string, we now compute the longest offset from the beginning of the pattern at which the must string might be found. If that offset is found to be infinite (through use of "+" or "*"), we set it to -1 to disable the heuristics applied later. After we are done with pre-matching, we use that offset and the point in the text at which the must string was found to compute the earliest point at which the pattern might be found. Special care should be taken here. The variable "start" is passed to the automata-processing functions fast() and slow() to indicate the point in the text at which they should start working from. The real beginning of the text is passed in a struct match variable m, which is used to check for anchors. That variable, though, is initialized with "start", so we must not adjust "start" before "m" is properly initialized. Simple tests showed a speed increase from 100% to 400%, but they were biased in that regexec() was called for the whole file instead of line by line, and parenthized subexpressions were not searched for. This change adds a single integer to the size of the "guts" structure, and does not change the ABI. Further improvements possible: Since the speed increase observed here is so huge, one intuitive optimization would be to introduce a bias in the function that computes the "must" string so as to prefer a smaller string with a finite offset over a larger one with an infinite offset. Tests have shown this to be a bad idea, though, as the cost of false pre-matches far outweights the benefits of a must offset, even in biased situations. A number of other improvements suggest themselves, though: * identify the cases where the pattern is identical to the must string, and avoid entering fast() and slow() in these cases. * compute the maximum offset from the must string to the end of the pattern, and use that to set the point at which fast() and slow() should give up trying to find a match, and return then return to pre-matching. * return all the way to pre-matching if a "match" was found and later invalidated by back reference processing. Since back references are evil and should be avoided anyway, this is of little use.
2000-07-02 10:58:07 +00:00
Add a couple of debugging statements.
2009-09-16 06:29:23 +00:00
SP("mloop", m->st, *start);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
/* this loop does only one repetition except for backrefs */
for (;;) {
regex(3): Refactor fast/slow stepping bits in the matching engine Adding features for matching is fairly straightforward, but this requires some duplication because of this fast/slow setup. They can be fairly trivially combined into a single walk(), so do it to make future additions less error prone. Reviewed by: cem (earlier version), emaste, pfg Approved by: emaste (mentor) Differential Revision: https://reviews.freebsd.org/D11233
2017-08-09 01:04:36 +00:00
endp = walk(m, start, stop, gf, gl, true);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
if (endp == NULL) { /* a miss */
Plug memory leak. PR: bin/75656 MFC after: 2 weeks
2004-12-30 14:27:57 +00:00
if (m->pmatch != NULL)
free((char *)m->pmatch);
if (m->lastpos != NULL)
free((char *)m->lastpos);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
STATETEARDOWN(m);
return(REG_NOMATCH);
}
if (nmatch == 0 && !g->backrefs)
break; /* no further info needed */
/* where? */
assert(m->coldp != NULL);
for (;;) {
NOTE("finding start");
regex(3): Refactor fast/slow stepping bits in the matching engine Adding features for matching is fairly straightforward, but this requires some duplication because of this fast/slow setup. They can be fairly trivially combined into a single walk(), so do it to make future additions less error prone. Reviewed by: cem (earlier version), emaste, pfg Approved by: emaste (mentor) Differential Revision: https://reviews.freebsd.org/D11233
2017-08-09 01:04:36 +00:00
endp = walk(m, m->coldp, stop, gf, gl, false);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
if (endp != NULL)
break;
assert(m->coldp < m->endp);
Make regular expression matching aware of multibyte characters. The general idea is that we perform multibyte->wide character conversion while parsing and compiling, then convert byte sequences to wide characters when they're needed for comparison and stepping through the string during execution. As with tr(1), the main complication is to efficiently represent sets of characters in bracket expressions. The old bitmap representation is replaced by a bitmap for the first 256 characters combined with a vector of individual wide characters, a vector of character ranges (for [A-Z] etc.), and a vector of character classes (for [[:alpha:]] etc.). One other point of interest is that although the Boyer-Moore algorithm had to be disabled in the general multibyte case, it is still enabled for UTF-8 because of its self-synchronizing nature. This greatly speeds up matching by reducing the number of multibyte conversions that need to be done.
2004-07-12 07:35:59 +00:00
m->coldp += XMBRTOWC(NULL, m->coldp,
m->endp - m->coldp, &m->mbs, 0);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
}
if (nmatch == 1 && !g->backrefs)
break; /* no further info needed */
/* oh my, he wants the subexpressions... */
if (m->pmatch == NULL)
m->pmatch = (regmatch_t *)malloc((m->g->nsub + 1) *
sizeof(regmatch_t));
if (m->pmatch == NULL) {
STATETEARDOWN(m);
return(REG_ESPACE);
}
for (i = 1; i <= m->g->nsub; i++)
m->pmatch[i].rm_so = m->pmatch[i].rm_eo = -1;
if (!g->backrefs && !(m->eflags&REG_BACKR)) {
NOTE("dissecting");
dp = dissect(m, m->coldp, endp, gf, gl);
} else {
if (g->nplus > 0 && m->lastpos == NULL)
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
m->lastpos = malloc((g->nplus+1) *
sizeof(const char *));
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
if (g->nplus > 0 && m->lastpos == NULL) {
free(m->pmatch);
STATETEARDOWN(m);
return(REG_ESPACE);
}
NOTE("backref dissect");
Only stop evaluation of a back reference if the match length is zero and the recursion level is too deep. Obtained from: OpenBSD
2007-03-05 09:43:55 +00:00
dp = backref(m, m->coldp, endp, gf, gl, (sopno)0, 0);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
}
if (dp != NULL)
break;
/* uh-oh... we couldn't find a subexpression-level match */
assert(g->backrefs); /* must be back references doing it */
assert(g->nplus == 0 || m->lastpos != NULL);
for (;;) {
if (dp != NULL || endp <= m->coldp)
break; /* defeat */
NOTE("backoff");
regex(3): Refactor fast/slow stepping bits in the matching engine Adding features for matching is fairly straightforward, but this requires some duplication because of this fast/slow setup. They can be fairly trivially combined into a single walk(), so do it to make future additions less error prone. Reviewed by: cem (earlier version), emaste, pfg Approved by: emaste (mentor) Differential Revision: https://reviews.freebsd.org/D11233
2017-08-09 01:04:36 +00:00
endp = walk(m, m->coldp, endp-1, gf, gl, false);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
if (endp == NULL)
break; /* defeat */
/* try it on a shorter possibility */
#ifndef NDEBUG
for (i = 1; i <= m->g->nsub; i++) {
assert(m->pmatch[i].rm_so == -1);
assert(m->pmatch[i].rm_eo == -1);
}
#endif
NOTE("backoff dissect");
Only stop evaluation of a back reference if the match length is zero and the recursion level is too deep. Obtained from: OpenBSD
2007-03-05 09:43:55 +00:00
dp = backref(m, m->coldp, endp, gf, gl, (sopno)0, 0);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
}
assert(dp == NULL || dp == endp);
if (dp != NULL) /* found a shorter one */
break;
/* despite initial appearances, there is no match here */
NOTE("false alarm");
Make regular expression matching aware of multibyte characters. The general idea is that we perform multibyte->wide character conversion while parsing and compiling, then convert byte sequences to wide characters when they're needed for comparison and stepping through the string during execution. As with tr(1), the main complication is to efficiently represent sets of characters in bracket expressions. The old bitmap representation is replaced by a bitmap for the first 256 characters combined with a vector of individual wide characters, a vector of character ranges (for [A-Z] etc.), and a vector of character classes (for [[:alpha:]] etc.). One other point of interest is that although the Boyer-Moore algorithm had to be disabled in the general multibyte case, it is still enabled for UTF-8 because of its self-synchronizing nature. This greatly speeds up matching by reducing the number of multibyte conversions that need to be done.
2004-07-12 07:35:59 +00:00
/* recycle starting later */
start = m->coldp + XMBRTOWC(NULL, m->coldp,
Fix computation of the 'n' argument to mbrtowc (through XMBRTOWC) to avoid reading past 'stop' in various places when converting multibyte characters. Reading too far caused truncation to not be detected when it should have been, eventually causing regexec() to loop infinitely in with certain combinations of patterns and strings in multibyte locales. PR: 74020 MFC after: 4 weeks
2004-11-21 03:14:32 +00:00
stop - m->coldp, &m->mbs, 0);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
assert(start <= stop);
}
/* fill in the details if requested */
if (nmatch > 0) {
pmatch[0].rm_so = m->coldp - m->offp;
pmatch[0].rm_eo = endp - m->offp;
}
if (nmatch > 1) {
assert(m->pmatch != NULL);
for (i = 1; i < nmatch; i++)
if (i <= m->g->nsub)
pmatch[i] = m->pmatch[i];
else {
pmatch[i].rm_so = -1;
pmatch[i].rm_eo = -1;
}
}
if (m->pmatch != NULL)
free((char *)m->pmatch);
if (m->lastpos != NULL)
free((char *)m->lastpos);
STATETEARDOWN(m);
return(0);
}
/*
- dissect - figure out what matched what, no back references
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
== static const char *dissect(struct match *m, const char *start, \
== const char *stop, sopno startst, sopno stopst);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
*/
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
static const char * /* == stop (success) always */
dissect(struct match *m,
const char *start,
const char *stop,
sopno startst,
sopno stopst)
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
{
Remove 'register' keyword.
2002-03-21 18:49:23 +00:00
int i;
sopno ss; /* start sop of current subRE */
sopno es; /* end sop of current subRE */
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
const char *sp; /* start of string matched by it */
const char *stp; /* string matched by it cannot pass here */
const char *rest; /* start of rest of string */
const char *tail; /* string unmatched by rest of RE */
Remove 'register' keyword.
2002-03-21 18:49:23 +00:00
sopno ssub; /* start sop of subsubRE */
sopno esub; /* end sop of subsubRE */
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
const char *ssp; /* start of string matched by subsubRE */
const char *sep; /* end of string matched by subsubRE */
const char *oldssp; /* previous ssp */
regex/engine.c: error: variable 'dp' set but not used The issue found with gcc6 build (originally on illumos, confirmed on FreeBSD). Mark it __unused. Differential Revision: https://reviews.freebsd.org/D13109
2018-07-14 09:29:45 +00:00
const char *dp __unused;
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
AT("diss", start, stop, startst, stopst);
sp = start;
for (ss = startst; ss < stopst; ss = es) {
/* identify end of subRE */
es = ss;
switch (OP(m->g->strip[es])) {
case OPLUS_:
case OQUEST_:
es += OPND(m->g->strip[es]);
break;
case OCH_:
regex(3): Resolve issues with higher WARNS levels libc is set for WARNS=2, but the incoming libregex will use WARNS=6. Sprinkle some casts and (void)bc's to alleviate the warnings that come along with the higher WARNS level. These 'bc' parameters could be outright removed, but as of right now they will be used in some parts of libregex land. Silence the warnings instead rather than flip-flopping.
2018-01-21 04:57:29 +00:00
while (OP(m->g->strip[es]) != (sop)O_CH)
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
es += OPND(m->g->strip[es]);
break;
}
es++;
/* figure out what it matched */
switch (OP(m->g->strip[ss])) {
case OEND:
assert(nope);
break;
case OCHAR:
Make regular expression matching aware of multibyte characters. The general idea is that we perform multibyte->wide character conversion while parsing and compiling, then convert byte sequences to wide characters when they're needed for comparison and stepping through the string during execution. As with tr(1), the main complication is to efficiently represent sets of characters in bracket expressions. The old bitmap representation is replaced by a bitmap for the first 256 characters combined with a vector of individual wide characters, a vector of character ranges (for [A-Z] etc.), and a vector of character classes (for [[:alpha:]] etc.). One other point of interest is that although the Boyer-Moore algorithm had to be disabled in the general multibyte case, it is still enabled for UTF-8 because of its self-synchronizing nature. This greatly speeds up matching by reducing the number of multibyte conversions that need to be done.
2004-07-12 07:35:59 +00:00
sp += XMBRTOWC(NULL, sp, stop - start, &m->mbs, 0);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
break;
case OBOL:
case OEOL:
case OBOW:
case OEOW:
break;
case OANY:
case OANYOF:
Make regular expression matching aware of multibyte characters. The general idea is that we perform multibyte->wide character conversion while parsing and compiling, then convert byte sequences to wide characters when they're needed for comparison and stepping through the string during execution. As with tr(1), the main complication is to efficiently represent sets of characters in bracket expressions. The old bitmap representation is replaced by a bitmap for the first 256 characters combined with a vector of individual wide characters, a vector of character ranges (for [A-Z] etc.), and a vector of character classes (for [[:alpha:]] etc.). One other point of interest is that although the Boyer-Moore algorithm had to be disabled in the general multibyte case, it is still enabled for UTF-8 because of its self-synchronizing nature. This greatly speeds up matching by reducing the number of multibyte conversions that need to be done.
2004-07-12 07:35:59 +00:00
sp += XMBRTOWC(NULL, sp, stop - start, &m->mbs, 0);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
break;
case OBACK_:
case O_BACK:
assert(nope);
break;
/* cases where length of match is hard to find */
case OQUEST_:
stp = stop;
for (;;) {
/* how long could this one be? */
regex(3): Refactor fast/slow stepping bits in the matching engine Adding features for matching is fairly straightforward, but this requires some duplication because of this fast/slow setup. They can be fairly trivially combined into a single walk(), so do it to make future additions less error prone. Reviewed by: cem (earlier version), emaste, pfg Approved by: emaste (mentor) Differential Revision: https://reviews.freebsd.org/D11233
2017-08-09 01:04:36 +00:00
rest = walk(m, sp, stp, ss, es, false);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
assert(rest != NULL); /* it did match */
/* could the rest match the rest? */
regex(3): Refactor fast/slow stepping bits in the matching engine Adding features for matching is fairly straightforward, but this requires some duplication because of this fast/slow setup. They can be fairly trivially combined into a single walk(), so do it to make future additions less error prone. Reviewed by: cem (earlier version), emaste, pfg Approved by: emaste (mentor) Differential Revision: https://reviews.freebsd.org/D11233
2017-08-09 01:04:36 +00:00
tail = walk(m, rest, stop, es, stopst, false);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
if (tail == stop)
break; /* yes! */
/* no -- try a shorter match for this one */
stp = rest - 1;
assert(stp >= sp); /* it did work */
}
ssub = ss + 1;
esub = es - 1;
/* did innards match? */
regex(3): Refactor fast/slow stepping bits in the matching engine Adding features for matching is fairly straightforward, but this requires some duplication because of this fast/slow setup. They can be fairly trivially combined into a single walk(), so do it to make future additions less error prone. Reviewed by: cem (earlier version), emaste, pfg Approved by: emaste (mentor) Differential Revision: https://reviews.freebsd.org/D11233
2017-08-09 01:04:36 +00:00
if (walk(m, sp, rest, ssub, esub, false) != NULL) {
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
dp = dissect(m, sp, rest, ssub, esub);
assert(dp == rest);
} else /* no */
assert(sp == rest);
sp = rest;
break;
case OPLUS_:
stp = stop;
for (;;) {
/* how long could this one be? */
regex(3): Refactor fast/slow stepping bits in the matching engine Adding features for matching is fairly straightforward, but this requires some duplication because of this fast/slow setup. They can be fairly trivially combined into a single walk(), so do it to make future additions less error prone. Reviewed by: cem (earlier version), emaste, pfg Approved by: emaste (mentor) Differential Revision: https://reviews.freebsd.org/D11233
2017-08-09 01:04:36 +00:00
rest = walk(m, sp, stp, ss, es, false);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
assert(rest != NULL); /* it did match */
/* could the rest match the rest? */
regex(3): Refactor fast/slow stepping bits in the matching engine Adding features for matching is fairly straightforward, but this requires some duplication because of this fast/slow setup. They can be fairly trivially combined into a single walk(), so do it to make future additions less error prone. Reviewed by: cem (earlier version), emaste, pfg Approved by: emaste (mentor) Differential Revision: https://reviews.freebsd.org/D11233
2017-08-09 01:04:36 +00:00
tail = walk(m, rest, stop, es, stopst, false);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
if (tail == stop)
break; /* yes! */
/* no -- try a shorter match for this one */
stp = rest - 1;
assert(stp >= sp); /* it did work */
}
ssub = ss + 1;
esub = es - 1;
ssp = sp;
oldssp = ssp;
for (;;) { /* find last match of innards */
regex(3): Refactor fast/slow stepping bits in the matching engine Adding features for matching is fairly straightforward, but this requires some duplication because of this fast/slow setup. They can be fairly trivially combined into a single walk(), so do it to make future additions less error prone. Reviewed by: cem (earlier version), emaste, pfg Approved by: emaste (mentor) Differential Revision: https://reviews.freebsd.org/D11233
2017-08-09 01:04:36 +00:00
sep = walk(m, ssp, rest, ssub, esub, false);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
if (sep == NULL || sep == ssp)
break; /* failed or matched null */
oldssp = ssp; /* on to next try */
ssp = sep;
}
if (sep == NULL) {
/* last successful match */
sep = ssp;
ssp = oldssp;
}
assert(sep == rest); /* must exhaust substring */
regex(3): Refactor fast/slow stepping bits in the matching engine Adding features for matching is fairly straightforward, but this requires some duplication because of this fast/slow setup. They can be fairly trivially combined into a single walk(), so do it to make future additions less error prone. Reviewed by: cem (earlier version), emaste, pfg Approved by: emaste (mentor) Differential Revision: https://reviews.freebsd.org/D11233
2017-08-09 01:04:36 +00:00
assert(walk(m, ssp, sep, ssub, esub, false) == rest);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
dp = dissect(m, ssp, sep, ssub, esub);
assert(dp == sep);
sp = rest;
break;
case OCH_:
stp = stop;
for (;;) {
/* how long could this one be? */
regex(3): Refactor fast/slow stepping bits in the matching engine Adding features for matching is fairly straightforward, but this requires some duplication because of this fast/slow setup. They can be fairly trivially combined into a single walk(), so do it to make future additions less error prone. Reviewed by: cem (earlier version), emaste, pfg Approved by: emaste (mentor) Differential Revision: https://reviews.freebsd.org/D11233
2017-08-09 01:04:36 +00:00
rest = walk(m, sp, stp, ss, es, false);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
assert(rest != NULL); /* it did match */
/* could the rest match the rest? */
regex(3): Refactor fast/slow stepping bits in the matching engine Adding features for matching is fairly straightforward, but this requires some duplication because of this fast/slow setup. They can be fairly trivially combined into a single walk(), so do it to make future additions less error prone. Reviewed by: cem (earlier version), emaste, pfg Approved by: emaste (mentor) Differential Revision: https://reviews.freebsd.org/D11233
2017-08-09 01:04:36 +00:00
tail = walk(m, rest, stop, es, stopst, false);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
if (tail == stop)
break; /* yes! */
/* no -- try a shorter match for this one */
stp = rest - 1;
assert(stp >= sp); /* it did work */
}
ssub = ss + 1;
esub = ss + OPND(m->g->strip[ss]) - 1;
assert(OP(m->g->strip[esub]) == OOR1);
for (;;) { /* find first matching branch */
regex(3): Refactor fast/slow stepping bits in the matching engine Adding features for matching is fairly straightforward, but this requires some duplication because of this fast/slow setup. They can be fairly trivially combined into a single walk(), so do it to make future additions less error prone. Reviewed by: cem (earlier version), emaste, pfg Approved by: emaste (mentor) Differential Revision: https://reviews.freebsd.org/D11233
2017-08-09 01:04:36 +00:00
if (walk(m, sp, rest, ssub, esub, false) == rest)
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
break; /* it matched all of it */
/* that one missed, try next one */
assert(OP(m->g->strip[esub]) == OOR1);
esub++;
assert(OP(m->g->strip[esub]) == OOR2);
ssub = esub + 1;
esub += OPND(m->g->strip[esub]);
regex(3): Resolve issues with higher WARNS levels libc is set for WARNS=2, but the incoming libregex will use WARNS=6. Sprinkle some casts and (void)bc's to alleviate the warnings that come along with the higher WARNS level. These 'bc' parameters could be outright removed, but as of right now they will be used in some parts of libregex land. Silence the warnings instead rather than flip-flopping.
2018-01-21 04:57:29 +00:00
if (OP(m->g->strip[esub]) == (sop)OOR2)
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
esub--;
else
assert(OP(m->g->strip[esub]) == O_CH);
}
dp = dissect(m, sp, rest, ssub, esub);
assert(dp == rest);
sp = rest;
break;
case O_PLUS:
case O_QUEST:
case OOR1:
case OOR2:
case O_CH:
assert(nope);
break;
case OLPAREN:
i = OPND(m->g->strip[ss]);
assert(0 < i && i <= m->g->nsub);
m->pmatch[i].rm_so = sp - m->offp;
break;
case ORPAREN:
i = OPND(m->g->strip[ss]);
assert(0 < i && i <= m->g->nsub);
m->pmatch[i].rm_eo = sp - m->offp;
break;
default: /* uh oh */
assert(nope);
break;
}
}
assert(sp == stop);
return(sp);
}
/*
- backref - figure out what matched what, figuring in back references
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
== static const char *backref(struct match *m, const char *start, \
== const char *stop, sopno startst, sopno stopst, sopno lev);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
*/
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
static const char * /* == stop (success) or NULL (failure) */
backref(struct match *m,
const char *start,
const char *stop,
sopno startst,
sopno stopst,
sopno lev, /* PLUS nesting level */
int rec)
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
{
Remove 'register' keyword.
2002-03-21 18:49:23 +00:00
int i;
sopno ss; /* start sop of current subRE */
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
const char *sp; /* start of string matched by it */
Remove 'register' keyword.
2002-03-21 18:49:23 +00:00
sopno ssub; /* start sop of subsubRE */
sopno esub; /* end sop of subsubRE */
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
const char *ssp; /* start of string matched by subsubRE */
const char *dp;
Remove 'register' keyword.
2002-03-21 18:49:23 +00:00
size_t len;
int hard;
sop s;
regoff_t offsave;
cset *cs;
Make regular expression matching aware of multibyte characters. The general idea is that we perform multibyte->wide character conversion while parsing and compiling, then convert byte sequences to wide characters when they're needed for comparison and stepping through the string during execution. As with tr(1), the main complication is to efficiently represent sets of characters in bracket expressions. The old bitmap representation is replaced by a bitmap for the first 256 characters combined with a vector of individual wide characters, a vector of character ranges (for [A-Z] etc.), and a vector of character classes (for [[:alpha:]] etc.). One other point of interest is that although the Boyer-Moore algorithm had to be disabled in the general multibyte case, it is still enabled for UTF-8 because of its self-synchronizing nature. This greatly speeds up matching by reducing the number of multibyte conversions that need to be done.
2004-07-12 07:35:59 +00:00
wint_t wc;
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
AT("back", start, stop, startst, stopst);
sp = start;
/* get as far as we can with easy stuff */
hard = 0;
for (ss = startst; !hard && ss < stopst; ss++)
switch (OP(s = m->g->strip[ss])) {
case OCHAR:
Make regular expression matching aware of multibyte characters. The general idea is that we perform multibyte->wide character conversion while parsing and compiling, then convert byte sequences to wide characters when they're needed for comparison and stepping through the string during execution. As with tr(1), the main complication is to efficiently represent sets of characters in bracket expressions. The old bitmap representation is replaced by a bitmap for the first 256 characters combined with a vector of individual wide characters, a vector of character ranges (for [A-Z] etc.), and a vector of character classes (for [[:alpha:]] etc.). One other point of interest is that although the Boyer-Moore algorithm had to be disabled in the general multibyte case, it is still enabled for UTF-8 because of its self-synchronizing nature. This greatly speeds up matching by reducing the number of multibyte conversions that need to be done.
2004-07-12 07:35:59 +00:00
if (sp == stop)
return(NULL);
sp += XMBRTOWC(&wc, sp, stop - sp, &m->mbs, BADCHAR);
if (wc != OPND(s))
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
return(NULL);
break;
case OANY:
if (sp == stop)
return(NULL);
Make regular expression matching aware of multibyte characters. The general idea is that we perform multibyte->wide character conversion while parsing and compiling, then convert byte sequences to wide characters when they're needed for comparison and stepping through the string during execution. As with tr(1), the main complication is to efficiently represent sets of characters in bracket expressions. The old bitmap representation is replaced by a bitmap for the first 256 characters combined with a vector of individual wide characters, a vector of character ranges (for [A-Z] etc.), and a vector of character classes (for [[:alpha:]] etc.). One other point of interest is that although the Boyer-Moore algorithm had to be disabled in the general multibyte case, it is still enabled for UTF-8 because of its self-synchronizing nature. This greatly speeds up matching by reducing the number of multibyte conversions that need to be done.
2004-07-12 07:35:59 +00:00
sp += XMBRTOWC(&wc, sp, stop - sp, &m->mbs, BADCHAR);
if (wc == BADCHAR)
return (NULL);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
break;
case OANYOF:
Make regular expression matching aware of multibyte characters. The general idea is that we perform multibyte->wide character conversion while parsing and compiling, then convert byte sequences to wide characters when they're needed for comparison and stepping through the string during execution. As with tr(1), the main complication is to efficiently represent sets of characters in bracket expressions. The old bitmap representation is replaced by a bitmap for the first 256 characters combined with a vector of individual wide characters, a vector of character ranges (for [A-Z] etc.), and a vector of character classes (for [[:alpha:]] etc.). One other point of interest is that although the Boyer-Moore algorithm had to be disabled in the general multibyte case, it is still enabled for UTF-8 because of its self-synchronizing nature. This greatly speeds up matching by reducing the number of multibyte conversions that need to be done.
2004-07-12 07:35:59 +00:00
if (sp == stop)
return (NULL);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
cs = &m->g->sets[OPND(s)];
Make regular expression matching aware of multibyte characters. The general idea is that we perform multibyte->wide character conversion while parsing and compiling, then convert byte sequences to wide characters when they're needed for comparison and stepping through the string during execution. As with tr(1), the main complication is to efficiently represent sets of characters in bracket expressions. The old bitmap representation is replaced by a bitmap for the first 256 characters combined with a vector of individual wide characters, a vector of character ranges (for [A-Z] etc.), and a vector of character classes (for [[:alpha:]] etc.). One other point of interest is that although the Boyer-Moore algorithm had to be disabled in the general multibyte case, it is still enabled for UTF-8 because of its self-synchronizing nature. This greatly speeds up matching by reducing the number of multibyte conversions that need to be done.
2004-07-12 07:35:59 +00:00
sp += XMBRTOWC(&wc, sp, stop - sp, &m->mbs, BADCHAR);
if (wc == BADCHAR || !CHIN(cs, wc))
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
return(NULL);
break;
case OBOL:
libc/regex: fix two buffer underruns. Fix some rather complex regex issues found on OpenBSD as part of some ongoing work to fix a sed(1) bug. Curiously the OpenBSD tests don't trigger segfaults on FreeBSD but the bugs were confirmed by running a port of FreeBSD's regex under OpenBSD's malloc. Huge thanks to Ingo for confirming the behavior. Taken from: Ingo Schwarze (through openbsd-tech 2016-05-15) MFC after: 1 week
2016-05-21 19:54:10 +00:00
if ((sp == m->beginp && !(m->eflags&REG_NOTBOL)) ||
(sp > m->offp && sp < m->endp &&
*(sp-1) == '\n' && (m->g->cflags&REG_NEWLINE)))
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
{ /* yes */ }
else
return(NULL);
break;
case OEOL:
if ( (sp == m->endp && !(m->eflags&REG_NOTEOL)) ||
(sp < m->endp && *sp == '\n' &&
(m->g->cflags&REG_NEWLINE)) )
{ /* yes */ }
else
return(NULL);
break;
case OBOW:
libc/regex: fix two buffer underruns. Fix some rather complex regex issues found on OpenBSD as part of some ongoing work to fix a sed(1) bug. Curiously the OpenBSD tests don't trigger segfaults on FreeBSD but the bugs were confirmed by running a port of FreeBSD's regex under OpenBSD's malloc. Huge thanks to Ingo for confirming the behavior. Taken from: Ingo Schwarze (through openbsd-tech 2016-05-15) MFC after: 1 week
2016-05-21 19:54:10 +00:00
if (sp < m->endp && ISWORD(*sp) &&
((sp == m->beginp && !(m->eflags&REG_NOTBOL)) ||
(sp > m->offp && !ISWORD(*(sp-1)))))
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
{ /* yes */ }
else
return(NULL);
break;
case OEOW:
if (( (sp == m->endp && !(m->eflags&REG_NOTEOL)) ||
(sp < m->endp && *sp == '\n' &&
(m->g->cflags&REG_NEWLINE)) ||
(sp < m->endp && !ISWORD(*sp)) ) &&
(sp > m->beginp && ISWORD(*(sp-1))) )
{ /* yes */ }
else
return(NULL);
break;
case O_QUEST:
break;
case OOR1: /* matches null but needs to skip */
ss++;
s = m->g->strip[ss];
do {
assert(OP(s) == OOR2);
ss += OPND(s);
regex(3): Resolve issues with higher WARNS levels libc is set for WARNS=2, but the incoming libregex will use WARNS=6. Sprinkle some casts and (void)bc's to alleviate the warnings that come along with the higher WARNS level. These 'bc' parameters could be outright removed, but as of right now they will be used in some parts of libregex land. Silence the warnings instead rather than flip-flopping.
2018-01-21 04:57:29 +00:00
} while (OP(s = m->g->strip[ss]) != (sop)O_CH);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
/* note that the ss++ gets us past the O_CH */
break;
default: /* have to make a choice */
hard = 1;
break;
}
if (!hard) { /* that was it! */
if (sp != stop)
return(NULL);
return(sp);
}
ss--; /* adjust for the for's final increment */
/* the hard stuff */
AT("hard", sp, stop, ss, stopst);
s = m->g->strip[ss];
switch (OP(s)) {
case OBACK_: /* the vilest depths */
i = OPND(s);
assert(0 < i && i <= m->g->nsub);
if (m->pmatch[i].rm_eo == -1)
return(NULL);
assert(m->pmatch[i].rm_so != -1);
len = m->pmatch[i].rm_eo - m->pmatch[i].rm_so;
Only stop evaluation of a back reference if the match length is zero and the recursion level is too deep. Obtained from: OpenBSD
2007-03-05 09:43:55 +00:00
if (len == 0 && rec++ > MAX_RECURSION)
Avoid infinite recursion on: echo "foo foo bar bar bar baz" | sed 's/\([^ ]*\)\( *\1\)*/\1/g' Obtained from: OpenBSD via NetBSD (rev. 1.18)
2007-03-05 03:07:36 +00:00
return(NULL);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
assert(stop - m->beginp >= len);
if (sp > stop - len)
return(NULL); /* not enough left to match */
ssp = m->offp + m->pmatch[i].rm_so;
if (memcmp(sp, ssp, len) != 0)
return(NULL);
regex(3): Resolve issues with higher WARNS levels libc is set for WARNS=2, but the incoming libregex will use WARNS=6. Sprinkle some casts and (void)bc's to alleviate the warnings that come along with the higher WARNS level. These 'bc' parameters could be outright removed, but as of right now they will be used in some parts of libregex land. Silence the warnings instead rather than flip-flopping.
2018-01-21 04:57:29 +00:00
while (m->g->strip[ss] != (sop)SOP(O_BACK, i))
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
ss++;
Only stop evaluation of a back reference if the match length is zero and the recursion level is too deep. Obtained from: OpenBSD
2007-03-05 09:43:55 +00:00
return(backref(m, sp+len, stop, ss+1, stopst, lev, rec));
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
case OQUEST_: /* to null or not */
Only stop evaluation of a back reference if the match length is zero and the recursion level is too deep. Obtained from: OpenBSD
2007-03-05 09:43:55 +00:00
dp = backref(m, sp, stop, ss+1, stopst, lev, rec);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
if (dp != NULL)
return(dp); /* not */
Only stop evaluation of a back reference if the match length is zero and the recursion level is too deep. Obtained from: OpenBSD
2007-03-05 09:43:55 +00:00
return(backref(m, sp, stop, ss+OPND(s)+1, stopst, lev, rec));
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
case OPLUS_:
assert(m->lastpos != NULL);
assert(lev+1 <= m->g->nplus);
m->lastpos[lev+1] = sp;
Only stop evaluation of a back reference if the match length is zero and the recursion level is too deep. Obtained from: OpenBSD
2007-03-05 09:43:55 +00:00
return(backref(m, sp, stop, ss+1, stopst, lev+1, rec));
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
case O_PLUS:
if (sp == m->lastpos[lev]) /* last pass matched null */
Only stop evaluation of a back reference if the match length is zero and the recursion level is too deep. Obtained from: OpenBSD
2007-03-05 09:43:55 +00:00
return(backref(m, sp, stop, ss+1, stopst, lev-1, rec));
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
/* try another pass */
m->lastpos[lev] = sp;
Only stop evaluation of a back reference if the match length is zero and the recursion level is too deep. Obtained from: OpenBSD
2007-03-05 09:43:55 +00:00
dp = backref(m, sp, stop, ss-OPND(s)+1, stopst, lev, rec);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
if (dp == NULL)
Only stop evaluation of a back reference if the match length is zero and the recursion level is too deep. Obtained from: OpenBSD
2007-03-05 09:43:55 +00:00
return(backref(m, sp, stop, ss+1, stopst, lev-1, rec));
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
else
return(dp);
case OCH_: /* find the right one, if any */
ssub = ss + 1;
esub = ss + OPND(s) - 1;
assert(OP(m->g->strip[esub]) == OOR1);
for (;;) { /* find first matching branch */
Only stop evaluation of a back reference if the match length is zero and the recursion level is too deep. Obtained from: OpenBSD
2007-03-05 09:43:55 +00:00
dp = backref(m, sp, stop, ssub, esub, lev, rec);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
if (dp != NULL)
return(dp);
/* that one missed, try next one */
regex(3): Resolve issues with higher WARNS levels libc is set for WARNS=2, but the incoming libregex will use WARNS=6. Sprinkle some casts and (void)bc's to alleviate the warnings that come along with the higher WARNS level. These 'bc' parameters could be outright removed, but as of right now they will be used in some parts of libregex land. Silence the warnings instead rather than flip-flopping.
2018-01-21 04:57:29 +00:00
if (OP(m->g->strip[esub]) == (sop)O_CH)
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
return(NULL); /* there is none */
esub++;
regex(3): Resolve issues with higher WARNS levels libc is set for WARNS=2, but the incoming libregex will use WARNS=6. Sprinkle some casts and (void)bc's to alleviate the warnings that come along with the higher WARNS level. These 'bc' parameters could be outright removed, but as of right now they will be used in some parts of libregex land. Silence the warnings instead rather than flip-flopping.
2018-01-21 04:57:29 +00:00
assert(OP(m->g->strip[esub]) == (sop)OOR2);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
ssub = esub + 1;
esub += OPND(m->g->strip[esub]);
regex(3): Resolve issues with higher WARNS levels libc is set for WARNS=2, but the incoming libregex will use WARNS=6. Sprinkle some casts and (void)bc's to alleviate the warnings that come along with the higher WARNS level. These 'bc' parameters could be outright removed, but as of right now they will be used in some parts of libregex land. Silence the warnings instead rather than flip-flopping.
2018-01-21 04:57:29 +00:00
if (OP(m->g->strip[esub]) == (sop)OOR2)
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
esub--;
else
assert(OP(m->g->strip[esub]) == O_CH);
}
regex: Remove some unreachable breaks. This is based on a much bigger cleanup done in Illumos. Reference: https://www.illumos.org/issues/2077 MFC after: 1 week
2014-05-01 23:34:14 +00:00
/* NOTREACHED */
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
break;
case OLPAREN: /* must undo assignment if rest fails */
i = OPND(s);
assert(0 < i && i <= m->g->nsub);
offsave = m->pmatch[i].rm_so;
m->pmatch[i].rm_so = sp - m->offp;
Only stop evaluation of a back reference if the match length is zero and the recursion level is too deep. Obtained from: OpenBSD
2007-03-05 09:43:55 +00:00
dp = backref(m, sp, stop, ss+1, stopst, lev, rec);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
if (dp != NULL)
return(dp);
m->pmatch[i].rm_so = offsave;
return(NULL);
case ORPAREN: /* must undo assignment if rest fails */
i = OPND(s);
assert(0 < i && i <= m->g->nsub);
offsave = m->pmatch[i].rm_eo;
m->pmatch[i].rm_eo = sp - m->offp;
Only stop evaluation of a back reference if the match length is zero and the recursion level is too deep. Obtained from: OpenBSD
2007-03-05 09:43:55 +00:00
dp = backref(m, sp, stop, ss+1, stopst, lev, rec);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
if (dp != NULL)
return(dp);
m->pmatch[i].rm_eo = offsave;
return(NULL);
default: /* uh oh */
assert(nope);
break;
}
/* "can't happen" */
assert(nope);
/* NOTREACHED */
More cleanup. Uhm, I also forgot: I took "EXTRA_SANITY" out of malloc.c
1995-10-22 14:40:55 +00:00
return "shut up gcc";
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
}
/*
regex(3): Refactor fast/slow stepping bits in the matching engine Adding features for matching is fairly straightforward, but this requires some duplication because of this fast/slow setup. They can be fairly trivially combined into a single walk(), so do it to make future additions less error prone. Reviewed by: cem (earlier version), emaste, pfg Approved by: emaste (mentor) Differential Revision: https://reviews.freebsd.org/D11233
2017-08-09 01:04:36 +00:00
- walk - step through the string either quickly or slowly
== static const char *walk(struct match *m, const char *start, \
== const char *stop, sopno startst, sopno stopst, bool fast);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
*/
regex(3): Refactor fast/slow stepping bits in the matching engine Adding features for matching is fairly straightforward, but this requires some duplication because of this fast/slow setup. They can be fairly trivially combined into a single walk(), so do it to make future additions less error prone. Reviewed by: cem (earlier version), emaste, pfg Approved by: emaste (mentor) Differential Revision: https://reviews.freebsd.org/D11233
2017-08-09 01:04:36 +00:00
static const char * /* where it ended, or NULL */
walk(struct match *m, const char *start, const char *stop, sopno startst,
sopno stopst, bool fast)
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
{
Remove 'register' keyword.
2002-03-21 18:49:23 +00:00
states st = m->st;
states fresh = m->fresh;
states empty = m->empty;
states tmp = m->tmp;
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
const char *p = start;
Make regular expression matching aware of multibyte characters. The general idea is that we perform multibyte->wide character conversion while parsing and compiling, then convert byte sequences to wide characters when they're needed for comparison and stepping through the string during execution. As with tr(1), the main complication is to efficiently represent sets of characters in bracket expressions. The old bitmap representation is replaced by a bitmap for the first 256 characters combined with a vector of individual wide characters, a vector of character ranges (for [A-Z] etc.), and a vector of character classes (for [[:alpha:]] etc.). One other point of interest is that although the Boyer-Moore algorithm had to be disabled in the general multibyte case, it is still enabled for UTF-8 because of its self-synchronizing nature. This greatly speeds up matching by reducing the number of multibyte conversions that need to be done.
2004-07-12 07:35:59 +00:00
wint_t c;
wint_t lastc; /* previous c */
wint_t flagch;
Remove 'register' keyword.
2002-03-21 18:49:23 +00:00
int i;
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
const char *matchp; /* last p at which a match ended */
Make regular expression matching aware of multibyte characters. The general idea is that we perform multibyte->wide character conversion while parsing and compiling, then convert byte sequences to wide characters when they're needed for comparison and stepping through the string during execution. As with tr(1), the main complication is to efficiently represent sets of characters in bracket expressions. The old bitmap representation is replaced by a bitmap for the first 256 characters combined with a vector of individual wide characters, a vector of character ranges (for [A-Z] etc.), and a vector of character classes (for [[:alpha:]] etc.). One other point of interest is that although the Boyer-Moore algorithm had to be disabled in the general multibyte case, it is still enabled for UTF-8 because of its self-synchronizing nature. This greatly speeds up matching by reducing the number of multibyte conversions that need to be done.
2004-07-12 07:35:59 +00:00
size_t clen;
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
AT("slow", start, stop, startst, stopst);
CLEAR(st);
SET1(st, startst);
SP("sstart", st, *p);
st = step(m->g, startst, stopst, st, NOTHING, st);
regex(3): Refactor fast/slow stepping bits in the matching engine Adding features for matching is fairly straightforward, but this requires some duplication because of this fast/slow setup. They can be fairly trivially combined into a single walk(), so do it to make future additions less error prone. Reviewed by: cem (earlier version), emaste, pfg Approved by: emaste (mentor) Differential Revision: https://reviews.freebsd.org/D11233
2017-08-09 01:04:36 +00:00
if (fast)
ASSIGN(fresh, st);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
matchp = NULL;
libc: regexec(3) adjustment. Change the behavior of when REG_STARTEND is combined with REG_NOTBOL. From the original posting[1]: "Enable the assumption that pmatch[0].rm_so is a continuation offset to a string and allows us to do a proper assessment of the character in regards to it's word position ('^' or '\<'), without risking going into unallocated memory." This change makes us similar to how glibc handles REG_STARTEND | REG_NOTBOL, and is closely related to a soon-to-land fix to sed. Special thanks to Martijn van Duren and Ingo Schwarze for working out some consistent behaviour. Differential Revision: https://reviews.freebsd.org/D6257 Taken from: openbsd-tech 2016-05-24 [1] (Martijn van Duren) Relnotes: yes MFC after: 1 month
2016-05-25 15:35:23 +00:00
if (start == m->offp || (start == m->beginp && !(m->eflags&REG_NOTBOL)))
Make regular expression matching aware of multibyte characters. The general idea is that we perform multibyte->wide character conversion while parsing and compiling, then convert byte sequences to wide characters when they're needed for comparison and stepping through the string during execution. As with tr(1), the main complication is to efficiently represent sets of characters in bracket expressions. The old bitmap representation is replaced by a bitmap for the first 256 characters combined with a vector of individual wide characters, a vector of character ranges (for [A-Z] etc.), and a vector of character classes (for [[:alpha:]] etc.). One other point of interest is that although the Boyer-Moore algorithm had to be disabled in the general multibyte case, it is still enabled for UTF-8 because of its self-synchronizing nature. This greatly speeds up matching by reducing the number of multibyte conversions that need to be done.
2004-07-12 07:35:59 +00:00
c = OUT;
else {
/*
* XXX Wrong if the previous character was multi-byte.
* Newline never is (in encodings supported by FreeBSD),
* so this only breaks the ISWORD tests below.
*/
c = (uch)*(start - 1);
}
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
for (;;) {
/* next character */
lastc = c;
Make regular expression matching aware of multibyte characters. The general idea is that we perform multibyte->wide character conversion while parsing and compiling, then convert byte sequences to wide characters when they're needed for comparison and stepping through the string during execution. As with tr(1), the main complication is to efficiently represent sets of characters in bracket expressions. The old bitmap representation is replaced by a bitmap for the first 256 characters combined with a vector of individual wide characters, a vector of character ranges (for [A-Z] etc.), and a vector of character classes (for [[:alpha:]] etc.). One other point of interest is that although the Boyer-Moore algorithm had to be disabled in the general multibyte case, it is still enabled for UTF-8 because of its self-synchronizing nature. This greatly speeds up matching by reducing the number of multibyte conversions that need to be done.
2004-07-12 07:35:59 +00:00
if (p == m->endp) {
c = OUT;
clen = 0;
} else
Fix a boundary condition error in slow() and fast() in multibyte locales: we must allow the character beginning at "p" to be converted to a wide character for the purposes of EOL processing and word-boundary matching.
2005-08-17 11:37:38 +00:00
clen = XMBRTOWC(&c, p, m->endp - p, &m->mbs, BADCHAR);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
regex(3): Refactor fast/slow stepping bits in the matching engine Adding features for matching is fairly straightforward, but this requires some duplication because of this fast/slow setup. They can be fairly trivially combined into a single walk(), so do it to make future additions less error prone. Reviewed by: cem (earlier version), emaste, pfg Approved by: emaste (mentor) Differential Revision: https://reviews.freebsd.org/D11233
2017-08-09 01:04:36 +00:00
if (fast && EQ(st, fresh))
matchp = p;
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
/* is there an EOL and/or BOL between lastc and c? */
flagch = '\0';
i = 0;
if ( (lastc == '\n' && m->g->cflags&REG_NEWLINE) ||
(lastc == OUT && !(m->eflags&REG_NOTBOL)) ) {
flagch = BOL;
i = m->g->nbol;
}
if ( (c == '\n' && m->g->cflags&REG_NEWLINE) ||
(c == OUT && !(m->eflags&REG_NOTEOL)) ) {
flagch = (flagch == BOL) ? BOLEOL : EOL;
i += m->g->neol;
}
if (i != 0) {
for (; i > 0; i--)
st = step(m->g, startst, stopst, st, flagch, st);
SP("sboleol", st, c);
}
/* how about a word boundary? */
if ( (flagch == BOL || (lastc != OUT && !ISWORD(lastc))) &&
(c != OUT && ISWORD(c)) ) {
flagch = BOW;
}
if ( (lastc != OUT && ISWORD(lastc)) &&
(flagch == EOL || (c != OUT && !ISWORD(c))) ) {
flagch = EOW;
}
if (flagch == BOW || flagch == EOW) {
st = step(m->g, startst, stopst, st, flagch, st);
SP("sboweow", st, c);
}
/* are we done? */
regex(3): Refactor fast/slow stepping bits in the matching engine Adding features for matching is fairly straightforward, but this requires some duplication because of this fast/slow setup. They can be fairly trivially combined into a single walk(), so do it to make future additions less error prone. Reviewed by: cem (earlier version), emaste, pfg Approved by: emaste (mentor) Differential Revision: https://reviews.freebsd.org/D11233
2017-08-09 01:04:36 +00:00
if (ISSET(st, stopst)) {
if (fast)
break;
else
matchp = p;
}
regex(3): Resolve issues with higher WARNS levels libc is set for WARNS=2, but the incoming libregex will use WARNS=6. Sprinkle some casts and (void)bc's to alleviate the warnings that come along with the higher WARNS level. These 'bc' parameters could be outright removed, but as of right now they will be used in some parts of libregex land. Silence the warnings instead rather than flip-flopping.
2018-01-21 04:57:29 +00:00
if (EQ(st, empty) || p == stop || clen > (size_t)(stop - p))
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
break; /* NOTE BREAK OUT */
/* no, we must deal with this character */
ASSIGN(tmp, st);
regex(3): Refactor fast/slow stepping bits in the matching engine Adding features for matching is fairly straightforward, but this requires some duplication because of this fast/slow setup. They can be fairly trivially combined into a single walk(), so do it to make future additions less error prone. Reviewed by: cem (earlier version), emaste, pfg Approved by: emaste (mentor) Differential Revision: https://reviews.freebsd.org/D11233
2017-08-09 01:04:36 +00:00
if (fast)
ASSIGN(st, fresh);
else
ASSIGN(st, empty);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
assert(c != OUT);
st = step(m->g, startst, stopst, tmp, c, st);
SP("saft", st, c);
assert(EQ(step(m->g, startst, stopst, st, NOTHING, st), st));
Make regular expression matching aware of multibyte characters. The general idea is that we perform multibyte->wide character conversion while parsing and compiling, then convert byte sequences to wide characters when they're needed for comparison and stepping through the string during execution. As with tr(1), the main complication is to efficiently represent sets of characters in bracket expressions. The old bitmap representation is replaced by a bitmap for the first 256 characters combined with a vector of individual wide characters, a vector of character ranges (for [A-Z] etc.), and a vector of character classes (for [[:alpha:]] etc.). One other point of interest is that although the Boyer-Moore algorithm had to be disabled in the general multibyte case, it is still enabled for UTF-8 because of its self-synchronizing nature. This greatly speeds up matching by reducing the number of multibyte conversions that need to be done.
2004-07-12 07:35:59 +00:00
p += clen;
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
}
regex(3): Refactor fast/slow stepping bits in the matching engine Adding features for matching is fairly straightforward, but this requires some duplication because of this fast/slow setup. They can be fairly trivially combined into a single walk(), so do it to make future additions less error prone. Reviewed by: cem (earlier version), emaste, pfg Approved by: emaste (mentor) Differential Revision: https://reviews.freebsd.org/D11233
2017-08-09 01:04:36 +00:00
if (fast) {
assert(matchp != NULL);
m->coldp = matchp;
if (ISSET(st, stopst))
return (p + XMBRTOWC(NULL, p, stop - p, &m->mbs, 0));
else
return (NULL);
} else
return (matchp);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
}
/*
- step - map set of states reachable before char to set reachable after
Remove 'register' keyword.
2002-03-21 18:49:23 +00:00
== static states step(struct re_guts *g, sopno start, sopno stop, \
== states bef, int ch, states aft);
Make regular expression matching aware of multibyte characters. The general idea is that we perform multibyte->wide character conversion while parsing and compiling, then convert byte sequences to wide characters when they're needed for comparison and stepping through the string during execution. As with tr(1), the main complication is to efficiently represent sets of characters in bracket expressions. The old bitmap representation is replaced by a bitmap for the first 256 characters combined with a vector of individual wide characters, a vector of character ranges (for [A-Z] etc.), and a vector of character classes (for [[:alpha:]] etc.). One other point of interest is that although the Boyer-Moore algorithm had to be disabled in the general multibyte case, it is still enabled for UTF-8 because of its self-synchronizing nature. This greatly speeds up matching by reducing the number of multibyte conversions that need to be done.
2004-07-12 07:35:59 +00:00
== #define BOL (OUT-1)
== #define EOL (BOL-1)
== #define BOLEOL (BOL-2)
== #define NOTHING (BOL-3)
== #define BOW (BOL-4)
== #define EOW (BOL-5)
== #define BADCHAR (BOL-6)
== #define NONCHAR(c) ((c) <= OUT)
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
*/
static states
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
step(struct re_guts *g,
sopno start, /* start state within strip */
sopno stop, /* state after stop state within strip */
states bef, /* states reachable before */
wint_t ch, /* character or NONCHAR code */
states aft) /* states already known reachable after */
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
{
Remove 'register' keyword.
2002-03-21 18:49:23 +00:00
cset *cs;
sop s;
sopno pc;
onestate here; /* note, macros know this name */
sopno look;
int i;
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
for (pc = start, INIT(here, pc); pc != stop; pc++, INC(here)) {
s = g->strip[pc];
switch (OP(s)) {
case OEND:
assert(pc == stop-1);
break;
case OCHAR:
/* only characters can match */
Make regular expression matching aware of multibyte characters. The general idea is that we perform multibyte->wide character conversion while parsing and compiling, then convert byte sequences to wide characters when they're needed for comparison and stepping through the string during execution. As with tr(1), the main complication is to efficiently represent sets of characters in bracket expressions. The old bitmap representation is replaced by a bitmap for the first 256 characters combined with a vector of individual wide characters, a vector of character ranges (for [A-Z] etc.), and a vector of character classes (for [[:alpha:]] etc.). One other point of interest is that although the Boyer-Moore algorithm had to be disabled in the general multibyte case, it is still enabled for UTF-8 because of its self-synchronizing nature. This greatly speeds up matching by reducing the number of multibyte conversions that need to be done.
2004-07-12 07:35:59 +00:00
assert(!NONCHAR(ch) || ch != OPND(s));
if (ch == OPND(s))
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
FWD(aft, bef, 1);
break;
case OBOL:
if (ch == BOL || ch == BOLEOL)
FWD(aft, bef, 1);
break;
case OEOL:
if (ch == EOL || ch == BOLEOL)
FWD(aft, bef, 1);
break;
case OBOW:
if (ch == BOW)
FWD(aft, bef, 1);
break;
case OEOW:
if (ch == EOW)
FWD(aft, bef, 1);
break;
case OANY:
if (!NONCHAR(ch))
FWD(aft, bef, 1);
break;
case OANYOF:
cs = &g->sets[OPND(s)];
if (!NONCHAR(ch) && CHIN(cs, ch))
FWD(aft, bef, 1);
break;
case OBACK_: /* ignored here */
case O_BACK:
FWD(aft, aft, 1);
break;
case OPLUS_: /* forward, this is just an empty */
FWD(aft, aft, 1);
break;
case O_PLUS: /* both forward and back */
FWD(aft, aft, 1);
i = ISSETBACK(aft, OPND(s));
BACK(aft, aft, OPND(s));
if (!i && ISSETBACK(aft, OPND(s))) {
/* oho, must reconsider loop body */
pc -= OPND(s) + 1;
INIT(here, pc);
}
break;
case OQUEST_: /* two branches, both forward */
FWD(aft, aft, 1);
FWD(aft, aft, OPND(s));
break;
case O_QUEST: /* just an empty */
FWD(aft, aft, 1);
break;
case OLPAREN: /* not significant here */
case ORPAREN:
FWD(aft, aft, 1);
break;
case OCH_: /* mark the first two branches */
FWD(aft, aft, 1);
regex(3): Resolve issues with higher WARNS levels libc is set for WARNS=2, but the incoming libregex will use WARNS=6. Sprinkle some casts and (void)bc's to alleviate the warnings that come along with the higher WARNS level. These 'bc' parameters could be outright removed, but as of right now they will be used in some parts of libregex land. Silence the warnings instead rather than flip-flopping.
2018-01-21 04:57:29 +00:00
assert(OP(g->strip[pc+OPND(s)]) == (sop)OOR2);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
FWD(aft, aft, OPND(s));
break;
case OOR1: /* done a branch, find the O_CH */
if (ISSTATEIN(aft, here)) {
for (look = 1;
regex(3): Resolve issues with higher WARNS levels libc is set for WARNS=2, but the incoming libregex will use WARNS=6. Sprinkle some casts and (void)bc's to alleviate the warnings that come along with the higher WARNS level. These 'bc' parameters could be outright removed, but as of right now they will be used in some parts of libregex land. Silence the warnings instead rather than flip-flopping.
2018-01-21 04:57:29 +00:00
OP(s = g->strip[pc+look]) != (sop)O_CH;
look += OPND(s))
assert(OP(s) == (sop)OOR2);
Fix an off-by-one error in the marking of the O_CH operator following an OOR2 operator. PR: 130504 MFC after: 2 weeks
2009-09-16 06:32:23 +00:00
FWD(aft, aft, look + 1);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
}
break;
case OOR2: /* propagate OCH_'s marking */
FWD(aft, aft, 1);
regex(3): Resolve issues with higher WARNS levels libc is set for WARNS=2, but the incoming libregex will use WARNS=6. Sprinkle some casts and (void)bc's to alleviate the warnings that come along with the higher WARNS level. These 'bc' parameters could be outright removed, but as of right now they will be used in some parts of libregex land. Silence the warnings instead rather than flip-flopping.
2018-01-21 04:57:29 +00:00
if (OP(g->strip[pc+OPND(s)]) != (sop)O_CH) {
assert(OP(g->strip[pc+OPND(s)]) == (sop)OOR2);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
FWD(aft, aft, OPND(s));
}
break;
case O_CH: /* just empty */
FWD(aft, aft, 1);
break;
default: /* ooooops... */
assert(nope);
break;
}
}
return(aft);
}
#ifdef REDEBUG
/*
- print - print a set of states
== #ifdef REDEBUG
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
== static void print(struct match *m, const char *caption, states st, \
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
== int ch, FILE *d);
== #endif
*/
static void
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
print(struct match *m,
const char *caption,
states st,
int ch,
FILE *d)
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
{
Remove 'register' keyword.
2002-03-21 18:49:23 +00:00
struct re_guts *g = m->g;
regex: prevent two improbable signed integer overflows. In matcher() we used an integer to index nsub of type size_t. In print() we used an integer to index nstates of type sopno, typedef'd long. In both cases the indexes never take negative values. Match the types to avoid any error. MFC after: 5 days
2016-04-23 20:45:09 +00:00
sopno i;
Remove 'register' keyword.
2002-03-21 18:49:23 +00:00
int first = 1;
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
if (!(m->eflags&REG_TRACE))
return;
fprintf(d, "%s", caption);
if (ch != '\0')
fprintf(d, " %s", pchar(ch));
for (i = 0; i < g->nstates; i++)
if (ISSET(st, i)) {
lib/libc/regex: fix build with REDEBUG defined Reviewed by: kevans Differential Revision: https://reviews.freebsd.org/D21760
2019-09-24 12:21:01 +00:00
fprintf(d, "%s%lu", (first) ? "\t" : ", ", i);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
first = 0;
}
fprintf(d, "\n");
}
Remove trailing whitespace.
1995-05-30 05:51:47 +00:00
/*
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
- at - print current situation
== #ifdef REDEBUG
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
== static void at(struct match *m, const char *title, const char *start, \
== const char *stop, sopno startst, sopno stopst);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
== #endif
*/
static void
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
at( struct match *m,
const char *title,
const char *start,
const char *stop,
sopno startst,
sopno stopst)
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
{
if (!(m->eflags&REG_TRACE))
return;
printf("%s %s-", title, pchar(*start));
printf("%s ", pchar(*stop));
printf("%ld-%ld\n", (long)startst, (long)stopst);
}
#ifndef PCHARDONE
#define PCHARDONE /* never again */
/*
- pchar - make a character printable
== #ifdef REDEBUG
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
== static const char *pchar(int ch);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
== #endif
*
* Is this identical to regchar() over in debug.c? Well, yes. But a
* duplicate here avoids having a debugging-capable regexec.o tied to
* a matching debug.o, and this is convenient. It all disappears in
* the non-debug compilation anyway, so it doesn't matter much.
*/
Const'ify and ANSIfy the internal interfaces of regex(3). This is the final change that makes libc to compile with WERROR on my amd64 crashbox.
2007-05-25 12:44:58 +00:00
static const char * /* -> representation */
pchar(int ch)
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
{
static char pbuf[10];
Use locale for character classes instead of hardcoded values Misc 8bit cleanup
1996-08-11 11:42:03 +00:00
if (isprint((uch)ch) || ch == ' ')
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
sprintf(pbuf, "%c", ch);
else
sprintf(pbuf, "\\%o", ch);
return(pbuf);
}
#endif
#endif
regexec: fix processing multibyte strings. Matcher function incorrectly assumed that moffset that we get from findmust is in bytes. Fix this by introducing a stepback function, taking short path if MB_CUR_MAX is 1, and going back byte-by-byte, checking if we have a legal character sequence otherwise. PR: 153502 Reviewed by: pfg, kevans Approved by: kib (mentor, implicit) Differential revision: https://reviews.freebsd.org/D18297
2018-11-23 15:49:18 +00:00
#undef stepback
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
#undef matcher
regex(3): Refactor fast/slow stepping bits in the matching engine Adding features for matching is fairly straightforward, but this requires some duplication because of this fast/slow setup. They can be fairly trivially combined into a single walk(), so do it to make future additions less error prone. Reviewed by: cem (earlier version), emaste, pfg Approved by: emaste (mentor) Differential Revision: https://reviews.freebsd.org/D11233
2017-08-09 01:04:36 +00:00
#undef walk
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
#undef dissect
#undef backref
#undef step
#undef print
#undef at
#undef match
Reference in New Issue Copy Permalink