diff options
Diffstat (limited to 'srclib')
| -rw-r--r-- | srclib/pcre/LICENCE | 24 | ||||
| -rw-r--r-- | srclib/pcre/doc/pcre.3 | 368 | ||||
| -rw-r--r-- | srclib/pcre/doc/pcre.html | 483 | ||||
| -rw-r--r-- | srclib/pcre/doc/pcre.txt | 426 | ||||
| -rw-r--r-- | srclib/pcre/doc/pcretest.txt | 532 | ||||
| -rw-r--r-- | srclib/pcre/pcre.c | 576 | ||||
| -rw-r--r-- | srclib/pcre/pcretest.c | 326 | ||||
| -rw-r--r-- | srclib/pcre/testdata/testinput3 | 16 | ||||
| -rw-r--r-- | srclib/pcre/testdata/testoutput1 | 116 | ||||
| -rw-r--r-- | srclib/pcre/testdata/testoutput2 | 318 | ||||
| -rw-r--r-- | srclib/pcre/testdata/testoutput3 | 28 | ||||
| -rw-r--r-- | srclib/pcre/testdata/testoutput4 | 3 |
12 files changed, 2670 insertions, 546 deletions
diff --git a/srclib/pcre/LICENCE b/srclib/pcre/LICENCE index 8422bd9ae6..8effa66492 100644 --- a/srclib/pcre/LICENCE +++ b/srclib/pcre/LICENCE @@ -9,7 +9,7 @@ Written by: Philip Hazel <ph10@cam.ac.uk> University of Cambridge Computing Service, Cambridge, England. Phone: +44 1223 334714. -Copyright (c) 1997-2000 University of Cambridge +Copyright (c) 1997-2001 University of Cambridge Permission is granted to anyone to use this software for any purpose on any computer system, and to redistribute it freely, subject to the following @@ -20,25 +20,31 @@ restrictions: MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. 2. The origin of this software must not be misrepresented, either by - explicit claim or by omission. In practice, this means you must put - a sentence like this + explicit claim or by omission. In practice, this means that if you use + PCRE in software which you distribute to others, commercially or + otherwise, you must put a sentence like this Regular expression support is provided by the PCRE library package, - which is open source software, copyright by the University of - Cambridge. + which is open source software, written by Philip Hazel, and copyright + by the University of Cambridge, England. somewhere reasonably visible in your documentation and in any relevant - files. A reference to the ftp site for the source should also be given + files or online help data or similar. A reference to the ftp site for + the source, that is, to ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/ - in the documentation. + should also be given in the documentation. 3. Altered versions must be plainly marked as such, and must not be misrepresented as being the original software. 4. If PCRE is embedded in any software that is released under the GNU - General Purpose Licence (GPL), then the terms of that licence shall - supersede any condition above with which it is incompatible. + General Purpose Licence (GPL), or Lesser General Purpose Licence (LGPL), + then the terms of that licence shall supersede any condition above with + which it is incompatible. + +The documentation for PCRE, supplied in the "doc" directory, is distributed +under the same terms as the software itself. End diff --git a/srclib/pcre/doc/pcre.3 b/srclib/pcre/doc/pcre.3 index 4334be2440..738f76b4a9 100644 --- a/srclib/pcre/doc/pcre.3 +++ b/srclib/pcre/doc/pcre.3 @@ -44,6 +44,12 @@ pcre - Perl-compatible regular expressions. .B int *\fIovector\fR, int \fIstringcount\fR, "const char ***\fIlistptr\fR);" .PP .br +.B void pcre_free_substring(const char *\fIstringptr\fR); +.PP +.br +.B void pcre_free_substring_list(const char **\fIstringptr\fR); +.PP +.br .B const unsigned char *pcre_maketables(void); .PP .br @@ -70,7 +76,9 @@ pcre - Perl-compatible regular expressions. The PCRE library is a set of functions that implement regular expression pattern matching using the same syntax and semantics as Perl 5, with just a few differences (see below). The current implementation corresponds to Perl 5.005, -with some additional features from the Perl development release. +with some additional features from later versions. This includes some +experimental, incomplete support for UTF-8 encoded strings. Details of exactly +what is and what is not supported are given below. PCRE has its own native API, which is described in this document. There is also a set of wrapper functions that correspond to the POSIX regular expression API. @@ -84,12 +92,18 @@ contain the major and minor release numbers for the library. Applications can use these to include support for different releases. The functions \fBpcre_compile()\fR, \fBpcre_study()\fR, and \fBpcre_exec()\fR -are used for compiling and matching regular expressions, while -\fBpcre_copy_substring()\fR, \fBpcre_get_substring()\fR, and +are used for compiling and matching regular expressions. A sample program that +demonstrates the simplest way of using them is given in the file +\fIpcredemo.c\fR. The last section of this man page describes how to run it. + +The functions \fBpcre_copy_substring()\fR, \fBpcre_get_substring()\fR, and \fBpcre_get_substring_list()\fR are convenience functions for extracting -captured substrings from a matched subject string. The function -\fBpcre_maketables()\fR is used (optionally) to build a set of character tables -in the current locale for passing to \fBpcre_compile()\fR. +captured substrings from a matched subject string; \fBpcre_free_substring()\fR +and \fBpcre_free_substring_list()\fR are also provided, to free the memory used +for extracted strings. + +The function \fBpcre_maketables()\fR is used (optionally) to build a set of +character tables in the current locale for passing to \fBpcre_compile()\fR. The function \fBpcre_fullinfo()\fR is used to find out information about a compiled pattern; \fBpcre_info()\fR is an obsolete version which returns only @@ -117,18 +131,22 @@ the same compiled pattern can safely be used by several threads at once. The function \fBpcre_compile()\fR is called to compile a pattern into an internal form. The pattern is a C string terminated by a binary zero, and is passed in the argument \fIpattern\fR. A pointer to a single block of memory -that is obtained via \fBpcre_malloc\fR is returned. This contains the -compiled code and related data. The \fBpcre\fR type is defined for this for -convenience, but in fact \fBpcre\fR is just a typedef for \fBvoid\fR, since the -contents of the block are not externally defined. It is up to the caller to -free the memory when it is no longer required. -.PP +that is obtained via \fBpcre_malloc\fR is returned. This contains the compiled +code and related data. The \fBpcre\fR type is defined for the returned block; +this is a typedef for a structure whose contents are not externally defined. It +is up to the caller to free the memory when it is no longer required. + +Although the compiled code of a PCRE regex is relocatable, that is, it does not +depend on memory location, the complete \fBpcre\fR data block is not +fully relocatable, because it contains a copy of the \fItableptr\fR argument, +which is an address (see below). + The size of a compiled pattern is roughly proportional to the length of the pattern string, except that each character class (other than those containing just a single character, negated or not) requires 33 bytes, and repeat quantifiers with a minimum greater than one or a bounded maximum cause the relevant portions of the compiled pattern to be replicated. -.PP + The \fIoptions\fR argument contains independent bits that affect the compilation. It should be zero if no options are required. Some of the options, in particular, those that are compatible with Perl, can also be set and unset @@ -137,19 +155,31 @@ below). For these options, the contents of the \fIoptions\fR argument specifies their initial settings at the start of compilation and execution. The PCRE_ANCHORED option can be set at the time of matching as well as at compile time. -.PP + If \fIerrptr\fR is NULL, \fBpcre_compile()\fR returns NULL immediately. Otherwise, if compilation of a pattern fails, \fBpcre_compile()\fR returns NULL, and sets the variable pointed to by \fIerrptr\fR to point to a textual error message. The offset from the start of the pattern to the character where the error was discovered is placed in the variable pointed to by \fIerroffset\fR, which must not be NULL. If it is, an immediate error is given. -.PP + If the final argument, \fItableptr\fR, is NULL, PCRE uses a default set of character tables which are built when it is compiled, using the default C locale. Otherwise, \fItableptr\fR must be the result of a call to \fBpcre_maketables()\fR. See the section on locale support below. -.PP + +This code fragment shows a typical straightforward call to \fBpcre_compile()\fR: + + pcre *re; + const char *error; + int erroffset; + re = pcre_compile( + "^A.*Z", /* the pattern */ + 0, /* default options */ + &error, /* for error message */ + &erroffset, /* for error offset */ + NULL); /* use default character tables */ + The following option bits are defined in the header file: PCRE_ANCHORED @@ -223,15 +253,23 @@ This option inverts the "greediness" of the quantifiers so that they are not greedy by default, but become greedy if followed by "?". It is not compatible with Perl. It can also be set by a (?U) option setting within the pattern. + PCRE_UTF8 + +This option causes PCRE to regard both the pattern and the subject as strings +of UTF-8 characters instead of just byte strings. However, it is available only +if PCRE has been built to include UTF-8 support. If not, the use of this option +provokes an error. Support for UTF-8 is new, experimental, and incomplete. +Details of exactly what it entails are given below. + .SH STUDYING A PATTERN When a pattern is going to be used several times, it is worth spending more time analyzing it in order to speed up the time taken for matching. The function \fBpcre_study()\fR takes a pointer to a compiled pattern as its first -argument, and returns a pointer to a \fBpcre_extra\fR block (another \fBvoid\fR -typedef) containing additional information about the pattern; this can be -passed to \fBpcre_exec()\fR. If no additional information is available, NULL -is returned. +argument, and returns a pointer to a \fBpcre_extra\fR block (another typedef +for a structure with hidden contents) containing additional information about +the pattern; this can be passed to \fBpcre_exec()\fR. If no additional +information is available, NULL is returned. The second argument contains option bits. At present, no options are defined for \fBpcre_study()\fR, and this argument should always be zero. @@ -240,6 +278,14 @@ The third argument for \fBpcre_study()\fR is a pointer to an error message. If studying succeeds (even if no data is returned), the variable it points to is set to NULL. Otherwise it points to a textual error message. +This is a typical call to \fBpcre_study\fR(): + + pcre_extra *pe; + pe = pcre_study( + re, /* result of pcre_compile() */ + 0, /* no options exist */ + &error); /* set to NULL or points to a message */ + At present, studying a pattern is useful only for non-anchored patterns that do not have a single fixed starting character. A bitmap of possible starting characters is created. @@ -289,13 +335,24 @@ the following negative numbers: PCRE_ERROR_BADMAGIC the "magic number" was not found PCRE_ERROR_BADOPTION the value of \fIwhat\fR was invalid +Here is a typical call of \fBpcre_fullinfo()\fR, to obtain the length of the +compiled pattern: + + int rc; + unsigned long int length; + rc = pcre_fullinfo( + re, /* result of pcre_compile() */ + pe, /* result of pcre_study(), or NULL */ + PCRE_INFO_SIZE, /* what is required */ + &length); /* where to put the data */ + The possible values for the third argument are defined in \fBpcre.h\fR, and are as follows: PCRE_INFO_OPTIONS Return a copy of the options with which the pattern was compiled. The fourth -argument should point to au \fBunsigned long int\fR variable. These option bits +argument should point to an \fBunsigned long int\fR variable. These option bits are those specified in the call to \fBpcre_compile()\fR, modified by any top-level option settings within the pattern itself, and with the PCRE_ANCHORED bit forcibly set if the form of the pattern implies that it can match only at @@ -376,6 +433,20 @@ pre-compiled pattern, which is passed in the \fIcode\fR argument. If the pattern has been studied, the result of the study should be passed in the \fIextra\fR argument. Otherwise this must be NULL. +Here is an example of a simple call to \fBpcre_exec()\fR: + + int rc; + int ovector[30]; + rc = pcre_exec( + re, /* result of pcre_compile() */ + NULL, /* we didn't study the pattern */ + "some string", /* the subject string */ + 11, /* the length of the subject string */ + 0, /* start at offset 0 in the subject */ + 0, /* default options */ + ovector, /* vector for substring information */ + 30); /* number of elements in the vector */ + The PCRE_ANCHORED option can be passed in the \fIoptions\fR argument, whose unused bits must be zero. However, if a pattern was compiled with PCRE_ANCHORED, or turned out to be anchored by virtue of its contents, it @@ -417,9 +488,9 @@ below) and trying an ordinary match again. The subject string is passed as a pointer in \fIsubject\fR, a length in \fIlength\fR, and a starting offset in \fIstartoffset\fR. Unlike the pattern -string, it may contain binary zero characters. When the starting offset is -zero, the search for a match starts at the beginning of the subject, and this -is by far the most common case. +string, the subject may contain binary zero characters. When the starting +offset is zero, the search for a match starts at the beginning of the subject, +and this is by far the most common case. A non-zero starting offset is useful when searching for another match in the same subject by calling \fBpcre_exec()\fR again after a previous success. @@ -558,7 +629,7 @@ extract a single substring, whose number is given as \fIstringnumber\fR. A value of zero extracts the substring that matched the entire pattern, while higher values extract the captured substrings. For \fBpcre_copy_substring()\fR, the string is placed in \fIbuffer\fR, whose length is given by -\fIbuffersize\fR, while for \fBpcre_get_substring()\fR a new block of store is +\fIbuffersize\fR, while for \fBpcre_get_substring()\fR a new block of memory is obtained via \fBpcre_malloc\fR, and its address is returned via \fIstringptr\fR. The yield of the function is the length of the string, not including the terminating zero, or one of @@ -590,6 +661,15 @@ string. This can be distinguished from a genuine zero-length substring by inspecting the appropriate offset in \fIovector\fR, which is negative for unset substrings. +The two convenience functions \fBpcre_free_substring()\fR and +\fBpcre_free_substring_list()\fR can be used to free the memory returned by +a previous call of \fBpcre_get_substring()\fR or +\fBpcre_get_substring_list()\fR, respectively. They do nothing more than call +the function pointed to by \fBpcre_free\fR, which of course could be called +directly from a C program. However, PCRE is used in some situations where it is +linked via a special interface to another programming language which cannot use +\fBpcre_free\fR directly; it is for these cases that the functions are +provided. .SH LIMITATIONS @@ -597,8 +677,9 @@ There are some size limitations in PCRE but it is hoped that they will never in practice be relevant. The maximum length of a compiled pattern is 65539 (sic) bytes. All values in repeating quantifiers must be less than 65536. -The maximum number of capturing subpatterns is 99. -The maximum number of all parenthesized subpatterns, including capturing +There maximum number of capturing subpatterns is 65535. +There is no limit to the number of non-capturing subpatterns, but the maximum +depth of nesting of all kinds of parenthesized subpattern, including capturing subpatterns, assertions, and other types of subpattern, is 200. The maximum length of a subject string is the largest positive number that an @@ -691,8 +772,14 @@ The syntax and semantics of the regular expressions supported by PCRE are described below. Regular expressions are also described in the Perl documentation and in a number of other books, some of which have copious examples. Jeffrey Friedl's "Mastering Regular Expressions", published by -O'Reilly (ISBN 1-56592-257), covers them in great detail. The description -here is intended as reference documentation. +O'Reilly (ISBN 1-56592-257), covers them in great detail. + +The description here is intended as reference documentation. The basic +operation of PCRE is on strings of bytes. However, there is the beginnings of +some support for UTF-8 character strings. To use this support you must +configure PCRE to include it, and then call \fBpcre_compile()\fR with the +PCRE_UTF8 option. How this affects the pattern matching is described in the +final section of this document. A regular expression is a pattern that is matched against a subject string from left to right. Most characters stand for themselves in a pattern, and match the @@ -914,7 +1001,7 @@ PCRE_MULTILINE is set. Note that the sequences \\A, \\Z, and \\z can be used to match the start and end of the subject in both modes, and if all branches of a pattern start with -\\A is it always anchored, whether PCRE_MULTILINE is set or not. +\\A it is always anchored, whether PCRE_MULTILINE is set or not. .SH FULL STOP (PERIOD, DOT) @@ -1018,7 +1105,7 @@ negation, which is indicated by a ^ character after the colon. For example, [12[:^digit:]] -matches "1", "2", or any non-digit. PCRE (and Perl) also recogize the POSIX +matches "1", "2", or any non-digit. PCRE (and Perl) also recognize the POSIX syntax [.ch.] and [=ch=] where "ch" is a "collating element", but these are not supported, and an error is given if they are encountered. @@ -1116,7 +1203,7 @@ For example, if the string "the red king" is matched against the pattern the ((red|white) (king|queen)) the captured substrings are "red king", "red", and "king", and are numbered 1, -2, and 3. +2, and 3, respectively. The fact that plain parentheses fulfil two functions is not always helpful. There are often times when a grouping subpattern is required without a @@ -1210,7 +1297,7 @@ to the string /* first command */ not comment /* second comment */ -fails, because it matches the entire string due to the greediness of the .* +fails, because it matches the entire string owing to the greediness of the .* item. However, if a quantifier is followed by a question mark, it ceases to be @@ -1311,7 +1398,7 @@ example, the pattern (a|b\\1)+ -matches any number of "a"s and also "aba", "ababaa" etc. At each iteration of +matches any number of "a"s and also "aba", "ababbaa" etc. At each iteration of the subpattern, the back reference matches the character string corresponding to the previous iteration. In order for this to work, the pattern must be such that the first iteration does not need to match the back reference. This can be @@ -1529,9 +1616,10 @@ subpattern, a compile-time error occurs. There are two kinds of condition. If the text between the parentheses consists of a sequence of digits, the condition is satisfied if the capturing subpattern -of that number has previously matched. Consider the following pattern, which -contains non-significant white space to make it more readable (assume the -PCRE_EXTENDED option) and to divide it into three parts for ease of discussion: +of that number has previously matched. The number must be greater than zero. +Consider the following pattern, which contains non-significant white space to +make it more readable (assume the PCRE_EXTENDED option) and to divide it into +three parts for ease of discussion: ( \\( )? [^()]+ (?(1) \\) ) @@ -1685,6 +1773,208 @@ with the pattern above. The former gives a failure almost instantly when applied to a whole line of "a" characters, whereas the latter takes an appreciable time with strings longer than about 20 characters. + +.SH UTF-8 SUPPORT +Starting at release 3.3, PCRE has some support for character strings encoded +in the UTF-8 format. This is incomplete, and is regarded as experimental. In +order to use it, you must configure PCRE to include UTF-8 support in the code, +and, in addition, you must call \fBpcre_compile()\fR with the PCRE_UTF8 option +flag. When you do this, both the pattern and any subject strings that are +matched against it are treated as UTF-8 strings instead of just strings of +bytes, but only in the cases that are mentioned below. + +If you compile PCRE with UTF-8 support, but do not use it at run time, the +library will be a bit bigger, but the additional run time overhead is limited +to testing the PCRE_UTF8 flag in several places, so should not be very large. + +PCRE assumes that the strings it is given contain valid UTF-8 codes. It does +not diagnose invalid UTF-8 strings. If you pass invalid UTF-8 strings to PCRE, +the results are undefined. + +Running with PCRE_UTF8 set causes these changes in the way PCRE works: + +1. In a pattern, the escape sequence \\x{...}, where the contents of the braces +is a string of hexadecimal digits, is interpreted as a UTF-8 character whose +code number is the given hexadecimal number, for example: \\x{1234}. This +inserts from one to six literal bytes into the pattern, using the UTF-8 +encoding. If a non-hexadecimal digit appears between the braces, the item is +not recognized. + +2. The original hexadecimal escape sequence, \\xhh, generates a two-byte UTF-8 +character if its value is greater than 127. + +3. Repeat quantifiers are NOT correctly handled if they follow a multibyte +character. For example, \\x{100}* and \\xc3+ do not work. If you want to +repeat such characters, you must enclose them in non-capturing parentheses, +for example (?:\\x{100}), at present. + +4. The dot metacharacter matches one UTF-8 character instead of a single byte. + +5. Unlike literal UTF-8 characters, the dot metacharacter followed by a +repeat quantifier does operate correctly on UTF-8 characters instead of +single bytes. + +4. Although the \\x{...} escape is permitted in a character class, characters +whose values are greater than 255 cannot be included in a class. + +5. A class is matched against a UTF-8 character instead of just a single byte, +but it can match only characters whose values are less than 256. Characters +with greater values always fail to match a class. + +6. Repeated classes work correctly on multiple characters. + +7. Classes containing just a single character whose value is greater than 127 +(but less than 256), for example, [\\x80] or [^\\x{93}], do not work because +these are optimized into single byte matches. In the first case, of course, +the class brackets are just redundant. + +8. Lookbehind assertions move backwards in the subject by a fixed number of +characters instead of a fixed number of bytes. Simple cases have been tested +to work correctly, but there may be hidden gotchas herein. + +9. The character types such as \\d and \\w do not work correctly with UTF-8 +characters. They continue to test a single byte. + +10. Anything not explicitly mentioned here continues to work in bytes rather +than in characters. + +The following UTF-8 features of Perl 5.6 are not implemented: + +1. The escape sequence \\C to match a single byte. + +2. The use of Unicode tables and properties and escapes \\p, \\P, and \\X. + + +.SH SAMPLE PROGRAM +The code below is a simple, complete demonstration program, to get you started +with using PCRE. This code is also supplied in the file \fIpcredemo.c\fR in the +PCRE distribution. + +The program compiles the regular expression that is its first argument, and +matches it against the subject string in its second argument. No options are +set, and default character tables are used. If matching succeeds, the program +outputs the portion of the subject that matched, together with the contents of +any captured substrings. + +On a Unix system that has PCRE installed in \fI/usr/local\fR, you can compile +the demonstration program using a command like this: + + gcc -o pcredemo pcredemo.c -I/usr/local/include -L/usr/local/lib -lpcre + +Then you can run simple tests like this: + + ./pcredemo 'cat|dog' 'the cat sat on the mat' + +Note that there is a much more comprehensive test program, called +\fBpcretest\fR, which supports many more facilities for testing regular +expressions. The \fBpcredemo\fR program is provided as a simple coding example. + +On some operating systems (e.g. Solaris) you may get an error like this when +you try to run \fBpcredemo\fR: + + ld.so.1: a.out: fatal: libpcre.so.0: open failed: No such file or directory + +This is caused by the way shared library support works on those systems. You +need to add + + -R/usr/local/lib + +to the compile command to get round this problem. Here's the code: + + #include <stdio.h> + #include <string.h> + #include <pcre.h> + + #define OVECCOUNT 30 /* should be a multiple of 3 */ + + int main(int argc, char **argv) + { + pcre *re; + const char *error; + int erroffset; + int ovector[OVECCOUNT]; + int rc, i; + + if (argc != 3) + { + printf("Two arguments required: a regex and a " + "subject string\\n"); + return 1; + } + + /* Compile the regular expression in the first argument */ + + re = pcre_compile( + argv[1], /* the pattern */ + 0, /* default options */ + &error, /* for error message */ + &erroffset, /* for error offset */ + NULL); /* use default character tables */ + + /* Compilation failed: print the error message and exit */ + + if (re == NULL) + { + printf("PCRE compilation failed at offset %d: %s\\n", + erroffset, error); + return 1; + } + + /* Compilation succeeded: match the subject in the second + argument */ + + rc = pcre_exec( + re, /* the compiled pattern */ + NULL, /* we didn't study the pattern */ + argv[2], /* the subject string */ + (int)strlen(argv[2]), /* the length of the subject */ + 0, /* start at offset 0 in the subject */ + 0, /* default options */ + ovector, /* vector for substring information */ + OVECCOUNT); /* number of elements in the vector */ + + /* Matching failed: handle error cases */ + + if (rc < 0) + { + switch(rc) + { + case PCRE_ERROR_NOMATCH: printf("No match\\n"); break; + /* + Handle other special cases if you like + */ + default: printf("Matching error %d\\n", rc); break; + } + return 1; + } + + /* Match succeded */ + + printf("Match succeeded\\n"); + + /* The output vector wasn't big enough */ + + if (rc == 0) + { + rc = OVECCOUNT/3; + printf("ovector only has room for %d captured " + substrings\\n", rc - 1); + } + + /* Show substrings stored in the output vector */ + + for (i = 0; i < rc; i++) + { + char *substring_start = argv[2] + ovector[2*i]; + int substring_length = ovector[2*i+1] - ovector[2*i]; + printf("%2d: %.*s\\n", i, substring_length, + substring_start); + } + + return 0; + } + + .SH AUTHOR Philip Hazel <ph10@cam.ac.uk> .br @@ -1696,6 +1986,6 @@ Cambridge CB2 3QG, England. .br Phone: +44 1223 334714 -Last updated: 27 January 2000 +Last updated: 15 August 2001 .br -Copyright (c) 1997-2000 University of Cambridge. +Copyright (c) 1997-2001 University of Cambridge. diff --git a/srclib/pcre/doc/pcre.html b/srclib/pcre/doc/pcre.html index 7eba9c35dd..3e9eb36b68 100644 --- a/srclib/pcre/doc/pcre.html +++ b/srclib/pcre/doc/pcre.html @@ -37,7 +37,9 @@ conversion went wrong. <LI><A NAME="TOC27" HREF="#SEC27">COMMENTS</A> <LI><A NAME="TOC28" HREF="#SEC28">RECURSIVE PATTERNS</A> <LI><A NAME="TOC29" HREF="#SEC29">PERFORMANCE</A> -<LI><A NAME="TOC30" HREF="#SEC30">AUTHOR</A> +<LI><A NAME="TOC30" HREF="#SEC30">UTF-8 SUPPORT</A> +<LI><A NAME="TOC31" HREF="#SEC31">SAMPLE PROGRAM</A> +<LI><A NAME="TOC32" HREF="#SEC32">AUTHOR</A> </UL> <LI><A NAME="SEC1" HREF="#TOC1">NAME</A> <P> @@ -76,6 +78,12 @@ pcre - Perl-compatible regular expressions. <B>int *<I>ovector</I>, int <I>stringcount</I>, const char ***<I>listptr</I>);</B> </P> <P> +<B>void pcre_free_substring(const char *<I>stringptr</I>);</B> +</P> +<P> +<B>void pcre_free_substring_list(const char **<I>stringptr</I>);</B> +</P> +<P> <B>const unsigned char *pcre_maketables(void);</B> </P> <P> @@ -100,7 +108,9 @@ pcre - Perl-compatible regular expressions. The PCRE library is a set of functions that implement regular expression pattern matching using the same syntax and semantics as Perl 5, with just a few differences (see below). The current implementation corresponds to Perl 5.005, -with some additional features from the Perl development release. +with some additional features from later versions. This includes some +experimental, incomplete support for UTF-8 encoded strings. Details of exactly +what is and what is not supported are given below. </P> <P> PCRE has its own native API, which is described in this document. There is also @@ -117,12 +127,20 @@ use these to include support for different releases. </P> <P> The functions <B>pcre_compile()</B>, <B>pcre_study()</B>, and <B>pcre_exec()</B> -are used for compiling and matching regular expressions, while -<B>pcre_copy_substring()</B>, <B>pcre_get_substring()</B>, and +are used for compiling and matching regular expressions. A sample program that +demonstrates the simplest way of using them is given in the file +<I>pcredemo.c</I>. The last section of this man page describes how to run it. +</P> +<P> +The functions <B>pcre_copy_substring()</B>, <B>pcre_get_substring()</B>, and <B>pcre_get_substring_list()</B> are convenience functions for extracting -captured substrings from a matched subject string. The function -<B>pcre_maketables()</B> is used (optionally) to build a set of character tables -in the current locale for passing to <B>pcre_compile()</B>. +captured substrings from a matched subject string; <B>pcre_free_substring()</B> +and <B>pcre_free_substring_list()</B> are also provided, to free the memory used +for extracted strings. +</P> +<P> +The function <B>pcre_maketables()</B> is used (optionally) to build a set of +character tables in the current locale for passing to <B>pcre_compile()</B>. </P> <P> The function <B>pcre_fullinfo()</B> is used to find out information about a @@ -153,11 +171,16 @@ the same compiled pattern can safely be used by several threads at once. The function <B>pcre_compile()</B> is called to compile a pattern into an internal form. The pattern is a C string terminated by a binary zero, and is passed in the argument <I>pattern</I>. A pointer to a single block of memory -that is obtained via <B>pcre_malloc</B> is returned. This contains the -compiled code and related data. The <B>pcre</B> type is defined for this for -convenience, but in fact <B>pcre</B> is just a typedef for <B>void</B>, since the -contents of the block are not externally defined. It is up to the caller to -free the memory when it is no longer required. +that is obtained via <B>pcre_malloc</B> is returned. This contains the compiled +code and related data. The <B>pcre</B> type is defined for the returned block; +this is a typedef for a structure whose contents are not externally defined. It +is up to the caller to free the memory when it is no longer required. +</P> +<P> +Although the compiled code of a PCRE regex is relocatable, that is, it does not +depend on memory location, the complete <B>pcre</B> data block is not +fully relocatable, because it contains a copy of the <I>tableptr</I> argument, +which is an address (see below). </P> <P> The size of a compiled pattern is roughly proportional to the length of the @@ -191,6 +214,22 @@ locale. Otherwise, <I>tableptr</I> must be the result of a call to <B>pcre_maketables()</B>. See the section on locale support below. </P> <P> +This code fragment shows a typical straightforward call to <B>pcre_compile()</B>: +</P> +<P> +<PRE> + pcre *re; + const char *error; + int erroffset; + re = pcre_compile( + "^A.*Z", /* the pattern */ + 0, /* default options */ + &error, /* for error message */ + &erroffset, /* for error offset */ + NULL); /* use default character tables */ +</PRE> +</P> +<P> The following option bits are defined in the header file: </P> <P> @@ -297,15 +336,27 @@ This option inverts the "greediness" of the quantifiers so that they are not greedy by default, but become greedy if followed by "?". It is not compatible with Perl. It can also be set by a (?U) option setting within the pattern. </P> +<P> +<PRE> + PCRE_UTF8 +</PRE> +</P> +<P> +This option causes PCRE to regard both the pattern and the subject as strings +of UTF-8 characters instead of just byte strings. However, it is available only +if PCRE has been built to include UTF-8 support. If not, the use of this option +provokes an error. Support for UTF-8 is new, experimental, and incomplete. +Details of exactly what it entails are given below. +</P> <LI><A NAME="SEC6" HREF="#TOC1">STUDYING A PATTERN</A> <P> When a pattern is going to be used several times, it is worth spending more time analyzing it in order to speed up the time taken for matching. The function <B>pcre_study()</B> takes a pointer to a compiled pattern as its first -argument, and returns a pointer to a <B>pcre_extra</B> block (another <B>void</B> -typedef) containing additional information about the pattern; this can be -passed to <B>pcre_exec()</B>. If no additional information is available, NULL -is returned. +argument, and returns a pointer to a <B>pcre_extra</B> block (another typedef +for a structure with hidden contents) containing additional information about +the pattern; this can be passed to <B>pcre_exec()</B>. If no additional +information is available, NULL is returned. </P> <P> The second argument contains option bits. At present, no options are defined @@ -317,6 +368,18 @@ studying succeeds (even if no data is returned), the variable it points to is set to NULL. Otherwise it points to a textual error message. </P> <P> +This is a typical call to <B>pcre_study</B>(): +</P> +<P> +<PRE> + pcre_extra *pe; + pe = pcre_study( + re, /* result of pcre_compile() */ + 0, /* no options exist */ + &error); /* set to NULL or points to a message */ +</PRE> +</P> +<P> At present, studying a pattern is useful only for non-anchored patterns that do not have a single fixed starting character. A bitmap of possible starting characters is created. @@ -376,6 +439,21 @@ the following negative numbers: </PRE> </P> <P> +Here is a typical call of <B>pcre_fullinfo()</B>, to obtain the length of the +compiled pattern: +</P> +<P> +<PRE> + int rc; + unsigned long int length; + rc = pcre_fullinfo( + re, /* result of pcre_compile() */ + pe, /* result of pcre_study(), or NULL */ + PCRE_INFO_SIZE, /* what is required */ + &length); /* where to put the data */ +</PRE> +</P> +<P> The possible values for the third argument are defined in <B>pcre.h</B>, and are as follows: </P> @@ -386,7 +464,7 @@ as follows: </P> <P> Return a copy of the options with which the pattern was compiled. The fourth -argument should point to au <B>unsigned long int</B> variable. These option bits +argument should point to an <B>unsigned long int</B> variable. These option bits are those specified in the call to <B>pcre_compile()</B>, modified by any top-level option settings within the pattern itself, and with the PCRE_ANCHORED bit forcibly set if the form of the pattern implies that it can match only at @@ -501,6 +579,24 @@ pattern has been studied, the result of the study should be passed in the <I>extra</I> argument. Otherwise this must be NULL. </P> <P> +Here is an example of a simple call to <B>pcre_exec()</B>: +</P> +<P> +<PRE> + int rc; + int ovector[30]; + rc = pcre_exec( + re, /* result of pcre_compile() */ + NULL, /* we didn't study the pattern */ + "some string", /* the subject string */ + 11, /* the length of the subject string */ + 0, /* start at offset 0 in the subject */ + 0, /* default options */ + ovector, /* vector for substring information */ + 30); /* number of elements in the vector */ +</PRE> +</P> +<P> The PCRE_ANCHORED option can be passed in the <I>options</I> argument, whose unused bits must be zero. However, if a pattern was compiled with PCRE_ANCHORED, or turned out to be anchored by virtue of its contents, it @@ -561,9 +657,9 @@ below) and trying an ordinary match again. <P> The subject string is passed as a pointer in <I>subject</I>, a length in <I>length</I>, and a starting offset in <I>startoffset</I>. Unlike the pattern -string, it may contain binary zero characters. When the starting offset is -zero, the search for a match starts at the beginning of the subject, and this -is by far the most common case. +string, the subject may contain binary zero characters. When the starting +offset is zero, the search for a match starts at the beginning of the subject, +and this is by far the most common case. </P> <P> A non-zero starting offset is useful when searching for another match in the @@ -743,7 +839,7 @@ extract a single substring, whose number is given as <I>stringnumber</I>. A value of zero extracts the substring that matched the entire pattern, while higher values extract the captured substrings. For <B>pcre_copy_substring()</B>, the string is placed in <I>buffer</I>, whose length is given by -<I>buffersize</I>, while for <B>pcre_get_substring()</B> a new block of store is +<I>buffersize</I>, while for <B>pcre_get_substring()</B> a new block of memory is obtained via <B>pcre_malloc</B>, and its address is returned via <I>stringptr</I>. The yield of the function is the length of the string, not including the terminating zero, or one of @@ -789,14 +885,26 @@ string. This can be distinguished from a genuine zero-length substring by inspecting the appropriate offset in <I>ovector</I>, which is negative for unset substrings. </P> +<P> +The two convenience functions <B>pcre_free_substring()</B> and +<B>pcre_free_substring_list()</B> can be used to free the memory returned by +a previous call of <B>pcre_get_substring()</B> or +<B>pcre_get_substring_list()</B>, respectively. They do nothing more than call +the function pointed to by <B>pcre_free</B>, which of course could be called +directly from a C program. However, PCRE is used in some situations where it is +linked via a special interface to another programming language which cannot use +<B>pcre_free</B> directly; it is for these cases that the functions are +provided. +</P> <LI><A NAME="SEC11" HREF="#TOC1">LIMITATIONS</A> <P> There are some size limitations in PCRE but it is hoped that they will never in practice be relevant. The maximum length of a compiled pattern is 65539 (sic) bytes. All values in repeating quantifiers must be less than 65536. -The maximum number of capturing subpatterns is 99. -The maximum number of all parenthesized subpatterns, including capturing +There maximum number of capturing subpatterns is 65535. +There is no limit to the number of non-capturing subpatterns, but the maximum +depth of nesting of all kinds of parenthesized subpattern, including capturing subpatterns, assertions, and other types of subpattern, is 200. </P> <P> @@ -908,8 +1016,15 @@ The syntax and semantics of the regular expressions supported by PCRE are described below. Regular expressions are also described in the Perl documentation and in a number of other books, some of which have copious examples. Jeffrey Friedl's "Mastering Regular Expressions", published by -O'Reilly (ISBN 1-56592-257), covers them in great detail. The description -here is intended as reference documentation. +O'Reilly (ISBN 1-56592-257), covers them in great detail. +</P> +<P> +The description here is intended as reference documentation. The basic +operation of PCRE is on strings of bytes. However, there is the beginnings of +some support for UTF-8 character strings. To use this support you must +configure PCRE to include it, and then call <B>pcre_compile()</B> with the +PCRE_UTF8 option. How this affects the pattern matching is described in the +final section of this document. </P> <P> A regular expression is a pattern that is matched against a subject string from @@ -1180,7 +1295,7 @@ PCRE_MULTILINE is set. <P> Note that the sequences \A, \Z, and \z can be used to match the start and end of the subject in both modes, and if all branches of a pattern start with -\A is it always anchored, whether PCRE_MULTILINE is set or not. +\A it is always anchored, whether PCRE_MULTILINE is set or not. </P> <LI><A NAME="SEC16" HREF="#TOC1">FULL STOP (PERIOD, DOT)</A> <P> @@ -1305,7 +1420,7 @@ negation, which is indicated by a ^ character after the colon. For example, </PRE> </P> <P> -matches "1", "2", or any non-digit. PCRE (and Perl) also recogize the POSIX +matches "1", "2", or any non-digit. PCRE (and Perl) also recognize the POSIX syntax [.ch.] and [=ch=] where "ch" is a "collating element", but these are not supported, and an error is given if they are encountered. </P> @@ -1437,7 +1552,7 @@ For example, if the string "the red king" is matched against the pattern </P> <P> the captured substrings are "red king", "red", and "king", and are numbered 1, -2, and 3. +2, and 3, respectively. </P> <P> The fact that plain parentheses fulfil two functions is not always helpful. @@ -1576,7 +1691,7 @@ to the string </PRE> </P> <P> -fails, because it matches the entire string due to the greediness of the .* +fails, because it matches the entire string owing to the greediness of the .* item. </P> <P> @@ -1718,7 +1833,7 @@ example, the pattern </PRE> </P> <P> -matches any number of "a"s and also "aba", "ababaa" etc. At each iteration of +matches any number of "a"s and also "aba", "ababbaa" etc. At each iteration of the subpattern, the back reference matches the character string corresponding to the previous iteration. In order for this to work, the pattern must be such that the first iteration does not need to match the back reference. This can be @@ -2033,9 +2148,10 @@ subpattern, a compile-time error occurs. <P> There are two kinds of condition. If the text between the parentheses consists of a sequence of digits, the condition is satisfied if the capturing subpattern -of that number has previously matched. Consider the following pattern, which -contains non-significant white space to make it more readable (assume the -PCRE_EXTENDED option) and to divide it into three parts for ease of discussion: +of that number has previously matched. The number must be greater than zero. +Consider the following pattern, which contains non-significant white space to +make it more readable (assume the PCRE_EXTENDED option) and to divide it into +three parts for ease of discussion: </P> <P> <PRE> @@ -2240,7 +2356,302 @@ with the pattern above. The former gives a failure almost instantly when applied to a whole line of "a" characters, whereas the latter takes an appreciable time with strings longer than about 20 characters. </P> -<LI><A NAME="SEC30" HREF="#TOC1">AUTHOR</A> +<LI><A NAME="SEC30" HREF="#TOC1">UTF-8 SUPPORT</A> +<P> +Starting at release 3.3, PCRE has some support for character strings encoded +in the UTF-8 format. This is incomplete, and is regarded as experimental. In +order to use it, you must configure PCRE to include UTF-8 support in the code, +and, in addition, you must call <B>pcre_compile()</B> with the PCRE_UTF8 option +flag. When you do this, both the pattern and any subject strings that are +matched against it are treated as UTF-8 strings instead of just strings of +bytes, but only in the cases that are mentioned below. +</P> +<P> +If you compile PCRE with UTF-8 support, but do not use it at run time, the +library will be a bit bigger, but the additional run time overhead is limited +to testing the PCRE_UTF8 flag in several places, so should not be very large. +</P> +<P> +PCRE assumes that the strings it is given contain valid UTF-8 codes. It does +not diagnose invalid UTF-8 strings. If you pass invalid UTF-8 strings to PCRE, +the results are undefined. +</P> +<P> +Running with PCRE_UTF8 set causes these changes in the way PCRE works: +</P> +<P> +1. In a pattern, the escape sequence \x{...}, where the contents of the braces +is a string of hexadecimal digits, is interpreted as a UTF-8 character whose +code number is the given hexadecimal number, for example: \x{1234}. This +inserts from one to six literal bytes into the pattern, using the UTF-8 +encoding. If a non-hexadecimal digit appears between the braces, the item is +not recognized. +</P> +<P> +2. The original hexadecimal escape sequence, \xhh, generates a two-byte UTF-8 +character if its value is greater than 127. +</P> +<P> +3. Repeat quantifiers are NOT correctly handled if they follow a multibyte +character. For example, \x{100}* and \xc3+ do not work. If you want to +repeat such characters, you must enclose them in non-capturing parentheses, +for example (?:\x{100}), at present. +</P> +<P> +4. The dot metacharacter matches one UTF-8 character instead of a single byte. +</P> +<P> +5. Unlike literal UTF-8 characters, the dot metacharacter followed by a +repeat quantifier does operate correctly on UTF-8 characters instead of +single bytes. +</P> +<P> +4. Although the \x{...} escape is permitted in a character class, characters +whose values are greater than 255 cannot be included in a class. +</P> +<P> +5. A class is matched against a UTF-8 character instead of just a single byte, +but it can match only characters whose values are less than 256. Characters +with greater values always fail to match a class. +</P> +<P> +6. Repeated classes work correctly on multiple characters. +</P> +<P> +7. Classes containing just a single character whose value is greater than 127 +(but less than 256), for example, [\x80] or [^\x{93}], do not work because +these are optimized into single byte matches. In the first case, of course, +the class brackets are just redundant. +</P> +<P> +8. Lookbehind assertions move backwards in the subject by a fixed number of +characters instead of a fixed number of bytes. Simple cases have been tested +to work correctly, but there may be hidden gotchas herein. +</P> +<P> +9. The character types such as \d and \w do not work correctly with UTF-8 +characters. They continue to test a single byte. +</P> +<P> +10. Anything not explicitly mentioned here continues to work in bytes rather +than in characters. +</P> +<P> +The following UTF-8 features of Perl 5.6 are not implemented: +</P> +<P> +1. The escape sequence \C to match a single byte. +</P> +<P> +2. The use of Unicode tables and properties and escapes \p, \P, and \X. +</P> +<LI><A NAME="SEC31" HREF="#TOC1">SAMPLE PROGRAM</A> +<P> +The code below is a simple, complete demonstration program, to get you started +with using PCRE. This code is also supplied in the file <I>pcredemo.c</I> in the +PCRE distribution. +</P> +<P> +The program compiles the regular expression that is its first argument, and +matches it against the subject string in its second argument. No options are +set, and default character tables are used. If matching succeeds, the program +outputs the portion of the subject that matched, together with the contents of +any captured substrings. +</P> +<P> +On a Unix system that has PCRE installed in <I>/usr/local</I>, you can compile +the demonstration program using a command like this: +</P> +<P> +<PRE> + gcc -o pcredemo pcredemo.c -I/usr/local/include -L/usr/local/lib -lpcre +</PRE> +</P> +<P> +Then you can run simple tests like this: +</P> +<P> +<PRE> + ./pcredemo 'cat|dog' 'the cat sat on the mat' +</PRE> +</P> +<P> +Note that there is a much more comprehensive test program, called +<B>pcretest</B>, which supports many more facilities for testing regular +expressions. The <B>pcredemo</B> program is provided as a simple coding example. +</P> +<P> +On some operating systems (e.g. Solaris) you may get an error like this when +you try to run <B>pcredemo</B>: +</P> +<P> +<PRE> + ld.so.1: a.out: fatal: libpcre.so.0: open failed: No such file or directory +</PRE> +</P> +<P> +This is caused by the way shared library support works on those systems. You +need to add +</P> +<P> +<PRE> + -R/usr/local/lib +</PRE> +</P> +<P> +to the compile command to get round this problem. Here's the code: +</P> +<P> +<PRE> + #include <stdio.h> + #include <string.h> + #include <pcre.h> +</PRE> +</P> +<P> +<PRE> + #define OVECCOUNT 30 /* should be a multiple of 3 */ +</PRE> +</P> +<P> +<PRE> + int main(int argc, char **argv) + { + pcre *re; + const char *error; + int erroffset; + int ovector[OVECCOUNT]; + int rc, i; +</PRE> +</P> +<P> +<PRE> + if (argc != 3) + { + printf("Two arguments required: a regex and a " + "subject string\n"); + return 1; + } +</PRE> +</P> +<P> +<PRE> + /* Compile the regular expression in the first argument */ +</PRE> +</P> +<P> +<PRE> + re = pcre_compile( + argv[1], /* the pattern */ + 0, /* default options */ + &error, /* for error message */ + &erroffset, /* for error offset */ + NULL); /* use default character tables */ +</PRE> +</P> +<P> +<PRE> + /* Compilation failed: print the error message and exit */ +</PRE> +</P> +<P> +<PRE> + if (re == NULL) + { + printf("PCRE compilation failed at offset %d: %s\n", + erroffset, error); + return 1; + } +</PRE> +</P> +<P> +<PRE> + /* Compilation succeeded: match the subject in the second + argument */ +</PRE> +</P> +<P> +<PRE> + rc = pcre_exec( + re, /* the compiled pattern */ + NULL, /* we didn't study the pattern */ + argv[2], /* the subject string */ + (int)strlen(argv[2]), /* the length of the subject */ + 0, /* start at offset 0 in the subject */ + 0, /* default options */ + ovector, /* vector for substring information */ + OVECCOUNT); /* number of elements in the vector */ +</PRE> +</P> +<P> +<PRE> + /* Matching failed: handle error cases */ +</PRE> +</P> +<P> +<PRE> + if (rc < 0) + { + switch(rc) + { + case PCRE_ERROR_NOMATCH: printf("No match\n"); break; + /* + Handle other special cases if you like + */ + default: printf("Matching error %d\n", rc); break; + } + return 1; + } +</PRE> +</P> +<P> +<PRE> + /* Match succeded */ +</PRE> +</P> +<P> +<PRE> + printf("Match succeeded\n"); +</PRE> +</P> +<P> +<PRE> + /* The output vector wasn't big enough */ +</PRE> +</P> +<P> +<PRE> + if (rc == 0) + { + rc = OVECCOUNT/3; + printf("ovector only has room for %d captured " + substrings\n", rc - 1); + } +</PRE> +</P> +<P> +<PRE> + /* Show substrings stored in the output vector */ +</PRE> +</P> +<P> +<PRE> + for (i = 0; i < rc; i++) + { + char *substring_start = argv[2] + ovector[2*i]; + int substring_length = ovector[2*i+1] - ovector[2*i]; + printf("%2d: %.*s\n", i, substring_length, + substring_start); + } +</PRE> +</P> +<P> +<PRE> + return 0; + } +</PRE> +</P> +<LI><A NAME="SEC32" HREF="#TOC1">AUTHOR</A> <P> Philip Hazel <ph10@cam.ac.uk> <BR> @@ -2253,6 +2664,6 @@ Cambridge CB2 3QG, England. Phone: +44 1223 334714 </P> <P> -Last updated: 27 January 2000 +Last updated: 15 August 2001 <BR> -Copyright (c) 1997-2000 University of Cambridge. +Copyright (c) 1997-2001 University of Cambridge. diff --git a/srclib/pcre/doc/pcre.txt b/srclib/pcre/doc/pcre.txt index b8106e4457..95f148f3de 100644 --- a/srclib/pcre/doc/pcre.txt +++ b/srclib/pcre/doc/pcre.txt @@ -28,6 +28,10 @@ SYNOPSIS int pcre_get_substring_list(const char *subject, int *ovector, int stringcount, const char ***listptr); + void pcre_free_substring(const char *stringptr); + + void pcre_free_substring_list(const char **stringptr); + const unsigned char *pcre_maketables(void); int pcre_fullinfo(const pcre *code, const pcre_extra *extra, @@ -48,9 +52,12 @@ DESCRIPTION The PCRE library is a set of functions that implement regu- lar expression pattern matching using the same syntax and semantics as Perl 5, with just a few differences (see + below). The current implementation corresponds to Perl - 5.005, with some additional features from the Perl develop- - ment release. + 5.005, with some additional features from later versions. + This includes some experimental, incomplete support for + UTF-8 encoded strings. Details of exactly what is and what + is not supported are given below. PCRE has its own native API, which is described in this document. There is also a set of wrapper functions that @@ -67,13 +74,21 @@ DESCRIPTION releases. The functions pcre_compile(), pcre_study(), and pcre_exec() - are used for compiling and matching regular expressions, - while pcre_copy_substring(), pcre_get_substring(), and - pcre_get_substring_list() are convenience functions for + are used for compiling and matching regular expressions. A + sample program that demonstrates the simplest way of using + them is given in the file pcredemo.c. The last section of + this man page describes how to run it. + + The functions pcre_copy_substring(), pcre_get_substring(), + and pcre_get_substring_list() are convenience functions for extracting captured substrings from a matched subject - string. The function pcre_maketables() is used (optionally) - to build a set of character tables in the current locale for - passing to pcre_compile(). + string; pcre_free_substring() and pcre_free_substring_list() + are also provided, to free the memory used for extracted + strings. + + The function pcre_maketables() is used (optionally) to build + a set of character tables in the current locale for passing + to pcre_compile(). The function pcre_fullinfo() is used to find out information about a compiled pattern; pcre_info() is an obsolete version @@ -103,18 +118,22 @@ MULTI-THREADING - COMPILING A PATTERN The function pcre_compile() is called to compile a pattern into an internal form. The pattern is a C string terminated by a binary zero, and is passed in the argument pattern. A pointer to a single block of memory that is obtained via pcre_malloc is returned. This contains the compiled code and - related data. The pcre type is defined for this for conveni- - ence, but in fact pcre is just a typedef for void, since the - contents of the block are not externally defined. It is up - to the caller to free the memory when it is no longer - required. + related data. The pcre type is defined for the returned + block; this is a typedef for a structure whose contents are + not externally defined. It is up to the caller to free the + memory when it is no longer required. + + Although the compiled code of a PCRE regex is relocatable, + that is, it does not depend on memory location, the complete + pcre data block is not fully relocatable, because it con- + tains a copy of the tableptr argument, which is an address + (see below). The size of a compiled pattern is roughly proportional to the length of the pattern string, except that each character @@ -149,6 +168,19 @@ COMPILING A PATTERN must be the result of a call to pcre_maketables(). See the section on locale support below. + This code fragment shows a typical straightforward call to + pcre_compile(): + + pcre *re; + const char *error; + int erroffset; + re = pcre_compile( + "^A.*Z", /* the pattern */ + 0, /* default options */ + &error, /* for error message */ + &erroffset, /* for error offset */ + NULL); /* use default character tables */ + The following option bits are defined in the header file: PCRE_ANCHORED @@ -235,6 +267,16 @@ COMPILING A PATTERN followed by "?". It is not compatible with Perl. It can also be set by a (?U) option setting within the pattern. + PCRE_UTF8 + + This option causes PCRE to regard both the pattern and the + subject as strings of UTF-8 characters instead of just byte + strings. However, it is available only if PCRE has been + built to include UTF-8 support. If not, the use of this + option provokes an error. Support for UTF-8 is new, experi- + mental, and incomplete. Details of exactly what it entails + are given below. + STUDYING A PATTERN @@ -242,10 +284,11 @@ STUDYING A PATTERN worth spending more time analyzing it in order to speed up the time taken for matching. The function pcre_study() takes a pointer to a compiled pattern as its first argument, and - returns a pointer to a pcre_extra block (another void - typedef) containing additional information about the pat- - tern; this can be passed to pcre_exec(). If no additional - information is available, NULL is returned. + returns a pointer to a pcre_extra block (another typedef for + a structure with hidden contents) containing additional + information about the pattern; this can be passed to + pcre_exec(). If no additional information is available, NULL + is returned. The second argument contains option bits. At present, no options are defined for pcre_study(), and this argument @@ -256,6 +299,14 @@ STUDYING A PATTERN the variable it points to is set to NULL. Otherwise it points to a textual error message. + This is a typical call to pcre_study(): + + pcre_extra *pe; + pe = pcre_study( + re, /* result of pcre_compile() */ + 0, /* no options exist */ + &error); /* set to NULL or points to a message */ + At present, studying a pattern is useful only for non- anchored patterns that do not have a single fixed starting character. A bitmap of possible starting characters is @@ -316,13 +367,24 @@ INFORMATION ABOUT A PATTERN PCRE_ERROR_BADMAGIC the "magic number" was not found PCRE_ERROR_BADOPTION the value of what was invalid + Here is a typical call of pcre_fullinfo(), to obtain the + length of the compiled pattern: + + int rc; + unsigned long int length; + rc = pcre_fullinfo( + re, /* result of pcre_compile() */ + pe, /* result of pcre_study(), or NULL */ + PCRE_INFO_SIZE, /* what is required */ + &length); /* where to put the data */ + The possible values for the third argument are defined in pcre.h, and are as follows: PCRE_INFO_OPTIONS Return a copy of the options with which the pattern was com- - piled. The fourth argument should point to au unsigned long + piled. The fourth argument should point to an unsigned long int variable. These option bits are those specified in the call to pcre_compile(), modified by any top-level option settings within the pattern itself, and with the @@ -409,11 +471,34 @@ INFORMATION ABOUT A PATTERN MATCHING A PATTERN The function pcre_exec() is called to match a subject string + + + + + +SunOS 5.8 Last change: 9 + + + against a pre-compiled pattern, which is passed in the code argument. If the pattern has been studied, the result of the study should be passed in the extra argument. Otherwise this must be NULL. + Here is an example of a simple call to pcre_exec(): + + int rc; + int ovector[30]; + rc = pcre_exec( + re, /* result of pcre_compile() */ + NULL, /* we didn't study the pattern */ + "some string", /* the subject string */ + 11, /* the length of the subject string */ + 0, /* start at offset 0 in the subject */ + 0, /* default options */ + ovector, /* vector for substring information */ + 30); /* number of elements in the vector */ + The PCRE_ANCHORED option can be passed in the options argu- ment, whose unused bits must be zero. However, if a pattern was compiled with PCRE_ANCHORED, or turned out to be @@ -464,10 +549,10 @@ MATCHING A PATTERN The subject string is passed as a pointer in subject, a length in length, and a starting offset in startoffset. - Unlike the pattern string, it may contain binary zero char- - acters. When the starting offset is zero, the search for a - match starts at the beginning of the subject, and this is by - far the most common case. + Unlike the pattern string, the subject may contain binary + zero characters. When the starting offset is zero, the + search for a match starts at the beginning of the subject, + and this is by far the most common case. A non-zero starting offset is useful when searching for another match in the same subject by calling pcre_exec() @@ -603,6 +688,7 @@ MATCHING A PATTERN + EXTRACTING CAPTURED SUBSTRINGS Captured substrings can be accessed directly by using the offsets returned by pcre_exec() in ovector. For convenience, @@ -631,7 +717,7 @@ EXTRACTING CAPTURED SUBSTRINGS the entire pattern, while higher values extract the captured substrings. For pcre_copy_substring(), the string is placed in buffer, whose length is given by buffersize, while for - pcre_get_substring() a new block of store is obtained via + pcre_get_substring() a new block of memory is obtained via pcre_malloc, and its address is returned via stringptr. The yield of the function is the length of the string, not including the terminating zero, or one of @@ -665,6 +751,16 @@ EXTRACTING CAPTURED SUBSTRINGS inspecting the appropriate offset in ovector, which is nega- tive for unset substrings. + The two convenience functions pcre_free_substring() and + pcre_free_substring_list() can be used to free the memory + returned by a previous call of pcre_get_substring() or + pcre_get_substring_list(), respectively. They do nothing + more than call the function pointed to by pcre_free, which + of course could be called directly from a C program. How- + ever, PCRE is used in some situations where it is linked via + a special interface to another programming language which + cannot use pcre_free directly; it is for these cases that + the functions are provided. @@ -672,10 +768,12 @@ LIMITATIONS There are some size limitations in PCRE but it is hoped that they will never in practice be relevant. The maximum length of a compiled pattern is 65539 (sic) bytes. All values in - repeating quantifiers must be less than 65536. The maximum - number of capturing subpatterns is 99. The maximum number - of all parenthesized subpatterns, including capturing sub- - patterns, assertions, and other types of subpattern, is 200. + repeating quantifiers must be less than 65536. There max- + imum number of capturing subpatterns is 65535. There is no + limit to the number of non-capturing subpatterns, but the + maximum depth of nesting of all kinds of parenthesized sub- + pattern, including capturing subpatterns, assertions, and + other types of subpattern, is 200. The maximum length of a subject string is the largest posi- tive number that an integer variable can hold. However, PCRE @@ -733,6 +831,7 @@ DIFFERENCES FROM PERL (?p{code}) constructions. However, there is some experimen- tal support for recursive patterns using the non-Perl item (?R). + 8. There are at the time of writing some oddities in Perl 5.005_02 concerned with the settings of captured strings when part of a pattern is repeated. For example, matching @@ -785,11 +884,17 @@ REGULAR EXPRESSION DETAILS The syntax and semantics of the regular expressions sup- ported by PCRE are described below. Regular expressions are also described in the Perl documentation and in a number of - other books, some of which have copious examples. Jeffrey Friedl's "Mastering Regular Expressions", published by - O'Reilly (ISBN 1-56592-257), covers them in great detail. + O'Reilly (ISBN 1-56592-257), covers them in great detail. + The description here is intended as reference documentation. + The basic operation of PCRE is on strings of bytes. However, + there is the beginnings of some support for UTF-8 character + strings. To use this support you must configure PCRE to + include it, and then call pcre_compile() with the PCRE_UTF8 + option. How this affects the pattern matching is described + in the final section of this document. A regular expression is a pattern that is matched against a subject string from left to right. Most characters stand for @@ -844,6 +949,7 @@ BACKSLASH The backslash character has several uses. Firstly, if it is followed by a non-alphameric character, it takes away any special meaning that character may have. This use of + backslash as an escape character applies both inside and outside character classes. @@ -1047,7 +1153,7 @@ CIRCUMFLEX AND DOLLAR Note that the sequences \A, \Z, and \z can be used to match the start and end of the subject in both modes, and if all - branches of a pattern start with \A is it always anchored, + branches of a pattern start with \A it is always anchored, whether PCRE_MULTILINE is set or not. @@ -1174,7 +1280,7 @@ POSIX CHARACTER CLASSES [12[:^digit:]] matches "1", "2", or any non-digit. PCRE (and Perl) also - recogize the POSIX syntax [.ch.] and [=ch=] where "ch" is a + recognize the POSIX syntax [.ch.] and [=ch=] where "ch" is a "collating element", but these are not supported, and an error is given if they are encountered. @@ -1293,7 +1399,7 @@ SUBPATTERNS the ((red|white) (king|queen)) the captured substrings are "red king", "red", and "king", - and are numbered 1, 2, and 3. + and are numbered 1, 2, and 3, respectively. The fact that plain parentheses fulfil two functions is not always helpful. There are often times when a grouping sub- @@ -1364,7 +1470,6 @@ REPETITION one that does not match the syntax of a quantifier, is taken as a literal character. For example, {,6} is not a quantif- ier, but a literal string of four characters. - The quantifier {0} is permitted, causing the expression to behave as if the previous item and the quantifier were not present. @@ -1403,7 +1508,7 @@ REPETITION /* first command */ not comment /* second comment */ - fails, because it matches the entire string due to the + fails, because it matches the entire string owing to the greediness of the .* item. However, if a quantifier is followed by a question mark, it @@ -1469,6 +1574,14 @@ REPETITION BACK REFERENCES Outside a character class, a backslash followed by a digit greater than 0 (and possibly further digits) is a back + + + + +SunOS 5.8 Last change: 30 + + + reference to a capturing subpattern earlier (i.e. to its left) in the pattern, provided there have been that many previous capturing left parentheses. @@ -1517,12 +1630,12 @@ BACK REFERENCES A back reference that occurs inside the parentheses to which it refers fails when the subpattern is first used, so, for example, (a\1) never matches. However, such references can - be useful inside repeated subpatterns. For example, the - pattern + be useful inside repeated subpatterns. For example, the pat- + tern (a|b\1)+ - matches any number of "a"s and also "aba", "ababaa" etc. At + matches any number of "a"s and also "aba", "ababbaa" etc. At each iteration of the subpattern, the back reference matches the character string corresponding to the previous itera- tion. In order for this to work, the pattern must be such @@ -1778,10 +1891,11 @@ CONDITIONAL SUBPATTERNS There are two kinds of condition. If the text between the parentheses consists of a sequence of digits, the condition is satisfied if the capturing subpattern of that number has - previously matched. Consider the following pattern, which - contains non-significant white space to make it more read- - able (assume the PCRE_EXTENDED option) and to divide it into - three parts for ease of discussion: + previously matched. The number must be greater than zero. + Consider the following pattern, which contains non- + significant white space to make it more readable (assume the + PCRE_EXTENDED option) and to divide it into three parts for + ease of discussion: ( \( )? [^()]+ (?(1) \) ) @@ -1966,6 +2080,230 @@ PERFORMANCE +UTF-8 SUPPORT + Starting at release 3.3, PCRE has some support for character + strings encoded in the UTF-8 format. This is incomplete, and + is regarded as experimental. In order to use it, you must + configure PCRE to include UTF-8 support in the code, and, in + addition, you must call pcre_compile() with the PCRE_UTF8 + option flag. When you do this, both the pattern and any sub- + ject strings that are matched against it are treated as + UTF-8 strings instead of just strings of bytes, but only in + the cases that are mentioned below. + + If you compile PCRE with UTF-8 support, but do not use it at + run time, the library will be a bit bigger, but the addi- + tional run time overhead is limited to testing the PCRE_UTF8 + flag in several places, so should not be very large. + + PCRE assumes that the strings it is given contain valid + UTF-8 codes. It does not diagnose invalid UTF-8 strings. If + you pass invalid UTF-8 strings to PCRE, the results are + undefined. + + Running with PCRE_UTF8 set causes these changes in the way + PCRE works: + + 1. In a pattern, the escape sequence \x{...}, where the + contents of the braces is a string of hexadecimal digits, is + interpreted as a UTF-8 character whose code number is the + given hexadecimal number, for example: \x{1234}. This + inserts from one to six literal bytes into the pattern, + using the UTF-8 encoding. If a non-hexadecimal digit appears + between the braces, the item is not recognized. + + 2. The original hexadecimal escape sequence, \xhh, generates + a two-byte UTF-8 character if its value is greater than 127. + + 3. Repeat quantifiers are NOT correctly handled if they fol- + low a multibyte character. For example, \x{100}* and \xc3+ + do not work. If you want to repeat such characters, you must + enclose them in non-capturing parentheses, for example + (?:\x{100}), at present. + + 4. The dot metacharacter matches one UTF-8 character instead + of a single byte. + + 5. Unlike literal UTF-8 characters, the dot metacharacter + followed by a repeat quantifier does operate correctly on + UTF-8 characters instead of single bytes. + + 4. Although the \x{...} escape is permitted in a character + class, characters whose values are greater than 255 cannot + be included in a class. + + 5. A class is matched against a UTF-8 character instead of + just a single byte, but it can match only characters whose + values are less than 256. Characters with greater values + always fail to match a class. + + 6. Repeated classes work correctly on multiple characters. + + 7. Classes containing just a single character whose value is + greater than 127 (but less than 256), for example, [\x80] or + [^\x{93}], do not work because these are optimized into sin- + gle byte matches. In the first case, of course, the class + brackets are just redundant. + + 8. Lookbehind assertions move backwards in the subject by a + fixed number of characters instead of a fixed number of + bytes. Simple cases have been tested to work correctly, but + there may be hidden gotchas herein. + + 9. The character types such as \d and \w do not work + correctly with UTF-8 characters. They continue to test a + single byte. + + 10. Anything not explicitly mentioned here continues to work + in bytes rather than in characters. + + The following UTF-8 features of Perl 5.6 are not imple- + mented: + + 1. The escape sequence \C to match a single byte. + + 2. The use of Unicode tables and properties and escapes \p, + \P, and \X. + + + +SAMPLE PROGRAM + The code below is a simple, complete demonstration program, + to get you started with using PCRE. This code is also sup- + plied in the file pcredemo.c in the PCRE distribution. + + The program compiles the regular expression that is its + first argument, and matches it against the subject string in + its second argument. No options are set, and default charac- + ter tables are used. If matching succeeds, the program out- + puts the portion of the subject that matched, together with + the contents of any captured substrings. + + On a Unix system that has PCRE installed in /usr/local, you + can compile the demonstration program using a command like + this: + + gcc -o pcredemo pcredemo.c -I/usr/local/include + -L/usr/local/lib -lpcre + + Then you can run simple tests like this: + + ./pcredemo 'cat|dog' 'the cat sat on the mat' + + Note that there is a much more comprehensive test program, + called pcretest, which supports many more facilities for + testing regular expressions. The pcredemo program is pro- + vided as a simple coding example. + + On some operating systems (e.g. Solaris) you may get an + error like this when you try to run pcredemo: + + ld.so.1: a.out: fatal: libpcre.so.0: open failed: No such + file or directory + + This is caused by the way shared library support works on + those systems. You need to add + + -R/usr/local/lib + + to the compile command to get round this problem. Here's the + code: + + #include <stdio.h> + #include <string.h> + #include <pcre.h> + + #define OVECCOUNT 30 /* should be a multiple of 3 */ + + int main(int argc, char **argv) + { + pcre *re; + const char *error; + int erroffset; + int ovector[OVECCOUNT]; + int rc, i; + + if (argc != 3) + { + printf("Two arguments required: a regex and a " + "subject string\n"); + return 1; + } + + /* Compile the regular expression in the first argument */ + + re = pcre_compile( + argv[1], /* the pattern */ + 0, /* default options */ + &error, /* for error message */ + &erroffset, /* for error offset */ + NULL); /* use default character tables */ + + /* Compilation failed: print the error message and exit */ + + if (re == NULL) + { + printf("PCRE compilation failed at offset %d: %s\n", + erroffset, error); + return 1; + } + + /* Compilation succeeded: match the subject in the second + argument */ + + rc = pcre_exec( + re, /* the compiled pattern */ + NULL, /* we didn't study the pattern */ + argv[2], /* the subject string */ + (int)strlen(argv[2]), /* the length of the subject */ + 0, /* start at offset 0 in the subject */ + 0, /* default options */ + ovector, /* vector for substring information */ + OVECCOUNT); /* number of elements in the vector */ + + /* Matching failed: handle error cases */ + + if (rc < 0) + { + switch(rc) + { + case PCRE_ERROR_NOMATCH: printf("No match\n"); break; + /* + Handle other special cases if you like + */ + default: printf("Matching error %d\n", rc); break; + } + return 1; + } + + /* Match succeded */ + + printf("Match succeeded\n"); + + /* The output vector wasn't big enough */ + + if (rc == 0) + { + rc = OVECCOUNT/3; + printf("ovector only has room for %d captured " + substrings\n", rc - 1); + } + + /* Show substrings stored in the output vector */ + + for (i = 0; i < rc; i++) + { + char *substring_start = argv[2] + ovector[2*i]; + int substring_length = ovector[2*i+1] - ovector[2*i]; + printf("%2d: %.*s\n", i, substring_length, + substring_start); + } + + return 0; + } + + + AUTHOR Philip Hazel <ph10@cam.ac.uk> University Computing Service, @@ -1973,5 +2311,5 @@ AUTHOR Cambridge CB2 3QG, England. Phone: +44 1223 334714 - Last updated: 27 January 2000 - Copyright (c) 1997-2000 University of Cambridge. + Last updated: 15 August 2001 + Copyright (c) 1997-2001 University of Cambridge. diff --git a/srclib/pcre/doc/pcretest.txt b/srclib/pcre/doc/pcretest.txt index 0e6783af0c..0e13b6c6c5 100644 --- a/srclib/pcre/doc/pcretest.txt +++ b/srclib/pcre/doc/pcretest.txt @@ -1,217 +1,319 @@ -The pcretest program --------------------- +NAME + pcretest - a program for testing Perl-compatible regular + expressions. -This program is intended for testing PCRE, but it can also be used for -experimenting with regular expressions. -If it is given two filename arguments, it reads from the first and writes to -the second. If it is given only one filename argument, it reads from that file -and writes to stdout. Otherwise, it reads from stdin and writes to stdout, and -prompts for each line of input, using "re>" to prompt for regular expressions, -and "data>" to prompt for data lines. - -The program handles any number of sets of input on a single input file. Each -set starts with a regular expression, and continues with any number of data -lines to be matched against the pattern. An empty line signals the end of the -data lines, at which point a new regular expression is read. The regular -expressions are given enclosed in any non-alphameric delimiters other than -backslash, for example - - /(a|bc)x+yz/ - -White space before the initial delimiter is ignored. A regular expression may -be continued over several input lines, in which case the newline characters are -included within it. See the test input files in the testdata directory for many -examples. It is possible to include the delimiter within the pattern by -escaping it, for example - - /abc\/def/ - -If you do so, the escape and the delimiter form part of the pattern, but since -delimiters are always non-alphameric, this does not affect its interpretation. -If the terminating delimiter is immediately followed by a backslash, for -example, - - /abc/\ - -then a backslash is added to the end of the pattern. This is done to provide a -way of testing the error condition that arises if a pattern finishes with a -backslash, because - - /abc\/ - -is interpreted as the first line of a pattern that starts with "abc/", causing -pcretest to read the next line as a continuation of the regular expression. - -The pattern may be followed by i, m, s, or x to set the PCRE_CASELESS, -PCRE_MULTILINE, PCRE_DOTALL, or PCRE_EXTENDED options, respectively. For -example: - - /caseless/i - -These modifier letters have the same effect as they do in Perl. There are -others which set PCRE options that do not correspond to anything in Perl: /A, -/E, and /X set PCRE_ANCHORED, PCRE_DOLLAR_ENDONLY, and PCRE_EXTRA respectively. - -Searching for all possible matches within each subject string can be requested -by the /g or /G modifier. After finding a match, PCRE is called again to search -the remainder of the subject string. The difference between /g and /G is that -the former uses the startoffset argument to pcre_exec() to start searching at -a new point within the entire string (which is in effect what Perl does), -whereas the latter passes over a shortened substring. This makes a difference -to the matching process if the pattern begins with a lookbehind assertion -(including \b or \B). - -If any call to pcre_exec() in a /g or /G sequence matches an empty string, the -next call is done with the PCRE_NOTEMPTY and PCRE_ANCHORED flags set in order -to search for another, non-empty, match at the same point. If this second match -fails, the start offset is advanced by one, and the normal match is retried. -This imitates the way Perl handles such cases when using the /g modifier or the -split() function. - -There are a number of other modifiers for controlling the way pcretest -operates. - -The /+ modifier requests that as well as outputting the substring that matched -the entire pattern, pcretest should in addition output the remainder of the -subject string. This is useful for tests where the subject contains multiple -copies of the same substring. - -The /L modifier must be followed directly by the name of a locale, for example, - - /pattern/Lfr - -For this reason, it must be the last modifier letter. The given locale is set, -pcre_maketables() is called to build a set of character tables for the locale, -and this is then passed to pcre_compile() when compiling the regular -expression. Without an /L modifier, NULL is passed as the tables pointer; that -is, /L applies only to the expression on which it appears. - -The /I modifier requests that pcretest output information about the compiled -expression (whether it is anchored, has a fixed first character, and so on). It -does this by calling pcre_fullinfo() after compiling an expression, and -outputting the information it gets back. If the pattern is studied, the results -of that are also output. - -The /D modifier is a PCRE debugging feature, which also assumes /I. It causes -the internal form of compiled regular expressions to be output after -compilation. - -The /S modifier causes pcre_study() to be called after the expression has been -compiled, and the results used when the expression is matched. - -The /M modifier causes the size of memory block used to hold the compiled -pattern to be output. - -Finally, the /P modifier causes pcretest to call PCRE via the POSIX wrapper API -rather than its native API. When this is done, all other modifiers except /i, -/m, and /+ are ignored. REG_ICASE is set if /i is present, and REG_NEWLINE is -set if /m is present. The wrapper functions force PCRE_DOLLAR_ENDONLY always, -and PCRE_DOTALL unless REG_NEWLINE is set. - -Before each data line is passed to pcre_exec(), leading and trailing whitespace -is removed, and it is then scanned for \ escapes. The following are recognized: - - \a alarm (= BEL) - \b backspace - \e escape - \f formfeed - \n newline - \r carriage return - \t tab - \v vertical tab - \nnn octal character (up to 3 octal digits) - \xhh hexadecimal character (up to 2 hex digits) - - \A pass the PCRE_ANCHORED option to pcre_exec() - \B pass the PCRE_NOTBOL option to pcre_exec() - \Cdd call pcre_copy_substring() for substring dd after a successful match - (any decimal number less than 32) - \Gdd call pcre_get_substring() for substring dd after a successful match - (any decimal number less than 32) - \L call pcre_get_substringlist() after a successful match - \N pass the PCRE_NOTEMPTY option to pcre_exec() - \Odd set the size of the output vector passed to pcre_exec() to dd - (any number of decimal digits) - \Z pass the PCRE_NOTEOL option to pcre_exec() - -A backslash followed by anything else just escapes the anything else. If the -very last character is a backslash, it is ignored. This gives a way of passing -an empty line as data, since a real empty line terminates the data input. - -If /P was present on the regex, causing the POSIX wrapper API to be used, only -\B, and \Z have any effect, causing REG_NOTBOL and REG_NOTEOL to be passed to -regexec() respectively. - -When a match succeeds, pcretest outputs the list of captured substrings that -pcre_exec() returns, starting with number 0 for the string that matched the -whole pattern. Here is an example of an interactive pcretest run. - - $ pcretest - PCRE version 2.06 08-Jun-1999 - - re> /^abc(\d+)/ - data> abc123 - 0: abc123 - 1: 123 - data> xyz - No match - -If the strings contain any non-printing characters, they are output as \0x -escapes. If the pattern has the /+ modifier, then the output for substring 0 is -followed by the the rest of the subject string, identified by "0+" like this: - - re> /cat/+ - data> cataract - 0: cat - 0+ aract - -If the pattern has the /g or /G modifier, the results of successive matching -attempts are output in sequence, like this: - - re> /\Bi(\w\w)/g - data> Mississippi - 0: iss - 1: ss - 0: iss - 1: ss - 0: ipp - 1: pp - -"No match" is output only if the first match attempt fails. - -If any of \C, \G, or \L are present in a data line that is successfully -matched, the substrings extracted by the convenience functions are output with -C, G, or L after the string number instead of a colon. This is in addition to -the normal full list. The string length (that is, the return from the -extraction function) is given in parentheses after each string for \C and \G. - -Note that while patterns can be continued over several lines (a plain ">" -prompt is used for continuations), data lines may not. However newlines can be -included in data by means of the \n escape. - -If the -p option is given to pcretest, it is equivalent to adding /P to each -regular expression: the POSIX wrapper API is used to call PCRE. None of the -following flags has any effect in this case. - -If the option -d is given to pcretest, it is equivalent to adding /D to each -regular expression: the internal form is output after compilation. - -If the option -i is given to pcretest, it is equivalent to adding /I to each -regular expression: information about the compiled pattern is given after -compilation. - -If the option -m is given to pcretest, it outputs the size of each compiled -pattern after it has been compiled. It is equivalent to adding /M to each -regular expression. For compatibility with earlier versions of pcretest, -s is -a synonym for -m. - -If the -t option is given, each compile, study, and match is run 20000 times -while being timed, and the resulting time per compile or match is output in -milliseconds. Do not set -t with -s, because you will then get the size output -20000 times and the timing will be distorted. If you want to change the number -of repetitions used for timing, edit the definition of LOOPREPEAT at the top of -pcretest.c - -Philip Hazel <ph10@cam.ac.uk> -January 2000 + +SYNOPSIS + pcretest [-d] [-i] [-m] [-o osize] [-p] [-t] [source] [des- + tination] + + pcretest was written as a test program for the PCRE regular + expression library itself, but it can also be used for + experimenting with regular expressions. This man page + describes the features of the test program; for details of + the regular expressions themselves, see the pcre man page. + + + +OPTIONS + -d Behave as if each regex had the /D modifier (see + below); the internal form is output after compila- + tion. + + -i Behave as if each regex had the /I modifier; + information about the compiled pattern is given + after compilation. + + -m Output the size of each compiled pattern after it + has been compiled. This is equivalent to adding /M + to each regular expression. For compatibility with + earlier versions of pcretest, -s is a synonym for + -m. + + -o osize Set the number of elements in the output vector + that is used when calling PCRE to be osize. The + default value is 45, which is enough for 14 cap- + turing subexpressions. The vector size can be + changed for individual matching calls by including + \O in the data line (see below). + + -p Behave as if each regex has /P modifier; the POSIX + wrapper API is used to call PCRE. None of the + other options has any effect when -p is set. + + -t Run each compile, study, and match 20000 times + with a timer, and output resulting time per com- + pile or match (in milliseconds). Do not set -t + with -m, because you will then get the size output + 20000 times and the timing will be distorted. + + + +DESCRIPTION + If pcretest is given two filename arguments, it reads from + the first and writes to the second. If it is given only one + + + + +SunOS 5.8 Last change: 1 + + + + filename argument, it reads from that file and writes to + stdout. Otherwise, it reads from stdin and writes to stdout, + and prompts for each line of input, using "re>" to prompt + for regular expressions, and "data>" to prompt for data + lines. + + The program handles any number of sets of input on a single + input file. Each set starts with a regular expression, and + continues with any number of data lines to be matched + against the pattern. An empty line signals the end of the + data lines, at which point a new regular expression is read. + The regular expressions are given enclosed in any non- + alphameric delimiters other than backslash, for example + + /(a|bc)x+yz/ + + White space before the initial delimiter is ignored. A regu- + lar expression may be continued over several input lines, in + which case the newline characters are included within it. It + is possible to include the delimiter within the pattern by + escaping it, for example + + /abc\/def/ + + If you do so, the escape and the delimiter form part of the + pattern, but since delimiters are always non-alphameric, + this does not affect its interpretation. If the terminating + delimiter is immediately followed by a backslash, for exam- + ple, + + /abc/\ + + then a backslash is added to the end of the pattern. This is + done to provide a way of testing the error condition that + arises if a pattern finishes with a backslash, because + + /abc\/ + + is interpreted as the first line of a pattern that starts + with "abc/", causing pcretest to read the next line as a + continuation of the regular expression. + + + +PATTERN MODIFIERS + The pattern may be followed by i, m, s, or x to set the + PCRE_CASELESS, PCRE_MULTILINE, PCRE_DOTALL, or PCRE_EXTENDED + options, respectively. For example: + + /caseless/i + + These modifier letters have the same effect as they do in + Perl. There are others which set PCRE options that do not + correspond to anything in Perl: /A, /E, and /X set + PCRE_ANCHORED, PCRE_DOLLAR_ENDONLY, and PCRE_EXTRA respec- + tively. + + Searching for all possible matches within each subject + string can be requested by the /g or /G modifier. After + finding a match, PCRE is called again to search the + remainder of the subject string. The difference between /g + and /G is that the former uses the startoffset argument to + pcre_exec() to start searching at a new point within the + entire string (which is in effect what Perl does), whereas + the latter passes over a shortened substring. This makes a + difference to the matching process if the pattern begins + with a lookbehind assertion (including \b or \B). + + If any call to pcre_exec() in a /g or /G sequence matches an + empty string, the next call is done with the PCRE_NOTEMPTY + and PCRE_ANCHORED flags set in order to search for another, + non-empty, match at the same point. If this second match + fails, the start offset is advanced by one, and the normal + match is retried. This imitates the way Perl handles such + cases when using the /g modifier or the split() function. + + There are a number of other modifiers for controlling the + way pcretest operates. + + The /+ modifier requests that as well as outputting the sub- + string that matched the entire pattern, pcretest should in + addition output the remainder of the subject string. This is + useful for tests where the subject contains multiple copies + of the same substring. + + The /L modifier must be followed directly by the name of a + locale, for example, + + /pattern/Lfr + + For this reason, it must be the last modifier letter. The + given locale is set, pcre_maketables() is called to build a + set of character tables for the locale, and this is then + passed to pcre_compile() when compiling the regular expres- + sion. Without an /L modifier, NULL is passed as the tables + pointer; that is, /L applies only to the expression on which + it appears. + + The /I modifier requests that pcretest output information + about the compiled expression (whether it is anchored, has a + fixed first character, and so on). It does this by calling + pcre_fullinfo() after compiling an expression, and output- + ting the information it gets back. If the pattern is stu- + died, the results of that are also output. + The /D modifier is a PCRE debugging feature, which also + assumes /I. It causes the internal form of compiled regular + expressions to be output after compilation. + + The /S modifier causes pcre_study() to be called after the + expression has been compiled, and the results used when the + expression is matched. + + The /M modifier causes the size of memory block used to hold + the compiled pattern to be output. + + The /P modifier causes pcretest to call PCRE via the POSIX + wrapper API rather than its native API. When this is done, + all other modifiers except /i, /m, and /+ are ignored. + REG_ICASE is set if /i is present, and REG_NEWLINE is set if + /m is present. The wrapper functions force + PCRE_DOLLAR_ENDONLY always, and PCRE_DOTALL unless + REG_NEWLINE is set. + + The /8 modifier causes pcretest to call PCRE with the + PCRE_UTF8 option set. This turns on the (currently incom- + plete) support for UTF-8 character handling in PCRE, pro- + vided that it was compiled with this support enabled. This + modifier also causes any non-printing characters in output + strings to be printed using the \x{hh...} notation if they + are valid UTF-8 sequences. + + + +DATA LINES + Before each data line is passed to pcre_exec(), leading and + trailing whitespace is removed, and it is then scanned for \ + escapes. The following are recognized: + + \a alarm (= BEL) + \b backspace + \e escape + \f formfeed + \n newline + \r carriage return + \t tab + \v vertical tab + \nnn octal character (up to 3 octal digits) + \xhh hexadecimal character (up to 2 hex digits) + \x{hh...} hexadecimal UTF-8 character + + \A pass the PCRE_ANCHORED option to pcre_exec() + \B pass the PCRE_NOTBOL option to pcre_exec() + \Cdd call pcre_copy_substring() for substring dd + after a successful match (any decimal number + less than 32) + \Gdd call pcre_get_substring() for substring dd + + after a successful match (any decimal number + less than 32) + \L call pcre_get_substringlist() after a + successful match + \N pass the PCRE_NOTEMPTY option to pcre_exec() + \Odd set the size of the output vector passed to + pcre_exec() to dd (any number of decimal + digits) + \Z pass the PCRE_NOTEOL option to pcre_exec() + + When \O is used, it may be higher or lower than the size set + by the -O option (or defaulted to 45); \O applies only to + the call of pcre_exec() for the line in which it appears. + + A backslash followed by anything else just escapes the any- + thing else. If the very last character is a backslash, it is + ignored. This gives a way of passing an empty line as data, + since a real empty line terminates the data input. + + If /P was present on the regex, causing the POSIX wrapper + API to be used, only B, and Z have any effect, causing + REG_NOTBOL and REG_NOTEOL to be passed to regexec() respec- + tively. + + The use of \x{hh...} to represent UTF-8 characters is not + dependent on the use of the /8 modifier on the pattern. It + is recognized always. There may be any number of hexadecimal + digits inside the braces. The result is from one to six + bytes, encoded according to the UTF-8 rules. + + + +OUTPUT FROM PCRETEST + When a match succeeds, pcretest outputs the list of captured + substrings that pcre_exec() returns, starting with number 0 + for the string that matched the whole pattern. Here is an + example of an interactive pcretest run. + + $ pcretest + PCRE version 2.06 08-Jun-1999 + + re> /^abc(\d+)/ + data> abc123 + 0: abc123 + 1: 123 + data> xyz + No match + + If the strings contain any non-printing characters, they are + output as \0x escapes, or as \x{...} escapes if the /8 + modifier was present on the pattern. If the pattern has the + /+ modifier, then the output for substring 0 is followed by + the the rest of the subject string, identified by "0+" like + this: + + re> /cat/+ + data> cataract + 0: cat + 0+ aract + + If the pattern has the /g or /G modifier, the results of + successive matching attempts are output in sequence, like + this: + + re> /\Bi(\w\w)/g + data> Mississippi + 0: iss + 1: ss + 0: iss + 1: ss + 0: ipp + 1: pp + + "No match" is output only if the first match attempt fails. + + If any of the sequences \C, \G, or \L are present in a data + line that is successfully matched, the substrings extracted + by the convenience functions are output with C, G, or L + after the string number instead of a colon. This is in addi- + tion to the normal full list. The string length (that is, + the return from the extraction function) is given in + parentheses after each string for \C and \G. + + Note that while patterns can be continued over several lines + (a plain ">" prompt is used for continuations), data lines + may not. However newlines can be included in data by means + of the \n escape. + + + +AUTHOR + Philip Hazel <ph10@cam.ac.uk> + University Computing Service, + New Museums Site, + Cambridge CB2 3QG, England. + Phone: +44 1223 334714 + + Last updated: 15 August 2001 + Copyright (c) 1997-2001 University of Cambridge. diff --git a/srclib/pcre/pcre.c b/srclib/pcre/pcre.c index e3fdde9114..ad3ddc7c57 100644 --- a/srclib/pcre/pcre.c +++ b/srclib/pcre/pcre.c @@ -9,7 +9,7 @@ the file Tech.Notes for some information on the internals. Written by: Philip Hazel <ph10@cam.ac.uk> - Copyright (c) 1997-2000 University of Cambridge + Copyright (c) 1997-2001 University of Cambridge ----------------------------------------------------------------------------- Permission is granted to anyone to use this software for any purpose on any @@ -60,12 +60,25 @@ the external pcre header. */ #endif -/* Number of items on the nested bracket stacks at compile time. This should -not be set greater than 200. */ +/* Maximum number of items on the nested bracket stacks at compile time. This +applies to the nesting of all kinds of parentheses. It does not limit +un-nested, non-capturing parentheses. This number can be made bigger if +necessary - it is used to dimension one int and one unsigned char vector at +compile time. */ #define BRASTACK_SIZE 200 +/* The number of bytes in a literal character string above which we can't add +any more is different when UTF-8 characters may be encountered. */ + +#ifdef SUPPORT_UTF8 +#define MAXLIT 250 +#else +#define MAXLIT 255 +#endif + + /* Min and max values for the common repeats; for the maxima, 0 => infinity */ static const char rep_min[] = { 0, 0, 1, 1, 0, 0 }; @@ -85,7 +98,7 @@ static const char *OP_names[] = { "class", "Ref", "Recurse", "Alt", "Ket", "KetRmax", "KetRmin", "Assert", "Assert not", "AssertB", "AssertB not", "Reverse", "Once", "Cond", "Cref", - "Brazero", "Braminzero", "Bra" + "Brazero", "Braminzero", "Branumber", "Bra" }; #endif @@ -101,9 +114,9 @@ static const short int escapes[] = { 0, 0, 0, 0, 0, 0, 0, 0, /* H - O */ 0, 0, 0, -ESC_S, 0, 0, 0, -ESC_W, /* P - W */ 0, 0, -ESC_Z, '[', '\\', ']', '^', '_', /* X - _ */ - '`', 7, -ESC_b, 0, -ESC_d, 27, '\f', 0, /* ` - g */ - 0, 0, 0, 0, 0, 0, '\n', 0, /* h - o */ - 0, 0, '\r', -ESC_s, '\t', 0, 0, -ESC_w, /* p - w */ + '`', 7, -ESC_b, 0, -ESC_d, ESC_E, ESC_F, 0, /* ` - g */ + 0, 0, 0, 0, 0, 0, ESC_N, 0, /* h - o */ + 0, 0, ESC_R, -ESC_s, ESC_T, 0, 0, -ESC_w, /* p - w */ 0, 0, -ESC_z /* x - z */ }; @@ -176,6 +189,64 @@ void (*pcre_free)(void *) = free; +/************************************************* +* Macros and tables for character handling * +*************************************************/ + +/* When UTF-8 encoding is being used, a character is no longer just a single +byte. The macros for character handling generate simple sequences when used in +byte-mode, and more complicated ones for UTF-8 characters. */ + +#ifndef SUPPORT_UTF8 +#define GETCHARINC(c, eptr) c = *eptr++; +#define GETCHARLEN(c, eptr, len) c = *eptr; +#define BACKCHAR(eptr) + +#else /* SUPPORT_UTF8 */ + +/* Get the next UTF-8 character, advancing the pointer */ + +#define GETCHARINC(c, eptr) \ + c = *eptr++; \ + if (md->utf8 && (c & 0xc0) == 0xc0) \ + { \ + int a = utf8_table4[c & 0x3f]; /* Number of additional bytes */ \ + int s = 6*a; \ + c = (c & utf8_table3[a]) << s; \ + while (a-- > 0) \ + { \ + s -= 6; \ + c |= (*eptr++ & 0x3f) << s; \ + } \ + } + +/* Get the next UTF-8 character, not advancing the pointer, setting length */ + +#define GETCHARLEN(c, eptr, len) \ + c = *eptr; \ + len = 1; \ + if (md->utf8 && (c & 0xc0) == 0xc0) \ + { \ + int i; \ + int a = utf8_table4[c & 0x3f]; /* Number of additional bytes */ \ + int s = 6*a; \ + c = (c & utf8_table3[a]) << s; \ + for (i = 1; i <= a; i++) \ + { \ + s -= 6; \ + c |= (eptr[i] & 0x3f) << s; \ + } \ + len += a; \ + } + +/* If the pointer is not at the start of a character, move it back until +it is. */ + +#define BACKCHAR(eptr) while((*eptr & 0xc0) == 0x80) eptr--; + +#endif + + /************************************************* * Default character tables * @@ -191,6 +262,66 @@ tables. */ +#ifdef SUPPORT_UTF8 +/************************************************* +* Tables for UTF-8 support * +*************************************************/ + +/* These are the breakpoints for different numbers of bytes in a UTF-8 +character. */ + +static int utf8_table1[] = { 0x7f, 0x7ff, 0xffff, 0x1fffff, 0x3ffffff, 0x7fffffff}; + +/* These are the indicator bits and the mask for the data bits to set in the +first byte of a character, indexed by the number of additional bytes. */ + +static int utf8_table2[] = { 0, 0xc0, 0xe0, 0xf0, 0xf8, 0xfc}; +static int utf8_table3[] = { 0xff, 0x1f, 0x0f, 0x07, 0x03, 0x01}; + +/* Table of the number of extra characters, indexed by the first character +masked with 0x3f. The highest number for a valid UTF-8 character is in fact +0x3d. */ + +static uschar utf8_table4[] = { + 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, + 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, + 2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2, + 3,3,3,3,3,3,3,3,4,4,4,4,5,5,5,5 }; + + +/************************************************* +* Convert character value to UTF-8 * +*************************************************/ + +/* This function takes an integer value in the range 0 - 0x7fffffff +and encodes it as a UTF-8 character in 0 to 6 bytes. + +Arguments: + cvalue the character value + buffer pointer to buffer for result - at least 6 bytes long + +Returns: number of characters placed in the buffer +*/ + +static int +ord2utf8(int cvalue, uschar *buffer) +{ +register int i, j; +for (i = 0; i < sizeof(utf8_table1)/sizeof(int); i++) + if (cvalue <= utf8_table1[i]) break; +buffer += i; +for (j = i; j > 0; j--) + { + *buffer-- = 0x80 | (cvalue & 0x3f); + cvalue >>= 6; + } +*buffer = utf8_table2[i] | cvalue; +return i + 1; +} +#endif + + + /************************************************* * Return version string * *************************************************/ @@ -349,9 +480,9 @@ while (length-- > 0) /* This function is called when a \ has been encountered. It either returns a positive value for a simple escape such as \n, or a negative value which -encodes one of the more complicated things such as \d. On entry, ptr is -pointing at the \. On exit, it is on the final character of the escape -sequence. +encodes one of the more complicated things such as \d. When UTF-8 is enabled, +a positive value greater than 255 may be returned. On entry, ptr is pointing at +the \. On exit, it is on the final character of the escape sequence. Arguments: ptrptr points to the pattern position pointer @@ -373,7 +504,9 @@ check_escape(const uschar **ptrptr, const char **errorptr, int bracount, const uschar *ptr = *ptrptr; int c, i; -c = *(++ptr) & 255; /* Ensure > 0 on signed-char systems */ +/* If backslash is at the end of the pattern, it's an error. */ + +c = *(++ptr); if (c == 0) *errorptr = ERR1; /* Digits or letters may have special meaning; all others are literals. */ @@ -433,18 +566,46 @@ else } /* \0 always starts an octal number, but we may drop through to here with a - larger first octal digit */ + larger first octal digit. */ case '0': c -= '0'; while(i++ < 2 && (cd->ctypes[ptr[1]] & ctype_digit) != 0 && ptr[1] != '8' && ptr[1] != '9') c = c * 8 + *(++ptr) - '0'; + c &= 255; /* Take least significant 8 bits */ break; - /* Special escapes not starting with a digit are straightforward */ + /* \x is complicated when UTF-8 is enabled. \x{ddd} is a character number + which can be greater than 0xff, but only if the ddd are hex digits. */ case 'x': +#ifdef SUPPORT_UTF8 + if (ptr[1] == '{' && (options & PCRE_UTF8) != 0) + { + const uschar *pt = ptr + 2; + register int count = 0; + c = 0; + while ((cd->ctypes[*pt] & ctype_xdigit) != 0) + { + count++; + c = c * 16 + cd->lcc[*pt] - + (((cd->ctypes[*pt] & ctype_digit) != 0)? '0' : 'W'); + pt++; + } + if (*pt == '}') + { + if (c < 0 || count > 8) *errorptr = ERR34; + ptr = pt; + break; + } + /* If the sequence of hex digits does not end with '}', then we don't + recognize this construct; fall through to the normal \x handling. */ + } +#endif + + /* Read just a single hex char */ + c = 0; while (i++ < 2 && (cd->ctypes[ptr[1]] & ctype_xdigit) != 0) { @@ -454,6 +615,8 @@ else } break; + /* Other special escapes not starting with a digit are straightforward */ + case 'c': c = *(++ptr); if (c == 0) @@ -591,12 +754,13 @@ if the length is fixed. This is needed for dealing with backward assertions. Arguments: code points to the start of the pattern (the bracket) + options the compiling options Returns: the fixed length, or -1 if there is no fixed length */ static int -find_fixedlength(uschar *code) +find_fixedlength(uschar *code, int options) { int length = -1; @@ -617,7 +781,7 @@ for (;;) case OP_BRA: case OP_ONCE: case OP_COND: - d = find_fixedlength(cc); + d = find_fixedlength(cc, options); if (d < 0) return -1; branchlength += d; do cc += (cc[1] << 8) + cc[2]; while (*cc == OP_ALT); @@ -653,10 +817,11 @@ for (;;) /* Skip over things that don't match chars */ case OP_REVERSE: + case OP_BRANUMBER: + case OP_CREF: cc++; /* Fall through */ - case OP_CREF: case OP_OPT: cc++; /* Fall through */ @@ -671,10 +836,17 @@ for (;;) cc++; break; - /* Handle char strings */ + /* Handle char strings. In UTF-8 mode we must count characters, not bytes. + This requires a scan of the string, unfortunately. We assume valid UTF-8 + strings, so all we do is reduce the length by one for byte whose bits are + 10xxxxxx. */ case OP_CHARS: branchlength += *(++cc); +#ifdef SUPPORT_UTF8 + for (d = 1; d <= *cc; d++) + if ((cc[d] & 0xc0) == 0x80) branchlength--; +#endif cc += *cc + 1; break; @@ -703,7 +875,7 @@ for (;;) /* Check a class for variable quantification */ case OP_CLASS: - cc += (*cc == OP_REF)? 2 : 33; + cc += 33; switch (*cc) { @@ -810,7 +982,7 @@ return -1; Arguments: options the option bits - brackets points to number of brackets used + brackets points to number of extracting brackets used code points to the pointer to the current code point ptrptr points to the current pattern pointer errorptr points to pointer to error message @@ -861,7 +1033,7 @@ for (;; ptr++) int class_charcount; int class_lastchar; int newoptions; - int condref; + int skipbytes; int subreqchar; c = *ptr; @@ -872,7 +1044,7 @@ for (;; ptr++) { /* The space before the ; is to avoid a warning on a silly compiler on the Macintosh. */ - while ((c = *(++ptr)) != 0 && c != '\n') ; + while ((c = *(++ptr)) != 0 && c != NEWLINE) ; continue; } } @@ -1054,7 +1226,17 @@ for (;; ptr++) goto FAILED; } } - /* Fall through if single character */ + + /* Fall through if single character, but don't at present allow + chars > 255 in UTF-8 mode. */ + +#ifdef SUPPORT_UTF8 + if (c > 255) + { + *errorptr = ERR33; + goto FAILED; + } +#endif } /* A single character may be followed by '-' to form a range. However, @@ -1074,17 +1256,29 @@ for (;; ptr++) } /* The second part of a range can be a single-character escape, but - not any of the other escapes. */ + not any of the other escapes. Perl 5.6 treats a hyphen as a literal + in such circumstances. */ if (d == '\\') { + const uschar *oldptr = ptr; d = check_escape(&ptr, errorptr, *brackets, options, TRUE, cd); + +#ifdef SUPPORT_UTF8 + if (d > 255) + { + *errorptr = ERR33; + goto FAILED; + } +#endif + /* \b is backslash; any other special means the '-' was literal */ + if (d < 0) { if (d == -ESC_b) d = '\b'; else { - *errorptr = ERR7; - goto FAILED; + ptr = oldptr - 2; + goto SINGLE_CHARACTER; /* A few lines below */ } } } @@ -1112,6 +1306,8 @@ for (;; ptr++) /* Handle a lone single character - we can get here for a normal non-escape char, or after \ that introduces a single character. */ + SINGLE_CHARACTER: + class [c/8] |= (1 << (c&7)); if ((options & PCRE_CASELESS) != 0) { @@ -1386,7 +1582,7 @@ for (;; ptr++) OP_BRAZERO in front of it, and because the group appears once in the data, whereas in other cases it appears the minimum number of times. For this reason, it is simplest to treat this case separately, as otherwise - the code gets far too mess. There are several special subcases when the + the code gets far too messy. There are several special subcases when the minimum is zero. */ if (repeat_min == 0) @@ -1537,7 +1733,7 @@ for (;; ptr++) case '(': newoptions = options; - condref = -1; + skipbytes = 0; if (*(++ptr) == '?') { @@ -1560,9 +1756,18 @@ for (;; ptr++) bravalue = OP_COND; /* Conditional group */ if ((cd->ctypes[*(++ptr)] & ctype_digit) != 0) { - condref = *ptr - '0'; + int condref = *ptr - '0'; while (*(++ptr) != ')') condref = condref*10 + *ptr - '0'; + if (condref == 0) + { + *errorptr = ERR35; + goto FAILED; + } ptr++; + code[3] = OP_CREF; + code[4] = condref >> 8; + code[5] = condref & 255; + skipbytes = 3; } else ptr--; break; @@ -1665,16 +1870,21 @@ for (;; ptr++) } } - /* Else we have a referencing group; adjust the opcode. */ + /* Else we have a referencing group; adjust the opcode. If the bracket + number is greater than EXTRACT_BASIC_MAX, we set the opcode one higher, and + arrange for the true number to follow later, in an OP_BRANUMBER item. */ else { - if (++(*brackets) > EXTRACT_MAX) + if (++(*brackets) > EXTRACT_BASIC_MAX) { - *errorptr = ERR13; - goto FAILED; + bravalue = OP_BRA + EXTRACT_BASIC_MAX + 1; + code[3] = OP_BRANUMBER; + code[4] = *brackets >> 8; + code[5] = *brackets & 255; + skipbytes = 3; } - bravalue = OP_BRA + *brackets; + else bravalue = OP_BRA + *brackets; } /* Process nested bracketed re. Assertions may not be repeated, but other @@ -1690,13 +1900,13 @@ for (;; ptr++) options | PCRE_INGROUP, /* Set for all nested groups */ ((options & PCRE_IMS) != (newoptions & PCRE_IMS))? newoptions & PCRE_IMS : -1, /* Pass ims options if changed */ - brackets, /* Bracket level */ + brackets, /* Extracting bracket count */ &tempcode, /* Where to put code (updated) */ &ptr, /* Input pointer (updated) */ errorptr, /* Where to put an error message */ (bravalue == OP_ASSERTBACK || bravalue == OP_ASSERTBACK_NOT), /* TRUE if back assert */ - condref, /* Condition reference number */ + skipbytes, /* Skip over OP_COND/OP_BRANUMBER */ &subreqchar, /* For possible last char */ &subcountlits, /* For literal count */ cd)) /* Tables block */ @@ -1710,7 +1920,7 @@ for (;; ptr++) /* If this is a conditional bracket, check that there are no more than two branches in the group. */ - if (bravalue == OP_COND) + else if (bravalue == OP_COND) { uschar *tc = code; condcount = 0; @@ -1777,9 +1987,11 @@ for (;; ptr++) { if (-c >= ESC_REF) { + int number = -c - ESC_REF; previous = code; *code++ = OP_REF; - *code++ = -c - ESC_REF; + *code++ = number >> 8; + *code++ = number & 255; } else { @@ -1814,7 +2026,7 @@ for (;; ptr++) { /* The space before the ; is to avoid a warning on a silly compiler on the Macintosh. */ - while ((c = *(++ptr)) != 0 && c != '\n') ; + while ((c = *(++ptr)) != 0 && c != NEWLINE) ; if (c == 0) break; continue; } @@ -1829,6 +2041,20 @@ for (;; ptr++) tempptr = ptr; c = check_escape(&ptr, errorptr, *brackets, options, FALSE, cd); if (c < 0) { ptr = tempptr; break; } + + /* If a character is > 127 in UTF-8 mode, we have to turn it into + two or more characters in the UTF-8 encoding. */ + +#ifdef SUPPORT_UTF8 + if (c > 127 && (options & PCRE_UTF8) != 0) + { + uschar buffer[8]; + int len = ord2utf8(c, buffer); + for (c = 0; c < len; c++) *code++ = buffer[c]; + length += len; + continue; + } +#endif } /* Ordinary character or single-char escape */ @@ -1839,7 +2065,7 @@ for (;; ptr++) /* This "while" is the end of the "do" above. */ - while (length < 255 && (cd->ctypes[c = *(++ptr)] & ctype_meta) == 0); + while (length < MAXLIT && (cd->ctypes[c = *(++ptr)] & ctype_meta) == 0); /* Update the last character and the count of literals */ @@ -1851,7 +2077,7 @@ for (;; ptr++) the next state. */ previous[1] = length; - if (length < 255) ptr--; + if (length < MAXLIT) ptr--; break; } } /* end of big loop */ @@ -1889,7 +2115,7 @@ Argument: ptrptr -> the address of the current pattern pointer errorptr -> pointer to error message lookbehind TRUE if this is a lookbehind assertion - condref > 0 for OPT_CREF setting at start of conditional group + skipbytes skip this many bytes at start (for OP_COND, OP_BRANUMBER) reqchar -> place to put the last required character, or a negative number countlits -> place to put the shortest literal count of any branch cd points to the data block with tables pointers @@ -1899,7 +2125,7 @@ Returns: TRUE on success static BOOL compile_regex(int options, int optchanged, int *brackets, uschar **codeptr, - const uschar **ptrptr, const char **errorptr, BOOL lookbehind, int condref, + const uschar **ptrptr, const char **errorptr, BOOL lookbehind, int skipbytes, int *reqchar, int *countlits, compile_data *cd) { const uschar *ptr = *ptrptr; @@ -1912,16 +2138,7 @@ int branchreqchar, branchcountlits; *reqchar = -1; *countlits = INT_MAX; -code += 3; - -/* At the start of a reference-based conditional group, insert the reference -number as an OP_CREF item. */ - -if (condref > 0) - { - *code++ = OP_CREF; - *code++ = condref; - } +code += 3 + skipbytes; /* Loop for each alternative branch */ @@ -1989,7 +2206,7 @@ for (;;) if (lookbehind) { *code = OP_END; - length = find_fixedlength(last_branch); + length = find_fixedlength(last_branch, options); DPRINTF(("fixed length = %d\n", length)); if (length < 0) { @@ -2073,7 +2290,8 @@ for (;;) break; case OP_CREF: - code += 2; + case OP_BRANUMBER: + code += 3; break; case OP_WORD_BOUNDARY: @@ -2280,6 +2498,16 @@ uschar bralenstack[BRASTACK_SIZE]; uschar *code_base, *code_end; #endif +/* Can't support UTF8 unless PCRE has been compiled to include the code. */ + +#ifndef SUPPORT_UTF8 +if ((options & PCRE_UTF8) != 0) + { + *errorptr = ERR32; + return NULL; + } +#endif + /* We can't pass back an error message if errorptr is NULL; I guess the best we can do is just return NULL. */ @@ -2326,6 +2554,7 @@ while ((c = *(++ptr)) != 0) { int min, max; int class_charcount; + int bracket_length; if ((options & PCRE_EXTENDED) != 0) { @@ -2334,7 +2563,7 @@ while ((c = *(++ptr)) != 0) { /* The space before the ; is to avoid a warning on a silly compiler on the Macintosh. */ - while ((c = *(++ptr)) != 0 && c != '\n') ; + while ((c = *(++ptr)) != 0 && c != NEWLINE) ; continue; } } @@ -2360,7 +2589,7 @@ while ((c = *(++ptr)) != 0) } length++; - /* A back reference needs an additional char, plus either one or 5 + /* A back reference needs an additional 2 bytes, plus either one or 5 bytes for a repeat. We also need to keep the value of the highest back reference. */ @@ -2368,7 +2597,7 @@ while ((c = *(++ptr)) != 0) { int refnum = -c - ESC_REF; if (refnum > top_backref) top_backref = refnum; - length++; /* For single back reference */ + length += 2; /* For single back reference */ if (ptr[1] == '{' && is_counted_repeat(ptr+2, &compile_block)) { ptr = read_repeat_counts(ptr+2, &min, &max, errorptr, &compile_block); @@ -2466,6 +2695,7 @@ while ((c = *(++ptr)) != 0) case '(': branch_newextra = 0; + bracket_length = 3; /* Handle special forms of bracket, which all start (? */ @@ -2533,7 +2763,7 @@ while ((c = *(++ptr)) != 0) if ((compile_block.ctypes[ptr[3]] & ctype_digit) != 0) { ptr += 4; - length += 2; + length += 3; while ((compile_block.ctypes[*ptr] & ctype_digit) != 0) ptr++; if (*ptr != ')') { @@ -2660,15 +2890,19 @@ while ((c = *(++ptr)) != 0) } /* Extracting brackets must be counted so we can process escapes in a - Perlish way. */ + Perlish way. If the number exceeds EXTRACT_BASIC_MAX we are going to + need an additional 3 bytes of store per extracting bracket. */ - else bracount++; + else + { + bracount++; + if (bracount > EXTRACT_BASIC_MAX) bracket_length += 3; + } - /* Non-special forms of bracket. Save length for computing whole length - at end if there's a repeat that requires duplication of the group. Also - save the current value of branch_extra, and start the new group with - the new value. If non-zero, this will either be 2 for a (?imsx: group, or 3 - for a lookbehind assertion. */ + /* Save length for computing whole length at end if there's a repeat that + requires duplication of the group. Also save the current value of + branch_extra, and start the new group with the new value. If non-zero, this + will either be 2 for a (?imsx: group, or 3 for a lookbehind assertion. */ if (brastackptr >= sizeof(brastack)/sizeof(int)) { @@ -2680,7 +2914,7 @@ while ((c = *(++ptr)) != 0) branch_extra = branch_newextra; brastack[brastackptr++] = length; - length += 3; + length += bracket_length; continue; /* Handle ket. Look for subsequent max/min; for certain sets of values we @@ -2760,7 +2994,7 @@ while ((c = *(++ptr)) != 0) { /* The space before the ; is to avoid a warning on a silly compiler on the Macintosh. */ - while ((c = *(++ptr)) != 0 && c != '\n') ; + while ((c = *(++ptr)) != 0 && c != NEWLINE) ; continue; } } @@ -2775,6 +3009,16 @@ while ((c = *(++ptr)) != 0) &compile_block); if (*errorptr != NULL) goto PCRE_ERROR_RETURN; if (c < 0) { ptr = saveptr; break; } + +#ifdef SUPPORT_UTF8 + if (c > 127 && (options & PCRE_UTF8) != 0) + { + int i; + for (i = 0; i < sizeof(utf8_table1)/sizeof(int); i++) + if (c <= utf8_table1[i]) break; + runlength += i; + } +#endif } /* Ordinary character or single-char escape */ @@ -2784,7 +3028,7 @@ while ((c = *(++ptr)) != 0) /* This "while" is the end of the "do" above. */ - while (runlength < 255 && + while (runlength < MAXLIT && (compile_block.ctypes[c = *(++ptr)] & ctype_meta) == 0); ptr--; @@ -2831,7 +3075,7 @@ ptr = (const uschar *)pattern; code = re->code; *code = OP_BRA; bracount = 0; -(void)compile_regex(options, -1, &bracount, &code, &ptr, errorptr, FALSE, -1, +(void)compile_regex(options, -1, &bracount, &code, &ptr, errorptr, FALSE, 0, &reqchar, &countlits, &compile_block); re->top_bracket = bracount; re->top_backref = top_backref; @@ -2945,7 +3189,10 @@ while (code < code_end) if (*code >= OP_BRA) { - printf("%3d Bra %d", (code[1] << 8) + code[2], *code - OP_BRA); + if (*code - OP_BRA > EXTRACT_BASIC_MAX) + printf("%3d Bra extra", (code[1] << 8) + code[2]); + else + printf("%3d Bra %d", (code[1] << 8) + code[2], *code - OP_BRA); code += 2; } @@ -2956,16 +3203,6 @@ while (code < code_end) code++; break; - case OP_COND: - printf("%3d Cond", (code[1] << 8) + code[2]); - code += 2; - break; - - case OP_CREF: - printf(" %.2d %s", code[1], OP_names[*code]); - code++; - break; - case OP_CHARS: charlength = *(++code); printf("%3d ", charlength); @@ -2982,11 +3219,10 @@ while (code < code_end) case OP_ASSERTBACK: case OP_ASSERTBACK_NOT: case OP_ONCE: - printf("%3d %s", (code[1] << 8) + code[2], OP_names[*code]); - code += 2; - break; - case OP_REVERSE: + case OP_BRANUMBER: + case OP_COND: + case OP_CREF: printf("%3d %s", (code[1] << 8) + code[2], OP_names[*code]); code += 2; break; @@ -3059,8 +3295,8 @@ while (code < code_end) break; case OP_REF: - printf(" \\%d", *(++code)); - code ++; + printf(" \\%d", (code[1] << 8) | code[2]); + code += 3; goto CLASS_REF_REPEAT; case OP_CLASS: @@ -3273,8 +3509,14 @@ for (;;) if (op > OP_BRA) { + int offset; int number = op - OP_BRA; - int offset = number << 1; + + /* For extended extraction brackets (large number), we have to fish out the + number from a dummy opcode at the start. */ + + if (number > EXTRACT_BASIC_MAX) number = (ecode[4] << 8) | ecode[5]; + offset = number << 1; #ifdef DEBUG printf("start bracket %d subject=", number); @@ -3304,6 +3546,7 @@ for (;;) md->offset_vector[offset] = save_offset1; md->offset_vector[offset+1] = save_offset2; md->offset_vector[md->offset_end - number] = save_offset3; + return FALSE; } @@ -3336,10 +3579,10 @@ for (;;) case OP_COND: if (ecode[3] == OP_CREF) /* Condition is extraction test */ { - int offset = ecode[4] << 1; /* Doubled reference number */ + int offset = (ecode[4] << 9) | (ecode[5] << 1); /* Doubled ref number */ return match(eptr, ecode + ((offset < offset_top && md->offset_vector[offset] >= 0)? - 5 : 3 + (ecode[1] << 8) + ecode[2]), + 6 : 3 + (ecode[1] << 8) + ecode[2]), offset_top, md, ims, eptrb, match_isgroup); } @@ -3359,10 +3602,12 @@ for (;;) } /* Control never reaches here */ - /* Skip over conditional reference data if encountered (should not be) */ + /* Skip over conditional reference or large extraction number data if + encountered. */ case OP_CREF: - ecode += 2; + case OP_BRANUMBER: + ecode += 3; break; /* End of the pattern. If PCRE_NOTEMPTY is set, fail if we have matched @@ -3429,10 +3674,21 @@ for (;;) /* Move the subject pointer back. This occurs only at the start of each branch of a lookbehind assertion. If we are too close to the start to - move back, this match function fails. */ + move back, this match function fails. When working with UTF-8 we move + back a number of characters, not bytes. */ case OP_REVERSE: +#ifdef SUPPORT_UTF8 + c = (ecode[1] << 8) + ecode[2]; + for (i = 0; i < c; i++) + { + eptr--; + BACKCHAR(eptr) + } +#else eptr -= (ecode[1] << 8) + ecode[2]; +#endif + if (eptr < md->start_subject) return FALSE; ecode += 3; break; @@ -3617,8 +3873,14 @@ for (;;) if (*prev != OP_COND) { + int offset; int number = *prev - OP_BRA; - int offset = number << 1; + + /* For extended extraction brackets (large number), we have to fish out + the number from a dummy opcode at the start. */ + + if (number > EXTRACT_BASIC_MAX) number = (prev[4] << 8) | prev[5]; + offset = number << 1; #ifdef DEBUG printf("end bracket %d", number); @@ -3678,7 +3940,7 @@ for (;;) if (md->notbol && eptr == md->start_subject) return FALSE; if ((ims & PCRE_MULTILINE) != 0) { - if (eptr != md->start_subject && eptr[-1] != '\n') return FALSE; + if (eptr != md->start_subject && eptr[-1] != NEWLINE) return FALSE; ecode++; break; } @@ -3697,7 +3959,7 @@ for (;;) case OP_DOLL: if ((ims & PCRE_MULTILINE) != 0) { - if (eptr < md->end_subject) { if (*eptr != '\n') return FALSE; } + if (eptr < md->end_subject) { if (*eptr != NEWLINE) return FALSE; } else { if (md->noteol) return FALSE; } ecode++; break; @@ -3708,7 +3970,7 @@ for (;;) if (!md->endonly) { if (eptr < md->end_subject - 1 || - (eptr == md->end_subject - 1 && *eptr != '\n')) return FALSE; + (eptr == md->end_subject - 1 && *eptr != NEWLINE)) return FALSE; ecode++; break; @@ -3727,7 +3989,7 @@ for (;;) case OP_EODN: if (eptr < md->end_subject - 1 || - (eptr == md->end_subject - 1 && *eptr != '\n')) return FALSE; + (eptr == md->end_subject - 1 && *eptr != NEWLINE)) return FALSE; ecode++; break; @@ -3749,9 +4011,13 @@ for (;;) /* Match a single character type; inline for speed */ case OP_ANY: - if ((ims & PCRE_DOTALL) == 0 && eptr < md->end_subject && *eptr == '\n') + if ((ims & PCRE_DOTALL) == 0 && eptr < md->end_subject && *eptr == NEWLINE) return FALSE; if (eptr++ >= md->end_subject) return FALSE; +#ifdef SUPPORT_UTF8 + if (md->utf8) + while (eptr < md->end_subject && (*eptr & 0xc0) == 0x80) eptr++; +#endif ecode++; break; @@ -3808,8 +4074,8 @@ for (;;) case OP_REF: { int length; - int offset = ecode[1] << 1; /* Doubled reference number */ - ecode += 2; /* Advance past the item */ + int offset = (ecode[1] << 9) | (ecode[2] << 1); /* Doubled ref number */ + ecode += 3; /* Advance past item */ /* If the reference is unset, set the length to be longer than the amount of subject left; this ensures that every attempt at a match fails. We @@ -3953,7 +4219,13 @@ for (;;) for (i = 1; i <= min; i++) { if (eptr >= md->end_subject) return FALSE; - c = *eptr++; + GETCHARINC(c, eptr) /* Get character; increment eptr */ + +#ifdef SUPPORT_UTF8 + /* We do not yet support class members > 255 */ + if (c > 255) return FALSE; +#endif + if ((data[c/8] & (1 << (c&7))) != 0) continue; return FALSE; } @@ -3973,7 +4245,12 @@ for (;;) if (match(eptr, ecode, offset_top, md, ims, eptrb, 0)) return TRUE; if (i >= max || eptr >= md->end_subject) return FALSE; - c = *eptr++; + GETCHARINC(c, eptr) /* Get character; increment eptr */ + +#ifdef SUPPORT_UTF8 + /* We do not yet support class members > 255 */ + if (c > 255) return FALSE; +#endif if ((data[c/8] & (1 << (c&7))) != 0) continue; return FALSE; } @@ -3985,17 +4262,29 @@ for (;;) else { const uschar *pp = eptr; - for (i = min; i < max; eptr++, i++) + int len = 1; + for (i = min; i < max; i++) { if (eptr >= md->end_subject) break; - c = *eptr; - if ((data[c/8] & (1 << (c&7))) != 0) continue; - break; + GETCHARLEN(c, eptr, len) /* Get character, set length if UTF-8 */ + +#ifdef SUPPORT_UTF8 + /* We do not yet support class members > 255 */ + if (c > 255) break; +#endif + if ((data[c/8] & (1 << (c&7))) == 0) break; + eptr += len; } while (eptr >= pp) + { if (match(eptr--, ecode, offset_top, md, ims, eptrb, 0)) return TRUE; + +#ifdef SUPPORT_UTF8 + BACKCHAR(eptr) +#endif + } return FALSE; } } @@ -4315,15 +4604,31 @@ for (;;) /* First, ensure the minimum number of matches are present. Use inline code for maximizing the speed, and do the type test once at the start - (i.e. keep it out of the loop). Also test that there are at least the - minimum number of characters before we start. */ + (i.e. keep it out of the loop). Also we can test that there are at least + the minimum number of bytes before we start, except when doing '.' in + UTF8 mode. Leave the test in in all cases; in the special case we have + to test after each character. */ if (min > md->end_subject - eptr) return FALSE; if (min > 0) switch(ctype) { case OP_ANY: +#ifdef SUPPORT_UTF8 + if (md->utf8) + { + for (i = 1; i <= min; i++) + { + if (eptr >= md->end_subject || + (*eptr++ == NEWLINE && (ims & PCRE_DOTALL) == 0)) + return FALSE; + while (eptr < md->end_subject && (*eptr & 0xc0) == 0x80) eptr++; + } + break; + } +#endif + /* Non-UTF8 can be faster */ if ((ims & PCRE_DOTALL) == 0) - { for (i = 1; i <= min; i++) if (*eptr++ == '\n') return FALSE; } + { for (i = 1; i <= min; i++) if (*eptr++ == NEWLINE) return FALSE; } else eptr += min; break; @@ -4378,7 +4683,11 @@ for (;;) switch(ctype) { case OP_ANY: - if ((ims & PCRE_DOTALL) == 0 && c == '\n') return FALSE; + if ((ims & PCRE_DOTALL) == 0 && c == NEWLINE) return FALSE; +#ifdef SUPPORT_UTF8 + if (md->utf8) + while (eptr < md->end_subject && (*eptr & 0xc0) == 0x80) eptr++; +#endif break; case OP_NOT_DIGIT: @@ -4418,11 +4727,38 @@ for (;;) switch(ctype) { case OP_ANY: + + /* Special code is required for UTF8, but when the maximum is unlimited + we don't need it. */ + +#ifdef SUPPORT_UTF8 + if (md->utf8 && max < INT_MAX) + { + if ((ims & PCRE_DOTALL) == 0) + { + for (i = min; i < max; i++) + { + if (eptr >= md->end_subject || *eptr++ == NEWLINE) break; + while (eptr < md->end_subject && (*eptr & 0xc0) == 0x80) eptr++; + } + } + else + { + for (i = min; i < max; i++) + { + eptr++; + while (eptr < md->end_subject && (*eptr & 0xc0) == 0x80) eptr++; + } + } + break; + } +#endif + /* Non-UTF8 can be faster */ if ((ims & PCRE_DOTALL) == 0) { for (i = min; i < max; i++) { - if (eptr >= md->end_subject || *eptr == '\n') break; + if (eptr >= md->end_subject || *eptr == NEWLINE) break; eptr++; } } @@ -4490,8 +4826,14 @@ for (;;) } while (eptr >= pp) + { if (match(eptr--, ecode, offset_top, md, ims, eptrb, 0)) return TRUE; +#ifdef SUPPORT_UTF8 + if (md->utf8) + while (eptr > pp && (*eptr & 0xc0) == 0x80) eptr--; +#endif + } return FALSE; } /* Control never gets here */ @@ -4557,8 +4899,8 @@ const uschar *req_char_ptr = start_match - 1; const real_pcre *re = (const real_pcre *)external_re; const real_pcre_extra *extra = (const real_pcre_extra *)external_extra; BOOL using_temporary_offsets = FALSE; -BOOL anchored = ((re->options | options) & PCRE_ANCHORED) != 0; -BOOL startline = (re->options & PCRE_STARTLINE) != 0; +BOOL anchored; +BOOL startline; if ((options & ~PUBLIC_EXEC_OPTIONS) != 0) return PCRE_ERROR_BADOPTION; @@ -4566,12 +4908,16 @@ if (re == NULL || subject == NULL || (offsets == NULL && offsetcount > 0)) return PCRE_ERROR_NULL; if (re->magic_number != MAGIC_NUMBER) return PCRE_ERROR_BADMAGIC; +anchored = ((re->options | options) & PCRE_ANCHORED) != 0; +startline = (re->options & PCRE_STARTLINE) != 0; + match_block.start_pattern = re->code; match_block.start_subject = (const uschar *)subject; match_block.end_subject = match_block.start_subject + length; end_subject = match_block.end_subject; match_block.endonly = (re->options & PCRE_DOLLAR_ENDONLY) != 0; +match_block.utf8 = (re->options & PCRE_UTF8) != 0; match_block.notbol = (options & PCRE_NOTBOL) != 0; match_block.noteol = (options & PCRE_NOTEOL) != 0; @@ -4693,7 +5039,7 @@ do { if (start_match > match_block.start_subject + start_offset) { - while (start_match < end_subject && start_match[-1] != '\n') + while (start_match < end_subject && start_match[-1] != NEWLINE) start_match++; } } @@ -4798,7 +5144,7 @@ do rc = match_block.offset_overflow? 0 : match_block.end_offset_top/2; - if (match_block.offset_end < 2) rc = 0; else + if (offsetcount < 2) rc = 0; else { offsets[0] = start_match - match_block.start_subject; offsets[1] = match_block.end_match_ptr - match_block.start_subject; diff --git a/srclib/pcre/pcretest.c b/srclib/pcre/pcretest.c index bbe9bdd109..f04443ab30 100644 --- a/srclib/pcre/pcretest.c +++ b/srclib/pcre/pcretest.c @@ -38,6 +38,114 @@ static size_t gotten_store; +static int utf8_table1[] = { + 0x0000007f, 0x000007ff, 0x0000ffff, 0x001fffff, 0x03ffffff, 0x7fffffff}; + +static int utf8_table2[] = { + 0, 0xc0, 0xe0, 0xf0, 0xf8, 0xfc}; + +static int utf8_table3[] = { + 0xff, 0x1f, 0x0f, 0x07, 0x03, 0x01}; + + +/************************************************* +* Convert character value to UTF-8 * +*************************************************/ + +/* This function takes an integer value in the range 0 - 0x7fffffff +and encodes it as a UTF-8 character in 0 to 6 bytes. + +Arguments: + cvalue the character value + buffer pointer to buffer for result - at least 6 bytes long + +Returns: number of characters placed in the buffer + -1 if input character is negative + 0 if input character is positive but too big (only when + int is longer than 32 bits) +*/ + +static int +ord2utf8(int cvalue, unsigned char *buffer) +{ +register int i, j; +for (i = 0; i < sizeof(utf8_table1)/sizeof(int); i++) + if (cvalue <= utf8_table1[i]) break; +if (i >= sizeof(utf8_table1)/sizeof(int)) return 0; +if (cvalue < 0) return -1; + +buffer += i; +for (j = i; j > 0; j--) + { + *buffer-- = 0x80 | (cvalue & 0x3f); + cvalue >>= 6; + } +*buffer = utf8_table2[i] | cvalue; +return i + 1; +} + + +/************************************************* +* Convert UTF-8 string to value * +*************************************************/ + +/* This function takes one or more bytes that represents a UTF-8 character, +and returns the value of the character. + +Argument: + buffer a pointer to the byte vector + vptr a pointer to an int to receive the value + +Returns: > 0 => the number of bytes consumed + -6 to 0 => malformed UTF-8 character at offset = (-return) +*/ + +int +utf82ord(unsigned char *buffer, int *vptr) +{ +int c = *buffer++; +int d = c; +int i, j, s; + +for (i = -1; i < 6; i++) /* i is number of additional bytes */ + { + if ((d & 0x80) == 0) break; + d <<= 1; + } + +if (i == -1) { *vptr = c; return 1; } /* ascii character */ +if (i == 0 || i == 6) return 0; /* invalid UTF-8 */ + +/* i now has a value in the range 1-5 */ + +s = 6*i; +d = (c & utf8_table3[i]) << s; + +for (j = 0; j < i; j++) + { + c = *buffer++; + if ((c & 0xc0) != 0x80) return -(j+1); + s -= 6; + d |= (c & 0x3f) << s; + } + +/* Check that encoding was the correct unique one */ + +for (j = 0; j < sizeof(utf8_table1)/sizeof(int); j++) + if (d <= utf8_table1[j]) break; +if (j != i) return -(i+1); + +/* Valid value */ + +*vptr = d; +return i+1; +} + + + + + + /* Debugging function to print the internal form of the regex. This is the same code as contained in pcre.c under the DEBUG macro. */ @@ -52,7 +160,7 @@ static const char *OP_names[] = { "class", "Ref", "Recurse", "Alt", "Ket", "KetRmax", "KetRmin", "Assert", "Assert not", "AssertB", "AssertB not", "Reverse", "Once", "Cond", "Cref", - "Brazero", "Braminzero", "Bra" + "Brazero", "Braminzero", "Branumber", "Bra" }; @@ -71,7 +179,10 @@ for(;;) if (*code >= OP_BRA) { - fprintf(outfile, "%3d Bra %d", (code[1] << 8) + code[2], *code - OP_BRA); + if (*code - OP_BRA > EXTRACT_BASIC_MAX) + fprintf(outfile, "%3d Bra extra", (code[1] << 8) + code[2]); + else + fprintf(outfile, "%3d Bra %d", (code[1] << 8) + code[2], *code - OP_BRA); code += 2; } @@ -87,16 +198,6 @@ for(;;) code++; break; - case OP_COND: - fprintf(outfile, "%3d Cond", (code[1] << 8) + code[2]); - code += 2; - break; - - case OP_CREF: - fprintf(outfile, " %.2d %s", code[1], OP_names[*code]); - code++; - break; - case OP_CHARS: charlength = *(++code); fprintf(outfile, "%3d ", charlength); @@ -114,11 +215,10 @@ for(;;) case OP_ASSERTBACK: case OP_ASSERTBACK_NOT: case OP_ONCE: - fprintf(outfile, "%3d %s", (code[1] << 8) + code[2], OP_names[*code]); - code += 2; - break; - + case OP_COND: + case OP_BRANUMBER: case OP_REVERSE: + case OP_CREF: fprintf(outfile, "%3d %s", (code[1] << 8) + code[2], OP_names[*code]); code += 2; break; @@ -191,8 +291,8 @@ for(;;) break; case OP_REF: - fprintf(outfile, " \\%d", *(++code)); - code++; + fprintf(outfile, " \\%d", (code[1] << 8) | code[2]); + code += 3; goto CLASS_REF_REPEAT; case OP_CLASS: @@ -265,14 +365,31 @@ for(;;) -/* Character string printing function. */ +/* Character string printing function. A "normal" and a UTF-8 version. */ -static void pchars(unsigned char *p, int length) +static void pchars(unsigned char *p, int length, int utf8) { int c; while (length-- > 0) + { + if (utf8) + { + int rc = utf82ord(p, &c); + if (rc > 0) + { + length -= rc - 1; + p += rc; + if (c < 256 && isprint(c)) fprintf(outfile, "%c", c); + else fprintf(outfile, "\\x{%02x}", c); + continue; + } + } + + /* Not UTF-8, or malformed UTF-8 */ + if (isprint(c = *(p++))) fprintf(outfile, "%c", c); else fprintf(outfile, "\\x%02x", c); + } } @@ -317,7 +434,12 @@ int op = 1; int timeit = 0; int showinfo = 0; int showstore = 0; +int size_offsets = 45; +int size_offsets_max; +int *offsets; +#if !defined NOPOSIX int posix = 0; +#endif int debug = 0; int done = 0; unsigned char buffer[30000]; @@ -331,27 +453,51 @@ outfile = stdout; while (argc > 1 && argv[op][0] == '-') { + char *endptr; + if (strcmp(argv[op], "-s") == 0 || strcmp(argv[op], "-m") == 0) showstore = 1; else if (strcmp(argv[op], "-t") == 0) timeit = 1; else if (strcmp(argv[op], "-i") == 0) showinfo = 1; else if (strcmp(argv[op], "-d") == 0) showinfo = debug = 1; + else if (strcmp(argv[op], "-o") == 0 && argc > 2 && + ((size_offsets = (int)strtoul(argv[op+1], &endptr, 10)), *endptr == 0)) + { + op++; + argc--; + } +#if !defined NOPOSIX else if (strcmp(argv[op], "-p") == 0) posix = 1; +#endif else { - printf("*** Unknown option %s\n", argv[op]); - printf("Usage: pcretest [-d] [-i] [-p] [-s] [-t] [<input> [<output>]]\n"); - printf(" -d debug: show compiled code; implies -i\n" - " -i show information about compiled pattern\n" - " -p use POSIX interface\n" - " -s output store information\n" - " -t time compilation and execution\n"); + printf("** Unknown or malformed option %s\n", argv[op]); + printf("Usage: pcretest [-d] [-i] [-o <n>] [-p] [-s] [-t] [<input> [<output>]]\n"); + printf(" -d debug: show compiled code; implies -i\n" + " -i show information about compiled pattern\n" + " -o <n> set size of offsets vector to <n>\n"); +#if !defined NOPOSIX + printf(" -p use POSIX interface\n"); +#endif + printf(" -s output store information\n" + " -t time compilation and execution\n"); return 1; } op++; argc--; } +/* Get the store for the offsets vector, and remember what it was */ + +size_offsets_max = size_offsets; +offsets = malloc(size_offsets_max * sizeof(int)); +if (offsets == NULL) + { + printf("** Failed to get %d bytes of memory for offsets vector\n", + size_offsets_max * sizeof(int)); + return 1; + } + /* Sort out the input and output files */ if (argc > 1) @@ -396,13 +542,14 @@ while (!done) const char *error; unsigned char *p, *pp, *ppp; - unsigned const char *tables = NULL; + const unsigned char *tables = NULL; int do_study = 0; int do_debug = debug; int do_G = 0; int do_g = 0; int do_showinfo = showinfo; int do_showrest = 0; + int utf8 = 0; int erroroffset, len, delimiter; if (infile == stdin) printf(" re> "); @@ -494,6 +641,7 @@ while (!done) case 'S': do_study = 1; break; case 'U': options |= PCRE_UNGREEDY; break; case 'X': options |= PCRE_EXTRA; break; + case '8': options |= PCRE_UTF8; utf8 = 1; break; case 'L': ppp = pp; @@ -594,13 +742,14 @@ while (!done) if (do_showinfo) { + unsigned long int get_options; int old_first_char, old_options, old_count; int count, backrefmax, first_char, need_char; size_t size; if (do_debug) print_internals(re); - new_info(re, NULL, PCRE_INFO_OPTIONS, &options); + new_info(re, NULL, PCRE_INFO_OPTIONS, &get_options); new_info(re, NULL, PCRE_INFO_SIZE, &size); new_info(re, NULL, PCRE_INFO_CAPTURECOUNT, &count); new_info(re, NULL, PCRE_INFO_BACKREFMAX, &backrefmax); @@ -620,9 +769,9 @@ while (!done) "First char disagreement: pcre_fullinfo=%d pcre_info=%d\n", first_char, old_first_char); - if (old_options != options) fprintf(outfile, - "Options disagreement: pcre_fullinfo=%d pcre_info=%d\n", options, - old_options); + if (old_options != (int)get_options) fprintf(outfile, + "Options disagreement: pcre_fullinfo=%ld pcre_info=%d\n", + get_options, old_options); } if (size != gotten_store) fprintf(outfile, @@ -632,16 +781,17 @@ while (!done) fprintf(outfile, "Capturing subpattern count = %d\n", count); if (backrefmax > 0) fprintf(outfile, "Max back reference = %d\n", backrefmax); - if (options == 0) fprintf(outfile, "No options\n"); - else fprintf(outfile, "Options:%s%s%s%s%s%s%s%s\n", - ((options & PCRE_ANCHORED) != 0)? " anchored" : "", - ((options & PCRE_CASELESS) != 0)? " caseless" : "", - ((options & PCRE_EXTENDED) != 0)? " extended" : "", - ((options & PCRE_MULTILINE) != 0)? " multiline" : "", - ((options & PCRE_DOTALL) != 0)? " dotall" : "", - ((options & PCRE_DOLLAR_ENDONLY) != 0)? " dollar_endonly" : "", - ((options & PCRE_EXTRA) != 0)? " extra" : "", - ((options & PCRE_UNGREEDY) != 0)? " ungreedy" : ""); + if (get_options == 0) fprintf(outfile, "No options\n"); + else fprintf(outfile, "Options:%s%s%s%s%s%s%s%s%s\n", + ((get_options & PCRE_ANCHORED) != 0)? " anchored" : "", + ((get_options & PCRE_CASELESS) != 0)? " caseless" : "", + ((get_options & PCRE_EXTENDED) != 0)? " extended" : "", + ((get_options & PCRE_MULTILINE) != 0)? " multiline" : "", + ((get_options & PCRE_DOTALL) != 0)? " dotall" : "", + ((get_options & PCRE_DOLLAR_ENDONLY) != 0)? " dollar_endonly" : "", + ((get_options & PCRE_EXTRA) != 0)? " extra" : "", + ((get_options & PCRE_UNGREEDY) != 0)? " ungreedy" : "", + ((get_options & PCRE_UTF8) != 0)? " utf8" : ""); if (((((real_pcre *)re)->options) & PCRE_ICHANGED) != 0) fprintf(outfile, "Case state changes\n"); @@ -744,6 +894,8 @@ while (!done) { unsigned char *q; unsigned char *bptr = dbuffer; + int *use_offsets = offsets; + int use_size_offsets = size_offsets; int count, c; int copystrings = 0; int getstrings = 0; @@ -751,8 +903,6 @@ while (!done) int gmatched = 0; int start_offset = 0; int g_notempty = 0; - int offsets[45]; - int size_offsets = sizeof(offsets)/sizeof(int); options = 0; @@ -796,6 +946,30 @@ while (!done) break; case 'x': + + /* Handle \x{..} specially - new Perl thing for utf8 */ + + if (*p == '{') + { + unsigned char *pt = p; + c = 0; + while (isxdigit(*(++pt))) + c = c * 16 + tolower(*pt) - ((isdigit(*pt))? '0' : 'W'); + if (*pt == '}') + { + unsigned char buffer[8]; + int ii, utn; + utn = ord2utf8(c, buffer); + for (ii = 0; ii < utn - 1; ii++) *q++ = buffer[ii]; + c = buffer[ii]; /* Last byte */ + p = pt + 1; + break; + } + /* Not correct form; fall through */ + } + + /* Ordinary \x */ + c = 0; while (i++ < 2 && isxdigit(*p)) { @@ -836,7 +1010,20 @@ while (!done) case 'O': while(isdigit(*p)) n = n * 10 + *p++ - '0'; - if (n <= (int)(sizeof(offsets)/sizeof(int))) size_offsets = n; + if (n > size_offsets_max) + { + size_offsets_max = n; + free(offsets); + use_offsets = offsets = malloc(size_offsets_max * sizeof(int)); + if (offsets == NULL) + { + printf("** Failed to get %d bytes of memory for offsets vector\n", + size_offsets_max * sizeof(int)); + return 1; + } + } + use_size_offsets = n; + if (n == 0) use_offsets = NULL; continue; case 'Z': @@ -856,11 +1043,11 @@ while (!done) { int rc; int eflags = 0; - regmatch_t pmatch[sizeof(offsets)/sizeof(int)]; + regmatch_t *pmatch = malloc(sizeof(regmatch_t) * use_size_offsets); if ((options & PCRE_NOTBOL) != 0) eflags |= REG_NOTBOL; if ((options & PCRE_NOTEOL) != 0) eflags |= REG_NOTEOL; - rc = regexec(&preg, (const char *)bptr, size_offsets, pmatch, eflags); + rc = regexec(&preg, (const char *)bptr, use_size_offsets, pmatch, eflags); if (rc != 0) { @@ -870,23 +1057,24 @@ while (!done) else { size_t i; - for (i = 0; i < size_offsets; i++) + for (i = 0; i < use_size_offsets; i++) { if (pmatch[i].rm_so >= 0) { fprintf(outfile, "%2d: ", (int)i); pchars(dbuffer + pmatch[i].rm_so, - pmatch[i].rm_eo - pmatch[i].rm_so); + pmatch[i].rm_eo - pmatch[i].rm_so, utf8); fprintf(outfile, "\n"); if (i == 0 && do_showrest) { fprintf(outfile, " 0+ "); - pchars(dbuffer + pmatch[i].rm_eo, len - pmatch[i].rm_eo); + pchars(dbuffer + pmatch[i].rm_eo, len - pmatch[i].rm_eo, utf8); fprintf(outfile, "\n"); } } } } + free(pmatch); } /* Handle matching via the native interface - repeats for /g and /G */ @@ -903,7 +1091,7 @@ while (!done) clock_t start_time = clock(); for (i = 0; i < LOOPREPEAT; i++) count = pcre_exec(re, extra, (char *)bptr, len, - start_offset, options | g_notempty, offsets, size_offsets); + start_offset, options | g_notempty, use_offsets, use_size_offsets); time_taken = clock() - start_time; fprintf(outfile, "Execute time %.3f milliseconds\n", ((double)time_taken * 1000.0)/ @@ -911,12 +1099,12 @@ while (!done) } count = pcre_exec(re, extra, (char *)bptr, len, - start_offset, options | g_notempty, offsets, size_offsets); + start_offset, options | g_notempty, use_offsets, use_size_offsets); if (count == 0) { fprintf(outfile, "Matched, but too many substrings\n"); - count = size_offsets/3; + count = use_size_offsets/3; } /* Matched */ @@ -926,19 +1114,19 @@ while (!done) int i; for (i = 0; i < count * 2; i += 2) { - if (offsets[i] < 0) + if (use_offsets[i] < 0) fprintf(outfile, "%2d: <unset>\n", i/2); else { fprintf(outfile, "%2d: ", i/2); - pchars(bptr + offsets[i], offsets[i+1] - offsets[i]); + pchars(bptr + use_offsets[i], use_offsets[i+1] - use_offsets[i], utf8); fprintf(outfile, "\n"); if (i == 0) { if (do_showrest) { fprintf(outfile, " 0+ "); - pchars(bptr + offsets[i+1], len - offsets[i+1]); + pchars(bptr + use_offsets[i+1], len - use_offsets[i+1], utf8); fprintf(outfile, "\n"); } } @@ -950,7 +1138,7 @@ while (!done) if ((copystrings & (1 << i)) != 0) { char copybuffer[16]; - int rc = pcre_copy_substring((char *)bptr, offsets, count, + int rc = pcre_copy_substring((char *)bptr, use_offsets, count, i, copybuffer, sizeof(copybuffer)); if (rc < 0) fprintf(outfile, "copy substring %d failed %d\n", i, rc); @@ -964,14 +1152,15 @@ while (!done) if ((getstrings & (1 << i)) != 0) { const char *substring; - int rc = pcre_get_substring((char *)bptr, offsets, count, + int rc = pcre_get_substring((char *)bptr, use_offsets, count, i, &substring); if (rc < 0) fprintf(outfile, "get substring %d failed %d\n", i, rc); else { fprintf(outfile, "%2dG %s (%d)\n", i, substring, rc); - free((void *)substring); + /* free((void *)substring); */ + pcre_free_substring(substring); } } } @@ -979,7 +1168,7 @@ while (!done) if (getlist) { const char **stringlist; - int rc = pcre_get_substring_list((char *)bptr, offsets, count, + int rc = pcre_get_substring_list((char *)bptr, use_offsets, count, &stringlist); if (rc < 0) fprintf(outfile, "get substring list failed %d\n", rc); @@ -989,7 +1178,8 @@ while (!done) fprintf(outfile, "%2dL %s\n", i, stringlist[i]); if (stringlist[i] != NULL) fprintf(outfile, "string list not terminated by NULL\n"); - free((void *)stringlist); + /* free((void *)stringlist); */ + pcre_free_substring_list(stringlist); } } } @@ -1004,8 +1194,8 @@ while (!done) { if (g_notempty != 0) { - offsets[0] = start_offset; - offsets[1] = start_offset + 1; + use_offsets[0] = start_offset; + use_offsets[1] = start_offset + 1; } else { @@ -1030,22 +1220,22 @@ while (!done) character. */ g_notempty = 0; - if (offsets[0] == offsets[1]) + if (use_offsets[0] == use_offsets[1]) { - if (offsets[0] == len) break; + if (use_offsets[0] == len) break; g_notempty = PCRE_NOTEMPTY | PCRE_ANCHORED; } /* For /g, update the start offset, leaving the rest alone */ - if (do_g) start_offset = offsets[1]; + if (do_g) start_offset = use_offsets[1]; /* For /G, update the pointer and length */ else { - bptr += offsets[1]; - len -= offsets[1]; + bptr += use_offsets[1]; + len -= use_offsets[1]; } } /* End of loop for /g and /G */ } /* End of loop for data lines */ diff --git a/srclib/pcre/testdata/testinput3 b/srclib/pcre/testdata/testinput3 index 67d39f3ac5..d3bd74fdd3 100644 --- a/srclib/pcre/testdata/testinput3 +++ b/srclib/pcre/testdata/testinput3 @@ -1707,4 +1707,18 @@ /a*/g abbab -/ End of test input / +/^[a-\d]/ + abcde + -things + 0digit + *** Failers + bcdef + +/^[\d-a]/ + abcde + -things + 0digit + *** Failers + bcdef + +/ End of testinput3 / diff --git a/srclib/pcre/testdata/testoutput1 b/srclib/pcre/testdata/testoutput1 index 3ead0561d1..f0047ffc40 100644 --- a/srclib/pcre/testdata/testoutput1 +++ b/srclib/pcre/testdata/testoutput1 @@ -1,4 +1,4 @@ -PCRE version 3.2 12-May-2000 +PCRE version 3.9 02-Jan-2002 /the quick brown fox/ the quick brown fox @@ -2079,15 +2079,6 @@ No match 0: abcABC 1: abc -/(main(O)?)+/ - mainmain - 0: mainmain - 1: main - mainOmain - 0: mainOmain - 1: main - 2: O - /ab{3cd/ ab{3cd 0: ab{3cd @@ -2920,5 +2911,108 @@ No match 0: 0: -/ End of test input / +/<tr([\w\W\s\d][^<>]{0,})><TD([\w\W\s\d][^<>]{0,})>([\d]{0,}\.)(.*)((<BR>([\w\W\s\d][^<>]{0,})|[\s]{0,}))<\/a><\/TD><TD([\w\W\s\d][^<>]{0,})>([\w\W\s\d][^<>]{0,})<\/TD><TD([\w\W\s\d][^<>]{0,})>([\w\W\s\d][^<>]{0,})<\/TD><\/TR>/is + <TR BGCOLOR='#DBE9E9'><TD align=left valign=top>43.<a href='joblist.cfm?JobID=94 6735&Keyword='>Word Processor<BR>(N-1286)</a></TD><TD align=left valign=top>Lega lstaff.com</TD><TD align=left valign=top>CA - Statewide</TD></TR> + 0: <TR BGCOLOR='#DBE9E9'><TD align=left valign=top>43.<a href='joblist.cfm?JobID=94 6735&Keyword='>Word Processor<BR>(N-1286)</a></TD><TD align=left valign=top>Lega lstaff.com</TD><TD align=left valign=top>CA - Statewide</TD></TR> + 1: BGCOLOR='#DBE9E9' + 2: align=left valign=top + 3: 43. + 4: <a href='joblist.cfm?JobID=94 6735&Keyword='>Word Processor<BR>(N-1286) + 5: + 6: + 7: <unset> + 8: align=left valign=top + 9: Lega lstaff.com +10: align=left valign=top +11: CA - Statewide + +/a[^a]b/ + acb + 0: acb + a\nb + 0: a\x0ab + +/a.b/ + acb + 0: acb + *** Failers +No match + a\nb +No match + +/a[^a]b/s + acb + 0: acb + a\nb + 0: a\x0ab + +/a.b/s + acb + 0: acb + a\nb + 0: a\x0ab + +/^(b+?|a){1,2}?c/ + bac + 0: bac + 1: a + bbac + 0: bbac + 1: a + bbbac + 0: bbbac + 1: a + bbbbac + 0: bbbbac + 1: a + bbbbbac + 0: bbbbbac + 1: a + +/^(b+|a){1,2}?c/ + bac + 0: bac + 1: a + bbac + 0: bbac + 1: a + bbbac + 0: bbbac + 1: a + bbbbac + 0: bbbbac + 1: a + bbbbbac + 0: bbbbbac + 1: a + +/(?!\A)x/m + x\nb\n +No match + a\bx\n + 0: x + +/\x0{ab}/ + \0{ab} + 0: \x00{ab} + +/(A|B)*?CD/ + CD + 0: CD + +/(A|B)*CD/ + CD + 0: CD + +/(AB)*?\1/ + ABABAB + 0: ABAB + 1: AB + +/(AB)*\1/ + ABABAB + 0: ABABAB + 1: AB + +/ End of testinput1 / diff --git a/srclib/pcre/testdata/testoutput2 b/srclib/pcre/testdata/testoutput2 index ba8cf0ed28..e8844d2aeb 100644 --- a/srclib/pcre/testdata/testoutput2 +++ b/srclib/pcre/testdata/testoutput2 @@ -1,4 +1,4 @@ -PCRE version 3.2 12-May-2000 +PCRE version 3.9 02-Jan-2002 /(a)b|/ Capturing subpattern count = 1 @@ -94,9 +94,6 @@ Failed: missing terminating ] for character class at offset 5 /[\B]/ Failed: invalid escape sequence in character class at offset 2 -/[a-\w]/ -Failed: invalid escape sequence in character class at offset 4 - /[z-a]/ Failed: range out of order in character class at offset 3 @@ -2064,7 +2061,318 @@ No match AB No match -/ End of test input / +/[\200-\410]/ +Failed: range out of order in character class at offset 9 + +/^(?(0)f|b)oo/ +Failed: invalid condition (?(0) at offset 5 + +/This one's here because of the large output vector needed/ +Capturing subpattern count = 0 +No options +First char = 'T' +Need char = 'd' + +/(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\d+(?:\s|$))(\w+)\s+(\270)/ +Capturing subpattern count = 271 +Max back reference = 270 +No options +No first char +No need char + \O900 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 ABC ABC + 0: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 ABC ABC + 1: 1 + 2: 2 + 3: 3 + 4: 4 + 5: 5 + 6: 6 + 7: 7 + 8: 8 + 9: 9 +10: 10 +11: 11 +12: 12 +13: 13 +14: 14 +15: 15 +16: 16 +17: 17 +18: 18 +19: 19 +20: 20 +21: 21 +22: 22 +23: 23 +24: 24 +25: 25 +26: 26 +27: 27 +28: 28 +29: 29 +30: 30 +31: 31 +32: 32 +33: 33 +34: 34 +35: 35 +36: 36 +37: 37 +38: 38 +39: 39 +40: 40 +41: 41 +42: 42 +43: 43 +44: 44 +45: 45 +46: 46 +47: 47 +48: 48 +49: 49 +50: 50 +51: 51 +52: 52 +53: 53 +54: 54 +55: 55 +56: 56 +57: 57 +58: 58 +59: 59 +60: 60 +61: 61 +62: 62 +63: 63 +64: 64 +65: 65 +66: 66 +67: 67 +68: 68 +69: 69 +70: 70 +71: 71 +72: 72 +73: 73 +74: 74 +75: 75 +76: 76 +77: 77 +78: 78 +79: 79 +80: 80 +81: 81 +82: 82 +83: 83 +84: 84 +85: 85 +86: 86 +87: 87 +88: 88 +89: 89 +90: 90 +91: 91 +92: 92 +93: 93 +94: 94 +95: 95 +96: 96 +97: 97 +98: 98 +99: 99 +100: 100 +101: 101 +102: 102 +103: 103 +104: 104 +105: 105 +106: 106 +107: 107 +108: 108 +109: 109 +110: 110 +111: 111 +112: 112 +113: 113 +114: 114 +115: 115 +116: 116 +117: 117 +118: 118 +119: 119 +120: 120 +121: 121 +122: 122 +123: 123 +124: 124 +125: 125 +126: 126 +127: 127 +128: 128 +129: 129 +130: 130 +131: 131 +132: 132 +133: 133 +134: 134 +135: 135 +136: 136 +137: 137 +138: 138 +139: 139 +140: 140 +141: 141 +142: 142 +143: 143 +144: 144 +145: 145 +146: 146 +147: 147 +148: 148 +149: 149 +150: 150 +151: 151 +152: 152 +153: 153 +154: 154 +155: 155 +156: 156 +157: 157 +158: 158 +159: 159 +160: 160 +161: 161 +162: 162 +163: 163 +164: 164 +165: 165 +166: 166 +167: 167 +168: 168 +169: 169 +170: 170 +171: 171 +172: 172 +173: 173 +174: 174 +175: 175 +176: 176 +177: 177 +178: 178 +179: 179 +180: 180 +181: 181 +182: 182 +183: 183 +184: 184 +185: 185 +186: 186 +187: 187 +188: 188 +189: 189 +190: 190 +191: 191 +192: 192 +193: 193 +194: 194 +195: 195 +196: 196 +197: 197 +198: 198 +199: 199 +200: 200 +201: 201 +202: 202 +203: 203 +204: 204 +205: 205 +206: 206 +207: 207 +208: 208 +209: 209 +210: 210 +211: 211 +212: 212 +213: 213 +214: 214 +215: 215 +216: 216 +217: 217 +218: 218 +219: 219 +220: 220 +221: 221 +222: 222 +223: 223 +224: 224 +225: 225 +226: 226 +227: 227 +228: 228 +229: 229 +230: 230 +231: 231 +232: 232 +233: 233 +234: 234 +235: 235 +236: 236 +237: 237 +238: 238 +239: 239 +240: 240 +241: 241 +242: 242 +243: 243 +244: 244 +245: 245 +246: 246 +247: 247 +248: 248 +249: 249 +250: 250 +251: 251 +252: 252 +253: 253 +254: 254 +255: 255 +256: 256 +257: 257 +258: 258 +259: 259 +260: 260 +261: 261 +262: 262 +263: 263 +264: 264 +265: 265 +266: 266 +267: 267 +268: 268 +269: 269 +270: ABC +271: ABC + +/This one's here because Perl does this differently and PCRE can't at present/ +Capturing subpattern count = 0 +No options +First char = 'T' +Need char = 't' + +/(main(O)?)+/ +Capturing subpattern count = 2 +No options +First char = 'm' +Need char = 'n' + mainmain + 0: mainmain + 1: main + mainOmain + 0: mainOmain + 1: main + 2: O + +/ End of testinput2 / Capturing subpattern count = 0 No options First char = ' ' diff --git a/srclib/pcre/testdata/testoutput3 b/srclib/pcre/testdata/testoutput3 index 0269f87034..cbe9aaa755 100644 --- a/srclib/pcre/testdata/testoutput3 +++ b/srclib/pcre/testdata/testoutput3 @@ -1,4 +1,4 @@ -PCRE version 3.2 12-May-2000 +PCRE version 3.9 02-Jan-2002 /(?<!bar)foo/ foo @@ -2963,5 +2963,29 @@ No match 0: 0: -/ End of test input / +/^[a-\d]/ + abcde + 0: a + -things + 0: - + 0digit + 0: 0 + *** Failers +No match + bcdef +No match + +/^[\d-a]/ + abcde + 0: a + -things + 0: - + 0digit + 0: 0 + *** Failers +No match + bcdef +No match + +/ End of testinput3 / diff --git a/srclib/pcre/testdata/testoutput4 b/srclib/pcre/testdata/testoutput4 index d285224515..df81a0f548 100644 --- a/srclib/pcre/testdata/testoutput4 +++ b/srclib/pcre/testdata/testoutput4 @@ -1,4 +1,4 @@ -PCRE version 3.2 12-May-2000 +PCRE version 3.9 02-Jan-2002 /^[\w]+/ *** Failers @@ -112,4 +112,5 @@ No match école No match +/ End of testinput4 / |