It has been effectively obsolete since Emacs 27, when the modern
extension mechanism was introduced.
* lisp/emacs-lisp/rx.el (rx-constituents): Make obsolete.
* test/lisp/emacs-lisp/rx-tests.el (rx-constituents): Suppress warning.
* etc/NEWS: Announce.
Translate or-patterns that (even partially) match single characters
into character alternatives which are more efficient in matching,
sometimes algorithmically so. Example:
(or "%" (in "a-z") space)
was previously translated to
"%\\|[a-z]\\|[[:space:]]"
but now becomes
"[%a-z[:space:]]"
Single-char patterns include `nonl` and `anychar`, which now can also
be used in set operations (union, complement and intersection), and
character classes. For example, `(or nonl "\n")` is now equivalent to
`anychar`.
* lisp/emacs-lisp/rx.el (rx--expand-def): Remove, split into...
(rx--expand-def-form, rx--expand-def-symbol): ...these.
(rx--translate-compat-symbol-entry)
(rx--translate-compat-form-entry): New functions for handling the
legacy extension mechanism.
(rx--normalise-or-arg): Renamed to...
(rx--normalise-char-pattern): ...this, and rewrite.
(rx--all-string-or-args): Remove, split into...
(rx--all-string-branches-p, rx--collect-or-strings): ...these.
(rx--char-alt-union, rx--intersection-intervals)
(rx--reduce-to-char-alt, rx--optimise-or-args)
(rx--translate-char-alt, rx--human-readable): New.
(rx--translate-or, rx--translate-not, rx--translate-intersection):
Rewrite.
(rx--charset-p, rx--intervals-to-alt, rx--charset-intervals)
(rx--charset-union, rx--charset-intersection, rx--charset-all)
(rx--translate-union): Remove.
(rx--generate-alt): Decide whether to generate a negated character
alternative.
(rx--complement-intervals, rx--intersect-intervals)
(rx--union-intervals): Rename to...
(rx--interval-set-complement, rx--interval-set-intersection)
(rx--interval-set-union): ...these.
(rx--translate-symbol, rx--translate-form): Refactor extension
processing. Handle synthetic `rx--char-alt` form.
* test/lisp/emacs-lisp/rx-tests.el (rx-or, rx-char-any-raw-byte)
(rx-any, rx-charset-or): Adapt to changes and extend.
* test/lisp/emacs-lisp/rx-tests.el (rx--complement-intervals)
(rx--union-intervals, rx--intersect-intervals): Rename to...
(rx--interval-set-complement, rx--interval-set-union)
(rx--interval-set-intersection): ...these.
* lisp/emacs-lisp/rx.el:
Add tables of legacy syntax.
(rx--translate-symbol):
Translate the legacy construct `not-wordchar` as (not wordchar), which
is more intuitively obvious.
* lisp/emacs-lisp/rx.el (rx--translate-syntax):
Generate the shorter \w and \W instead of \sw and \Sw.
* test/lisp/emacs-lisp/rx-tests.el (rx-atoms, rx-syntax, rx-not):
Adapt tests.
* lisp/emacs-lisp/rx.el (rx--generate-alt):
Treat the intervals and classes lists separately without joining,
to reduce allocation. Handle special cases first.
(rx--union-intervals):
Implement directly instead of using intersection and complement.
* test/lisp/emacs-lisp/rx-tests.el (rx-any): Adapt test, as some
character alternatives are now slightly different.
(rx--complement-intervals, rx--union-intervals)
(rx--intersect-intervals): New unit tests.
(rx (in (?^ . ?a))) was incorrectly translated to "[^-a]".
Change it so that we get "[_-a^]" instead.
* lisp/emacs-lisp/rx.el (rx--generate-alt): Split ranges starting with
`^` occurring first in a non-negated character alternative.
* test/lisp/emacs-lisp/rx-tests.el (rx-any): Add and adapt tests.
(cherry picked from commit 5f5d668ac7)
(rx (in (?^ . ?a))) was incorrectly translated to "[^-a]".
Change it so that we get "[_-a^]" instead.
* lisp/emacs-lisp/rx.el (rx--generate-alt): Split ranges starting with
`^` occurring first in a non-negated character alternative.
* test/lisp/emacs-lisp/rx-tests.el (rx-any): Add and adapt tests.
The Emacs regexp engine interprets character ranges from ASCII to raw
bytes, such as [a-\xfe], as not including non-ASCII Unicode at all;
ranges from non-ACII Unicode to raw bytes, such as [ü-\x91], are
ignored entirely.
To make rx produce a translation that works as intended, split ranges
that that go from ordinary characters to raw bytes. Such ranges may
appear from set manipulation and regexp optimisation.
* lisp/emacs-lisp/rx.el (rx--generate-alt): Split intervals that
straddle the char-raw boundary when rendering a string regexp from an
interval set.
* test/lisp/emacs-lisp/rx-tests.el (rx-char-any-raw-byte):
Add test cases.
Reported by Daniel Pittman.
* lisp/emacs-lisp/rx.el (rx): Move binding of rx--local-definitions...
(rx--to-expr): ...here.
* test/lisp/emacs-lisp/rx-tests.el (rx-let-pcase): New test.
pcase 'rx' patterns with a single named submatch, like
(rx (let x "a"))
would always succeed because of an over-optimistic transformation.
Patterns with 0 or more than 1 named submatches were not affected.
Reported by Philipp Stephani.
* lisp/emacs-lisp/rx.el (rx--pcase-macroexpander):
Special case for a single named submatch.
* test/lisp/emacs-lisp/rx-tests.el (rx-pcase): Add tests.
The pcase 'rx' pattern would in some cases allow the match data to be
clobbered before it is read. For example:
(pcase "PQR"
((and (rx (let a nonl)) (rx ?z)) (list 'one a))
((rx (let b ?Q)) (list 'two b)))
The above returned (two "P") instead of the correct (two "Q").
This occurred because the calls to string-match and match-string were
presented as separate patterns to pcase, which would interleave them
with other patterns.
As a remedy, combine string matching and match-data extraction into a
single pcase pattern. This introduces a slight inefficiency for two
or more submatches as they are grouped into a list structure which
then has to be destructured.
Found by Stefan Monnier. See discussion at
https://lists.gnu.org/archive/html/emacs-devel/2021-02/msg02010.html
* lisp/emacs-lisp/rx.el (rx--reduce-right): New helper.
(rx [pcase macro]): Combine string-match and match-string calls into a
single pcase pattern.
* test/lisp/emacs-lisp/rx-tests.el (rx-pcase): Add test cases.
Two unrelated bugs: A missing type check caused an error in rx
patterns for non-string match targets, and rx patterns did not work at
all in pcase-let or pcase-let*.
Second bug reported by Basil Contovounesios and Ag Ibragimov; fixes
proposed by Stefan Monnier. Discussion and explanation in thread at
https://lists.gnu.org/archive/html/emacs-devel/2021-02/msg01924.html
* lisp/emacs-lisp/rx.el (rx): Add (pred stringp) to avoid type errors,
and replace the `pred` clause for the actual match with something that
works with pcase-let(*) without being optimised away.
* test/lisp/emacs-lisp/rx-tests.el (rx-pcase): Add test cases.
* lisp/progmodes/cperl-mode.el (cperl--package-regexp):
Avoid double repetition; cperl--ws-or-comment-regexp is already
repeated with 1+.
* test/lisp/textmodes/dns-mode-tests.el
(dns-mode-tests-dns-mode-soa-increment-serial): Escape literal '$'.
* test/lisp/emacs-lisp/rx-tests.el (rx-regexp): Modify test to not
trigger a linting warning while retaining its testing power.
The argument of the rx `regexp` form is assumed to evaluate to a valid
regexp, but certain kinds of deprecated but still accepted usage were
not handled correctly, such as unescaped literal (special) characters:
(rx "a" (regexp "*")) => "a*" which is wrong.
Handle these cases; there is no extra trouble.
* lisp/emacs-lisp/rx.el (rx--translate-regexp): Force bracketing
of single special characters.
* test/lisp/emacs-lisp/rx-tests.el (rx-regexp): Add test case.
* test/lisp/emacs-lisp/rx-tests.el (rx-compat): Use with-no-warnings
instead of with-suppressed-warnings which complains when running
the test interactively.
This is a regression from Emacs 26.
Reported by Phillip Stephani.
* lisp/emacs-lisp/rx.el (rx--pcase-transform): Process ? and ?? correctly.
* test/lisp/emacs-lisp/rx-tests.el (rx-pcase): Add test case.
The ? and ?? rx operators are special in that they can be written as
characters (space and '?' respectively). This confused the definition
look-up mechanism in rare cases.
* lisp/emacs-lisp/rx.el (rx--expand-def): Don't look up non-symbols.
* test/lisp/emacs-lisp/rx-tests.el (rx-charset-or): Test.
Perform 'regexp-opt' on nested 'or' forms, and after expansion of
user-defined and 'eval' forms. Characters are now turned into strings
for wider 'regexp-opt' scope. This preserves the longest-match
semantics for string in 'or' forms over composition.
* doc/lispref/searching.texi (Rx Constructs): Document.
* lisp/emacs-lisp/rx.el (rx--normalise-or-arg)
(rx--all-string-or-args): New.
(rx--translate-or): Normalise arguments first, and check for strings
in subforms.
(rx--expand-eval): Extracted from rx--translate-eval.
(rx--translate-eval): Call rx--expand-eval.
* test/lisp/emacs-lisp/rx-tests.el (rx-or, rx-def-in-or): Add tests.
* etc/NEWS: Announce.
Revert to the Emacs 26 semantics that always gave the longest match
for rx 'or' forms with only string arguments. This guarantee was
never well documented, but it is useful and people likely have come to
rely on it. For example, prior to this change,
(rx (or ">" ">="))
matched ">" even if the text contained ">=".
* lisp/emacs-lisp/rx.el (rx--translate-or): Don't tell regexp-opt to
preserve the matching order.
* doc/lispref/searching.texi (Rx Constructs): Document the
longest-match guarantee for all-string 'or' forms.
* test/lisp/emacs-lisp/rx-tests.el (rx-or): Update test.
The `not' and `intersection' forms, and `or' inside these forms,
now accept characters and single-character strings as arguments.
Previously, they had to be wrapped in `any' forms.
This does not add expressive power but is a convenience and is easily
understood.
* doc/lispref/searching.texi (Rx Constructs): Amend the documentation.
* etc/NEWS: Announce the change.
* lisp/emacs-lisp/rx.el (rx--charset-p, rx--translate-not)
(rx--charset-intervals, rx): Accept characters and 1-char strings in
more places.
* test/lisp/emacs-lisp/rx-tests.el (rx-not, rx-charset-or)
(rx-def-in-charset-or, rx-intersection): Test the change.
These character set operations, together with `not' for set
complement, improve the compositionality of rx, and reduce duplication
in complicated cases. Named character classes are not permitted in
set operations.
* lisp/emacs-lisp/rx.el (rx--translate-any): Split into multiple
functions.
(rx--foldl, rx--parse-any, rx--generate-alt, rx--intervals-to-alt)
(rx--complement-intervals, rx--intersect-intervals)
(rx--union-intervals, rx--charset-intervals, rx--charset-union)
(rx--charset-all, rx--charset-intersection, rx--translate-union)
(rx--translate-intersection): New.
(rx--translate-not, rx--translate-form, rx--builtin-forms, rx):
Add `union' and `intersection'.
* test/lisp/emacs-lisp/rx-tests.el (rx-union ,rx-def-in-union)
(rx-intersection, rx-def-in-intersection): New tests.
* doc/lispref/searching.texi (Rx Constructs):
* etc/NEWS:
Document `union' and `intersection'.
For example, (any digit digit) should produce "[[:digit:]]",
not "[[:digit:][:digit:]]".
* lisp/emacs-lisp/rx.el (rx--translate-any): Deduplicate character classes.
* test/lisp/emacs-lisp/rx-tests.el (rx-any): Add test case.
* lisp/emacs-lisp/rx.el (rx--translate-symbol):
* test/lisp/emacs-lisp/rx-tests.el (rx-any, rx-atoms):
Use [^z-a] instead of ".\\|\n" for anychar.
The new expression is faster (about 2×) and does not allocate regexp
stack space. For example, (0+ anychar) now matches strings of any
size (bug#37659).
* lisp/emacs-lisp/rx.el (rx--translate-symbol, rx--builtin-symbols, rx):
* test/lisp/emacs-lisp/rx-tests.el (rx-atoms):
* doc/lispref/searching.texi (Rx Constructs):
* etc/NEWS:
Add `anychar', an alias for `anything'. Since `anychar' is more
descriptive (and slightly shorter), treat it as the preferred name.
It was an internal symbol in the old `rx' implementation, used in old
versions of the `flycheck' package.
* lisp/emacs-lisp/rx.el (rx-submatch-n): Alias of `rx-to-string'.
* test/lisp/emacs-lisp/rx-tests.el (rx-compat): Test it.
* lisp/emacs-lisp/rx.el:
* test/lisp/emacs-lisp/rx-tests.el:
* doc/lispref/searching.texi (Rx Constructs):
Rewrite rx for correctness, clarity, and performance. The new
implementation retains full compatibility and has more comprehensive
tests.
* lisp/emacs-lisp/re-builder.el (reb-rx-font-lock-keywords):
Adapt to changes in internal variables in rx.el.
* lisp/emacs-lisp/rx.el (rx-regexp, rx-literal): Check the cadr of the
form for stringness, not the form itself.
* test/lisp/emacs-lisp/rx-tests.el (rx-to-string-lisp-forms): New test.
* lisp/emacs-lisp/rx.el (rx-regexp): Allow non-string forms.
(rx-constituents): Add literal constituent, which is like a plain
STRING form, but allows arbitrary lisp expressions.
(rx-literal): New function.
(rx-compile-to-lisp): New variable.
(rx--subforms): New helper function for handling subforms, including
non-constant case.
(rx-group-if, rx-and, rx-or, rx-=, rx->=, rx-repeat, rx-submatch)
(rx-submatch-n, rx-kleene, rx-atomic-p): Use it to handle non-constant
subforms.
(rx): Document new form, wrap non-constant forms with concat call.
* test/lisp/emacs-lisp/rx-tests.el (rx-tests--match): New macro.
(rx-nonstring-expr, rx-nonstring-expr-non-greedy): New tests.
* etc/NEWS: Announce changes.
Make the rx `or' and `seq' forms accept zero arguments to produce a
never-matching regexp and an empty string, respectively.
* lisp/emacs-lisp/rx.el: Require cl-extra.
(rx-constituents, rx-or): Permit zero args.
(rx): Amend doc string for `or' and `seq'.
* test/lisp/emacs-lisp/rx-tests.el (rx-or, rx-seq): Test the change.
* etc/NEWS (Changes in Specialized Modes and Packages): Mention the change.
Make the rx `or' and `seq' forms accept zero arguments to produce a
never-matching regexp and an empty string, respectively.
* lisp/emacs-lisp/rx.el (rx-constituents, rx-or): Permit zero args.
(rx): Amend doc string for `or' and `seq'.
* test/lisp/emacs-lisp/rx-tests.el (rx-or, rx-seq): Test the change.
* etc/NEWS (Changes in Specialized Modes and Packages): Mention the change.
(any "a-Z0-9") generated "[0-9]", and (any (?9 . ?0)) generated "[9-0]".
Reversed ranges are either mistakes or abuse. Neither should be allowed.
etc/NEWS: Explain the change.
lisp/emacs-lisp/rx.el (rx): Document.
(rx-check-any-string, rx-check-any): Add error checks for reversed ranges.
test/lisp/emacs-lisp/rx-tests.el (rx-char-any-range-bad): New test.
Deal with lone backslashes that have no effect in string literals,
but indicate that something is amiss.
* lisp/auth-source-pass.el (auth-source-pass-entries):
* lisp/textmodes/artist.el (artist-figlet-get-font-list-windows):
* lisp/org/ob-abc.el (org-babel-expand-body:abc, org-babel-execute:abc):
* lisp/org/ob-forth.el (org-babel-forth-session-execute):
* lisp/vc/vc-git.el (vc-git--program-version):
Add backslash in regexp for correctness.
* lisp/gnus/nnmail.el (nnmail-split-abbrev-alist):
Replace `\||' with `\\|' to follow the obvious regexp intent.
* lisp/org/org-list.el (org-plain-list-ordered-item-terminator):
Add backslash in doc comment so that it appears as intended.
* lisp/progmodes/cc-engine.el (c-forward-decl-or-cast-1, c-end-of-decl-1):
* lisp/progmodes/f90.el (f90-font-lock-keywords-2):
* lisp/progmodes/etags.el (etags-tags-completion-table):
* lisp/progmodes/ruby-mode.el (ruby-syntax-propertize):
* test/lisp/emacs-lisp/cl-print-tests.el (cl-print-tests-1):
Remove superfluous backslashes from regexp.
* test/lisp/emacs-lisp/rx-tests.el (rx-char-any):
Remove superfluous backslash from doc comment.
The rx `or' form may reorder its arguments in an unpredictable way,
contrary to user expectation, since it sometimes uses `regexp-opt'.
Add a NOREORDER option to `regexp-opt' for preventing it from
producing a reordered regexp (Bug#34641).
* doc/lispref/searching.texi (Regular Expression Functions):
* etc/NEWS (Lisp Changes in Emacs 27.1):
Describe the new regexp-opt NOREORDER argument.
* lisp/emacs-lisp/regexp-opt.el (regexp-opt): Add NOREORDER.
Make no attempt at regexp improvement if the set of strings contains
a prefix of another string.
(regexp-opt--contains-prefix): New.
* lisp/emacs-lisp/rx.el (rx-or): Call regexp-opt with NOREORDER.
* test/lisp/emacs-lisp/rx-tests.el: Test rx `or' form match order.
`rx' incorrectly considers character ranges between ASCII and raw bytes to
cover all codes in-between, which includes all non-ASCII Unicode chars.
This causes (any "\000-\377" ?Å) to be simplified to (any "\000-\377"),
which is not at all the same thing: [\000-\377] really means
[\000-\177\200-\377] (Bug#34492).
* lisp/emacs-lisp/rx.el (rx-any-condense-range): Split ranges going
from ASCII to raw bytes.
* test/lisp/emacs-lisp/rx-tests.el (rx-char-any-raw-byte): Add test case.
* etc/NEWS: Mention the overall change (Bug#33205).
