Lazily convert numeric strings to Lisp numbers in Eshell

This should reduce the number of issues with Eshell converting strings
to numbers too aggressively and losing information (e.g. "001" -> 1)
while still allowing almost all of the beneficial uses, like summing a
list of numeric strings with '+'.

* lisp/eshell/esh-util.el (eshell--do-mark-numeric-string): New
function.
(eshell-convert-to-number): Make obsolete in favor of...
(eshell-mark-numeric-string): ... this.  Update callers.

* lisp/eshell/esh-arg.el (eshell--numberlike-p): New function...
(eshell-concat-1): ... use it.

* test/lisp/eshell/esh-util-tests.el:  Reimplement type conversion tests
to use 'eshell-convertible-to-number-p' instead.

* test/lisp/eshell/esh-var-tests.el
(esh-var-test/interp-var-splice-concat, esh-var-test/interp-concat-cmd)
(esh-var-test/interp-convert-var-split-indices)
(esh-var-test/interp-convert-quoted-var-split-indices)
(esh-var-test/interp-convert-cmd-multiline)
(esh-var-test/interp-convert-cmd-split-indices): Adjust tests to check
the new behavior.

* doc/misc/eshell.texi (Type Conversion): New section.
(Expansion): Clarify concatenation behavior.
This commit is contained in:
Jim Porter 2024-10-19 11:52:42 -07:00
parent c2a9f519f7
commit 43d5b7a04c
6 changed files with 155 additions and 85 deletions

View file

@ -432,16 +432,6 @@ This command writes a list of all files matching the glob pattern
@node Arguments
@section Arguments
Ordinarily, Eshell parses arguments in command form as either strings
or numbers, depending on what the parser thinks they look like. To
specify an argument of some other data type, you can use a Lisp form
(@pxref{Invocation}):
@example
~ $ echo (list 1 2 3)
(1 2 3)
@end example
When calling external commands (and many built-in Eshell commands,
too) Eshell will flatten the arguments the command receives, so
passing a list as an argument will ``spread'' the elements into
@ -454,13 +444,15 @@ multiple arguments:
3
@end example
@subsection Quoting and escaping
@subsection Quoting and Escaping
As with other shells, you can escape special characters and spaces by
prefixing the character with a backslash (@samp{\}), or by surrounding
the string with apostrophes (@samp{''}) or double quotes (@samp{""}).
This is needed especially for file names with special characters like
pipe (@samp{|}) or square brackets (@samp{[} or @samp{]}), which could
be part of remote file names.
be part of remote file names. In addition, quoting or escaping an
argument will prevent it from being converted to a number when passed to
a Lisp function.
When you escape a character with @samp{\} outside of any quotes, the
result is the literal character immediately following it. For
@ -495,7 +487,46 @@ When using expansions (@pxref{Expansion}) in an Eshell command, the
result may potentially be of any data type. To ensure that the result
is always a string, the expansion can be surrounded by double quotes.
@subsection Special argument types
@subsection Type Conversion
When invoking a Lisp function via command form, Eshell automatically
converts string arguments that look like numbers to actual Lisp
numbers in order to make it easier to work with numeric values. You can
prevent this conversion on a case-by-case basis by quoting or escaping
the argument:
@example
~ $ type-of 1
integer
~ $ type-of "1"
string
@end example
When invoking a subcommand in command form, Eshell will split the output
line-by-line into a list. Additionally, if every line looks like a
number, then Eshell will mark them as numeric so that passing them to a
Lisp function will convert them to Lisp numbers:
@example
~ $ cat numbers.txt
01
02
03
~ $ + $@@@{cat numbers.txt@}
6
@end example
If you find this behavior inconvenient for certain functions, you can
tell Eshell not to perform this conversion for that function:
@example
(put \\='find-file \\='eshell-no-numeric-conversions t)
@end example
@vindex eshell-convert-numeric-arguments
You can also disable this conversion behavior entirely by setting
@code{eshell-convert-numeric-arguments} to @code{nil}.
@subsection Special Argument Types
In addition to strings and numbers, Eshell supports a number of
special argument types. These let you refer to various other Emacs
Lisp data types, such as lists or buffers.
@ -1764,8 +1795,8 @@ behavior depends on the types of each value being concatenated:
Concatenate both values together.
@item one or both numbers
Concatenate the string representation of each value, converting back to
a number if possible.
Concatenate the string representation of each value. If either value is
numeric, mark the concatenated value as numeric if possible.
@item one or both (non-@code{nil}) lists
Concatenate ``adjacent'' elements of each value (possibly converting

View file

@ -271,15 +271,21 @@ would produce (\"abc\" \"d\")."
(t
(setq result (eshell-concat-1 quoted result i))))))))
(defsubst eshell--numberlike-p (object)
(or (numberp object)
(and (stringp object) (get-text-property 0 'number object))))
(defun eshell-concat-1 (quoted first second)
"Concatenate FIRST and SECOND.
If QUOTED is nil and either FIRST or SECOND are numbers, try to
convert the result to a number as well."
If QUOTED is nil and either FIRST or SECOND are numberlike, try to mark
the result as a number as well."
(let ((result (concat (eshell-stringify first) (eshell-stringify second))))
(if (and (not quoted)
(or (numberp first) (numberp second)))
(eshell-convert-to-number result)
result)))
(remove-text-properties 0 (length result) '(number) result)
(when (and (not quoted)
(or (eshell--numberlike-p first)
(eshell--numberlike-p second)))
(eshell-mark-numeric-string result))
result))
(defun eshell-concat-groups (quoted &rest args)
"Concatenate groups of arguments in ARGS and return the result.

View file

@ -343,11 +343,22 @@ If `eshell-convert-numeric-arguments', always return nil."
(concat "\\`\\s-*" eshell-number-regexp "\\s-*\\'")
string)))
(defsubst eshell--do-mark-numeric-string (string)
(put-text-property 0 (length string) 'number t string))
(defun eshell-mark-numeric-string (string)
"If STRING is convertible to a number, add a text property indicating so.
See `eshell-convertible-to-number-p'."
(when (eshell-convertible-to-number-p string)
(eshell--do-mark-numeric-string string))
string)
(defun eshell-convert-to-number (string)
"Try to convert STRING to a number.
If STRING doesn't look like a number (or
`eshell-convert-numeric-arguments' is nil), just return STRING
unchanged."
(declare (obsolete 'eshell-mark-numeric-string "31.1"))
(if (eshell-convertible-to-number-p string)
(string-to-number string)
string))
@ -376,10 +387,10 @@ trailing newlines removed. Otherwise, this behaves as follows:
(setq string (substring string 0 (1- len))))
(if (string-search "\n" string)
(let ((lines (split-string string "\n")))
(if (seq-every-p #'eshell-convertible-to-number-p lines)
(mapcar #'string-to-number lines)
lines))
(eshell-convert-to-number string)))))))
(when (seq-every-p #'eshell-convertible-to-number-p lines)
(mapc #'eshell--do-mark-numeric-string lines))
lines)
(eshell-mark-numeric-string string)))))))
(defvar-local eshell-path-env (getenv "PATH")
"Content of $PATH.

View file

@ -765,11 +765,10 @@ Otherwise, each INT-OR-NAME refers to an element of the list value.
Integers imply a direct index, and names, an associate lookup using
`assoc'.
If QUOTED is non-nil, this was invoked inside double-quotes.
This affects the behavior of splitting strings: without quoting,
the split values are converted to numbers via
`eshell-convert-to-number' if possible; with quoting, they're
left as strings.
If QUOTED is non-nil, this was invoked inside double-quotes. This
affects the behavior of splitting strings: without quoting, the split
values are marked as numbers via `eshell-mark-numeric-string' if
possible; with quoting, they're left as plain strings.
For example, to retrieve the second element of a user's record in
'/etc/passwd', the variable reference would look like:
@ -785,7 +784,7 @@ For example, to retrieve the second element of a user's record in
refs (cdr refs)))
(setq value (split-string value separator))
(unless quoted
(setq value (mapcar #'eshell-convert-to-number value)))))
(setq value (mapcar #'eshell-mark-numeric-string value)))))
(cond
((< (length refs) 0)
(error "Invalid array variable index: %s"

View file

@ -66,70 +66,70 @@
"Test that `eshell-stringify' correctly stringifies complex objects."
(should (equal (eshell-stringify (list 'quote 'hello)) "'hello")))
(ert-deftest esh-util-test/eshell-convert-to-number/integer ()
"Test that `eshell-convert-to-number' correctly converts integers."
(should (equal (eshell-convert-to-number "123") 123))
(should (equal (eshell-convert-to-number "-123") -123))
(ert-deftest esh-util-test/eshell-convertible-to-number-p/integer ()
"Test that `eshell-convertible-to-number-p' matches integers."
(should (eshell-convertible-to-number-p "123"))
(should (eshell-convertible-to-number-p "-123"))
;; These are technially integers, since Emacs Lisp requires at least
;; one digit after the "." to be a float:
(should (equal (eshell-convert-to-number "123.") 123))
(should (equal (eshell-convert-to-number "-123.") -123)))
(should (eshell-convertible-to-number-p "123."))
(should (eshell-convertible-to-number-p "-123.")))
(ert-deftest esh-util-test/eshell-convert-to-number/floating-point ()
"Test that `eshell-convert-to-number' correctly converts floats."
(should (equal (eshell-convert-to-number "1.23") 1.23))
(should (equal (eshell-convert-to-number "-1.23") -1.23))
(should (equal (eshell-convert-to-number ".1") 0.1))
(should (equal (eshell-convert-to-number "-.1") -0.1)))
(ert-deftest esh-util-test/eshell-convertible-to-number-p/float ()
"Test that `eshell-convertible-to-number-p' matches floats."
(should (eshell-convertible-to-number-p "1.23"))
(should (eshell-convertible-to-number-p "-1.23"))
(should (eshell-convertible-to-number-p ".1"))
(should (eshell-convertible-to-number-p "-.1")))
(ert-deftest esh-util-test/eshell-convert-to-number/floating-point-exponent ()
"Test that `eshell-convert-to-number' correctly converts exponent notation."
(ert-deftest esh-util-test/eshell-convertible-to-number-p/float-exponent ()
"Test that `eshell-convertible-to-number-p' matches exponent notation."
;; Positive exponent:
(dolist (exp '("e2" "e+2" "E2" "E+2"))
(should (equal (eshell-convert-to-number (concat "123" exp)) 12300.0))
(should (equal (eshell-convert-to-number (concat "-123" exp)) -12300.0))
(should (equal (eshell-convert-to-number (concat "1.23" exp)) 123.0))
(should (equal (eshell-convert-to-number (concat "-1.23" exp)) -123.0))
(should (equal (eshell-convert-to-number (concat "1." exp)) 100.0))
(should (equal (eshell-convert-to-number (concat "-1." exp)) -100.0))
(should (equal (eshell-convert-to-number (concat ".1" exp)) 10.0))
(should (equal (eshell-convert-to-number (concat "-.1" exp)) -10.0)))
(should (eshell-convertible-to-number-p (concat "123" exp)))
(should (eshell-convertible-to-number-p (concat "-123" exp)))
(should (eshell-convertible-to-number-p (concat "1.23" exp)))
(should (eshell-convertible-to-number-p (concat "-1.23" exp)))
(should (eshell-convertible-to-number-p (concat "1." exp)))
(should (eshell-convertible-to-number-p (concat "-1." exp)))
(should (eshell-convertible-to-number-p (concat ".1" exp)))
(should (eshell-convertible-to-number-p (concat "-.1" exp))))
;; Negative exponent:
(dolist (exp '("e-2" "E-2"))
(should (equal (eshell-convert-to-number (concat "123" exp)) 1.23))
(should (equal (eshell-convert-to-number (concat "-123" exp)) -1.23))
(should (equal (eshell-convert-to-number (concat "1.23" exp)) 0.0123))
(should (equal (eshell-convert-to-number (concat "-1.23" exp)) -0.0123))
(should (equal (eshell-convert-to-number (concat "1." exp)) 0.01))
(should (equal (eshell-convert-to-number (concat "-1." exp)) -0.01))
(should (equal (eshell-convert-to-number (concat ".1" exp)) 0.001))
(should (equal (eshell-convert-to-number (concat "-.1" exp)) -0.001))))
(should (eshell-convertible-to-number-p (concat "123" exp)))
(should (eshell-convertible-to-number-p (concat "-123" exp)))
(should (eshell-convertible-to-number-p (concat "1.23" exp)))
(should (eshell-convertible-to-number-p (concat "-1.23" exp)))
(should (eshell-convertible-to-number-p (concat "1." exp)))
(should (eshell-convertible-to-number-p (concat "-1." exp)))
(should (eshell-convertible-to-number-p (concat ".1" exp)))
(should (eshell-convertible-to-number-p (concat "-.1" exp)))))
(ert-deftest esh-util-test/eshell-convert-to-number/floating-point/infinite ()
"Test that `eshell-convert-to-number' correctly converts infinite floats."
(should (equal (eshell-convert-to-number "1.0e+INF") 1.0e+INF))
(should (equal (eshell-convert-to-number "2.e+INF") 1.0e+INF))
(should (equal (eshell-convert-to-number "-1.0e+INF") -1.0e+INF))
(should (equal (eshell-convert-to-number "-2.e+INF") -1.0e+INF)))
(ert-deftest esh-util-test/eshell-convertible-to-number-p/float/infinite ()
"Test that `eshell-convertible-to-number-p' matches infinite floats."
(should (eshell-convertible-to-number-p "1.0e+INF"))
(should (eshell-convertible-to-number-p "2.e+INF"))
(should (eshell-convertible-to-number-p "-1.0e+INF"))
(should (eshell-convertible-to-number-p "-2.e+INF")))
(ert-deftest esh-util-test/eshell-convert-to-number/floating-point/nan ()
"Test that `eshell-convert-to-number' correctly converts NaNs."
(should (equal (eshell-convert-to-number "1.0e+NaN") 1.0e+NaN))
(should (equal (eshell-convert-to-number "2.e+NaN") 2.0e+NaN))
(should (equal (eshell-convert-to-number "-1.0e+NaN") -1.0e+NaN))
(should (equal (eshell-convert-to-number "-2.e+NaN") -2.0e+NaN)))
(ert-deftest esh-util-test/eshell-convertible-to-number-p/float/nan ()
"Test that `eshell-convertible-to-number-p' matches NaNs."
(should (eshell-convertible-to-number-p "1.0e+NaN"))
(should (eshell-convertible-to-number-p "2.e+NaN"))
(should (eshell-convertible-to-number-p "-1.0e+NaN"))
(should (eshell-convertible-to-number-p "-2.e+NaN")))
(ert-deftest esh-util-test/eshell-convert-to-number/non-numeric ()
"Test that `eshell-convert-to-number' does nothing to non-numeric values."
(should (equal (eshell-convert-to-number "foo") "foo"))
(should (equal (eshell-convert-to-number "") ""))
(should (equal (eshell-convert-to-number "123foo") "123foo")))
(ert-deftest esh-util-test/eshell-convertible-to-number-p/non-numeric ()
"Test that `eshell-convertible-to-number-p' returns nil for non-numerics."
(should-not (eshell-convertible-to-number-p "foo"))
(should-not (eshell-convertible-to-number-p ""))
(should-not (eshell-convertible-to-number-p "123foo")))
(ert-deftest esh-util-test/eshell-convert-to-number/no-convert ()
"Test that `eshell-convert-to-number' does nothing when disabled."
(ert-deftest esh-util-test/eshell-convertible-to-number-p/no-convert ()
"Test that `eshell-convertible-to-number-p' returns nil when disabled."
(let ((eshell-convert-numeric-arguments nil))
(should (equal (eshell-convert-to-number "123") "123"))
(should (equal (eshell-convert-to-number "1.23") "1.23"))))
(should-not (eshell-convertible-to-number-p "123"))
(should-not (eshell-convertible-to-number-p "1.23"))))
(ert-deftest esh-util-test/eshell-printable-size ()
(should (equal (eshell-printable-size (expt 2 16)) "65536"))

View file

@ -231,7 +231,7 @@ nil, use FUNCTION instead."
;; into the first value of the non-spliced list.
(eshell-command-result-equal
"echo it is $@'eshell-test-value'$eshell-test-value"
'("it" "is" 1 2 (31 2 3)))))
'("it" "is" 1 2 ("31" 2 3)))))
(ert-deftest esh-var-test/interp-lisp ()
"Interpolate Lisp form evaluation."
@ -280,12 +280,16 @@ nil, use FUNCTION instead."
(eshell-command-result-equal "+ ${+ 1 2}3 3" 36)
(eshell-command-result-equal "echo ${*echo \"foo\nbar\"}-baz"
'("foo" "bar-baz"))
;; Concatenating to a number in a list should produce a number...
;; Concatenating to a number in a list should produce a numeric value...
(eshell-command-result-equal "echo ${*echo \"1\n2\"}3"
'("1" "23"))
(eshell-command-result-equal "echo $@{*echo \"1\n2\"}3"
'(1 23))
;; ... but concatenating to a string that looks like a number in a list
;; should produce a string.
(eshell-command-result-equal "echo ${*echo \"hi\n2\"}3"
'("hi" "23"))
(eshell-command-result-equal "echo $@{*echo \"hi\n2\"}3"
'("hi" "23")))
(ert-deftest esh-var-test/interp-concat-cmd2 ()
@ -491,11 +495,16 @@ nil, use FUNCTION instead."
(ert-deftest esh-var-test/interp-convert-var-split-indices ()
"Interpolate and convert string variable with indices."
;; Check that numeric forms are converted to numbers.
;; Check that numeric forms are marked as numeric.
(let ((eshell-test-value "000 010 020 030 040"))
;; `eshell/echo' converts numeric strings to Lisp numbers...
(eshell-command-result-equal "echo $eshell-test-value[0]"
0)
;; ... but not lists of numeric strings...
(eshell-command-result-equal "echo $eshell-test-value[0 2]"
'("000" "020"))
;; ... unless each element is a separate argument to `eshell/echo'.
(eshell-command-result-equal "echo $@eshell-test-value[0 2]"
'(0 20)))
;; Check that multiline forms are preserved as-is.
(let ((eshell-test-value "foo\nbar:baz\n"))
@ -515,9 +524,14 @@ nil, use FUNCTION instead."
(ert-deftest esh-var-test/interp-convert-quoted-var-split-indices ()
"Interpolate and convert quoted string variable with indices."
(let ((eshell-test-value "000 010 020 030 040"))
;; `eshell/echo' converts numeric strings to Lisp numbers...
(eshell-command-result-equal "echo $'eshell-test-value'[0]"
0)
;; ... but not lists of numeric strings...
(eshell-command-result-equal "echo $'eshell-test-value'[0 2]"
'("000" "020"))
;; ... unless each element is a separate argument to `eshell/echo'.
(eshell-command-result-equal "echo $@'eshell-test-value'[0 2]"
'(0 20))))
(ert-deftest esh-var-test/interp-convert-cmd-string-newline ()
@ -530,9 +544,13 @@ nil, use FUNCTION instead."
'("foo" "bar"))
;; Numeric output should be converted to numbers...
(eshell-command-result-equal "echo ${echo \"01\n02\n03\"}"
'("01" "02" "03"))
(eshell-command-result-equal "echo $@{echo \"01\n02\n03\"}"
'(1 2 3))
;; ... but only if every line is numeric.
(eshell-command-result-equal "echo ${echo \"01\n02\nhi\"}"
'("01" "02" "hi"))
(eshell-command-result-equal "echo $@{echo \"01\n02\nhi\"}"
'("01" "02" "hi")))
(ert-deftest esh-var-test/interp-convert-cmd-number ()
@ -541,9 +559,14 @@ nil, use FUNCTION instead."
(ert-deftest esh-var-test/interp-convert-cmd-split-indices ()
"Interpolate command result with indices."
;; `eshell/echo' converts numeric strings to Lisp numbers...
(eshell-command-result-equal "echo ${echo \"000 010 020\"}[0]"
0)
;; ... but not lists of numeric strings...
(eshell-command-result-equal "echo ${echo \"000 010 020\"}[0 2]"
'("000" "020"))
;; ... unless each element is a separate argument to `eshell/echo'.
(eshell-command-result-equal "echo $@{echo \"000 010 020\"}[0 2]"
'(0 20)))
(ert-deftest esh-var-test/quoted-interp-convert-var-number ()