This fixes bug #59213.
* lisp/emacs-lisp/cconv.el (cconv-dont-trim-unused-variables): New variable.
(cconv-fv, cconv-make-interpreted-closure): Add/amend doc strings.
(cconv-make-interpreted-closure): Test cconv-dont-trim-unused-variables, and
if non-nil, don't "optimize" the lexical environment.
* lisp/emacs-lisp/edebug.el (edebug-make-enter-wrapper): Compile a binding of
cconv-dont-trim-unused-variables to t around the call of edebug-enter.
* lisp/emacs-lisp/testconver.el (testcover-analyze-coverage): Add a new arm to
the pcase form to handle the new form of edebug-enter.
* lisp/emacs-lisp/cconv.el (cconv-convert, cconv-analyze-form):
Since we can't always remove the wrapper, make sure the wrapper
accepts the expected calling convention (i.e. called with no args).
Advising interactive forms relies on the ability to distinguish
interactive forms that do nothing else than return a function.
So, be careful to preserve this info.
Furthermore, interactive forms are expected to be evaluated in
the lexical context captured by the closure to which they belong,
so be careful to preserve that context when manipulating those forms.
* lisp/emacs-lisp/cconv.el (cconv-convert, cconv-analyze-form) <lambda>:
Preserve the info that an interactive form does nothing else than
return a function.
* lisp/emacs-lisp/nadvice.el (advice--interactive-form-1): New function.
(advice--interactive-form): Use it.
(advice--make-interactive-form): Refine to also accept function values
quoted with `quote`. Remove obsolete TODO.
* test/lisp/emacs-lisp/nadvice-tests.el: Don't disallow byte-compilation.
(advice-test-bug61179): New test.
* lisp/emacs-lisp/oclosure.el (cconv--interactive-helper): Allow
the `if` arg to be a form.
* lisp/simple.el (oclosure-interactive-form): Adjust accordingly.
The new code to make interpreted closures safe-for-space introduced
a regression in `cconv-tests-interactive-closure-bug51695`, only seen
when using TEST_LOAD_EL.
A few other issues were found and fixed along the way.
* lisp/emacs-lisp/cconv.el (cconv-fv): Change calling convention and
focus on finding the free variables.
(cconv-make-interpreted-closure): New function.
* lisp/loadup.el: Use `compiled-function-p` rather than
`byte-code-function-p` so we also use safe-for-space interpreted
closures when we build with native compilation.
(internal-make-interpreted-closure-function):
Use `cconv-make-interpreted-closure`.
* src/eval.c (syms_of_eval): Rename `internal-filter-closure-env-function`
to `internal-make-interpreted-closure-function`.
(Ffunction): Let that new var build the actual closure.
* test/lisp/emacs-lisp/cconv-tests.el
(cconv-tests-interactive-closure-bug51695): Test specifically the
interpreted case.
Interpreted closures currently just grab a reference to the complete
lexical environment, so (lambda (x) (+ x y)) can end up looking like
(closure ((foo ...) (y 7) (bar ...) ...)
(x) (+ x y))
where the foo/bar/... bindings are not only useless but can prevent
the GC from collecting that memory (i.e. it's a representation that is
not "safe for space") and it can also make that closure "unwritable"
(or more specifically, it can cause the closure's print
representation to be u`read`able).
Compiled closures don't suffer from this problem because `cconv.el`
actually looks at the code and only stores in the compiled closure
those variables which are actually used.
So, we fix this discrepancy by letting the existing code in `cconv.el` tell
`Ffunction` which variables are actually used by the body of the
function such that it can filter out the irrelevant elements and
return a closure of the form:
(closure ((y 7)) (x) (+ x y))
* lisp/loadup.el: Preload `cconv` and set
`internal-filter-closure-env-function` once we have a usable `cconv-fv`.
* lisp/emacs-lisp/bytecomp.el (byte-compile-preprocess): Adjust to new
calling convention of `cconv-closure-convert`.
(byte-compile-not-lexical-var-p): Delete function, moved to `cconv.el`.
(byte-compile-bind): Use `cconv--not-lexical-var-p`.
* lisp/emacs-lisp/cconv.el (cconv--dynbound-variables): New var.
(cconv-closure-convert): New arg `dynbound-vars`
(cconv--warn-unused-msg): Remove special case for `ignored`,
so we don't get confused when a function uses an argument called
`ignored`, e.g. holding a list of things that it should ignore.
(cconv--not-lexical-var-p): New function, moved from `bytecomp.el`.
Don't special case keywords and `nil` and `t` since they are already
`special-variable-p`.
(cconv--analyze-function): Use `cconv--not-lexical-var-p`.
(cconv--dynbindings): New dynbound var.
(cconv-analyze-form): Use `cconv--not-lexical-var-p`.
Remember in `cconv--dynbindings` the vars for which we used
dynamic scoping.
(cconv-analyze-form): Use `cconv--dynbound-variables` rather than
`byte-compile-bound-variables`.
(cconv-fv): New function.
* src/eval.c (Fsetq, eval_sub): Remove optimization designed when
`lexical-binding == nil` was the common case.
(Ffunction): Use `internal-filter-closure-env-function` when available.
(eval_sub, Ffuncall): Improve error info for `excessive_lisp_nesting`.
(internal-filter-closure-env-function): New defvar.
Make cconv.el detect when a closure's interactive form needs to
capture variables from the context and tweak the code accordingly
if so.
* lisp/emacs-lisp/cconv.el (cconv--interactive-form-funs): New var.
(cconv-convert): Handle the case where the interactive form captures
vars from the surrounding context. Remove left over handling of
`declare` which was already removed from the cconv-analyze` phase.
(cconv-analyze-form): Adjust analysis of interactive forms accordingly.
* lisp/emacs-lisp/oclosure.el (cconv--interactive-helper): New type and
function.
* lisp/simple.el (function-documentation, oclosure-interactive-form):
Add methods for it.
* test/lisp/emacs-lisp/cconv-tests.el
(cconv-tests-interactive-closure-bug51695): New test.
Make cconv work for dynamically bound code and always run it.
This allows later stages to benefit from transformations and
normalisations in cconv.
* lisp/emacs-lisp/bytecomp.el (byte-compile-preprocess): Always run
cconv.
* lisp/emacs-lisp/cconv.el (cconv--analyze-function)
(cconv-analyze-form): In dynbound code, treat all variable bindings
as dynamic (lambda, let, let* and condition-case).
Early normalisation of setq during macroexpand-all allows later
stages, cconv, byte-opt and codegen, to be simplified and duplicated
checks to be eliminated.
* lisp/emacs-lisp/macroexp.el (macroexp--expand-all):
Normalise all setq forms to a sequence of (setq VAR EXPR).
Emit warnings if necessary.
* lisp/emacs-lisp/cconv.el (cconv-convert, cconv-analyze-form):
* lisp/emacs-lisp/byte-opt.el (byte-optimize-form-code-walker):
* lisp/emacs-lisp/bytecomp.el (byte-compile-setq):
Simplify.
* test/lisp/emacs-lisp/bytecomp-tests.el: Adapt and add tests.
* test/lisp/emacs-lisp/bytecomp-resources/warn-variable-setq-nonvariable.el;
* test/lisp/emacs-lisp/bytecomp-resources/warn-variable-setq-odd.el:
New files.
These are symbols with position from source code, which should not be replaced
by bare symbols in, e.g., optimization functions.
* lisp/Makefile.in: (BYTE_COMPILE_FLAGS, compile-first case): Set
max-specpdl-size to 5000 for the benefit of lisp/emacs-lisp/comp.el.
* lisp/emacs-lisp/byte-opt.el (byte-optimize-form-code-walker)
(byte-optimize--rename-var, byte-optimize-if, byte-optimize-letX)
* lisp/emacs-lisp/bytecomp.el (byte-compile-recurse-toplevel)
(byte-compile-lambda)
* lisp/emacs-lisp/cconv.el (cconv-convert)
* lisp/emacs-lisp/macroexp.el (macroexp--expand-all): Preserve, e.g., (car
form) in the byte compiler, when this form's car is a symbol with position of
a special form, rather than replacing the symbol with a bare symbol, e.g.
'cond.
We here just add the new type. It is not fully self-contained.
It requires cooperation from `cconv.el` on the one hand, and it
hijacks the docstring info to hold the type of OClosure objects.
This does imply that OClosures can't have docstrings, tho this
limitation will be lifted in subsequent patches.
* lisp/emacs-lisp/oclosure.el: New file.
* test/lisp/emacs-lisp/oclosure-tests.el: New file.
* doc/lispref/functions.texi (OClosures): New section.
* src/eval.c (Ffunction): Accept symbols instead of strings for docstrings.
* src/doc.c (store_function_docstring): Avoid overwriting an OClosure type.
* lisp/emacs-lisp/cconv.el (cconv--convert-function): Tweak ordering of
captured variables.
(cconv-convert): Add case for `oclosure--fix-type`.
* lisp/emacs-lisp/cconv.el (cconv--convert-funcbody): Change the position
argument to macroexp--warn-wrap from BODY to ARG, to give correct positions in
warning messages.
* test/lisp/emacs-lisp/cconv-tests.el: Remove `:expected-result :failed`
from the bug#28557 tests.
(cconv-tests-cl-function-:documentation): Account for the presence of
the arglist (aka "usage") in the docstring.
* lisp/emacs-lisp/cl-macs.el (cl--transform-lambda):
* lisp/emacs-lisp/cl-generic.el (cl-defgeneric):
Handle non-constant `:documentation`.
* lisp/emacs-lisp/generator.el (iter-lambda):
* lisp/emacs-lisp/cconv.el (cconv--convert-funcbody):
Use `macroexp-parse-body`.
The position return by read-positioning-symbols is now the position in the
buffer, rather than the offset from the start of a form, enabling warning
positions in other parts of the buffer to be output.
* src/lisp.h (lisp_h_EQ): Add XLI casts so that it compiles cleanly.
* src/data.c (Fremove_pos_from_symbol): New DEFUN.
* src/lread.c (readchar_count): renamed to readchar_offset.
(read_internal_start) Initialize readchar_offset to the buffer's point when
STREAM is a buffer.
* lisp/emacs-lisp/bytecomp.el (byte-compile-warning-prefix): Amend to use
OFFSET as a buffer position, not an offset from the start of a form.
(byte-compile-warn): Remove symbol positions from any shape of ARGS, not just
a symbol with position.
* lisp/emacs-lisp/cconv.c (cconv-convert): In the :unused case, position the
new IGNORE symbol with the VAR it has replaced.
* lisp/emacs-lisp/macroexp.el (macroexp--warn-wrap, macroexp-warn-and-return):
Add an extra position parameter to each.
* lisp/emacs-lisp/bindat.el (bindat-type), lisp/emacs-lisp/byte-run.el
(defmacro, defun), lisp/emacs-lisp/cconv.el (cconv--convert-func-body)
(cconv-convert), lisp/emacs-lisp/cl-generic.el (cl-defmethod),
lisp/emacs-lisp/cl-macs.el (cl-symbol-macrolet, cl-defstruct),
lisp/emacs-lisp/easy-mmode.el (define-minor-mode),
lisp/emacs-lisp/eieio-core.el (eieio-oref, eieio-oref-default)
(eieio-oset-default), lisp/emacs-lisp/eieio.el (defclass),
lisp/emacs-lisp/gv.el (gv-ref), lisp/emacs-lisp/macroexp.el
(macroexp-macroexpand, macroexp--unfold-lambda, macroexp--expand-all),
lisp/emacs-lisp/pcase.el (pcase-compile-patterns, pcase--u1): Add an extra
position argument to each call of macroexp-warn-and-return.
Lambda-lifted variables (ones passed explicitly to lambda-lifted
functions) that are also captured in an outer closure and shadowed
were renamed incorrectly (bug#51982).
Reported by Paul Pogonyshev.
* lisp/emacs-lisp/cconv.el (cconv--lifted-arg): New.
(cconv-convert): Provide correct definiens for the closed-over
variable.
* test/lisp/emacs-lisp/bytecomp-tests.el (bytecomp-tests--test-cases):
* test/lisp/emacs-lisp/cconv-tests.el (cconv-tests--intern-all)
(cconv-closure-convert-remap-var): Add tests.
This branch is intended to generate correct position information in warning
and error messages from the byte compiler, and is intended thereby to fix bugs
It introduces a new mechanism, the symbol with position. This is taken over
from the previous git branch scratch/accurate-warning-pos which was abandoned
for being too slow. The main difference in the current branch is that the
symbol `nil' is never given a position, thus speeding up NILP markedly.
* lisp/emacs-lisp/byte-opt.el (byte-compile-inline-expand)
(byte-optimize-form-code-walker, byte-optimize-let-form, byte-optimize-while)
(byte-optimize-apply): Use byte-compile-warn-x in place of byte-compile-warn.
* lisp/emacs-lisp/bytecomp.el (byte-compile--form-stack): New variable.
(byte-compile-strip-s-p-1, byte-compile-strip-symbol-positions): New
functions.
(byte-compile-recurse-toplevel, byte-compile-initial-macro-environment)
(byte-compile-preprocess, byte-compile-macroexpand-declare-function): Bind
print-symbols-bare to non-nil.
(byte-compile--first-symbol, byte-compile--warning-source-offset): New
functions.
(byte-compile-warning-prefix): Modify to output two sets of position
information, the old (incorrect) set and the new set.
(byte-compile-warn): Strip positions from symbols before outputting.
(byte-compile-warn-x): New function which outputs a correct position supplied
in an argument.
(byte-compile-warn-obsolete, byte-compile-emit-callargs-warn)
(byte-compile-format-warn, byte-compile-nogroup-warn)
(byte-compile-arglist-warn, byte-compile-docstring-length-warn)
(byte-compile-warn-about-unresolved-functions, byte-compile-file)
(byte-compile--check-prefixed-var, byte-compile--declare-var)
(byte-compile-file-form-defvar-function, byte-compile-file-form-defmumble)
(byte-compile-check-lambda-list, byte-compile--warn-lexical-dynamic)
(byte-compile-lambda, byte-compile-form, byte-compile-normal-call)
(byte-compile-check-variable, byte-compile-free-vars-warn)
(byte-compile-subr-wrong-args, byte-compile-fset, byte-compile-set-default)
(byte-compile-condition-case, byte-compile-save-excursion)
(byte-compile-defvar, byte-compile-autoload)
(byte-compile-make-variable-buffer-local, byte-compile-define-symbol-prop)
(byte-compile-define-keymap): Replace byte-compile-warn with
byte-compile-warn-x.
(byte-compile-file, compile-defun): Bind symbols-with-pos-enabled to non-nil.
(compile-defun, byte-compile-from-buffer): Use `read-positioning-symbols'
rather than plain `read'.
(byte-compile-toplevel-file-form, byte-compile-form): Dynamically bind
byte-compile--form-stack.
(byte-compile-file-form-autoload, byte-compile-file-form-defvar)
(byte-compile-file-form-make-obsolete, byte-compile-lambda)
(byte-compile-push-constant, byte-compile-cond-jump-table)
(byte-compile-define-keymap, byte-compile-annotate-call-tree):
Strip positions from symbols where they are unwanted.
(byte-compile-file-form-defvar): Strip positions from symbols using
`bare-symbol'.
(byte-compile-file-form-defmumble): New variable bare-name, a version of name
without its position.
(byte-compile-lambda): Similarly, new variable bare-arglist.
(byte-compile-free-vars-warn): New argument arg supplying position information
to byte-compile-warn-x.
(byte-compile-push-constant): Manipulation of symbol positions.
(display-call-tree): Strip positions from symbols.
* lisp/emacs-lisp/cconv.el (cconv-convert, cconv--analyze-use)
(cconv--analyze-function, cconv-analyze-form): Replace use of
byte-compile-warn with byte-compile-warn-x.
* lisp/emacs-lisp/cl-generic.el (cl-defmethod): New variable org-name which
will supply position information to a new macroexp-warn-and-return.
* lisp/emacs-lisp/cl-macs.el (cl-macs--strip-s-p-1)
(cl-macs--strip-symbol-positions): New functions to strip positions from
symbols in an expression. These duplicaate similarly named functions in
bytecomp.el.
* lisp/emacs-lisp/macroexpand.el (macroexp--warn-wrap): Calls
byte-compile-warn-x in place of byte-compile-warn.
(macroexp-warn-and-return): Commented out new position parameter _arg.
* src/.gdbinit: Add in code to handle symbols with position.
* src/alloc.c (XPNTR, set_symbol_name, valid_lisp_object_p, purecopy)
(mark_char_table, mark_object, survives_gc_p, symbol_uses_obj): Use
BARE_SYMBOL_P and XBARE_SYMBOL in place of the former SYMBOLP and XSYMBOL.
(build_symbol_with_pos): New function.
(Fgarbage_collect): Bind Qsymbols_with_pos_enabled to nil around the call to
garbage_collect.
* src/data.c (Ftype_of): Add case for PVEC_SYMBOL_WITH_POS.
(Fbare_symbol_p, Fsymbol_with_pos_p, Fbare_symbol, Fsymbol_with_pos_pos)
(Fposition_symbol): New functions.
(symbols_with_pos_enabled): New boolean variable.
* src/fns.c (internal_equal, hash_lookup): Handle symbols with position.
* src/keyboard.c (recursive_edit_1): Bind Qsymbols_with_pos_enabled and
Qprint_symbols_bare to nil.
* src/lisp.h (lisp_h_PSEUDOVECTORP): New macro.
(lisp_h_BASE_EQ): New name for the former lisp_h_EQ.
(lisp_h_EQ): Extended to handle symbols with position.
(lisp_h_NILP): Now uses BASE_EQ rather than EQ.
(lisp_h_SYMBOL_WITH_POS_P, lisp_h_BARE_SYMBOL_P): New macros.
(lisp_h_SYMBOLP): Redefined to handle symbols with position.
(BARE_SYMBOL_P, BASE_EQ): New macros.
(SYMBOLP (macro)): Removed.
(SYMBOLP (function), XSYMBOL, make_lisp_symbol, builtin_lisp_symbol)
(c_symbol_p): Moved to later in file.
(struct Lisp_Symbol_With_Pos): New data type.
(pvec_type): PVEC_SYMBOL_WITH_POS: New type code.
(PSEUDOVECTORP): Redefined to use the lisp_h_PSEUDOVECTORP.
(BARE_SYMBOL_P, SYMBOL_WITH_POS_P, SYMBOLP, XSYMBOL_WITH_POS, XBARE_SYMBOL)
(XSYMBOL, make_lisp_symbol, builtin_lisp_symbol, c_symbol_p, CHECK_SYMBOL)
(BASE_EQ): New functions, or functions moved from earlier in the file.
(SYMBOL_WITH_POS_SYM, SYMBOL_WITH_POS_POS): New INLINE functions.
* src/lread.c (read0, read1, read_list, read_vector, read_internal_start)
(list2): Add a new bool parameter locate_syms.
(Fread_positioning_symbols): New function.
(Fread_from_string, read_internal_start, read0, read1, read_list): Pass around
suitable values for locate_syms.
(read1): Build symbols with position when locate_syms is true.
* src/print.c (print_vectorlike): Add handling for PVEC_SYMBOL_WITH_POS.
(print_object): Replace EQ with BASE_EQ.
(print_symbols_bare): New boolean variable.
* lisp/emacs-lisp/bytecomp.el (byte-compile-warning-types): Allow
inhibiting the `not-unused' warning (bug#31641). (There has been
some discussion about removing the `not-unused' warning, but it's
still in there, so making it possible to inhibit it seems like the
right thing to do.)
* lisp/emacs-lisp/cconv.el (cconv--analyze-use): Don't warn about
`not-unused'.
* lisp/emacs-lisp/cconv.el (cconv-analyze-form): Improve error
messaging when byte-compiling dotted lists (bug#35186).
Test case -- byte-compile the following file:
;;; -*- lexical-binding: t -*-
(defun foo ()
(+ 1 2)
(a . b))
Ensure in cconv that let-bindings have the normal form (VAR EXPR)
where VAR is a valid variable name, so that we don't need to keep
re-checking this all the time in the optimiser.
* lisp/emacs-lisp/byte-opt.el
(byte-optimize-enable-variable-constprop)
(byte-optimize-warn-eliminated-variable): Remove; these were mainly
used for debugging.
* lisp/emacs-lisp/byte-opt.el (byte-optimize-let-form):
Assume normalised let-bindings (with lexical-binding).
Stop using the variables removed above.
* lisp/emacs-lisp/cconv.el (cconv-convert): Ensure normalised
let-bindings. Malformed bindings are dropped after warning.
remove byte-optimize-warn-eliminated-variable
* lisp/emacs-lisp/byte-opt.el (byte-optimize-let-form): Move warnings...
* lisp/emacs-lisp/cconv.el (cconv-convert): ...here, which is an
overall better place (closer to the front-end).
Use it to obey `byte-compile-warnings`.
(macroexp--warn-wrap): Add arg `category`.
(macroexp-macroexpand, macroexp--expand-all): Use it.
* lisp/emacs-lisp/cconv.el (cconv--convert-funcbody, cconv-convert):
Mark the warnings as `lexical`.
* lisp/emacs-lisp/eieio-core.el (eieio-oref, eieio-oref-default)
(eieio-oset-default):
* lisp/emacs-lisp/eieio.el (defclass): Adjust to new calling convention.
(cconv--convert-funcbody): Check there's something after a docstring.
* test/lisp/emacs-lisp/bytecomp-tests.el (bytecomp-string-vs-docstring):
New corresponding test.
Instead of warning about unused vars during the analysis phase of
closure conversion, do it in the actual closure conversion by
annotating the code with "unused" warnings, so that the warnings
get emitted later by the bytecomp phase, like all other warnings,
at which point the line-number info is a bit less imprecise.
Take advantage of this change to wrap the expressions of unused
let-bound vars inside (ignore ...) so the byte-compiler can better
optimize them away.
Finally, promote `macroexp--warn-and-return` to "official" status
by removing its "--" marker.
(cconv-captured+mutated, cconv-lambda-candidates): Remove vars.
(cconv-var-classification): New var to replace them.
(cconv-warnings-only): Delete function.
(cconv--warn-unused-msg, cconv--var-classification): New functions.
(cconv--convert-funcbody): Add warnings for unused args.
(cconv-convert): Add warnings for unused vars in `let` and `condition-case`.
(cconv--analyze-use): Don't emit an "unused var" warning any more,
but instead remember the fact in `cconv-var-classification`.
* lisp/emacs-lisp/bytecomp.el (byte-compile-force-lexical-warnings):
Remove variable.
(byte-compile-preprocess): Remove corresponding case.
* lisp/emacs-lisp/pcase.el (pcase--if): Don't throw away `test` effects.
(\`):
* lisp/emacs-lisp/cl-macs.el (cl--do-arglist): Use `car-safe` instead
of `car`, so it can more easily be removed by the optimizer if the
result is not used.
* lisp/emacs-lisp/macroexp.el (macroexp--warn-wrap): New function.
(macroexp-warn-and-return): Rename from `macroexp--warn-and-return`.
Add some hints to the message for byte compiler free & unused variable
warnings, and 'void-variable' errors where the variable has confusable
quote characters in it.
* lisp/help.el (uni-confusables), uni-confusables-regexp): New
constants.
(help-command-error-confusable-suggestions): New function, added to
`command-error-function'.
(help-uni-confusable-suggestions): New function.
* lisp/emacs-lisp/bytecomp.el (byte-compile-variable-ref):
* lisp/emacs-lisp/cconv.el (cconv--analyze-use): Use it.
* lisp/emacs-lisp/lisp-mode.el
(lisp--match-confusable-symbol-character): New function.
(lisp-fdefs): Use it to fontify confusable characters with
font-lock-warning-face when they occur in symbol names.
* doc/lispref/modes.texi (Faces for Font Lock):
* doc/lispref/objects.texi (Basic Char Syntax): Recommend backslash
escaping of confusable characters, and mention new fontification.
* etc/NEWS: Announce the new fontification behavior.
* test/lisp/emacs-lisp/lisp-mode-tests.el (lisp-fontify-confusables):
New test.
Restore lines saying "Maintainer: emacs-devel@gnu.org" when there is
no special maintainer for a file. Although this wasn't documented
it was common practice and removing the lines didn't have consensus.
