Make cconv work for dynamically bound code and always run it.
This allows later stages to benefit from transformations and
normalisations in cconv.
* lisp/emacs-lisp/bytecomp.el (byte-compile-preprocess): Always run
cconv.
* lisp/emacs-lisp/cconv.el (cconv--analyze-function)
(cconv-analyze-form): In dynbound code, treat all variable bindings
as dynamic (lambda, let, let* and condition-case).
Early normalisation of setq during macroexpand-all allows later
stages, cconv, byte-opt and codegen, to be simplified and duplicated
checks to be eliminated.
* lisp/emacs-lisp/macroexp.el (macroexp--expand-all):
Normalise all setq forms to a sequence of (setq VAR EXPR).
Emit warnings if necessary.
* lisp/emacs-lisp/cconv.el (cconv-convert, cconv-analyze-form):
* lisp/emacs-lisp/byte-opt.el (byte-optimize-form-code-walker):
* lisp/emacs-lisp/bytecomp.el (byte-compile-setq):
Simplify.
* test/lisp/emacs-lisp/bytecomp-tests.el: Adapt and add tests.
* test/lisp/emacs-lisp/bytecomp-resources/warn-variable-setq-nonvariable.el;
* test/lisp/emacs-lisp/bytecomp-resources/warn-variable-setq-odd.el:
New files.
These are symbols with position from source code, which should not be replaced
by bare symbols in, e.g., optimization functions.
* lisp/Makefile.in: (BYTE_COMPILE_FLAGS, compile-first case): Set
max-specpdl-size to 5000 for the benefit of lisp/emacs-lisp/comp.el.
* lisp/emacs-lisp/byte-opt.el (byte-optimize-form-code-walker)
(byte-optimize--rename-var, byte-optimize-if, byte-optimize-letX)
* lisp/emacs-lisp/bytecomp.el (byte-compile-recurse-toplevel)
(byte-compile-lambda)
* lisp/emacs-lisp/cconv.el (cconv-convert)
* lisp/emacs-lisp/macroexp.el (macroexp--expand-all): Preserve, e.g., (car
form) in the byte compiler, when this form's car is a symbol with position of
a special form, rather than replacing the symbol with a bare symbol, e.g.
'cond.
We here just add the new type. It is not fully self-contained.
It requires cooperation from `cconv.el` on the one hand, and it
hijacks the docstring info to hold the type of OClosure objects.
This does imply that OClosures can't have docstrings, tho this
limitation will be lifted in subsequent patches.
* lisp/emacs-lisp/oclosure.el: New file.
* test/lisp/emacs-lisp/oclosure-tests.el: New file.
* doc/lispref/functions.texi (OClosures): New section.
* src/eval.c (Ffunction): Accept symbols instead of strings for docstrings.
* src/doc.c (store_function_docstring): Avoid overwriting an OClosure type.
* lisp/emacs-lisp/cconv.el (cconv--convert-function): Tweak ordering of
captured variables.
(cconv-convert): Add case for `oclosure--fix-type`.
* lisp/emacs-lisp/cconv.el (cconv--convert-funcbody): Change the position
argument to macroexp--warn-wrap from BODY to ARG, to give correct positions in
warning messages.
* test/lisp/emacs-lisp/cconv-tests.el: Remove `:expected-result :failed`
from the bug#28557 tests.
(cconv-tests-cl-function-:documentation): Account for the presence of
the arglist (aka "usage") in the docstring.
* lisp/emacs-lisp/cl-macs.el (cl--transform-lambda):
* lisp/emacs-lisp/cl-generic.el (cl-defgeneric):
Handle non-constant `:documentation`.
* lisp/emacs-lisp/generator.el (iter-lambda):
* lisp/emacs-lisp/cconv.el (cconv--convert-funcbody):
Use `macroexp-parse-body`.
The position return by read-positioning-symbols is now the position in the
buffer, rather than the offset from the start of a form, enabling warning
positions in other parts of the buffer to be output.
* src/lisp.h (lisp_h_EQ): Add XLI casts so that it compiles cleanly.
* src/data.c (Fremove_pos_from_symbol): New DEFUN.
* src/lread.c (readchar_count): renamed to readchar_offset.
(read_internal_start) Initialize readchar_offset to the buffer's point when
STREAM is a buffer.
* lisp/emacs-lisp/bytecomp.el (byte-compile-warning-prefix): Amend to use
OFFSET as a buffer position, not an offset from the start of a form.
(byte-compile-warn): Remove symbol positions from any shape of ARGS, not just
a symbol with position.
* lisp/emacs-lisp/cconv.c (cconv-convert): In the :unused case, position the
new IGNORE symbol with the VAR it has replaced.
* lisp/emacs-lisp/macroexp.el (macroexp--warn-wrap, macroexp-warn-and-return):
Add an extra position parameter to each.
* lisp/emacs-lisp/bindat.el (bindat-type), lisp/emacs-lisp/byte-run.el
(defmacro, defun), lisp/emacs-lisp/cconv.el (cconv--convert-func-body)
(cconv-convert), lisp/emacs-lisp/cl-generic.el (cl-defmethod),
lisp/emacs-lisp/cl-macs.el (cl-symbol-macrolet, cl-defstruct),
lisp/emacs-lisp/easy-mmode.el (define-minor-mode),
lisp/emacs-lisp/eieio-core.el (eieio-oref, eieio-oref-default)
(eieio-oset-default), lisp/emacs-lisp/eieio.el (defclass),
lisp/emacs-lisp/gv.el (gv-ref), lisp/emacs-lisp/macroexp.el
(macroexp-macroexpand, macroexp--unfold-lambda, macroexp--expand-all),
lisp/emacs-lisp/pcase.el (pcase-compile-patterns, pcase--u1): Add an extra
position argument to each call of macroexp-warn-and-return.
Lambda-lifted variables (ones passed explicitly to lambda-lifted
functions) that are also captured in an outer closure and shadowed
were renamed incorrectly (bug#51982).
Reported by Paul Pogonyshev.
* lisp/emacs-lisp/cconv.el (cconv--lifted-arg): New.
(cconv-convert): Provide correct definiens for the closed-over
variable.
* test/lisp/emacs-lisp/bytecomp-tests.el (bytecomp-tests--test-cases):
* test/lisp/emacs-lisp/cconv-tests.el (cconv-tests--intern-all)
(cconv-closure-convert-remap-var): Add tests.
This branch is intended to generate correct position information in warning
and error messages from the byte compiler, and is intended thereby to fix bugs
It introduces a new mechanism, the symbol with position. This is taken over
from the previous git branch scratch/accurate-warning-pos which was abandoned
for being too slow. The main difference in the current branch is that the
symbol `nil' is never given a position, thus speeding up NILP markedly.
* lisp/emacs-lisp/byte-opt.el (byte-compile-inline-expand)
(byte-optimize-form-code-walker, byte-optimize-let-form, byte-optimize-while)
(byte-optimize-apply): Use byte-compile-warn-x in place of byte-compile-warn.
* lisp/emacs-lisp/bytecomp.el (byte-compile--form-stack): New variable.
(byte-compile-strip-s-p-1, byte-compile-strip-symbol-positions): New
functions.
(byte-compile-recurse-toplevel, byte-compile-initial-macro-environment)
(byte-compile-preprocess, byte-compile-macroexpand-declare-function): Bind
print-symbols-bare to non-nil.
(byte-compile--first-symbol, byte-compile--warning-source-offset): New
functions.
(byte-compile-warning-prefix): Modify to output two sets of position
information, the old (incorrect) set and the new set.
(byte-compile-warn): Strip positions from symbols before outputting.
(byte-compile-warn-x): New function which outputs a correct position supplied
in an argument.
(byte-compile-warn-obsolete, byte-compile-emit-callargs-warn)
(byte-compile-format-warn, byte-compile-nogroup-warn)
(byte-compile-arglist-warn, byte-compile-docstring-length-warn)
(byte-compile-warn-about-unresolved-functions, byte-compile-file)
(byte-compile--check-prefixed-var, byte-compile--declare-var)
(byte-compile-file-form-defvar-function, byte-compile-file-form-defmumble)
(byte-compile-check-lambda-list, byte-compile--warn-lexical-dynamic)
(byte-compile-lambda, byte-compile-form, byte-compile-normal-call)
(byte-compile-check-variable, byte-compile-free-vars-warn)
(byte-compile-subr-wrong-args, byte-compile-fset, byte-compile-set-default)
(byte-compile-condition-case, byte-compile-save-excursion)
(byte-compile-defvar, byte-compile-autoload)
(byte-compile-make-variable-buffer-local, byte-compile-define-symbol-prop)
(byte-compile-define-keymap): Replace byte-compile-warn with
byte-compile-warn-x.
(byte-compile-file, compile-defun): Bind symbols-with-pos-enabled to non-nil.
(compile-defun, byte-compile-from-buffer): Use `read-positioning-symbols'
rather than plain `read'.
(byte-compile-toplevel-file-form, byte-compile-form): Dynamically bind
byte-compile--form-stack.
(byte-compile-file-form-autoload, byte-compile-file-form-defvar)
(byte-compile-file-form-make-obsolete, byte-compile-lambda)
(byte-compile-push-constant, byte-compile-cond-jump-table)
(byte-compile-define-keymap, byte-compile-annotate-call-tree):
Strip positions from symbols where they are unwanted.
(byte-compile-file-form-defvar): Strip positions from symbols using
`bare-symbol'.
(byte-compile-file-form-defmumble): New variable bare-name, a version of name
without its position.
(byte-compile-lambda): Similarly, new variable bare-arglist.
(byte-compile-free-vars-warn): New argument arg supplying position information
to byte-compile-warn-x.
(byte-compile-push-constant): Manipulation of symbol positions.
(display-call-tree): Strip positions from symbols.
* lisp/emacs-lisp/cconv.el (cconv-convert, cconv--analyze-use)
(cconv--analyze-function, cconv-analyze-form): Replace use of
byte-compile-warn with byte-compile-warn-x.
* lisp/emacs-lisp/cl-generic.el (cl-defmethod): New variable org-name which
will supply position information to a new macroexp-warn-and-return.
* lisp/emacs-lisp/cl-macs.el (cl-macs--strip-s-p-1)
(cl-macs--strip-symbol-positions): New functions to strip positions from
symbols in an expression. These duplicaate similarly named functions in
bytecomp.el.
* lisp/emacs-lisp/macroexpand.el (macroexp--warn-wrap): Calls
byte-compile-warn-x in place of byte-compile-warn.
(macroexp-warn-and-return): Commented out new position parameter _arg.
* src/.gdbinit: Add in code to handle symbols with position.
* src/alloc.c (XPNTR, set_symbol_name, valid_lisp_object_p, purecopy)
(mark_char_table, mark_object, survives_gc_p, symbol_uses_obj): Use
BARE_SYMBOL_P and XBARE_SYMBOL in place of the former SYMBOLP and XSYMBOL.
(build_symbol_with_pos): New function.
(Fgarbage_collect): Bind Qsymbols_with_pos_enabled to nil around the call to
garbage_collect.
* src/data.c (Ftype_of): Add case for PVEC_SYMBOL_WITH_POS.
(Fbare_symbol_p, Fsymbol_with_pos_p, Fbare_symbol, Fsymbol_with_pos_pos)
(Fposition_symbol): New functions.
(symbols_with_pos_enabled): New boolean variable.
* src/fns.c (internal_equal, hash_lookup): Handle symbols with position.
* src/keyboard.c (recursive_edit_1): Bind Qsymbols_with_pos_enabled and
Qprint_symbols_bare to nil.
* src/lisp.h (lisp_h_PSEUDOVECTORP): New macro.
(lisp_h_BASE_EQ): New name for the former lisp_h_EQ.
(lisp_h_EQ): Extended to handle symbols with position.
(lisp_h_NILP): Now uses BASE_EQ rather than EQ.
(lisp_h_SYMBOL_WITH_POS_P, lisp_h_BARE_SYMBOL_P): New macros.
(lisp_h_SYMBOLP): Redefined to handle symbols with position.
(BARE_SYMBOL_P, BASE_EQ): New macros.
(SYMBOLP (macro)): Removed.
(SYMBOLP (function), XSYMBOL, make_lisp_symbol, builtin_lisp_symbol)
(c_symbol_p): Moved to later in file.
(struct Lisp_Symbol_With_Pos): New data type.
(pvec_type): PVEC_SYMBOL_WITH_POS: New type code.
(PSEUDOVECTORP): Redefined to use the lisp_h_PSEUDOVECTORP.
(BARE_SYMBOL_P, SYMBOL_WITH_POS_P, SYMBOLP, XSYMBOL_WITH_POS, XBARE_SYMBOL)
(XSYMBOL, make_lisp_symbol, builtin_lisp_symbol, c_symbol_p, CHECK_SYMBOL)
(BASE_EQ): New functions, or functions moved from earlier in the file.
(SYMBOL_WITH_POS_SYM, SYMBOL_WITH_POS_POS): New INLINE functions.
* src/lread.c (read0, read1, read_list, read_vector, read_internal_start)
(list2): Add a new bool parameter locate_syms.
(Fread_positioning_symbols): New function.
(Fread_from_string, read_internal_start, read0, read1, read_list): Pass around
suitable values for locate_syms.
(read1): Build symbols with position when locate_syms is true.
* src/print.c (print_vectorlike): Add handling for PVEC_SYMBOL_WITH_POS.
(print_object): Replace EQ with BASE_EQ.
(print_symbols_bare): New boolean variable.
* lisp/emacs-lisp/bytecomp.el (byte-compile-warning-types): Allow
inhibiting the `not-unused' warning (bug#31641). (There has been
some discussion about removing the `not-unused' warning, but it's
still in there, so making it possible to inhibit it seems like the
right thing to do.)
* lisp/emacs-lisp/cconv.el (cconv--analyze-use): Don't warn about
`not-unused'.
* lisp/emacs-lisp/cconv.el (cconv-analyze-form): Improve error
messaging when byte-compiling dotted lists (bug#35186).
Test case -- byte-compile the following file:
;;; -*- lexical-binding: t -*-
(defun foo ()
(+ 1 2)
(a . b))
Ensure in cconv that let-bindings have the normal form (VAR EXPR)
where VAR is a valid variable name, so that we don't need to keep
re-checking this all the time in the optimiser.
* lisp/emacs-lisp/byte-opt.el
(byte-optimize-enable-variable-constprop)
(byte-optimize-warn-eliminated-variable): Remove; these were mainly
used for debugging.
* lisp/emacs-lisp/byte-opt.el (byte-optimize-let-form):
Assume normalised let-bindings (with lexical-binding).
Stop using the variables removed above.
* lisp/emacs-lisp/cconv.el (cconv-convert): Ensure normalised
let-bindings. Malformed bindings are dropped after warning.
remove byte-optimize-warn-eliminated-variable
* lisp/emacs-lisp/byte-opt.el (byte-optimize-let-form): Move warnings...
* lisp/emacs-lisp/cconv.el (cconv-convert): ...here, which is an
overall better place (closer to the front-end).
Use it to obey `byte-compile-warnings`.
(macroexp--warn-wrap): Add arg `category`.
(macroexp-macroexpand, macroexp--expand-all): Use it.
* lisp/emacs-lisp/cconv.el (cconv--convert-funcbody, cconv-convert):
Mark the warnings as `lexical`.
* lisp/emacs-lisp/eieio-core.el (eieio-oref, eieio-oref-default)
(eieio-oset-default):
* lisp/emacs-lisp/eieio.el (defclass): Adjust to new calling convention.
(cconv--convert-funcbody): Check there's something after a docstring.
* test/lisp/emacs-lisp/bytecomp-tests.el (bytecomp-string-vs-docstring):
New corresponding test.
Instead of warning about unused vars during the analysis phase of
closure conversion, do it in the actual closure conversion by
annotating the code with "unused" warnings, so that the warnings
get emitted later by the bytecomp phase, like all other warnings,
at which point the line-number info is a bit less imprecise.
Take advantage of this change to wrap the expressions of unused
let-bound vars inside (ignore ...) so the byte-compiler can better
optimize them away.
Finally, promote `macroexp--warn-and-return` to "official" status
by removing its "--" marker.
(cconv-captured+mutated, cconv-lambda-candidates): Remove vars.
(cconv-var-classification): New var to replace them.
(cconv-warnings-only): Delete function.
(cconv--warn-unused-msg, cconv--var-classification): New functions.
(cconv--convert-funcbody): Add warnings for unused args.
(cconv-convert): Add warnings for unused vars in `let` and `condition-case`.
(cconv--analyze-use): Don't emit an "unused var" warning any more,
but instead remember the fact in `cconv-var-classification`.
* lisp/emacs-lisp/bytecomp.el (byte-compile-force-lexical-warnings):
Remove variable.
(byte-compile-preprocess): Remove corresponding case.
* lisp/emacs-lisp/pcase.el (pcase--if): Don't throw away `test` effects.
(\`):
* lisp/emacs-lisp/cl-macs.el (cl--do-arglist): Use `car-safe` instead
of `car`, so it can more easily be removed by the optimizer if the
result is not used.
* lisp/emacs-lisp/macroexp.el (macroexp--warn-wrap): New function.
(macroexp-warn-and-return): Rename from `macroexp--warn-and-return`.
Add some hints to the message for byte compiler free & unused variable
warnings, and 'void-variable' errors where the variable has confusable
quote characters in it.
* lisp/help.el (uni-confusables), uni-confusables-regexp): New
constants.
(help-command-error-confusable-suggestions): New function, added to
`command-error-function'.
(help-uni-confusable-suggestions): New function.
* lisp/emacs-lisp/bytecomp.el (byte-compile-variable-ref):
* lisp/emacs-lisp/cconv.el (cconv--analyze-use): Use it.
* lisp/emacs-lisp/lisp-mode.el
(lisp--match-confusable-symbol-character): New function.
(lisp-fdefs): Use it to fontify confusable characters with
font-lock-warning-face when they occur in symbol names.
* doc/lispref/modes.texi (Faces for Font Lock):
* doc/lispref/objects.texi (Basic Char Syntax): Recommend backslash
escaping of confusable characters, and mention new fontification.
* etc/NEWS: Announce the new fontification behavior.
* test/lisp/emacs-lisp/lisp-mode-tests.el (lisp-fontify-confusables):
New test.
Restore lines saying "Maintainer: emacs-devel@gnu.org" when there is
no special maintainer for a file. Although this wasn't documented
it was common practice and removing the lines didn't have consensus.
* lisp/emacs-lisp/cconv.el (cconv--convert-funcbody): New function.
(cconv--convert-function): Extracted from here.
(cconv-convert): Also use it here, in the lambda lifted case, so that
mutated args are properly accounted for.
* test/lisp/emacs-lisp/cconv-tests.el: New test.
Most of this change is to boilerplate commentary such as license URLs.
This change was prompted by ftp://ftp.gnu.org's going-away party,
planned for November. Change these FTP URLs to https://ftp.gnu.org
instead. Make similar changes for URLs to other organizations moving
away from FTP. Also, change HTTP to HTTPS for URLs to gnu.org and
fsf.org when this works, as this will further help defend against
man-in-the-middle attacks (for this part I omitted the MS-DOS and
MS-Windows sources and the test tarballs to keep the workload down).
HTTPS is not fully working to lists.gnu.org so I left those URLs alone
for now.
For errors, use ‘byte-compile-report-error’ instead so that the error
is registered and causes compilation to fail (Bug#24359).
For warnings, use ‘byte-compile-warn’ instead so that
‘byte-compile-error-on-warn’ is honored (Bug#24360).
* lisp/emacs-lisp/macroexp.el (macroexp--funcall-if-compiled)
(macroexp--warn-and-return): Use ‘byte-compile-warn’ instead of
‘byte-compile-log-warning’.
* lisp/emacs-lisp/bytecomp.el (byte-compile-form, byte-compile-unfold-bcf)
(byte-compile-setq, byte-compile-funcall): Use
‘byte-compile-report-error’ instead of ‘byte-compile-log-warning’.
(byte-compile-log-warning): Convert comment to documentation
string. Explain that the function shouldn’t be called directly.
(byte-compile-report-error): Add optional FILL argument.
* lisp/emacs-lisp/cconv.el (cconv-convert, cconv--analyze-use)
(cconv--analyze-function, cconv-analyze-form): Use
‘byte-compile-warn’ instead of ‘byte-compile-log-warning’.
* lisp/emacs-lisp/byte-opt.el (byte-compile-inline-expand): Use
‘byte-compile-warn’ instead of ‘byte-compile-log-warning’.
* lisp/subr.el (add-to-list): Use ‘byte-compile-report-error’ instead
of ‘byte-compile-log-warning’.
(do-after-load-evaluation): Use ‘byte-compile-warn’ instead of
‘byte-compile-log-warning’.
Change the code which detects and circumvents the case where one of the
variables used in λ-lifting is shadowed, so that it also works when the
shadowing comes before the λ-lifted function (bug#24171).
(cconv--remap-llv): New function, extracted from cconv-convert.
(cconv-convert): Use it, but differently for `let' and `let*'.
