* lisp/url/url-http.el (url-http-parse-headers): Parse redirect
URIs more like other web browsers (bug#42382).
RFC 7231 the Location header is defined to carry a URI-reference.
According to RFC 3986 it should be percent-encoded and thus should not
contain spaces. However, there are HTTP server implementation (notably
nginx) that do not do that. This makes Emacs url-http.el behave like
most other HTTP client implementatios. Also remove the stripping of
angle bracket quotes as they are not valid according to the RFCs.
Copyright-paperwork-exempt: yes
* lisp/url/url-http.el (url-http-create-request): Do not recreate
full URL for proxied HTTPS requests.
(url-https-proxy-after-change-function): Do not bind
url-http-proxy to nil before calling url-http-create-request.
(Bug#35969)
* lisp/url/url-http.el (url-http-parse-headers): Narrow the
buffer to the headers at the beginning to make sure
url-handle-content-transfer-encoding uses the correct
headers. (Bug#37023)
It is now called `byte-count-to-string-function', and used instead of
calling `file-size-human-readable' directly where appropriate.
* lisp/files.el (file-size-human-readable-iec): New.
(file-size-function): Rename to byte-count-to-string-function. Better
default value. Eliminate lambda. Better default for custom choice.
Put in group `files'. More descriptive doc string. Move.
(out-of-memory-warning-percentage, warn-maybe-out-of-memory)
(get-free-disk-space):
* lisp/dired.el (dired-number-of-marked-files):
* lisp/url/url-http.el (url-http-simple-after-change-function)
(url-http-content-length-after-change-function):
Use byte-count-to-string-function.
* test/lisp/files-test.el (files-test-file-size-human-readable):
Test file-size-human-readable-iec.
* lisp/url/url-http.el (url-handle-content-transfer-encoding): Modify
the message headers as well as the message body to reflect
decompression.
* lisp/mail/mail-utils.el (mail-fetch-field): Add DELETE argument, to
delete header lines included in the result.
* lisp/url/url-http.el (url-http-handle-authentication): Bail out
if the wrong credentials were passed to the server instead of
inflooping (bug#27022).
To improve readability of strings produced by
`file-size-human-readable', add two optional arguments:
- SPACE, to provide a string (typically a space or non-breaking space)
to put between the number and unit. For compatibility, the default is
an empty string.
- UNIT, a string to use as unit. For compatibility, the default is
"B" in `iec' mode and the empty string otherwise.
Also fix a glitch with small numbers in `iec' mode which caused a
stray "i" in the result.
* lisp/files.el (file-size-human-readable):
Add optional SPACE and UNIT arguments and handle small numbers correctly.
(files--ask-user-about-large-file, warn-maybe-out-of-memory):
Call with `iec' and space.
* test/lisp/files-tests.el (files-test-file-size-human-readable): New test.
* lisp/url/url-http.el (url-http-simple-after-change-function)
(url-http-content-length-after-change-function): Call with `iec' and space.
* etc/NEWS (Lisp Changes): Mention the change.
Restore lines saying "Maintainer: emacs-devel@gnu.org" when there is
no special maintainer for a file. Although this wasn't documented
it was common practice and removing the lines didn't have consensus.
This is a different fix for bug#34909, which should also fix bug#35739.
Our downloading code used to automatically decode the result according
to the usual heuristics for files. This caused problems when we later
needed to save the data in a file that needed to be byte-for-byte
equal to the original in order to pass the signature verification,
especially because we didn't keep track of which coding-system was
used to decode the data.
(package--unless-error): New macro extracted from
package--with-response-buffer-1, so that we can specify edebug and
indent specs.
(package--with-response-buffer-1): Use it. More importantly, change
code so it runs `body` in a unibyte buffer with undecoded data.
(package--download-one-archive): Don't encode with utf-8 since the data
is not decoded yet.
(describe-package-1): Explicitly decode the readem.txt files here.
* lisp/url/url-handlers.el (url-insert-file-contents): Use it.
(url-insert): Don't decode if buffer is unibyte.
* lisp/url/url-http.el (url-http--insert-file-helper): New function,
extracted from url-insert-file-contents.
* lisp/url/url-http.el
(url-http-chunked-encoding-after-change-function): Ensure that we
parse the entire initial chunked header as the length (bug#35658).
* lisp/url/url-http.el (url-http-debug):
* lisp/url/url-util.el (url-debug): Don't signal an error if
quit-flag is non-nil, but not t. This could happen because
some unrelated code is running inside while-no-input.
(Bug#34763)
* lisp/obsolete/tls.el: Moved from lisp/net/tls.el.
* lisp/gnus/nnimap.el:
* lisp/url/url-http.el: Don't require tls, since it's obsolete.
* lisp/net/network-stream.el: Only require tls if we actually try to
use it (i.e., when (gnutls-available-p) returns nil). Declare some
functions to fix compilation warnings.
* lisp/obsolete/starttls.el: Moved from lisp/net/starttls.el.
* lisp/net/sieve-manage.el:
* lisp/net/network-stream.el: Don't require `starttls' at the
top-level, declare the variables and functions used instead.
(network-stream-open-starttls): Only require `starttls' if
needed (i.e., gnutls-available-p fails).
* etc/NEWS: Announce obsoletion.
* lisp/net/gnutls.el (open-gnutls-stream): IDNA-encode hostnames
before passing them on to gnutls for verification.
* lisp/net/network-stream.el (network-stream-open-starttls): Ditto.
* lisp/url/url-http.el (url-http--get-referer): Be IDNA-aware.
(url-http-create-request): Don't de-Unicodify host names, because
they may be IDNA names (that are later encoded).
* lisp/url/url-util.el (url-domain): Be IDNA-aware when doing
domain name computations.
* doc/misc/url.texi (Customization): Describe the new user
option url-lastloc-privacy-level.
* lisp/net/eww.el (eww-render): Set url-current-lastloc to the
url we are rendering, to get the referer header right on
subsequent requests.
* lisp/url/url-http.el (url-http--get-referer): New function
to determine which referer to send, if any, considering the
users privacy settings and the target url we are visiting.
(url-http-referer): New variable keeping track of the referer
computed by url-http--get-referer
(url-http-create-request): Use url-http-referer instead of the
optional argument to set up the referer header. Leave
checking of privacy settings to url-http--get-referer.
(url-http): Set up url-http-referer by using
url-http--get-referer.
* lisp/url/url-queue.el (url-queue): New struct member
context-buffer for keeping track of the context a queued job
started from.
(url-queue-retrieve): Store the current buffer in the queue
object.
(url-queue-start-retrieve): Make sure url-retrieve is called
in the context of the original buffer, if available.
* lisp/url/url-util.el (url-domain): New function to determine
the domain of a given URL.
* lisp/url/url-vars.el (url-current-lastloc): New variable to
keep track of the desired "last location" (referer header).
(url-lastloc-privacy-level): New custom setting for more
fine-grained control over how lastloc (referer) is sent to
servers (Bug#27012).
Most of this change is to boilerplate commentary such as license URLs.
This change was prompted by ftp://ftp.gnu.org's going-away party,
planned for November. Change these FTP URLs to https://ftp.gnu.org
instead. Make similar changes for URLs to other organizations moving
away from FTP. Also, change HTTP to HTTPS for URLs to gnu.org and
fsf.org when this works, as this will further help defend against
man-in-the-middle attacks (for this part I omitted the MS-DOS and
MS-Windows sources and the test tarballs to keep the workload down).
HTTPS is not fully working to lists.gnu.org so I left those URLs alone
for now.
* lisp/url/url.el, lisp/url/url-cache.el, lisp/url/url-dav.el,
lisp/url/url-expand.el, lisp/url/url-file.el, lisp/url/url-imap.el,
lisp/url/url-ldap.el: Use `url-p' instead of `vectorp'.
* lisp/url/url-http.el (url-http): Check for type `url' instead of
`vector'.
