Support casing characters which map into multiple code points (bug#24603)
Implement unconditional special casing rules defined in Unicode standard. Among other things, they deal with cases when a single code point is replaced by multiple ones because single character does not exist (e.g. ‘fi’ ligature turning into ‘FL’) or is not commonly used (e.g. ß turning into SS). * admin/unidata/SpecialCasing.txt: New data file pulled from Unicode standard distribution. * admin/unidata/README: Mention SpecialCasing.txt. * admin/unidata/unidata-get.el (unidata-gen-table-special-casing, unidata-gen-table-special-casing--do-load): New functions generating ‘special-uppercase’, ‘special-lowercase’ and ‘special-titlecase’ character Unicode properties built from the SpecialCasing.txt Unicode data file. * src/casefiddle.c (struct casing_str_buf): New structure for representing short strings used to handle one-to-many character mappings. (case_character_imlp): New function which can handle one-to-many character mappings. (case_character, case_single_character): Wrappers for the above functions. The former may map one character to multiple (or no) code points while the latter does what the former used to do (i.e. handles one-to-one mappings only). (do_casify_natnum, do_casify_unibyte_string, do_casify_unibyte_region): Use case_single_character. (do_casify_multibyte_string, do_casify_multibyte_region): Support new features of case_character. * (do_casify_region): Updated to reflact do_casify_multibyte_string changes. (casify_word): Handle situation when one character-length of a word can change affecting where end of the word is. (upcase, capitalize, upcase-initials): Update documentation to mention limitations when working on characters. * test/src/casefiddle-tests.el (casefiddle-tests-char-properties): Add test cases for the newly introduced character properties. (casefiddle-tests-casing): Update test cases which are now passing. * test/lisp/char-fold-tests.el (char-fold--ascii-upcase, char-fold--ascii-downcase): New functions which behave like old ‘upcase’ and ‘downcase’. (char-fold--test-match-exactly): Use the new functions. This is needed because otherwise fi and similar characters are turned into their multi- -character representation. * doc/lispref/strings.texi: Describe issue with casing characters versus strings. * doc/lispref/nonascii.texi: Describe the new character properties.
This commit is contained in:
Michal Nazarewicz
parent
2c87dabd04
commit
b3b9b258c4
9 changed files with 679 additions and 128 deletions
|
|
@ -24,3 +24,7 @@ http://www.unicode.org/Public/8.0.0/ucd/Blocks.txt
|
|||
NormalizationTest.txt
|
||||
http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt
|
||||
2016-07-16
|
||||
|
||||
SpecialCasing.txt
|
||||
http://unicode.org/Public/UNIDATA/SpecialCasing.txt
|
||||
2016-03-03
|
||||
|
|
|
|||
281
admin/unidata/SpecialCasing.txt
Normal file
281
admin/unidata/SpecialCasing.txt
Normal file
|
|
@ -0,0 +1,281 @@
|
|||
# SpecialCasing-9.0.0.txt
|
||||
# Date: 2016-03-02, 18:55:13 GMT
|
||||
# © 2016 Unicode®, Inc.
|
||||
# Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the U.S. and other countries.
|
||||
# For terms of use, see http://www.unicode.org/terms_of_use.html
|
||||
#
|
||||
# Unicode Character Database
|
||||
# For documentation, see http://www.unicode.org/reports/tr44/
|
||||
#
|
||||
# Special Casing
|
||||
#
|
||||
# This file is a supplement to the UnicodeData.txt file. It does not define any
|
||||
# properties, but rather provides additional information about the casing of
|
||||
# Unicode characters, for situations when casing incurs a change in string length
|
||||
# or is dependent on context or locale. For compatibility, the UnicodeData.txt
|
||||
# file only contains simple case mappings for characters where they are one-to-one
|
||||
# and independent of context and language. The data in this file, combined with
|
||||
# the simple case mappings in UnicodeData.txt, defines the full case mappings
|
||||
# Lowercase_Mapping (lc), Titlecase_Mapping (tc), and Uppercase_Mapping (uc).
|
||||
#
|
||||
# Note that the preferred mechanism for defining tailored casing operations is
|
||||
# the Unicode Common Locale Data Repository (CLDR). For more information, see the
|
||||
# discussion of case mappings and case algorithms in the Unicode Standard.
|
||||
#
|
||||
# All code points not listed in this file that do not have a simple case mappings
|
||||
# in UnicodeData.txt map to themselves.
|
||||
# ================================================================================
|
||||
# Format
|
||||
# ================================================================================
|
||||
# The entries in this file are in the following machine-readable format:
|
||||
#
|
||||
# <code>; <lower>; <title>; <upper>; (<condition_list>;)? # <comment>
|
||||
#
|
||||
# <code>, <lower>, <title>, and <upper> provide the respective full case mappings
|
||||
# of <code>, expressed as character values in hex. If there is more than one character,
|
||||
# they are separated by spaces. Other than as used to separate elements, spaces are
|
||||
# to be ignored.
|
||||
#
|
||||
# The <condition_list> is optional. Where present, it consists of one or more language IDs
|
||||
# or casing contexts, separated by spaces. In these conditions:
|
||||
# - A condition list overrides the normal behavior if all of the listed conditions are true.
|
||||
# - The casing context is always the context of the characters in the original string,
|
||||
# NOT in the resulting string.
|
||||
# - Case distinctions in the condition list are not significant.
|
||||
# - Conditions preceded by "Not_" represent the negation of the condition.
|
||||
# The condition list is not represented in the UCD as a formal property.
|
||||
#
|
||||
# A language ID is defined by BCP 47, with '-' and '_' treated equivalently.
|
||||
#
|
||||
# A casing context for a character is defined by Section 3.13 Default Case Algorithms
|
||||
# of The Unicode Standard.
|
||||
#
|
||||
# Parsers of this file must be prepared to deal with future additions to this format:
|
||||
# * Additional contexts
|
||||
# * Additional fields
|
||||
# ================================================================================
|
||||
|
||||
# ================================================================================
|
||||
# Unconditional mappings
|
||||
# ================================================================================
|
||||
|
||||
# The German es-zed is special--the normal mapping is to SS.
|
||||
# Note: the titlecase should never occur in practice. It is equal to titlecase(uppercase(<es-zed>))
|
||||
|
||||
00DF; 00DF; 0053 0073; 0053 0053; # LATIN SMALL LETTER SHARP S
|
||||
|
||||
# Preserve canonical equivalence for I with dot. Turkic is handled below.
|
||||
|
||||
0130; 0069 0307; 0130; 0130; # LATIN CAPITAL LETTER I WITH DOT ABOVE
|
||||
|
||||
# Ligatures
|
||||
|
||||
FB00; FB00; 0046 0066; 0046 0046; # LATIN SMALL LIGATURE FF
|
||||
FB01; FB01; 0046 0069; 0046 0049; # LATIN SMALL LIGATURE FI
|
||||
FB02; FB02; 0046 006C; 0046 004C; # LATIN SMALL LIGATURE FL
|
||||
FB03; FB03; 0046 0066 0069; 0046 0046 0049; # LATIN SMALL LIGATURE FFI
|
||||
FB04; FB04; 0046 0066 006C; 0046 0046 004C; # LATIN SMALL LIGATURE FFL
|
||||
FB05; FB05; 0053 0074; 0053 0054; # LATIN SMALL LIGATURE LONG S T
|
||||
FB06; FB06; 0053 0074; 0053 0054; # LATIN SMALL LIGATURE ST
|
||||
|
||||
0587; 0587; 0535 0582; 0535 0552; # ARMENIAN SMALL LIGATURE ECH YIWN
|
||||
FB13; FB13; 0544 0576; 0544 0546; # ARMENIAN SMALL LIGATURE MEN NOW
|
||||
FB14; FB14; 0544 0565; 0544 0535; # ARMENIAN SMALL LIGATURE MEN ECH
|
||||
FB15; FB15; 0544 056B; 0544 053B; # ARMENIAN SMALL LIGATURE MEN INI
|
||||
FB16; FB16; 054E 0576; 054E 0546; # ARMENIAN SMALL LIGATURE VEW NOW
|
||||
FB17; FB17; 0544 056D; 0544 053D; # ARMENIAN SMALL LIGATURE MEN XEH
|
||||
|
||||
# No corresponding uppercase precomposed character
|
||||
|
||||
0149; 0149; 02BC 004E; 02BC 004E; # LATIN SMALL LETTER N PRECEDED BY APOSTROPHE
|
||||
0390; 0390; 0399 0308 0301; 0399 0308 0301; # GREEK SMALL LETTER IOTA WITH DIALYTIKA AND TONOS
|
||||
03B0; 03B0; 03A5 0308 0301; 03A5 0308 0301; # GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND TONOS
|
||||
01F0; 01F0; 004A 030C; 004A 030C; # LATIN SMALL LETTER J WITH CARON
|
||||
1E96; 1E96; 0048 0331; 0048 0331; # LATIN SMALL LETTER H WITH LINE BELOW
|
||||
1E97; 1E97; 0054 0308; 0054 0308; # LATIN SMALL LETTER T WITH DIAERESIS
|
||||
1E98; 1E98; 0057 030A; 0057 030A; # LATIN SMALL LETTER W WITH RING ABOVE
|
||||
1E99; 1E99; 0059 030A; 0059 030A; # LATIN SMALL LETTER Y WITH RING ABOVE
|
||||
1E9A; 1E9A; 0041 02BE; 0041 02BE; # LATIN SMALL LETTER A WITH RIGHT HALF RING
|
||||
1F50; 1F50; 03A5 0313; 03A5 0313; # GREEK SMALL LETTER UPSILON WITH PSILI
|
||||
1F52; 1F52; 03A5 0313 0300; 03A5 0313 0300; # GREEK SMALL LETTER UPSILON WITH PSILI AND VARIA
|
||||
1F54; 1F54; 03A5 0313 0301; 03A5 0313 0301; # GREEK SMALL LETTER UPSILON WITH PSILI AND OXIA
|
||||
1F56; 1F56; 03A5 0313 0342; 03A5 0313 0342; # GREEK SMALL LETTER UPSILON WITH PSILI AND PERISPOMENI
|
||||
1FB6; 1FB6; 0391 0342; 0391 0342; # GREEK SMALL LETTER ALPHA WITH PERISPOMENI
|
||||
1FC6; 1FC6; 0397 0342; 0397 0342; # GREEK SMALL LETTER ETA WITH PERISPOMENI
|
||||
1FD2; 1FD2; 0399 0308 0300; 0399 0308 0300; # GREEK SMALL LETTER IOTA WITH DIALYTIKA AND VARIA
|
||||
1FD3; 1FD3; 0399 0308 0301; 0399 0308 0301; # GREEK SMALL LETTER IOTA WITH DIALYTIKA AND OXIA
|
||||
1FD6; 1FD6; 0399 0342; 0399 0342; # GREEK SMALL LETTER IOTA WITH PERISPOMENI
|
||||
1FD7; 1FD7; 0399 0308 0342; 0399 0308 0342; # GREEK SMALL LETTER IOTA WITH DIALYTIKA AND PERISPOMENI
|
||||
1FE2; 1FE2; 03A5 0308 0300; 03A5 0308 0300; # GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND VARIA
|
||||
1FE3; 1FE3; 03A5 0308 0301; 03A5 0308 0301; # GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND OXIA
|
||||
1FE4; 1FE4; 03A1 0313; 03A1 0313; # GREEK SMALL LETTER RHO WITH PSILI
|
||||
1FE6; 1FE6; 03A5 0342; 03A5 0342; # GREEK SMALL LETTER UPSILON WITH PERISPOMENI
|
||||
1FE7; 1FE7; 03A5 0308 0342; 03A5 0308 0342; # GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND PERISPOMENI
|
||||
1FF6; 1FF6; 03A9 0342; 03A9 0342; # GREEK SMALL LETTER OMEGA WITH PERISPOMENI
|
||||
|
||||
# IMPORTANT-when iota-subscript (0345) is uppercased or titlecased,
|
||||
# the result will be incorrect unless the iota-subscript is moved to the end
|
||||
# of any sequence of combining marks. Otherwise, the accents will go on the capital iota.
|
||||
# This process can be achieved by first transforming the text to NFC before casing.
|
||||
# E.g. <alpha><iota_subscript><acute> is uppercased to <ALPHA><acute><IOTA>
|
||||
|
||||
# The following cases are already in the UnicodeData.txt file, so are only commented here.
|
||||
|
||||
# 0345; 0345; 0345; 0399; # COMBINING GREEK YPOGEGRAMMENI
|
||||
|
||||
# All letters with YPOGEGRAMMENI (iota-subscript) or PROSGEGRAMMENI (iota adscript)
|
||||
# have special uppercases.
|
||||
# Note: characters with PROSGEGRAMMENI are actually titlecase, not uppercase!
|
||||
|
||||
1F80; 1F80; 1F88; 1F08 0399; # GREEK SMALL LETTER ALPHA WITH PSILI AND YPOGEGRAMMENI
|
||||
1F81; 1F81; 1F89; 1F09 0399; # GREEK SMALL LETTER ALPHA WITH DASIA AND YPOGEGRAMMENI
|
||||
1F82; 1F82; 1F8A; 1F0A 0399; # GREEK SMALL LETTER ALPHA WITH PSILI AND VARIA AND YPOGEGRAMMENI
|
||||
1F83; 1F83; 1F8B; 1F0B 0399; # GREEK SMALL LETTER ALPHA WITH DASIA AND VARIA AND YPOGEGRAMMENI
|
||||
1F84; 1F84; 1F8C; 1F0C 0399; # GREEK SMALL LETTER ALPHA WITH PSILI AND OXIA AND YPOGEGRAMMENI
|
||||
1F85; 1F85; 1F8D; 1F0D 0399; # GREEK SMALL LETTER ALPHA WITH DASIA AND OXIA AND YPOGEGRAMMENI
|
||||
1F86; 1F86; 1F8E; 1F0E 0399; # GREEK SMALL LETTER ALPHA WITH PSILI AND PERISPOMENI AND YPOGEGRAMMENI
|
||||
1F87; 1F87; 1F8F; 1F0F 0399; # GREEK SMALL LETTER ALPHA WITH DASIA AND PERISPOMENI AND YPOGEGRAMMENI
|
||||
1F88; 1F80; 1F88; 1F08 0399; # GREEK CAPITAL LETTER ALPHA WITH PSILI AND PROSGEGRAMMENI
|
||||
1F89; 1F81; 1F89; 1F09 0399; # GREEK CAPITAL LETTER ALPHA WITH DASIA AND PROSGEGRAMMENI
|
||||
1F8A; 1F82; 1F8A; 1F0A 0399; # GREEK CAPITAL LETTER ALPHA WITH PSILI AND VARIA AND PROSGEGRAMMENI
|
||||
1F8B; 1F83; 1F8B; 1F0B 0399; # GREEK CAPITAL LETTER ALPHA WITH DASIA AND VARIA AND PROSGEGRAMMENI
|
||||
1F8C; 1F84; 1F8C; 1F0C 0399; # GREEK CAPITAL LETTER ALPHA WITH PSILI AND OXIA AND PROSGEGRAMMENI
|
||||
1F8D; 1F85; 1F8D; 1F0D 0399; # GREEK CAPITAL LETTER ALPHA WITH DASIA AND OXIA AND PROSGEGRAMMENI
|
||||
1F8E; 1F86; 1F8E; 1F0E 0399; # GREEK CAPITAL LETTER ALPHA WITH PSILI AND PERISPOMENI AND PROSGEGRAMMENI
|
||||
1F8F; 1F87; 1F8F; 1F0F 0399; # GREEK CAPITAL LETTER ALPHA WITH DASIA AND PERISPOMENI AND PROSGEGRAMMENI
|
||||
1F90; 1F90; 1F98; 1F28 0399; # GREEK SMALL LETTER ETA WITH PSILI AND YPOGEGRAMMENI
|
||||
1F91; 1F91; 1F99; 1F29 0399; # GREEK SMALL LETTER ETA WITH DASIA AND YPOGEGRAMMENI
|
||||
1F92; 1F92; 1F9A; 1F2A 0399; # GREEK SMALL LETTER ETA WITH PSILI AND VARIA AND YPOGEGRAMMENI
|
||||
1F93; 1F93; 1F9B; 1F2B 0399; # GREEK SMALL LETTER ETA WITH DASIA AND VARIA AND YPOGEGRAMMENI
|
||||
1F94; 1F94; 1F9C; 1F2C 0399; # GREEK SMALL LETTER ETA WITH PSILI AND OXIA AND YPOGEGRAMMENI
|
||||
1F95; 1F95; 1F9D; 1F2D 0399; # GREEK SMALL LETTER ETA WITH DASIA AND OXIA AND YPOGEGRAMMENI
|
||||
1F96; 1F96; 1F9E; 1F2E 0399; # GREEK SMALL LETTER ETA WITH PSILI AND PERISPOMENI AND YPOGEGRAMMENI
|
||||
1F97; 1F97; 1F9F; 1F2F 0399; # GREEK SMALL LETTER ETA WITH DASIA AND PERISPOMENI AND YPOGEGRAMMENI
|
||||
1F98; 1F90; 1F98; 1F28 0399; # GREEK CAPITAL LETTER ETA WITH PSILI AND PROSGEGRAMMENI
|
||||
1F99; 1F91; 1F99; 1F29 0399; # GREEK CAPITAL LETTER ETA WITH DASIA AND PROSGEGRAMMENI
|
||||
1F9A; 1F92; 1F9A; 1F2A 0399; # GREEK CAPITAL LETTER ETA WITH PSILI AND VARIA AND PROSGEGRAMMENI
|
||||
1F9B; 1F93; 1F9B; 1F2B 0399; # GREEK CAPITAL LETTER ETA WITH DASIA AND VARIA AND PROSGEGRAMMENI
|
||||
1F9C; 1F94; 1F9C; 1F2C 0399; # GREEK CAPITAL LETTER ETA WITH PSILI AND OXIA AND PROSGEGRAMMENI
|
||||
1F9D; 1F95; 1F9D; 1F2D 0399; # GREEK CAPITAL LETTER ETA WITH DASIA AND OXIA AND PROSGEGRAMMENI
|
||||
1F9E; 1F96; 1F9E; 1F2E 0399; # GREEK CAPITAL LETTER ETA WITH PSILI AND PERISPOMENI AND PROSGEGRAMMENI
|
||||
1F9F; 1F97; 1F9F; 1F2F 0399; # GREEK CAPITAL LETTER ETA WITH DASIA AND PERISPOMENI AND PROSGEGRAMMENI
|
||||
1FA0; 1FA0; 1FA8; 1F68 0399; # GREEK SMALL LETTER OMEGA WITH PSILI AND YPOGEGRAMMENI
|
||||
1FA1; 1FA1; 1FA9; 1F69 0399; # GREEK SMALL LETTER OMEGA WITH DASIA AND YPOGEGRAMMENI
|
||||
1FA2; 1FA2; 1FAA; 1F6A 0399; # GREEK SMALL LETTER OMEGA WITH PSILI AND VARIA AND YPOGEGRAMMENI
|
||||
1FA3; 1FA3; 1FAB; 1F6B 0399; # GREEK SMALL LETTER OMEGA WITH DASIA AND VARIA AND YPOGEGRAMMENI
|
||||
1FA4; 1FA4; 1FAC; 1F6C 0399; # GREEK SMALL LETTER OMEGA WITH PSILI AND OXIA AND YPOGEGRAMMENI
|
||||
1FA5; 1FA5; 1FAD; 1F6D 0399; # GREEK SMALL LETTER OMEGA WITH DASIA AND OXIA AND YPOGEGRAMMENI
|
||||
1FA6; 1FA6; 1FAE; 1F6E 0399; # GREEK SMALL LETTER OMEGA WITH PSILI AND PERISPOMENI AND YPOGEGRAMMENI
|
||||
1FA7; 1FA7; 1FAF; 1F6F 0399; # GREEK SMALL LETTER OMEGA WITH DASIA AND PERISPOMENI AND YPOGEGRAMMENI
|
||||
1FA8; 1FA0; 1FA8; 1F68 0399; # GREEK CAPITAL LETTER OMEGA WITH PSILI AND PROSGEGRAMMENI
|
||||
1FA9; 1FA1; 1FA9; 1F69 0399; # GREEK CAPITAL LETTER OMEGA WITH DASIA AND PROSGEGRAMMENI
|
||||
1FAA; 1FA2; 1FAA; 1F6A 0399; # GREEK CAPITAL LETTER OMEGA WITH PSILI AND VARIA AND PROSGEGRAMMENI
|
||||
1FAB; 1FA3; 1FAB; 1F6B 0399; # GREEK CAPITAL LETTER OMEGA WITH DASIA AND VARIA AND PROSGEGRAMMENI
|
||||
1FAC; 1FA4; 1FAC; 1F6C 0399; # GREEK CAPITAL LETTER OMEGA WITH PSILI AND OXIA AND PROSGEGRAMMENI
|
||||
1FAD; 1FA5; 1FAD; 1F6D 0399; # GREEK CAPITAL LETTER OMEGA WITH DASIA AND OXIA AND PROSGEGRAMMENI
|
||||
1FAE; 1FA6; 1FAE; 1F6E 0399; # GREEK CAPITAL LETTER OMEGA WITH PSILI AND PERISPOMENI AND PROSGEGRAMMENI
|
||||
1FAF; 1FA7; 1FAF; 1F6F 0399; # GREEK CAPITAL LETTER OMEGA WITH DASIA AND PERISPOMENI AND PROSGEGRAMMENI
|
||||
1FB3; 1FB3; 1FBC; 0391 0399; # GREEK SMALL LETTER ALPHA WITH YPOGEGRAMMENI
|
||||
1FBC; 1FB3; 1FBC; 0391 0399; # GREEK CAPITAL LETTER ALPHA WITH PROSGEGRAMMENI
|
||||
1FC3; 1FC3; 1FCC; 0397 0399; # GREEK SMALL LETTER ETA WITH YPOGEGRAMMENI
|
||||
1FCC; 1FC3; 1FCC; 0397 0399; # GREEK CAPITAL LETTER ETA WITH PROSGEGRAMMENI
|
||||
1FF3; 1FF3; 1FFC; 03A9 0399; # GREEK SMALL LETTER OMEGA WITH YPOGEGRAMMENI
|
||||
1FFC; 1FF3; 1FFC; 03A9 0399; # GREEK CAPITAL LETTER OMEGA WITH PROSGEGRAMMENI
|
||||
|
||||
# Some characters with YPOGEGRAMMENI also have no corresponding titlecases
|
||||
|
||||
1FB2; 1FB2; 1FBA 0345; 1FBA 0399; # GREEK SMALL LETTER ALPHA WITH VARIA AND YPOGEGRAMMENI
|
||||
1FB4; 1FB4; 0386 0345; 0386 0399; # GREEK SMALL LETTER ALPHA WITH OXIA AND YPOGEGRAMMENI
|
||||
1FC2; 1FC2; 1FCA 0345; 1FCA 0399; # GREEK SMALL LETTER ETA WITH VARIA AND YPOGEGRAMMENI
|
||||
1FC4; 1FC4; 0389 0345; 0389 0399; # GREEK SMALL LETTER ETA WITH OXIA AND YPOGEGRAMMENI
|
||||
1FF2; 1FF2; 1FFA 0345; 1FFA 0399; # GREEK SMALL LETTER OMEGA WITH VARIA AND YPOGEGRAMMENI
|
||||
1FF4; 1FF4; 038F 0345; 038F 0399; # GREEK SMALL LETTER OMEGA WITH OXIA AND YPOGEGRAMMENI
|
||||
|
||||
1FB7; 1FB7; 0391 0342 0345; 0391 0342 0399; # GREEK SMALL LETTER ALPHA WITH PERISPOMENI AND YPOGEGRAMMENI
|
||||
1FC7; 1FC7; 0397 0342 0345; 0397 0342 0399; # GREEK SMALL LETTER ETA WITH PERISPOMENI AND YPOGEGRAMMENI
|
||||
1FF7; 1FF7; 03A9 0342 0345; 03A9 0342 0399; # GREEK SMALL LETTER OMEGA WITH PERISPOMENI AND YPOGEGRAMMENI
|
||||
|
||||
# ================================================================================
|
||||
# Conditional Mappings
|
||||
# The remainder of this file provides conditional casing data used to produce
|
||||
# full case mappings.
|
||||
# ================================================================================
|
||||
# Language-Insensitive Mappings
|
||||
# These are characters whose full case mappings do not depend on language, but do
|
||||
# depend on context (which characters come before or after). For more information
|
||||
# see the header of this file and the Unicode Standard.
|
||||
# ================================================================================
|
||||
|
||||
# Special case for final form of sigma
|
||||
|
||||
03A3; 03C2; 03A3; 03A3; Final_Sigma; # GREEK CAPITAL LETTER SIGMA
|
||||
|
||||
# Note: the following cases for non-final are already in the UnicodeData.txt file.
|
||||
|
||||
# 03A3; 03C3; 03A3; 03A3; # GREEK CAPITAL LETTER SIGMA
|
||||
# 03C3; 03C3; 03A3; 03A3; # GREEK SMALL LETTER SIGMA
|
||||
# 03C2; 03C2; 03A3; 03A3; # GREEK SMALL LETTER FINAL SIGMA
|
||||
|
||||
# Note: the following cases are not included, since they would case-fold in lowercasing
|
||||
|
||||
# 03C3; 03C2; 03A3; 03A3; Final_Sigma; # GREEK SMALL LETTER SIGMA
|
||||
# 03C2; 03C3; 03A3; 03A3; Not_Final_Sigma; # GREEK SMALL LETTER FINAL SIGMA
|
||||
|
||||
# ================================================================================
|
||||
# Language-Sensitive Mappings
|
||||
# These are characters whose full case mappings depend on language and perhaps also
|
||||
# context (which characters come before or after). For more information
|
||||
# see the header of this file and the Unicode Standard.
|
||||
# ================================================================================
|
||||
|
||||
# Lithuanian
|
||||
|
||||
# Lithuanian retains the dot in a lowercase i when followed by accents.
|
||||
|
||||
# Remove DOT ABOVE after "i" with upper or titlecase
|
||||
|
||||
0307; 0307; ; ; lt After_Soft_Dotted; # COMBINING DOT ABOVE
|
||||
|
||||
# Introduce an explicit dot above when lowercasing capital I's and J's
|
||||
# whenever there are more accents above.
|
||||
# (of the accents used in Lithuanian: grave, acute, tilde above, and ogonek)
|
||||
|
||||
0049; 0069 0307; 0049; 0049; lt More_Above; # LATIN CAPITAL LETTER I
|
||||
004A; 006A 0307; 004A; 004A; lt More_Above; # LATIN CAPITAL LETTER J
|
||||
012E; 012F 0307; 012E; 012E; lt More_Above; # LATIN CAPITAL LETTER I WITH OGONEK
|
||||
00CC; 0069 0307 0300; 00CC; 00CC; lt; # LATIN CAPITAL LETTER I WITH GRAVE
|
||||
00CD; 0069 0307 0301; 00CD; 00CD; lt; # LATIN CAPITAL LETTER I WITH ACUTE
|
||||
0128; 0069 0307 0303; 0128; 0128; lt; # LATIN CAPITAL LETTER I WITH TILDE
|
||||
|
||||
# ================================================================================
|
||||
|
||||
# Turkish and Azeri
|
||||
|
||||
# I and i-dotless; I-dot and i are case pairs in Turkish and Azeri
|
||||
# The following rules handle those cases.
|
||||
|
||||
0130; 0069; 0130; 0130; tr; # LATIN CAPITAL LETTER I WITH DOT ABOVE
|
||||
0130; 0069; 0130; 0130; az; # LATIN CAPITAL LETTER I WITH DOT ABOVE
|
||||
|
||||
# When lowercasing, remove dot_above in the sequence I + dot_above, which will turn into i.
|
||||
# This matches the behavior of the canonically equivalent I-dot_above
|
||||
|
||||
0307; ; 0307; 0307; tr After_I; # COMBINING DOT ABOVE
|
||||
0307; ; 0307; 0307; az After_I; # COMBINING DOT ABOVE
|
||||
|
||||
# When lowercasing, unless an I is before a dot_above, it turns into a dotless i.
|
||||
|
||||
0049; 0131; 0049; 0049; tr Not_Before_Dot; # LATIN CAPITAL LETTER I
|
||||
0049; 0131; 0049; 0049; az Not_Before_Dot; # LATIN CAPITAL LETTER I
|
||||
|
||||
# When uppercasing, i turns into a dotted capital I
|
||||
|
||||
0069; 0069; 0130; 0130; tr; # LATIN SMALL LETTER I
|
||||
0069; 0069; 0130; 0130; az; # LATIN SMALL LETTER I
|
||||
|
||||
# Note: the following case is already in the UnicodeData.txt file.
|
||||
|
||||
# 0131; 0131; 0049; 0049; tr; # LATIN SMALL LETTER DOTLESS I
|
||||
|
||||
# EOF
|
||||
|
||||
|
|
@ -268,6 +268,42 @@ Property value is a character or nil.
|
|||
The value nil means that the actual property value of a character
|
||||
is the character itself."
|
||||
string)
|
||||
(special-uppercase
|
||||
2 unidata-gen-table-special-casing "uni-special-uppercase.el"
|
||||
"Unicode unconditional special casing mapping.
|
||||
|
||||
Property value is (possibly empty) string or nil. The value nil denotes that
|
||||
`uppercase' property should be consulted instead. A string denotes what
|
||||
sequence of characters given character maps into.
|
||||
|
||||
This mapping includes language- and context-independent special casing rules
|
||||
defined by Unicode only. It also does not include association which would
|
||||
duplicate information from `uppercase' property."
|
||||
nil)
|
||||
(special-lowercase
|
||||
0 unidata-gen-table-special-casing "uni-special-lowercase.el"
|
||||
"Unicode unconditional special casing mapping.
|
||||
|
||||
Property value is (possibly empty) string or nil. The value nil denotes that
|
||||
`lowercase' property should be consulted instead. A string denotes what
|
||||
sequence of characters given character maps into.
|
||||
|
||||
This mapping includes language- and context-independent special casing rules
|
||||
defined by Unicode only. It also does not include association which would
|
||||
duplicate information from `lowercase' property."
|
||||
nil)
|
||||
(special-titlecase
|
||||
1 unidata-gen-table-special-casing "uni-special-titlecase.el"
|
||||
"Unicode unconditional special casing mapping.
|
||||
|
||||
Property value is (possibly empty) string or nil. The value nil denotes that
|
||||
`titlecase' property should be consulted instead. A string denotes what
|
||||
sequence of characters given character maps into.
|
||||
|
||||
This mapping includes language- and context-independent special casing rules
|
||||
defined by Unicode only. It also does not include association which would
|
||||
duplicate information from `titlecase' property."
|
||||
nil)
|
||||
(mirroring
|
||||
unidata-gen-mirroring-list unidata-gen-table-character "uni-mirrored.el"
|
||||
"Unicode bidi-mirroring characters.
|
||||
|
|
@ -1083,6 +1119,51 @@ Property value is a symbol `o' (Open), `c' (Close), or `n' (None)."
|
|||
table))
|
||||
|
||||
|
||||
|
||||
|
||||
(defvar unidata-gen-table-special-casing--cache nil
|
||||
"Cached value for `unidata-gen-table-special-casing' function.")
|
||||
|
||||
(defun unidata-gen-table-special-casing--do-load ()
|
||||
(let (result)
|
||||
(with-temp-buffer
|
||||
(insert-file-contents (expand-file-name "SpecialCasing.txt" unidata-dir))
|
||||
(goto-char (point-min))
|
||||
(while (not (eobp))
|
||||
;; Ignore empty lines and comments.
|
||||
(unless (or (eq (char-after) ?\n) (eq (char-after) ?#))
|
||||
(let ((line (split-string
|
||||
(buffer-substring (point) (progn (end-of-line) (point)))
|
||||
";" "")))
|
||||
;; Ignore entries with conditions, i.e. those with six values.
|
||||
(when (= (length line) 5)
|
||||
(let ((ch (string-to-number (pop line) 16)))
|
||||
(setcdr (cddr line) nil) ; strip comment
|
||||
(push
|
||||
(cons ch
|
||||
(mapcar (lambda (entry)
|
||||
(mapcar (lambda (n) (string-to-number n 16))
|
||||
(split-string entry)))
|
||||
line))
|
||||
result)))))
|
||||
(forward-line)))
|
||||
result))
|
||||
|
||||
(defun unidata-gen-table-special-casing (prop &rest ignore)
|
||||
(let ((table (make-char-table 'char-code-property-table))
|
||||
(prop-idx (unidata-prop-index prop)))
|
||||
(set-char-table-extra-slot table 0 prop)
|
||||
(mapc (lambda (entry)
|
||||
(let ((ch (car entry)) (v (nth prop-idx (cdr entry))))
|
||||
;; If character maps to a single character, the mapping is already
|
||||
;; covered by regular casing property. Don’t store those.
|
||||
(when (/= (length v) 1)
|
||||
(set-char-table-range table ch (apply 'string v)))))
|
||||
(or unidata-gen-table-special-casing--cache
|
||||
(setq unidata-gen-table-special-casing--cache
|
||||
(unidata-gen-table-special-casing--do-load))))
|
||||
table))
|
||||
|
||||
|
||||
(defun unidata-describe-general-category (val)
|
||||
(cdr (assq val
|
||||
|
|
|
|||
|
|
@ -619,6 +619,29 @@ Corresponds to the Unicode @code{Simple_Titlecase_Mapping} property.
|
|||
character of a word needs to be capitalized. The value of this
|
||||
property is a single character. For unassigned codepoints, the value
|
||||
is @code{nil}, which means the character itself.
|
||||
|
||||
@item special-uppercase
|
||||
Corresponds to Unicode language- and context-independent special upper-casing
|
||||
rules. The value of this property is a string (which may be empty). For
|
||||
example mapping for @code{U+00DF} (@sc{latin small letter sharp s}) is
|
||||
@code{"SS"}. For characters with no special mapping, the value is @code{nil}
|
||||
which means @code{uppercase} property needs to be consulted instead.
|
||||
|
||||
@item special-lowercase
|
||||
Corresponds to Unicode language- and context-independent special lower-casing
|
||||
rules. The value of this property is a string (which may be empty). For
|
||||
example mapping for @code{U+0130} (@sc{latin capital letter i with dot above})
|
||||
the value is @code{"i\u0307"} (i.e. 2-character string consisting of @sc{latin
|
||||
small letter i} followed by @sc{combining dot above}). For characters with no
|
||||
special mapping, the value is @code{nil} which means @code{lowercase} property
|
||||
needs to be consulted instead.
|
||||
|
||||
@item special-titlecase
|
||||
Corresponds to Unicode unconditional special title-casing rules. The value of
|
||||
this property is a string (which may be empty). For example mapping for
|
||||
@code{U+FB01} (@sc{latin small ligature fi}) the value is @code{"Fi"}. For
|
||||
characters with no special mapping, the value is @code{nil} which means
|
||||
@code{titlecase} property needs to be consulted instead.
|
||||
@end table
|
||||
|
||||
@defun get-char-code-property char propname
|
||||
|
|
|
|||
|
|
@ -1177,6 +1177,33 @@ When the argument to @code{upcase-initials} is a character,
|
|||
@end example
|
||||
@end defun
|
||||
|
||||
Note that case conversion is not a one-to-one mapping of codepoints
|
||||
and length of the result may differ from length of the argument.
|
||||
Furthermore, because passing a character forces return type to be
|
||||
a character, functions are unable to perform proper substitution and
|
||||
result may differ compared to treating a one-character string. For
|
||||
example:
|
||||
|
||||
@example
|
||||
@group
|
||||
(upcase "fi") ; note: single character, ligature "fi"
|
||||
@result{} "FI"
|
||||
@end group
|
||||
@group
|
||||
(upcase ?fi)
|
||||
@result{} 64257 ; i.e. ?fi
|
||||
@end group
|
||||
@end example
|
||||
|
||||
To avoid this, a character must first be converted into a string,
|
||||
using @code{string} function, before being passed to one of the casing
|
||||
functions. Of course, no assumptions on the length of the result may
|
||||
be made.
|
||||
|
||||
Mapping for such special cases are taken from
|
||||
@code{special-uppercase}, @code{special-lowercase} and
|
||||
@code{special-titlecase} @xref{Character Properties}.
|
||||
|
||||
@xref{Text Comparison}, for functions that compare strings; some of
|
||||
them ignore case differences, or can optionally ignore case differences.
|
||||
|
||||
|
|
|
|||
17
etc/NEWS
17
etc/NEWS
|
|
@ -355,12 +355,17 @@ same as in modes where the character is not whitespace.
|
|||
Instead of only checking the modification time, Emacs now also checks
|
||||
the file's actual content before prompting the user.
|
||||
|
||||
** Title case characters are properly cased (from and into).
|
||||
'upcase', 'upcase-region' et al. convert title case characters (such
|
||||
as the single character "Dz") into their upper case form (such as "DZ").
|
||||
As a downside, 'capitalize' and 'upcase-initials' produce awkward
|
||||
words where first character is upper rather than title case, e.g.,
|
||||
"DŽungla" instead of "Džungla".
|
||||
** Various casing improvements.
|
||||
|
||||
*** 'upcase', 'upcase-region' et al. convert title case characters
|
||||
(such as Dz) into their upper case form (such as DZ).
|
||||
|
||||
*** 'capitalize', 'upcase-initials' et al. make use of title-case forms
|
||||
of initial characters (correctly producing for example Džungla instead
|
||||
of incorrect DŽungla).
|
||||
|
||||
*** Characters which turn into multiple ones when cased are correctly handled.
|
||||
For example, fi ligature is converted to FI when upper cased.
|
||||
|
||||
|
||||
* Changes in Specialized Modes and Packages in Emacs 26.1
|
||||
|
|
|
|||
295
src/casefiddle.c
295
src/casefiddle.c
|
|
@ -1,3 +1,4 @@
|
|||
/* -*- coding: utf-8 -*- */
|
||||
/* GNU Emacs case conversion functions.
|
||||
|
||||
Copyright (C) 1985, 1994, 1997-1999, 2001-2017 Free Software Foundation,
|
||||
|
|
@ -36,6 +37,9 @@ struct casing_context {
|
|||
/* A char-table with title-case character mappings or nil. Non-nil implies
|
||||
flag is CASE_CAPITALIZE or CASE_CAPITALIZE_UP. */
|
||||
Lisp_Object titlecase_char_table;
|
||||
/* The unconditional special-casing Unicode property char tables for upper
|
||||
casing, lower casing and title casing respectively. */
|
||||
Lisp_Object specialcase_char_tables[3];
|
||||
/* User-requested action. */
|
||||
enum case_action flag;
|
||||
/* If true, function operates on a buffer as opposed to a string or character.
|
||||
|
|
@ -58,6 +62,13 @@ prepare_casing_context (struct casing_context *ctx,
|
|||
ctx->inword = flag == CASE_DOWN;
|
||||
ctx->titlecase_char_table = (int)flag < (int)CASE_CAPITALIZE ? Qnil :
|
||||
uniprop_table (intern_c_string ("titlecase"));
|
||||
ctx->specialcase_char_tables[CASE_UP] = flag == CASE_DOWN ? Qnil :
|
||||
uniprop_table (intern_c_string ("special-uppercase"));
|
||||
ctx->specialcase_char_tables[CASE_DOWN] = flag == CASE_UP ? Qnil :
|
||||
uniprop_table (intern_c_string ("special-lowercase"));
|
||||
ctx->specialcase_char_tables[CASE_CAPITALIZE] =
|
||||
(int)flag < (int)CASE_CAPITALIZE ? Qnil :
|
||||
uniprop_table (intern_c_string ("special-titlecase"));
|
||||
|
||||
/* If the case table is flagged as modified, rescan it. */
|
||||
if (NILP (XCHAR_TABLE (BVAR (current_buffer, downcase_table))->extras[1]))
|
||||
|
|
@ -67,25 +78,98 @@ prepare_casing_context (struct casing_context *ctx,
|
|||
SETUP_BUFFER_SYNTAX_TABLE (); /* For syntax_prefix_flag_p. */
|
||||
}
|
||||
|
||||
/* Based on CTX, case character CH accordingly. Update CTX as necessary.
|
||||
Return cased character. */
|
||||
struct casing_str_buf {
|
||||
unsigned char data[MAX_MULTIBYTE_LENGTH > 6 ? MAX_MULTIBYTE_LENGTH : 6];
|
||||
unsigned char len_chars;
|
||||
unsigned char len_bytes;
|
||||
};
|
||||
|
||||
/* Based on CTX, case character CH. If BUF is NULL, return cased character.
|
||||
Otherwise, if BUF is non-NULL, save result in it and return whether the
|
||||
character has been changed.
|
||||
|
||||
Since meaning of return value depends on arguments, it’s more convenient to
|
||||
use case_single_character or case_character instead. */
|
||||
static int
|
||||
case_character (struct casing_context *ctx, int ch)
|
||||
case_character_impl (struct casing_str_buf *buf,
|
||||
struct casing_context *ctx, int ch)
|
||||
{
|
||||
enum case_action flag;
|
||||
Lisp_Object prop;
|
||||
bool was_inword;
|
||||
int cased;
|
||||
|
||||
if (ctx->inword)
|
||||
ch = ctx->flag == CASE_CAPITALIZE_UP ? ch : downcase (ch);
|
||||
else if (!NILP (ctx->titlecase_char_table) &&
|
||||
CHARACTERP (prop = CHAR_TABLE_REF (ctx->titlecase_char_table, ch)))
|
||||
ch = XFASTINT (prop);
|
||||
else
|
||||
ch = upcase(ch);
|
||||
|
||||
/* Update inword state */
|
||||
was_inword = ctx->inword;
|
||||
if ((int) ctx->flag >= (int) CASE_CAPITALIZE)
|
||||
ctx->inword = SYNTAX (ch) == Sword &&
|
||||
(!ctx->inbuffer || ctx->inword || !syntax_prefix_flag_p (ch));
|
||||
return ch;
|
||||
(!ctx->inbuffer || was_inword || !syntax_prefix_flag_p (ch));
|
||||
|
||||
/* Normalise flag so its one of CASE_UP, CASE_DOWN or CASE_CAPITALIZE. */
|
||||
if (!was_inword)
|
||||
flag = ctx->flag == CASE_UP ? CASE_UP : CASE_CAPITALIZE;
|
||||
else if (ctx->flag != CASE_CAPITALIZE_UP)
|
||||
flag = CASE_DOWN;
|
||||
else
|
||||
{
|
||||
cased = ch;
|
||||
goto done;
|
||||
}
|
||||
|
||||
/* Look through the special casing entries. */
|
||||
if (buf && !NILP(ctx->specialcase_char_tables[(int)flag]))
|
||||
{
|
||||
prop = CHAR_TABLE_REF(ctx->specialcase_char_tables[(int)flag], ch);
|
||||
if (STRINGP(prop))
|
||||
{
|
||||
struct Lisp_String *str = XSTRING(prop);
|
||||
if (STRING_BYTES(str) <= sizeof buf->data)
|
||||
{
|
||||
buf->len_chars = str->size;
|
||||
buf->len_bytes = STRING_BYTES(str);
|
||||
memcpy(buf->data, str->data, buf->len_bytes);
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* Handle simple, one-to-one case. */
|
||||
if (flag == CASE_DOWN)
|
||||
cased = downcase (ch);
|
||||
else if (!NILP (ctx->titlecase_char_table) &&
|
||||
CHARACTERP (prop = CHAR_TABLE_REF (ctx->titlecase_char_table, ch)))
|
||||
cased = XFASTINT (prop);
|
||||
else
|
||||
cased = upcase(ch);
|
||||
|
||||
/* And we’re done. */
|
||||
done:
|
||||
if (!buf)
|
||||
return cased;
|
||||
buf->len_chars = 1;
|
||||
buf->len_bytes = CHAR_STRING (cased, buf->data);
|
||||
return cased != ch;
|
||||
}
|
||||
|
||||
/* Based on CTX, case character CH accordingly. Update CTX as necessary.
|
||||
Return cased character.
|
||||
|
||||
Special casing rules (such as upcase(fi) = FI) are not handled. For
|
||||
characters whose casing results in multiple code points, the character is
|
||||
returned unchanged. */
|
||||
static inline int
|
||||
case_single_character (struct casing_context *ctx, int ch)
|
||||
{
|
||||
return case_character_impl (NULL, ctx, ch);
|
||||
}
|
||||
|
||||
/* Save in BUF result of casing character CH. Return whether casing changed the
|
||||
character. This is like case_single_character but also handles one-to-many
|
||||
casing rules. */
|
||||
static inline bool
|
||||
case_character (struct casing_str_buf *buf, struct casing_context *ctx, int ch)
|
||||
{
|
||||
return case_character_impl (buf, ctx, ch);
|
||||
}
|
||||
|
||||
static Lisp_Object
|
||||
|
|
@ -112,7 +196,7 @@ do_casify_natnum (struct casing_context *ctx, Lisp_Object obj)
|
|||
|| !NILP (BVAR (current_buffer, enable_multibyte_characters));
|
||||
if (! multibyte)
|
||||
MAKE_CHAR_MULTIBYTE (ch);
|
||||
cased = case_character (ctx, ch);
|
||||
cased = case_single_character (ctx, ch);
|
||||
if (cased == ch)
|
||||
return obj;
|
||||
|
||||
|
|
@ -125,25 +209,34 @@ do_casify_natnum (struct casing_context *ctx, Lisp_Object obj)
|
|||
static Lisp_Object
|
||||
do_casify_multibyte_string (struct casing_context *ctx, Lisp_Object obj)
|
||||
{
|
||||
ptrdiff_t i, i_byte, size = SCHARS (obj);
|
||||
int len, ch, cased;
|
||||
/* We assume data is the first member of casing_str_buf structure so that if
|
||||
we cast a (char *) into (struct casing_str_buf *) the representation of the
|
||||
character is at the beginning of the buffer. This is why we don’t need
|
||||
separate struct casing_str_buf object but rather write directly to o. */
|
||||
typedef char static_assertion[offsetof(struct casing_str_buf, data) ? -1 : 1];
|
||||
|
||||
ptrdiff_t size = SCHARS (obj), n;
|
||||
int ch;
|
||||
USE_SAFE_ALLOCA;
|
||||
ptrdiff_t o_size;
|
||||
if (INT_MULTIPLY_WRAPV (size, MAX_MULTIBYTE_LENGTH, &o_size))
|
||||
o_size = PTRDIFF_MAX;
|
||||
unsigned char *dst = SAFE_ALLOCA (o_size);
|
||||
if (INT_MULTIPLY_WRAPV (size, MAX_MULTIBYTE_LENGTH, &n) ||
|
||||
INT_ADD_WRAPV (n, sizeof(struct casing_str_buf), &n))
|
||||
n = PTRDIFF_MAX;
|
||||
unsigned char *const dst = SAFE_ALLOCA (n), *const dst_end = dst + n;
|
||||
unsigned char *o = dst;
|
||||
|
||||
for (i = i_byte = 0; i < size; i++, i_byte += len)
|
||||
const unsigned char *src = SDATA (obj);
|
||||
|
||||
for (n = 0; size; --size)
|
||||
{
|
||||
if (o_size - MAX_MULTIBYTE_LENGTH < o - dst)
|
||||
if (dst_end - o < sizeof(struct casing_str_buf))
|
||||
string_overflow ();
|
||||
ch = STRING_CHAR_AND_LENGTH (SDATA (obj) + i_byte, len);
|
||||
cased = case_character (ctx, ch);
|
||||
o += CHAR_STRING (cased, o);
|
||||
ch = STRING_CHAR_ADVANCE (src);
|
||||
case_character ((void *)o, ctx, ch);
|
||||
n += ((struct casing_str_buf *)o)->len_chars;
|
||||
o += ((struct casing_str_buf *)o)->len_bytes;
|
||||
}
|
||||
eassert (o - dst <= o_size);
|
||||
obj = make_multibyte_string ((char *) dst, size, o - dst);
|
||||
eassert (o <= dst_end);
|
||||
obj = make_multibyte_string ((char *) dst, n, o - dst);
|
||||
SAFE_FREE ();
|
||||
return obj;
|
||||
}
|
||||
|
|
@ -159,7 +252,7 @@ do_casify_unibyte_string (struct casing_context *ctx, Lisp_Object obj)
|
|||
{
|
||||
ch = SREF (obj, i);
|
||||
MAKE_CHAR_MULTIBYTE (ch);
|
||||
cased = case_character (ctx, ch);
|
||||
cased = case_single_character (ctx, ch);
|
||||
if (ch == cased)
|
||||
continue;
|
||||
MAKE_CHAR_UNIBYTE (cased);
|
||||
|
|
@ -191,7 +284,9 @@ casify_object (enum case_action flag, Lisp_Object obj)
|
|||
DEFUN ("upcase", Fupcase, Supcase, 1, 1, 0,
|
||||
doc: /* Convert argument to upper case and return that.
|
||||
The argument may be a character or string. The result has the same type.
|
||||
The argument object is not altered--the value is a copy.
|
||||
The argument object is not altered--the value is a copy. If argument
|
||||
is a character, characters which map to multiple code points when
|
||||
cased, e.g. fi, are returned unchanged.
|
||||
See also `capitalize', `downcase' and `upcase-initials'. */)
|
||||
(Lisp_Object obj)
|
||||
{
|
||||
|
|
@ -212,7 +307,9 @@ DEFUN ("capitalize", Fcapitalize, Scapitalize, 1, 1, 0,
|
|||
This means that each word's first character is converted to either
|
||||
title case or upper case, and the rest to lower case.
|
||||
The argument may be a character or string. The result has the same type.
|
||||
The argument object is not altered--the value is a copy. */)
|
||||
The argument object is not altered--the value is a copy. If argument
|
||||
is a character, characters which map to multiple code points when
|
||||
cased, e.g. fi, are returned unchanged. */)
|
||||
(Lisp_Object obj)
|
||||
{
|
||||
return casify_object (CASE_CAPITALIZE, obj);
|
||||
|
|
@ -225,21 +322,28 @@ DEFUN ("upcase-initials", Fupcase_initials, Supcase_initials, 1, 1, 0,
|
|||
This means that each word's first character is converted to either
|
||||
title case or upper case, and the rest are left unchanged.
|
||||
The argument may be a character or string. The result has the same type.
|
||||
The argument object is not altered--the value is a copy. */)
|
||||
The argument object is not altered--the value is a copy. If argument
|
||||
is a character, characters which map to multiple code points when
|
||||
cased, e.g. fi, are returned unchanged. */)
|
||||
(Lisp_Object obj)
|
||||
{
|
||||
return casify_object (CASE_CAPITALIZE_UP, obj);
|
||||
}
|
||||
|
||||
/* Based on CTX, case region in a unibyte buffer from POS to *ENDP. Return
|
||||
first position that has changed and save last position in *ENDP. If no
|
||||
characters were changed, return -1 and *ENDP is unspecified. */
|
||||
/* Based on CTX, case region in a unibyte buffer from *STARTP to *ENDP.
|
||||
|
||||
Save first and last positions that has changed in *STARTP and *ENDP
|
||||
respectively. If no characters were changed, save -1 to *STARTP and leave
|
||||
*ENDP unspecified.
|
||||
|
||||
Always return 0. This is so that interface of this function is the same as
|
||||
do_casify_multibyte_region. */
|
||||
static ptrdiff_t
|
||||
do_casify_unibyte_region (struct casing_context *ctx,
|
||||
ptrdiff_t pos, ptrdiff_t *endp)
|
||||
ptrdiff_t *startp, ptrdiff_t *endp)
|
||||
{
|
||||
ptrdiff_t first = -1, last = -1; /* Position of first and last changes. */
|
||||
ptrdiff_t end = *endp;
|
||||
ptrdiff_t pos = *startp, end = *endp;
|
||||
int ch, cased;
|
||||
|
||||
for (; pos < end; ++pos)
|
||||
|
|
@ -247,11 +351,11 @@ do_casify_unibyte_region (struct casing_context *ctx,
|
|||
ch = FETCH_BYTE (pos);
|
||||
MAKE_CHAR_MULTIBYTE (ch);
|
||||
|
||||
cased = case_character (ctx, ch);
|
||||
cased = case_single_character (ctx, ch);
|
||||
if (cased == ch)
|
||||
continue;
|
||||
|
||||
last = pos;
|
||||
last = pos + 1;
|
||||
if (first < 0)
|
||||
first = pos;
|
||||
|
||||
|
|
@ -259,88 +363,107 @@ do_casify_unibyte_region (struct casing_context *ctx,
|
|||
FETCH_BYTE (pos) = cased;
|
||||
}
|
||||
|
||||
*endp = last + 1;
|
||||
return first;
|
||||
*startp = first;
|
||||
*endp = last;
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Based on CTX, case region in a multibyte buffer from POS to *ENDP. Return
|
||||
first position that has changed and save last position in *ENDP. If no
|
||||
characters were changed, return -1 and *ENDP is unspecified. */
|
||||
/* Based on CTX, case region in a multibyte buffer from *STARTP to *ENDP.
|
||||
|
||||
Return number of added characters (may be negative if more characters were
|
||||
deleted then inserted), save first and last positions that has changed in
|
||||
*STARTP and *ENDP respectively. If no characters were changed, return 0,
|
||||
save -1 to *STARTP and leave *ENDP unspecified. */
|
||||
static ptrdiff_t
|
||||
do_casify_multibyte_region (struct casing_context *ctx,
|
||||
ptrdiff_t pos, ptrdiff_t *endp)
|
||||
ptrdiff_t *startp, ptrdiff_t *endp)
|
||||
{
|
||||
ptrdiff_t first = -1, last = -1; /* Position of first and last changes. */
|
||||
ptrdiff_t pos_byte = CHAR_TO_BYTE (pos), end = *endp;
|
||||
ptrdiff_t opoint = PT;
|
||||
ptrdiff_t pos = *startp, pos_byte = CHAR_TO_BYTE (pos), size = *endp - pos;
|
||||
ptrdiff_t opoint = PT, added = 0;
|
||||
struct casing_str_buf buf;
|
||||
int ch, cased, len;
|
||||
|
||||
while (pos < end)
|
||||
for (; size; --size)
|
||||
{
|
||||
ch = STRING_CHAR_AND_LENGTH (BYTE_POS_ADDR (pos_byte), len);
|
||||
cased = case_character (ctx, ch);
|
||||
if (cased != ch)
|
||||
if (!case_character (&buf, ctx, ch))
|
||||
{
|
||||
last = pos;
|
||||
if (first < 0)
|
||||
first = pos;
|
||||
|
||||
if (ASCII_CHAR_P (cased) && ASCII_CHAR_P (ch))
|
||||
FETCH_BYTE (pos_byte) = cased;
|
||||
else
|
||||
{
|
||||
unsigned char str[MAX_MULTIBYTE_LENGTH];
|
||||
int totlen = CHAR_STRING (cased, str);
|
||||
if (len == totlen)
|
||||
memcpy (BYTE_POS_ADDR (pos_byte), str, len);
|
||||
else
|
||||
/* Replace one character with the other(s), keeping text
|
||||
properties the same. */
|
||||
replace_range_2 (pos, pos_byte, pos + 1, pos_byte + len,
|
||||
(char *) str, 9, totlen, 0);
|
||||
len = totlen;
|
||||
}
|
||||
pos_byte += len;
|
||||
++pos;
|
||||
continue;
|
||||
}
|
||||
pos++;
|
||||
pos_byte += len;
|
||||
|
||||
last = pos + buf.len_chars;
|
||||
if (first < 0)
|
||||
first = pos;
|
||||
|
||||
if (buf.len_chars == 1 && buf.len_bytes == len)
|
||||
memcpy (BYTE_POS_ADDR (pos_byte), buf.data, len);
|
||||
else
|
||||
{
|
||||
/* Replace one character with the other(s), keeping text
|
||||
properties the same. */
|
||||
replace_range_2 (pos, pos_byte, pos + 1, pos_byte + len,
|
||||
(const char *) buf.data, buf.len_chars,
|
||||
buf.len_bytes,
|
||||
0);
|
||||
added += (ptrdiff_t) buf.len_chars - 1;
|
||||
if (opoint > pos)
|
||||
opoint += (ptrdiff_t) buf.len_chars - 1;
|
||||
}
|
||||
|
||||
pos_byte += buf.len_bytes;
|
||||
pos += buf.len_chars;
|
||||
}
|
||||
|
||||
if (PT != opoint)
|
||||
TEMP_SET_PT_BOTH (opoint, CHAR_TO_BYTE (opoint));
|
||||
|
||||
*startp = first;
|
||||
*endp = last;
|
||||
return first;
|
||||
return added;
|
||||
}
|
||||
|
||||
/* flag is CASE_UP, CASE_DOWN or CASE_CAPITALIZE or CASE_CAPITALIZE_UP.
|
||||
b and e specify range of buffer to operate on. */
|
||||
static void
|
||||
/* flag is CASE_UP, CASE_DOWN or CASE_CAPITALIZE or CASE_CAPITALIZE_UP. b and
|
||||
e specify range of buffer to operate on. Return character position of the
|
||||
end of the region after changes. */
|
||||
static ptrdiff_t
|
||||
casify_region (enum case_action flag, Lisp_Object b, Lisp_Object e)
|
||||
{
|
||||
ptrdiff_t start, end, orig_end, added;
|
||||
struct casing_context ctx;
|
||||
ptrdiff_t start, end;
|
||||
|
||||
if (EQ (b, e))
|
||||
/* Not modifying because nothing marked */
|
||||
return;
|
||||
|
||||
validate_region (&b, &e);
|
||||
start = XFASTINT (b);
|
||||
end = XFASTINT (e);
|
||||
if (start == end)
|
||||
/* Not modifying because nothing marked */
|
||||
return end;
|
||||
modify_text (start, end);
|
||||
record_change (start, end - start);
|
||||
prepare_casing_context (&ctx, flag, true);
|
||||
|
||||
orig_end = end;
|
||||
record_delete (start, make_buffer_string (start, end, true), false);
|
||||
if (NILP (BVAR (current_buffer, enable_multibyte_characters)))
|
||||
start = do_casify_unibyte_region (&ctx, start, &end);
|
||||
{
|
||||
record_insert (start, end - start);
|
||||
added = do_casify_unibyte_region (&ctx, &start, &end);
|
||||
}
|
||||
else
|
||||
start = do_casify_multibyte_region (&ctx, start, &end);
|
||||
{
|
||||
ptrdiff_t len = end - start, ostart = start;
|
||||
added = do_casify_multibyte_region (&ctx, &start, &end);
|
||||
record_insert (ostart, len + added);
|
||||
}
|
||||
|
||||
if (start >= 0)
|
||||
{
|
||||
signal_after_change (start, end + 1 - start, end + 1 - start);
|
||||
update_compositions (start, end + 1, CHECK_ALL);
|
||||
signal_after_change (start, end - start - added, end - start);
|
||||
update_compositions (start, end, CHECK_ALL);
|
||||
}
|
||||
|
||||
return orig_end + added;
|
||||
}
|
||||
|
||||
DEFUN ("upcase-region", Fupcase_region, Supcase_region, 2, 3,
|
||||
|
|
@ -432,9 +555,7 @@ casify_word (enum case_action flag, Lisp_Object arg)
|
|||
ptrdiff_t farend = scan_words (PT, XINT (arg));
|
||||
if (!farend)
|
||||
farend = XINT (arg) <= 0 ? BEGV : ZV;
|
||||
ptrdiff_t newpoint = max (PT, farend);
|
||||
casify_region (flag, make_number (PT), make_number (farend));
|
||||
SET_PT (newpoint);
|
||||
SET_PT (casify_region (flag, make_number (PT), make_number (farend)));
|
||||
return Qnil;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -54,6 +54,14 @@
|
|||
(concat w1 "\s\n\s\t\f\t\n\r\t" w2)
|
||||
(concat w1 (make-string 10 ?\s) w2)))))
|
||||
|
||||
(defun char-fold--ascii-upcase (string)
|
||||
"Like `upcase' but acts on ASCII characters only."
|
||||
(replace-regexp-in-string "[a-z]+" 'upcase string))
|
||||
|
||||
(defun char-fold--ascii-downcase (string)
|
||||
"Like `downcase' but acts on ASCII characters only."
|
||||
(replace-regexp-in-string "[a-z]+" 'downcase string))
|
||||
|
||||
(defun char-fold--test-match-exactly (string &rest strings-to-match)
|
||||
(let ((re (concat "\\`" (char-fold-to-regexp string) "\\'")))
|
||||
(dolist (it strings-to-match)
|
||||
|
|
@ -61,8 +69,8 @@
|
|||
;; Case folding
|
||||
(let ((case-fold-search t))
|
||||
(dolist (it strings-to-match)
|
||||
(should (string-match (upcase re) (downcase it)))
|
||||
(should (string-match (downcase re) (upcase it)))))))
|
||||
(should (string-match (char-fold--ascii-upcase re) (downcase it)))
|
||||
(should (string-match (char-fold--ascii-downcase re) (upcase it)))))))
|
||||
|
||||
(ert-deftest char-fold--test-some-defaults ()
|
||||
(dolist (it '(("ffl" . "ffl") ("ffi" . "ffi")
|
||||
|
|
|
|||
|
|
@ -24,36 +24,40 @@
|
|||
|
||||
(ert-deftest casefiddle-tests-char-properties ()
|
||||
"Sanity check of character Unicode properties."
|
||||
(should-not
|
||||
(let (errors)
|
||||
;; character uppercase lowercase titlecase
|
||||
(dolist (test '((?A nil ?a nil)
|
||||
(?a ?A nil ?A)
|
||||
(?Ł nil ?ł nil)
|
||||
(?ł ?Ł nil ?Ł)
|
||||
(let ((props '(uppercase lowercase titlecase
|
||||
special-uppercase special-lowercase special-titlecase))
|
||||
(tests '((?A nil ?a nil nil nil nil)
|
||||
(?a ?A nil ?A nil nil nil)
|
||||
(?Ł nil ?ł nil nil nil nil)
|
||||
(?ł ?Ł nil ?Ł nil nil nil)
|
||||
|
||||
(?DŽ nil ?dž ?Dž)
|
||||
(?Dž ?DŽ ?dž ?Dž)
|
||||
(?dž ?DŽ nil ?Dž)
|
||||
(?DŽ nil ?dž ?Dž nil nil nil)
|
||||
(?Dž ?DŽ ?dž ?Dž nil nil nil)
|
||||
(?dž ?DŽ nil ?Dž nil nil nil)
|
||||
|
||||
(?Σ nil ?σ nil)
|
||||
(?σ ?Σ nil ?Σ)
|
||||
(?ς ?Σ nil ?Σ)
|
||||
(?Σ nil ?σ nil nil nil nil)
|
||||
(?σ ?Σ nil ?Σ nil nil nil)
|
||||
(?ς ?Σ nil ?Σ nil nil nil)
|
||||
|
||||
(?ⅷ ?Ⅷ nil ?Ⅷ)
|
||||
(?Ⅷ nil ?ⅷ nil)))
|
||||
(let ((ch (car test))
|
||||
(expected (cdr test))
|
||||
(props '(uppercase lowercase titlecase)))
|
||||
(while props
|
||||
(let ((got (get-char-code-property ch (car props))))
|
||||
(unless (equal (car expected) got)
|
||||
(push (format "\n%c %s; expected: %s but got: %s"
|
||||
ch (car props) (car expected) got)
|
||||
errors)))
|
||||
(setq props (cdr props) expected (cdr expected)))))
|
||||
(when errors
|
||||
(mapconcat (lambda (line) line) (nreverse errors) "")))))
|
||||
(?ⅷ ?Ⅷ nil ?Ⅷ nil nil nil)
|
||||
(?Ⅷ nil ?ⅷ nil nil nil nil)
|
||||
|
||||
(?fi nil nil nil "FI" nil "Fi")
|
||||
(?ß nil nil nil "SS" nil "Ss")
|
||||
(?İ nil ?i nil nil "i\u0307" nil)))
|
||||
errors)
|
||||
(dolist (test tests)
|
||||
(let ((ch (car test))
|
||||
(expected (cdr test)))
|
||||
(dolist (prop props)
|
||||
(let ((got (get-char-code-property ch prop)))
|
||||
(unless (equal (car expected) got)
|
||||
(push (format "\n%c %s; expected: %s but got: %s"
|
||||
ch prop (car expected) got)
|
||||
errors)))
|
||||
(setq expected (cdr expected)))))
|
||||
(when errors
|
||||
(ert-fail (mapconcat (lambda (line) line) (nreverse errors) "")))))
|
||||
|
||||
|
||||
(defconst casefiddle-tests--characters
|
||||
|
|
@ -188,16 +192,13 @@
|
|||
("DŽUNGLA" "DŽUNGLA" "džungla" "Džungla" "DžUNGLA")
|
||||
("Džungla" "DŽUNGLA" "džungla" "Džungla" "Džungla")
|
||||
("džungla" "DŽUNGLA" "džungla" "Džungla" "Džungla")
|
||||
("define" "DEFINE" "define" "Define" "Define")
|
||||
("fish" "FISH" "fish" "Fish" "Fish")
|
||||
("Straße" "STRASSE" "straße" "Straße" "Straße")
|
||||
;; FIXME(bug#24603): Everything below is broken at the moment.
|
||||
;; Here’s what should happen:
|
||||
;;("define" "DEFINE" "define" "Define" "Define")
|
||||
;;("fish" "FIsh" "fish" "Fish" "Fish")
|
||||
;;("Straße" "STRASSE" "straße" "Straße" "Straße")
|
||||
;;("ΌΣΟΣ" "ΌΣΟΣ" "όσος" "Όσος" "Όσος")
|
||||
;; And here’s what is actually happening:
|
||||
("define" "DEfiNE" "define" "Define" "Define")
|
||||
("fish" "fiSH" "fish" "fish" "fish")
|
||||
("Straße" "STRAßE" "straße" "Straße" "Straße")
|
||||
("ΌΣΟΣ" "ΌΣΟΣ" "όσοσ" "Όσοσ" "ΌΣΟΣ")
|
||||
|
||||
("όσος" "ΌΣΟΣ" "όσος" "Όσος" "Όσος"))))))
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue