Preliminary Collection of Data on Cross-script Homoglyphs

Note to Reviewers:
While this document is formatted like an LGR, it does not define a repertoire or Label Generation Rules. Instead, it uses the variant formalism to present sets of code points that are identical across scripts. Doing so allowed the use of existing tools for verification and cross-checking tasks, as well as formatting the data into HTML-based tables.
Review is solicited on whether the code points constitute cross-script homoglyphs and whether having such a data collection is considered useful.

Overview

This file presents a collection of code points and code point sequences that could be considered cross-script homoglyphs. The focus is on code points that cannot be distinguished, because they are shown with identical glyphs in most or all fonts.

This file was generated by starting with Intentional.txt from Version 10.0.0, filtered to exclude DISALLOWED code points from IDNA 2008 (the latter set is based on Unicode version 6.3.0). Also filtered out were any "in-script" homoglyphs, that is any code point that are identical to another code point of the same script, and not also a cross-script homoglyph.

In addition, code points that where cited as cross-script homoglyphs in relevant LGR proposals have been added and referenced to the corresponding proposal

This list is augmented by adding a few code points that are not intentionally the same, but effectively identical. Those code points may have a nominally distinct shape, as shown in the code charts, and while some fonts may make that distinction, many or most common fonts do not. In some cases a code point has two common glyph shape and one or both may be identical to the shape for another code point.

There is a much larger set of code points that are pairwise similar, sometimes confusingly so; compare the well known examples of DIGIT ONE (1) and SMALL L (l) that predates the development of IDNs. These "confusables" are not considered true homoglyphs and are excluded here.

Classification by similarity

The following taxonomy can be defined to classify the similarity in the glyph shape used for code points:

This classification overlaps with a different categorization that pertains to the degree that one can expect for a visible distinction to be presented to the user,

Obsolete and not widely used code points

A number of code points in this file are for scripts that are not in widespread modern use or they are deprecated or obsolete code points in otherwise modern scripts. For these code points, known cross-script homoglyphs have nevertheless been listed. For deprecated and known obsolete code points, the conservative approach would be to not allow them, which also removes the problem of their homoglyph relations. For cases where the script itself is in limited or not in modern use, it should be noted that these may not be well-understood enough to be sure that all cross script homoglyph relations are known. They may also have other as-yet-unidentified problems for use with identifiers.

The data on ancient scripts is very selective and generally inadequate. This was known when "intentional.txt" was created. For example, the Cuneiform entries are just some obvious examples, not a well-researched set.

Code points in modern use

Code points referenced with [100] are included in the MSR-2. The MSR-2 is limited to code points in widespread modern use. This does not mean that all of these code points are not "safe", but at least they are moderately well understood, and information about them is available in this and similar data collections.

A note on Intentional.txt: the methodology for that file normalizes to NFD (fully decomposed) not NFC (composed) which is the way IDNs are normalized. Accordingly, precomposed code points corresponding to NFD sequences differing only by codepoints that are listed in Intentional.txt are also considered "intentional" here and are referenced as [151].

Change History

1

Original draft