Shaping check roadmap

[MrBrezina]

Here is a humble start:

• mark-to-base placement: automatic based on combinations provided
• Semitic scripts:
  • Arabic positional alternates (joining)
  • Syriac positional alternates (joining)
  • Hebrew positional alternates and mark positioning
• Brahmi-based syllabic shaping:
  • Devanagari (Hindi, Marathi, Nepali, Sanskrit, …)
  • Gujarati
  • Gurmukhi (Punjabi)
  • Bengali (Bengali, Assamese)
  • Odia/Oriya
  • Kannada
  • Telugu
  • Tamil
  • Malayalam
  • Meitei Mayek (Manipuri)
  • Kaithi
  • Tibetan
• SEA scripts:
  • Thai syllables + mark placement
  • Khmer syllables + mark placement
  • Lao syllables + mark placement
  • Burmese syllables + mark placement
  • Tham
  • Tai Viet
  • Kayah Li
  • Chakma
  • Javanese
  • Balinese
  • other smaller scripts that might be already included
• African scripts:
  • N'Ko

Not sure:

• Geʽez
• Buginese
• Hanunoo
• Sundanese
• Thaana
• Baybayin
• Batak
• Bamum

Scripts that do NOT seem to require any additional shaping checks:

• Chinese
• Hiragana, Katakana, Kanji (Japanese)
• Hangul (Korean)
• Hanja (Korean)
• Latin
• Cyrillic
• Armenian
• Georgian
• Greek
• Coptic
• Tifinagh
• Akkadian
• Summerian
• Linear A/B
• Ancient South Arabian script
• Avestan
• Vai
• Modern Yi
• Ojibwe Syllabics
• Inuktitut Syllabics
• Cree
• Cherokee
• Egyptian Hieroglyphs (for now)

Currently not in the database, but will need a shaping check:

• Mongolian

Please, respond if you think of some checks I have not considered. This is to set out a road map for what needs to be eventually implemented. I am currently including only scripts for languages included in the database.

A shaping check is anything that can be checked automatically using the OpenType font and Harfbuzz.

[kontur]

Using the unicode joining types table and looking at what scripts those chars are I get these as requiring joining checks:

Adlm: Adlam*
Arab: Arabic
Chrs: Chorasmian*
Mand: Mandaic*
Mani: Manichaean*
Mong: Mongolian*
Nkoo: Nko
Ougr: Old Uyghur*
Phag: Phags Pa*
Phlp: Psalter Pahlavi*
Rohg: Hanifi Rohingya*
Sogd: Sogdian*
Syrc: Syriac
Zinh: Inherited # ignore
Zyyy: Common # ignore

I marked * which are not in Hyperglot at all, yet. The remaining are listed already above. I'll tweak the actual check (and its name) to opt-in for those scripts, that should automagically give us checking for those, since the actual check logic simply observes shaping based on joining side (L, R, D_ual_) with ZWJ and doesn't required (afaik) any additional logic.

[kontur]

Not sure what to make of N'Ko unjoined (e.g. Noto N'Ko Unjoined) for our purposes of language detection.

It would be possible to have a more specific joining check for N'Ko in particular that only opts-in when certain shaping features are present in the font, assuming that any N'Ko font (unicodes) without them is deliberately the "Unjoined" kind and doesn't require shaping checks. In a way it would be odd if e.g. Noto N'Ko Unjoined was not detected as supporting N'Ko, but also it's an stylistic edge case. 9 / 10 N'Ko fonts (and those may be total numbers ;-) ) should be checked for correct joining behaviour, and enforcing this might be a lot more beneficial for the overall purpose of better quality fonts for all scripts in Hyperglot.

[khaledhosny]

There are unjoined Arabic fonts as well (e.g. many of the fonts in https://www.myfonts.com/collections/arabetics-foundry?sort_by=price-ascending&pageType=foundryPage).

[kontur]

There are unjoined Arabic fonts as well (e.g. many of the fonts in https://www.myfonts.com/collections/arabetics-foundry?sort_by=price-ascending&pageType=foundryPage).

Good to know. I am inclined to think those are edge-cases that don't represent Arabic script fonts in a general sense, but it would be interesting to get a better sense of how widespread and accepted such fonts are. Also re the N'Ko case mentioned above. There also is the --no-shaping flag in the CLI / API equivalent to forego shaping checks entirely (it could be more specific, also, e.g. --unjoined). We could theoretically check and say if a font covers Arabic codepoints largely but does not contain any init/medi/fina/isol features at all, we assume it's such an unjoined font... but then what if such a unjoined font uses those features for some combinations somehow. Hard to make save guesses here, but could inquire from such a foundry that uses this style quite some.

I wonder what would be the split: False negatives for unjoined fonts, vs false positives for joined Arabic fonts failing to actually shape. I think this tool should err on the side of false negatives. If Hyperglot detects support, it should absolutely support a language. Maybe the CLI could provide better feedback, e.g. in the above case of a failed Arabic unjoined font with unicode coverage but absent positional features, output something like "The checked font appears to cover Arabic (or XXX) unicode ranges but lacks positional shaping features — if you are checking an unjoined font for language support, use the --no-shaping option".