Custom segmentation for words.separated.like.this?

[recatek]

I'm trying to replicate the behavior seen in code editors, browser bars, etc. where words.separated.like.this can be effectively segmented for navigation and selection, either as words|.|separated|.|like|.|this (most text editors) or as words.|separated.|like.|this (browser bar). However, I noticed that by default WordSegmenter treats this as a single word unless the period is followed by a space. Is there a way to override this behavior?

[recatek]

Some additional digging on this myself:

It looks like the language ID "en-US-POSIX" exists with this behavior, as demonstrated here: https://icu4c-demos.unicode.org/icu-bin/icusegments#1/en_US_POSIX (it's called "English (United States, Computer)" here).

Typing in words.separated.like.this yields breakpoints words|.|separated|.|like|.|this. However, running this Rust code does not replicate that behavior:

use icu::locale::langid;
use icu::segmenter::WordSegmenter;
use icu::segmenter::options::WordBreakOptions;

fn main() {
    let mut options = WordBreakOptions::default();
    let langid = &langid!("en-US-POSIX");
    options.content_locale = Some(langid);
    let segmenter = WordSegmenter::try_new_auto(options).unwrap();

    let breakpoints: Vec<usize> = segmenter
        .as_borrowed()
        .segment_str("words.separated.like.this")
        .collect();

    println!("{:?}", breakpoints);
}

This prints [0, 25] like other language IDs. Is there something I'm doing wrong here? Or is there a better/alternative way?

[sffc]

CC @macchiati

There's been discussion about segmentation profiles, which could tweak behavior for cases like this. Browser engines like Chrome already implement the behavior you requested:

[...new Intl.Segmenter("en", { granularity: "word" }).segment("hello aaa.bb.ccc world")]
// returns an array with 9 segments

I've been wanting to get Chrome's tailorings upstreamed into CLDR as a segmentation profile. Here is the ticket you can watch:

https://unicode-org.atlassian.net/browse/CLDR-15839

As far as en-US-POSIX, I do see the tailoring here:

https://github.com/unicode-org/cldr/blob/main/common/segments/en_US_POSIX.xml

I'm not sure why ICU4X isn't implementing it. CC @aethanyc @makotokato

[recatek]

I'm not sure why ICU4X isn't implementing it.

Looking a little further into it, it looks like it's missing in a few places. It looks like this uses cldr-json rather than the XML data, and I'm not seeing en-us-POSIX in cldr-json/cldr-segments-full. Additionally, I don't think this is reading the segment data at all, since there doesn't appear to be a segments data deserializer in cldr_serde.

I'm definitely not an expert on these crates nor the CLDR format/structure itself, so I could be missing things myself.

[sffc]

ICU4X does support locale tailorings, like fi/sw, but you're right that they don't seem to come from CLDR Segments.