In Langium/Xtext, to combine multiple tokens into one, one needs to write a data type parser rule:
Combination returns string: "a" | "b" | ID;
This has a bunch of implications, mainly slow performance (multiple reasons, that I won't go into here).
Inspired by chevrotain, we could collect multiple tokens into one category that acts as a "super" token that can be used in the grammar like the Combination parser rule:
token category Combination {
// Order shouldn't matter
"a", "b", ID
}
Hints for the implementation can be taken from chevrotain as well.
In Langium/Xtext, to combine multiple tokens into one, one needs to write a data type parser rule:
This has a bunch of implications, mainly slow performance (multiple reasons, that I won't go into here).
Inspired by chevrotain, we could collect multiple tokens into one category that acts as a "super" token that can be used in the grammar like the
Combinationparser rule:Hints for the implementation can be taken from chevrotain as well.