New metrics for Language models: corpus coverage

[Trondtr]

We want to add corpus coverage as ab evaluation for our language models, this is perhaps the best evaluation measure.
have
I suggest doing the following, for fsts

1. Count the number of words or tokens (probably tokens) in the open corpus (evt. both open and closed, but open is more transparent)
2. Run them through the descriptive analyser
3. In the table under Language Models, add the coverage as percentage
4. in the documentation page (next to the map) add the same percentage, and also the size of the corpus

The table under Language Models is crowded, I suggest to save space, as follows:

1. to remove the version number column from the table and only keep it next to the map, not in the table
2. If needed: publish the corpus coverage in the table only as percenage, not as percentage and number of šord for corpus.
3. On the page for each language there is more space, here we may have both percentage and number.

[flammie]

This is something I've had on my wishlist too, I think it should eventually run under CI and probably also / or on corpus push CI, i.e. updates to corpus also show up in coverage? But we can start with having it work same as those speller badge datas that anyone can commit.

[snomos]

I definitely agree with both of you regarding the desire to have this metric. But I disagree with Trond that we should remove the version column to make room for the new information.

My plan is to stop using the manual classification into production, beta, alpha and experimental we have now, and instead have automatic classification based upon the following metrics:

• Version number
• lemma count
• coverage as you describe it in this issue

That is, I want us to start to take version numbers seriously, and release analyser tools like we release spellers and other tools. Hiding the version number will only make us forget it again.