special_tokens, 'list' object has no attribute 'keys'

[Ratinax]

System Info

In tokenization_utils_base.py line 1190, there is a special_tokens.keys() done. But special_tokens is typed as List[str].

And when using it, I have special_tokens being equal to ['<s>NOTUSED', '</s>NOTUSED', '<unk>NOTUSED'] this lead to the error 'list' object has no attribute 'keys'

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

use AutoTokenizer.from_pretrained(model_path) with a valid model_path and a valid content

Expected behavior

not raising an error and working

[sebastianvlad1]

I would like to take this.

I’ll reproduce it on current main, then add a narrow load time normalization in tokenization_utils_base.py so list-valued model_specific_special_tokens is treated as extra_special_tokens (where lists belong), with a local AutoTokenizer.from_pretrained(...) regression test in tests/tokenization/test_tokenization_utils.py.

[AnayGarodia]

Could not reproduce this on v5.13.0: AutoTokenizer.from_pretrained("almanach/camembert-base") (which has the NOTUSED style tokens) loads fine. The only special_tokens.keys() call at current HEAD is in _set_model_specific_special_tokens, and every path into it guards against list values, so this looks version-dependent. Could you share your transformers version and the tokenizer_config.json / special_tokens_map.json of the failing model? Happy to dig further with that.