Repository navigation
SentencePiece Tokenizer does not encode </s> as special token #383
Description
Activity
I think this is to be expected, see
- SentencePieceProcessor.encode() breaks down special tokens that exist in vocabulary google/sentencepiece#777
- Unable to tokenize <s>, <pad>, </s>, and <unk> correctly in Python google/sentencepiece#667
- What is the difference between --user_defined_symbols and --control_symbols google/sentencepiece#215
In modalities, we put the tokenizer's EOD token between documents here.
This was a good hint here:
google/sentencepiece#667 (comment)I wonder if this creates issues for our instruction tuning, where we do play around with the special tokens.
What's your opinion on this @lllAlexanderlll ?Yes, that is problematic, if we want to use the token - we could use any special token e.g. the "placeholder_tokens"
However, adding new tokens is not possible yet due to #208. Currently, a given single token is marked as special and hence tokenized prior to non-special tokens, which is needed to reliably not resolve prefixed tokens and special tokens into a single token.
When tokenizing the
</s>token, it does not get tokenized into a single special token, instead it gets split into multiple subtokens.