Iʻm struggling to get the ʻokina, the Hawaiian glottal stop character (U+02BB), treated as a letter and not as punctuation in SentencePiece subword tokenization, whether BPE or Unigram. Can someone tell me how I can achieve that please?
This is what I attempted:
spm_train --input=tgt-train.txt --model_prefix=data/tgt_spm --vocab_size=32000 --model_type=bpe --character_coverage=1.0 --output_format=piece --input_sentence_size=1000000 --user_defined_symbols=ʻa,ʻe,ʻi,ʻo,ʻu,ʻā,ʻē,ʻī,ʻō,ʻū
However, only these specific tokens above are listed in the created vocab file, when I want all tokens coming from a word that contains the glottal stop to carry the glottal stop.