There are many tutorial using add_tokens(special_tokens=True)
, but I read the source code, and find that add_special_tokens
will do more thing than add_tokens
, Which is prefer?
There are many tutorial using add_tokens(special_tokens=True)
, but I read the source code, and find that add_special_tokens
will do more thing than add_tokens
, Which is prefer?