innat
March 25, 2023, 6:24am
1
I’ve searched on doc but couldn’t find any hint.
Generally, detokenize is the inverse of the tokenize method, and can be used to reconstrct a string from a set of tokens.
from transformers import TFBertTokenizer
tf_tokenizer = TFBertTokenizer.from_pretrained("bert-base-uncased")
# something like
tf_tokenizer.encode([string]) # o/p: ids / token
tf_tokenizer.decode([1,2,3]) # o/p: string
Available in some extent
# Copyright 2023 The KerasNLP Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Tests for BERT tokenizer."""
import os
import pytest
import tensorflow as tf
from absl.testing import parameterized
show original
... def detokenize(self, input):
... return tf.strings.reduce_join(input, axis=-1, separator=" ")
>>> text = tf.ragged.constant([["hello", "world"], ["a", "b", "c"]])
>>> print(SimpleDetokenizer().detokenize(text))
tf.Tensor([b'hello world' b'a b c'], shape=(2,), dtype=string)
"""
__metaclass__ = abc.ABCMeta
@abc.abstractmethod
def detokenize(self, input): # pylint: disable=redefined-builtin
"""Assembles the tokens in the input tensor into a string.
Generally, `detokenize` is the inverse of the `tokenize` method, and can
be used to reconstrct a string from a set of tokens. This is especially
helpful in cases where the tokens are integer ids, such as indexes into a
vocabulary table -- in that case, the tokenized encoding is not very
human-readable (since it's just a list of integers), so the `detokenize`
method can be used to turn it back into something that's more readable.
Args:
Hi @innat , and sorry for the delay! I don’t think our TF in-graph tokenizers support decoding/detokenization. However, our main tokenizers do. So you could do something like
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
tokenizer.decode([1, 2, 3])
This should work for most purposes - do you have a usecase for wanting to do de tokenization inside a TF graph? We’re very interested if so, because we assumed people would generally not need that!
1 Like