Using transformers (BERT, RoBERTa) without embedding layer

agemagician · December 16, 2020, 7:55pm

You can send “top_k” parameter to “fill-mask” method, to return more/all tokens.
Check here:

huggingface/transformers/blob/1c1a2ffbff2052100053cddb3a87d45fb9d210ca/src/transformers/pipelines.py#L1184


"""
def __init__(
    self,
    model: Union["PreTrainedModel", "TFPreTrainedModel"],
    tokenizer: PreTrainedTokenizer,
    modelcard: Optional[ModelCard] = None,
    framework: Optional[str] = None,
    args_parser: ArgumentHandler = None,
    device: int = -1,
    top_k=5,
    task: str = "",
):
    super().__init__(
        model=model,
        tokenizer=tokenizer,
        modelcard=modelcard,
        framework=framework,
        args_parser=args_parser,
        device=device,
        binary_output=True,

If it is still doesn’t fit your use-case, then you have to implement it your self.

Topic		Replies	Views
How could protein language models generate outputs for natural language input texts? 🤗Transformers	4	419	November 21, 2023
Transformers with protein data Beginners	0	323	July 6, 2022
PreTrain ProteinBERT from scratch Flax/JAX Projects	5	2319	July 6, 2022
How to Fine-tune Rostlab/prot_t5_xl_uniref50 Model for Sequence Generation Beginners	0	389	April 5, 2023
Unmasker probabilities for all tokens in sequence 🤗Transformers	0	223	December 23, 2022

Using transformers (BERT, RoBERTa) without embedding layer

Related topics