Saving tokenizer's configuration

Amalq · February 21, 2022, 3:39am

Hi,

I tried to fine-tuned language model using the following code

huggingface/notebooks/blob/master/examples/language_modeling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "X4cRE8IbIrIV"
   },
   "source": [
    "If you're opening this Notebook on colab, you will probably need to install 🤗 Transformers and 🤗 Datasets. Uncomment the following cell and run it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "MOsHUjgdIrIW",

This file has been truncated. show original

The issue I face is that the model is saved using (trainer.save_mode()), but I don’t know how save tokenizer’s results as well?

I’ve tried to add tokenizer=tokenizer to trainer, but it ariesed error.
any help is really appreciated.

tntchung · February 24, 2022, 4:22pm

hi,

As the tokenizer is AutoTokenizer, you can use tokenizer.save_pretrained(your path) to save it.

Topic		Replies	Views
"How to train a new language model from scratch using Transformers and Tokenizers" possibly requiring an update Site Feedback	4	2464	November 1, 2022
How to save tokenizer after finetunning distilgpt2 model Beginners	2	446	March 18, 2022
How to save my tokenizer using save_pretrained? Beginners	5	16746	August 13, 2021
Tokenizer Saving Issues, Wrapper Issues and Push to Hub issues Beginners	2	933	December 28, 2022
Two approaches to training a tokenizer Beginners	0	740	March 6, 2023

Saving tokenizer's configuration

Related Topics