How to construct Chinese dataset with gpt2 fine tune

LycheeX · February 14, 2023, 5:52am

what's the format of my own datasets when running language-modeling with gpt2

opened 05:38AM - 14 Feb 23 UTC

### System Info - `transformers` version: 4.26.0.dev0 - Platform: Linux-3.10.0…-1160.81.1.el7.x86_64-x86_64-with-glibc2.17 - Python version: 3.9.15 - Huggingface_hub version: 0.11.1 - PyTorch version (GPU?): 1.13.1+cu116 (True) - Tensorflow version (GPU?): not installed (NA) - Flax version (CPU?/GPU?/TPU?): not installed (NA) - Jax version: not installed - JaxLib version: not installed - Using GPU in script?: <fill in> - Using distributed or parallel set-up in script?: <fill in> ### Who can help? @vanpelt @pvl @arfon @xeb @kashif @sgugger ### Information - [X] The official example scripts - [ ] My own modified scripts ### Tasks - [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...) - [X] My own task or dataset (give details below) ### Reproduction want to know how to construct my own data, and does it support chinese words with gpt2, my dataset is Chinese sentence , the format is txt, and no label could you pls help me ? ### Expected behavior must have [CLS] in the beginning of the sentence ? and what's the meaning of the "==" as follow (from wiki.test.raw) : = Robert Boulter = Robert Boulter is an English film , television and theatre actor . He had a guest @-@ starring role on the television series The Bill in 2000 . This was followed by a starring role in the play Herons written by Simon Stephens , which was performed in 2001 at the Royal Court Theatre . He had a guest role in the television series Judge John Deed in 2002 . In 2004 Boulter landed a role as " Craig " in the episode " Teddy 's Story " of the television series The Long Firm ; he starred alongside actors Mark Strong and Derek Jacobi . He was cast in the 2005 theatre productions of the Philip Ridley play Mercury Fur , which was performed at the Drum Theatre in Plymouth and the Menier Chocolate Factory in London . He was directed by John Tiffany and starred alongside Ben Whishaw , Shane Zaza , Harry Kent , Fraser Ayres , Sophie Stanton and Dominic Hall . In 2006 , Boulter starred alongside Whishaw in the play Citizenship written by Mark Ravenhill . He appeared on a 2006 episode of the television series , Doctors , followed by a role in the 2007 theatre production of How to Curse directed by Josie Rourke . How to Curse was performed at Bush Theatre in the London Borough of Hammersmith and Fulham . Boulter starred in two films in 2008 , Daylight Robbery by filmmaker Paris Leonti , and Donkey Punch directed by Olly Blackburn . In May 2008 , Boulter made a guest appearance on a two @-@ part episode arc of the television series Waking the Dead , followed by an appearance on the television series Survivors in November 2008 . He had a recurring role in ten episodes of the television series Casualty in 2010 , as " Kieron Fletcher " . Boulter starred in the 2011 film Mercenaries directed by Paris Leonti . = = Career = =

want to use my own dataset, run clm
must have [cls] in the beginning ?

Topic		Replies	Views
GPT-2 fine-tuning Beginners	0	1574	June 12, 2023
Train gpt-2 from scratch in Italian Beginners	0	878	September 8, 2022
Language-modeling script "killed" when fine-tuning gpt2-medium Beginners	3	6863	May 19, 2023
Defining a custom dataset for fine-tuning translation Beginners	4	5061	July 10, 2021
Guidance Needed on Choosing the Right Dataset Format for Transformer Model Training 🤗Datasets	1	1743	December 8, 2023

How to construct Chinese dataset with gpt2 fine tune

Related topics