Hello!
I am trying to train distilgpt2 with my own data on colab. The problem is, even after installing the packages from requirements.txt, it is running into an import error. I am reproducing the entire error below.
Traceback (most recent call last):
File “train.py”, line 28, in
from distiller import Distiller
File “/content/transformers/examples/distillation/distiller.py”, line 31, in
from grouped_batch_sampler import GroupedBatchSampler, create_lengths_groups
File “/content/transformers/examples/distillation/grouped_batch_sampler.py”, line 24, in
from utils import logger
File “/content/transformers/examples/distillation/utils.py”, line 23, in
import git
File “/usr/local/lib/python3.6/dist-packages/git/init.py”, line 38, in
from git.exc import * # @NoMove @IgnorePep8
File “/usr/local/lib/python3.6/dist-packages/git/exc.py”, line 9, in
from git.compat import UnicodeMixin, safe_decode, string_types
File “/usr/local/lib/python3.6/dist-packages/git/compat.py”, line 16, in
from gitdb.utils.compat import (
ModuleNotFoundError: No module named ‘gitdb.utils.compat’
Any help is appreciated. Is this because of an improper version of python?
NB : I have restarted the runtime after installing packages from requirements.txt , but still the error persists.
Thank you for your time!
Are you sure you ran
pip install -r requirements.txt
in the distillation subfolder of the repo? It looks like you are missing some required packages.
Yes. I have installed all packages using the given command and restarted the runtime to use them all.
Is this a problem pertaining to google-colab?
I am reproducing the output of the pip install requirements command.
Requirement already satisfied: transformers in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 1)) (3.0.2)
Collecting gitpython==3.0.2
Downloading https://files.pythonhosted.org/packages/8e/c7/70bd352e8a561a9b6d1cde9aa313b9d7c871b0c94c3821f44c01f3187e1d/GitPython-3.0.2-py3-none-any.whl (453kB)
|████████████████████████████████| 460kB 6.3MB/s
Requirement already satisfied: tensorboard>=1.14.0 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 4)) (2.2.2)
Collecting tensorboardX==1.8
Downloading https://files.pythonhosted.org/packages/c3/12/dcaf67e1312475b26db9e45e7bb6f32b540671a9ee120b3a72d9e09bc517/tensorboardX-1.8-py2.py3-none-any.whl (216kB)
|████████████████████████████████| 225kB 16.4MB/s
Collecting psutil==5.6.6
Downloading https://files.pythonhosted.org/packages/5f/dc/edf6758183afc7591a16bd4b8a44d8eea80aca1327ea60161dd3bad9ad22/psutil-5.6.6.tar.gz (447kB)
|████████████████████████████████| 450kB 10.7MB/s
Collecting scipy==1.3.1
Downloading https://files.pythonhosted.org/packages/29/50/a552a5aff252ae915f522e44642bb49a7b7b31677f9580cfd11bcc869976/scipy-1.3.1-cp36-cp36m-manylinux1_x86_64.whl (25.2MB)
|████████████████████████████████| 25.2MB 129kB/s
Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from transformers->-r requirements.txt (line 1)) (20.4)
Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from transformers->-r requirements.txt (line 1)) (1.18.5)
Requirement already satisfied: sacremoses in /usr/local/lib/python3.6/dist-packages (from transformers->-r requirements.txt (line 1)) (0.0.43)
Requirement already satisfied: tokenizers==0.8.1.rc1 in /usr/local/lib/python3.6/dist-packages (from transformers->-r requirements.txt (line 1)) (0.8.1rc1)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.6/dist-packages (from transformers->-r requirements.txt (line 1)) (2019.12.20)
Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.6/dist-packages (from transformers->-r requirements.txt (line 1)) (4.41.1)
Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from transformers->-r requirements.txt (line 1)) (2.23.0)
Requirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (from transformers->-r requirements.txt (line 1)) (3.0.12)
Requirement already satisfied: sentencepiece!=0.1.92 in /usr/local/lib/python3.6/dist-packages (from transformers->-r requirements.txt (line 1)) (0.1.91)
Requirement already satisfied: dataclasses; python_version < "3.7" in /usr/local/lib/python3.6/dist-packages (from transformers->-r requirements.txt (line 1)) (0.7)
Collecting gitdb2>=2.0.0
Downloading https://files.pythonhosted.org/packages/52/7e/59f96b47f671b3fe0aa0c1b609531a540434b719a10c417581e25b383909/gitdb2-4.0.2-py3-none-any.whl
Requirement already satisfied: grpcio>=1.24.3 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14.0->-r requirements.txt (line 4)) (1.30.0)
Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14.0->-r requirements.txt (line 4)) (1.17.2)
Requirement already satisfied: wheel>=0.26; python_version >= "3" in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14.0->-r requirements.txt (line 4)) (0.34.2)
Requirement already satisfied: protobuf>=3.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14.0->-r requirements.txt (line 4)) (3.12.2)
Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14.0->-r requirements.txt (line 4)) (1.12.0)
Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14.0->-r requirements.txt (line 4)) (0.9.0)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14.0->-r requirements.txt (line 4)) (1.7.0)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14.0->-r requirements.txt (line 4)) (0.4.1)
Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14.0->-r requirements.txt (line 4)) (1.0.1)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14.0->-r requirements.txt (line 4)) (3.2.2)
Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14.0->-r requirements.txt (line 4)) (49.1.0)
Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->transformers->-r requirements.txt (line 1)) (2.4.7)
Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers->-r requirements.txt (line 1)) (0.16.0)
Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers->-r requirements.txt (line 1)) (7.1.2)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->transformers->-r requirements.txt (line 1)) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->transformers->-r requirements.txt (line 1)) (2.10)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->transformers->-r requirements.txt (line 1)) (1.24.3)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->transformers->-r requirements.txt (line 1)) (2020.6.20)
Collecting gitdb>=4.0.1
Downloading https://files.pythonhosted.org/packages/48/11/d1800bca0a3bae820b84b7d813ad1eff15a48a64caea9c823fc8c1b119e8/gitdb-4.0.5-py3-none-any.whl (63kB)
|████████████████████████████████| 71kB 9.5MB/s
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=1.14.0->-r requirements.txt (line 4)) (0.2.8)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=1.14.0->-r requirements.txt (line 4)) (4.1.1)
Requirement already satisfied: rsa<5,>=3.1.4; python_version >= "3" in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=1.14.0->-r requirements.txt (line 4)) (4.6)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=1.14.0->-r requirements.txt (line 4)) (1.3.0)
Requirement already satisfied: importlib-metadata; python_version < "3.8" in /usr/local/lib/python3.6/dist-packages (from markdown>=2.6.8->tensorboard>=1.14.0->-r requirements.txt (line 4)) (1.7.0)
Collecting smmap<4,>=3.0.1
Downloading https://files.pythonhosted.org/packages/b0/9a/4d409a6234eb940e6a78dfdfc66156e7522262f5f2fecca07dc55915952d/smmap-3.0.4-py2.py3-none-any.whl
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.6/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard>=1.14.0->-r requirements.txt (line 4)) (0.4.8)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=1.14.0->-r requirements.txt (line 4)) (3.1.0)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < "3.8"->markdown>=2.6.8->tensorboard>=1.14.0->-r requirements.txt (line 4)) (3.1.0)
Building wheels for collected packages: psutil
Building wheel for psutil (setup.py) ... done
Created wheel for psutil: filename=psutil-5.6.6-cp36-cp36m-linux_x86_64.whl size=274397 sha256=c857e5eafb2cdb184bdd46c7ec670c3baaa970e8ca8e2f619ede2e35657c66ed
Stored in directory: /root/.cache/pip/wheels/d2/24/03/45099ba838d8bfe67d44d36581135a5b59b71c6c8620f5575a
Successfully built psutil
ERROR: tensorflow 2.2.0 has requirement scipy==1.4.1; python_version >= "3", but you'll have scipy 1.3.1 which is incompatible.
ERROR: albumentations 0.1.12 has requirement imgaug<0.2.7,>=0.2.5, but you'll have imgaug 0.2.9 which is incompatible.
Installing collected packages: smmap, gitdb, gitdb2, gitpython, tensorboardX, psutil, scipy
Found existing installation: psutil 5.4.8
Uninstalling psutil-5.4.8:
Successfully uninstalled psutil-5.4.8
Found existing installation: scipy 1.4.1
Uninstalling scipy-1.4.1:
Successfully uninstalled scipy-1.4.1
Successfully installed gitdb-4.0.5 gitdb2-4.0.2 gitpython-3.0.2 psutil-5.6.6 scipy-1.3.1 smmap-3.0.4 tensorboardX-1.8
Thanks for your help!
is there any command line interface available to do so.
kindly pls suggest.
and how to upload python files and folders… as in import feature.
& even suggest how to write following command in requirements if cmd line interface is not available
git clone -b opencv-2.4 --single-branch https://github.com/Itseez/opencv.git