Way to train a basic Transformer

Beginners
1

Is there any way to train a basic Transformer (the transformer architecture presented in the original Attention Is All You Need paper). I’m trying to set up a translation baseline on my custom dataset (previously trained models do not exist for the language). If anyone can point out a ready-made implementation of just the basic Transformer it would be very helpful.

Thanks!

1 Like
2

I think OpenNMT is the better framework to do something like that. It allows to train a model that you describe easily. transformers is more directed towards specialized, specific transformer architectures.

1 Like
3

Thanks! I used fairseq a few times. I think OpenNMT is kinda similar, they seem more black box like not full control on the entire pipeline. That’s why I wanted to know if such a thing is possible with transformers because this is native Python and not command-line tools.

4

Have you seen this: https://pytorch.org/tutorials/beginner/transformer_tutorial.html ?

2 Likes
5

And this is TF2-Keras equivalent to @abhishek’s recommendation :smiley:

1 Like
6

Thank you! I implemented mine from scratch in Pytorch and Lightning with the help of bentrevett’s notebook and source code of BART in the transformers library.

7

Thank you for the resources.

1 Like