So last week we shared the first feedback request on transformers. The community was pretty amazingly involved in this with close to 900 detailed feedback forms to analyze and dive into, containing more than 50 thousand words of open answers
In this post, I would like, first, to deeply thank the community for this amazing feedback and for the carefully crafted answers, people were so keen to share. It’s really humbling to read all these encouraging and more critical words, from the thank-you to the detailed critics of the library in which people tried to be so helpful I wanted to thank you all one by one if only I had all your email addresses.
In the following post, I’d like to try to summarize and share the most interesting takeaways from this huge corpus.
Let me first start by our users
transformers has three big user communities of roughly equal sizes (in the respondents):
- Researchers (blue)
- ML engineers (red)
- Data scientists (green)
Comparing the answers from each community can give some hints:
Researchers :
- The oldest and core users
- They probably often develop or study models
- They are more often using directly master and want verbose and easy to customize model and examples. Some would like to be able to train from scratch. They usually don’t want to use high-level training abstraction and encapsulated examples.
ML engineers :
- They often joined a bit after the researchers community
- They probably often push models in production applications
- They are more often using recent PyPi versions. They are interested in fast inference, fp16, ONNX, TPU, multi-GPU, production-ready models. Some of them like training abstractions, some don’t.
Data scientists :
- The most recent community of users (many have been using it since less than 3 months)
- They probably often use models for data-analytics (i.e. no strong reqs for perf)
- They are often beginning to use transformers models. They are interested in fast prototyping tool and examples that they can easily and quickly adapt to their use cases. They often don’t care much about the model internals or having access to training details but they are very interested in diving into/mastering data-processing.
There are also a lot of common points between these communities (they have mostly common interests) so don’t be distracted by these apparent differences, almost all of our users want recent (SOTA) models in an easy-to-use API
For how long have our users been using the library
There is a significant influx of new users (green + purple are < 3 months users). One-third of the respondents have been using the library for less than 3 months!
The longest users are researchers followed by ML engineers. Data scientists are more recent users (40% of them have been using transformers for less than 3 months).
work or
fun
Most users are using transformers for work (80% overall)
Researchers are somewhat the more serious community (>90% using it for work)
Data scientists are using it more for fun than the other communities at the moment (maybe also because they are still discovering it).
Which version
Many users are on the latest or two latest versions (blue+red+green+purple)
Researchers use master (red) more than the other communities (maybe because they tweak the models more).
ML engineers are somewhat a bit more conservative (more users on 2.11.X (green))
Data scientists tend to use master (red) less than the other communities (maybe because they customize the models less than the other groups)
Would you recommend the library?
Importance of various features in the examples
User specific interests:
Three features are rated as most essential in the examples overall by all user-communities:
- Full/transparent access to data-processing
- Full access to training loop
- Simple data-downloading and processing
People care less about TPU support
Some more community-specific interests:
- Researchers want more strongly to avoid encapsulated training than the other communities
- ML engineers are more interested in FP16, multi-GPU, TPU than the other communities
- Data Scientists care less about training and optimization than the other communities (more ok with encapsulated trainer logic) and care more about the data processing
What to prioritize
Users ask for more priority on:
- Adding examples for NLP tasks + easier to adapt to more real-life scenarios
- Keep adding SOTA models
Less interesting for most users:
- Refactor the code toward modularization
What do you like the most
Most frequent reasons are:
- Easy to use and simplicity
- Many SOTA pretrained models
- Community
Short summary of the top 300 strongest likes (apart from above mentioned top 3):
- Pipelines <= many people like them
- AutoModels <= many people like them as well
- Easy to tweak - self-contained-verbose - non-modularization - no code reuse
- Good doc
- Model hub
- PyTorch and TF interopt
- Transparency
What do you dislike the most
Some top dislikes are:
- Need more examples, more real-use-cases and on how to load your own datasets for certain tasks - More examples on using transformers for text or token classification
- More tutorials and lack of guidance for simple examples
- Too frequent breaking changes
- Too much modularization - Model code is spread across too many files
- Examples are too encapsulated – examples are hard to unpack sometimes
- Trainer
- Less support for Tensorflow 2.0
- model hub is a bit messy, it’s hard to find the right model
- …
Open-feedback
The most noticeable one:
Thanks! <=
…