When i want to use tensor parallelism during the model inference , I find the parallelism is supported on training. How to customize tensor parallelism？
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| Tensor parallelism for customized model | 0 | 49 | September 2, 2024 | |
| Data Parallelism for multi-GPUs Inference | 0 | 494 | October 26, 2022 | |
| Trainer.predict in parallel not supported! | 2 | 611 | November 2, 2022 | |
| How can I use Inference API with my model? | 0 | 146 | February 24, 2024 | |
| GPU inference slows down if done in a loop | 1 | 1505 | July 20, 2020 |