This will not work for gpt4-x-alpaca-13b-native-4bit-128g since it requires the GPTQ package. Therefore you need to create a custom infernece.py script and add the latest transforemrs version + gptq with a requirements.txt
1 Like