Dockerfile for deploying Qwen QwQ 32B on A10Gs , L4s or L40S

Show and Tell
1

Adding a Dockerfile here that can be used to deploy Qwen on any machine which has a combined GPU RAM of ~80GBs. The below Dockerfile is for multi-GPU L4 instances as L4s are the cheapest ones on AWS, feel free to make changes to try it on L40S, A10Gs, A100s etc. Soon will follow up with metrics around single request tokens / sec and throughput.

# Dockerfile for Qwen QwQ 32B

FROM vllm/vllm-openai:latest

# Enable HF Hub Transfer for faster downloads
ENV HF_HUB_ENABLE_HF_TRANSFER 1

# Expose port 80
EXPOSE 80

# Entrypoint with API key
ENTRYPOINT ["python3", "-m", "vllm.entrypoints.openai.api_server", \
            # name of the model
           "--model", "Qwen/QwQ-32B", \
           # set the data type to bfloat16 - requires ~1400GB GPU memory
           "--dtype", "bfloat16", \
           "--trust-remote-code", \
           # below runs the model on 4 GPUs
           "--tensor-parallel-size","4", \
           # Maximum number of tokens, can lead to OOM if overestimated
           "--max-model-len", "8192", \
           # Port on which to run the vLLM server
           "--port", "80", \
           # CPU offload in GB. Need this as 8 H100s are not sufficient
           "--cpu-offload-gb", "80", \
           "--gpu-memory-utilization", "0.95", \
           # API key for authentication to the server stored in Tensorfuse secrets
           "--api-key", "${VLLM_API_KEY}"]

You can use the following commands to build and run the above Dockerfile.

docker build -t qwen-qwq-32b .

followed by

docker run --gpus all --shm-size=2g -p 80:80 -e VLLM_API_KEY=YOUR_API_KEY qwen-qwq-32b

Originally posted here: -

1 Like