What's the best way to run AWQ models?
#1
by Winnougan - opened
Do you have any github repo that you use to run your models? Thanks
Thank you for your interest in the model.
Please install vllm from source, and the model should work!
git clone https://github.com/vllm-project/vllm.git
cd vllm
VLLM_USE_PRECOMPILED=1 pip install -e .
And then
vllm serve cyankiwi/Qwen3.5-9B-AWQ-BF16-INT8