Instructions to use amd/Qwen2.5-VL-7B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use amd/Qwen2.5-VL-7B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="amd/Qwen2.5-VL-7B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("amd/Qwen2.5-VL-7B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0")
model = AutoModelForMultimodalLM.from_pretrained("amd/Qwen2.5-VL-7B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use amd/Qwen2.5-VL-7B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "amd/Qwen2.5-VL-7B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/Qwen2.5-VL-7B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/amd/Qwen2.5-VL-7B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0

SGLang

How to use amd/Qwen2.5-VL-7B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "amd/Qwen2.5-VL-7B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/Qwen2.5-VL-7B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "amd/Qwen2.5-VL-7B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/Qwen2.5-VL-7B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use amd/Qwen2.5-VL-7B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0 with Docker Model Runner:
```
docker model run hf.co/amd/Qwen2.5-VL-7B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0
```

Qwen2.5-VL-7B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0

Model Overview

Model Architecture: Qwen2_5_VLForConditionalGeneration
- Input: Text
- Output: Text
Source Model: Qwen2.5-VL-7B-Instruct
Supported Hardware: AMD EPYC (CPU inference)
Preferred Operating System: Linux
Inference Engine: vLLM v0.20.2
Quantization Framework: TorchAO v0.17.0
Quantization Method: 4-bit Weight-Only (W4A16), Symmetric Per-Group
Compatible Stack:
- ZenDNN v6.0.0
- ZenTorch v2.11.0.1
- PyTorch v2.11.0
- TorchAO v0.17.0
- vLLM v0.20.2

This is a quantized version of Qwen2.5-VL-7B-Instruct created by AMD using TorchAO for ZenDNN-optimized CPU inference.

Quantization

The model was quantized from Qwen2.5-VL-7B-Instruct using TorchAO v0.17.0.

Method: 4-bit Weight-Only (W4A16), Symmetric Per-Group
Config: Int4WeightOnlyConfig(group_size=128, mapping_type=MappingType.SYMMETRIC)
Quantized Layers: All linear layers excluding lm_head and embed_tokens

Quantization script:

python woq_sym_group.py \
    --model_name Qwen/Qwen2.5-VL-7B-Instruct \
    --output_dir ./Qwen2.5-VL-7B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0

Quick Start

Use with vLLM

from vllm import LLM, SamplingParams

model = LLM(
    model="amd/Qwen2.5-VL-7B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0",
    dtype="bfloat16",
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = model.generate(["Hello, how are you?"], sampling_params)
print(outputs[0].outputs[0].text)

Requirements

torch==2.11.0
torchao==0.17.0
zentorch==2.11.0.1
vllm==0.20.2

OpenMP Setup

For optimal performance, set LD_PRELOAD with libomp.so (LLVM OpenMP) or libiomp5.so (Intel OpenMP):

# Using LLVM OpenMP (llvmopenmp)
export LD_PRELOAD=$(find /path/to/env -name "libomp.so" | head -1)

# Or using Intel OpenMP (libiomp)
export LD_PRELOAD=$(find /path/to/env -name "libiomp5.so" | head -1)

Note: Set LD_PRELOAD before launching vLLM or any inference script.

Evaluation

The model was evaluated against the BF16 (unquantized) baseline on standard benchmarks using lm-evaluation-harness with the vLLM engine.

Benchmark	BF16 Baseline	W4A16-TAO-SymGroup (this model)	Recovery
MMLU (5-shot)	-	-	-
GSM8K_COT (8-shot)	-	-	-
Perplexity (wikitext2)	-	-	-

Evaluation results will be updated after benchmarking.

Reproduction

lm_eval \
    --model vllm \
    --model_args pretrained="amd/Qwen2.5-VL-7B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0" \
    --tasks mmlu \
    --num_fewshot 5 \
    --batch_size auto

Limitations

Version Lock: This model is quantized with TorchAO v0.17.0 and is compatible only with PyTorch v2.11.0 / ZenDNN v6.0.0. It will not load correctly on other PyTorch versions.
CPU Only: This model is optimized for AMD EPYC CPU inference via ZenDNN. It is not intended for GPU inference.

License

This model is distributed under the same license as the source model. See the LICENSE file for details.

Downloads last month: 29

Model tree for amd/Qwen2.5-VL-7B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0

Base model

Qwen/Qwen2.5-VL-7B-Instruct

Quantized

(143)

this model

Collection including amd/Qwen2.5-VL-7B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0

zentorch TorchAO Quantized Models - PyTorch 2.11

Collection

TorchAO v0.17.0 quantized models for AMD EPYC CPU inference. • 26 items • Updated 6 days ago