Instructions to use deepseek-ai/DeepSeek-R1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use deepseek-ai/DeepSeek-R1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="deepseek-ai/DeepSeek-R1", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("deepseek-ai/DeepSeek-R1", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use deepseek-ai/DeepSeek-R1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "deepseek-ai/DeepSeek-R1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-R1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/deepseek-ai/DeepSeek-R1

SGLang

How to use deepseek-ai/DeepSeek-R1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "deepseek-ai/DeepSeek-R1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-R1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "deepseek-ai/DeepSeek-R1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-R1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use deepseek-ai/DeepSeek-R1 with Docker Model Runner:
```
docker model run hf.co/deepseek-ai/DeepSeek-R1
```

ImportError: cannot import name 'is_torch_greater_or_equal_than_1_13' from 'transformers.pytorch_utils'

#67

by bashir-abubakar - opened Jan 27, 2025

Discussion

bashir-abubakar

Jan 27, 2025

ImportError: cannot import name 'is_torch_greater_or_equal_than_1_13' from 'transformers.pytorch_utils' (C:\Users\ASUS\AppData\Local\Programs\Python\Python312\Lib\site-packages\transformers\pytorch_utils.py)

sgurwinderr

Jan 28, 2025

Have you tried changing that to is_torch_greater_or_equal_than_2_1?

GerwinVanGiessen

Jan 28, 2025

This comment has been hidden (marked as Resolved)

adi441994

Jan 28, 2025

Having the same issue. Any direction on this will be great. Thanks .

shreeshailvitkar

Jan 28, 2025

This PR will help https://github.com/huggingface/transformers/pull/35734

mrnithisht

Jan 29, 2025

•

edited Jan 29, 2025

@GerwinVanGiessen There will me a models/{models_name}/modeling_deepseek.py

circuitsage007

Jan 29, 2025

hey @shreeshailvitkar

I am facing the similar issue with transformer - 4.48.1 and pytorch - 2.5.1 . I did pull latest change from GIT code with changes - https://github.com/huggingface/transformers/pull/35734 . your input much appreciated. Thanks

Error:
ImportError: cannot import name 'is_torch_greater_or_equal_than_1_13' from 'transformers.pytorch_utils' (C:\ProgramData\anaconda3\Lib\site-packages\transformers\pytorch_utils.py). Did you mean: 'is_torch_greater_or_equal_than_2_1'?

grewe

Jan 30, 2025

I am getting the same error "ImportError: cannot import name 'is_torch_greater_or_equal_than_1_13' from 'transformers.pytorch_utils' (/usr/local/lib/python3.11/dist-packages/transformers/pytorch_utils.py)" and I have the following versions:

Transformers version: 4.48.1
Torch version: 2.6.0+cu118

mmalek06

Jan 30, 2025

Same here

Transformers version: 4.48.1
Torch version: 2.6.0+cu124

ManasNair1209

Jan 30, 2025

Same here

dcgod

Jan 30, 2025

•

edited Jan 30, 2025

Just go in and edit it yourself then it'll find a match.
(eg)

.... // pytorch_utils.py
is_torch_greater_or_equal_than_2_1 = parsed_torch_version_base >= version.parse("2.1")
is_torch_greater_or_equal_than_1_13 = parsed_torch_version_base >= version.parse("1.13")

bingqsong

Feb 28, 2025

Transformers version 4.49.0 may help to address the issue.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment