Instructions to use QuantFactory/sqlcoder-7b-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use QuantFactory/sqlcoder-7b-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="QuantFactory/sqlcoder-7b-GGUF",
	filename="sqlcoder-7b.Q2_K.gguf",
)

output = llm(
	"Once upon a time,",
	max_tokens=512,
	echo=True
)
print(output)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use QuantFactory/sqlcoder-7b-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf QuantFactory/sqlcoder-7b-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf QuantFactory/sqlcoder-7b-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf QuantFactory/sqlcoder-7b-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf QuantFactory/sqlcoder-7b-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf QuantFactory/sqlcoder-7b-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf QuantFactory/sqlcoder-7b-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf QuantFactory/sqlcoder-7b-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf QuantFactory/sqlcoder-7b-GGUF:Q4_K_M

Use Docker

docker model run hf.co/QuantFactory/sqlcoder-7b-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use QuantFactory/sqlcoder-7b-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "QuantFactory/sqlcoder-7b-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantFactory/sqlcoder-7b-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/QuantFactory/sqlcoder-7b-GGUF:Q4_K_M

Ollama
How to use QuantFactory/sqlcoder-7b-GGUF with Ollama:
```
ollama run hf.co/QuantFactory/sqlcoder-7b-GGUF:Q4_K_M
```

Unsloth Studio new

How to use QuantFactory/sqlcoder-7b-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for QuantFactory/sqlcoder-7b-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for QuantFactory/sqlcoder-7b-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for QuantFactory/sqlcoder-7b-GGUF to start chatting

Docker Model Runner
How to use QuantFactory/sqlcoder-7b-GGUF with Docker Model Runner:
```
docker model run hf.co/QuantFactory/sqlcoder-7b-GGUF:Q4_K_M
```

Lemonade

How to use QuantFactory/sqlcoder-7b-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull QuantFactory/sqlcoder-7b-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.sqlcoder-7b-GGUF-Q4_K_M

List all available models

lemonade list

sqlcoder-7b-GGUF / README.md

aashish1904

Upload README.md with huggingface_hub

5b3538b verified almost 2 years ago

preview code

raw

history blame contribute delete

4.94 kB


	---

	license: cc-by-sa-4.0
	language:
	- en
	pipeline_tag: text-generation
	tags:
	- code

	---

	![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXeiuCm7c8lEwEJuRey9kiVZsRn2W-b4pWlu3-X534V3YmVuVc2ZL-NXg2RkzSOOS2JXGHutDuyyNAUtdJI65jGTo8jT9Y99tMi4H4MqL44Uc5QKG77B0d6-JfIkZHFaUA71-RtjyYZWVIhqsNZcx8-OMaA?key=xt3VSDoCbmTY7o-cwwOFwQ)

	# QuantFactory/sqlcoder-7b-GGUF
	This is quantized version of [defog/sqlcoder-7b](https://huggingface.co/defog/sqlcoder-7b) created using llama.cpp

	# Original Model Card

	# IMPORTANT
	This model is now outdated. Please use [`defog/sqlcoder-7b-2`](https://huggingface.co/defog/sqlcoder-7b-2) for much better performance!

	# Defog SQLCoder
	Defog's SQLCoder is a state-of-the-art LLM for converting natural language questions to SQL queries.

	[Interactive Demo](https://defog.ai/sqlcoder-demo/) \| [🤗 HF Repo](https://huggingface.co/defog/sqlcoder2) \| [♾️ Colab](https://colab.research.google.com/drive/1z4rmOEiFkxkMiecAWeTUlPl0OmKgfEu7?usp=sharing) \| [🐦 Twitter](https://twitter.com/defogdata)

	## TL;DR
	SQLCoder-7B is a 7B parameter model that outperforms `gpt-3.5-turbo` for natural language to SQL generation tasks on our [sql-eval](https://github.com/defog-ai/sql-eval) framework, and significantly outperforms all popular open-source models. When fine-tuned on a given schema, it also outperforms `gpt-4`

	SQLCoder-7B is fine-tuned on a base Mistral-7B model.

	## Results on novel datasets not seen in training
	\| model \| perc_correct \|
	\|-\|-\|
	\| gpt4-2023-10-04 \| 82.0 \|
	\| defog-sqlcoder2 \| 74.5 \|
	\| gpt4-2023-08-28 \| 74.0 \|
	\| defog-sqlcoder-7b \| 71.0 \|
	\| gpt-3.5-2023-10-04 \| 66.0 \|
	\| claude-2 \| 64.5 \|
	\| gpt-3.5-2023-08-28 \| 61.0 \|
	\| claude_instant_1 \| 61.0 \|
	\| text-davinci-003 \| 52.5 \|

	## License
	The code in this repo (what little there is of it) is Apache-2 licensed. The model weights have a `CC BY-SA 4.0` license. The TL;DR is that you can use and modify the model for any purpose – including commercial use. However, if you modify the weights (for example, by fine-tuning), you must open-source your modified weights under the same license terms.

	## Training
	SQLCoder was trained on more than 20,000 human-curated questions. These questions were based on 10 different schemas. None of the schemas in the training data were included in our evaluation framework.

	You can read more about our [training approach](https://defog.ai/blog/open-sourcing-sqlcoder2-7b/) and [evaluation framework](https://defog.ai/blog/open-sourcing-sqleval/).

	## Results by question category
	We classified each generated question into one of 5 categories. The table displays the percentage of questions answered correctly by each model, broken down by category.
	\| query_category \| gpt-4 \| sqlcoder2-15b \| sqlcoder-7b \| gpt-3.5 \| claude-2 \| claude-instant \| gpt-3 \|
	\|:-----------------\|--------:\|----------------:\|--------------:\|----------:\|-----------:\|-----------------:\|--------:\|
	\| date \| 72 \| 76 \| 64 \| 68 \| 52 \| 48 \| 32 \|
	\| group_by \| 91.4 \| 80 \| 82.9 \| 77.1 \| 71.4 \| 71.4 \| 71.4 \|
	\| order_by \| 82.9 \| 77.1 \| 74.3 \| 68.6 \| 74.3 \| 74.3 \| 68.6 \|
	\| ratio \| 80 \| 60 \| 54.3 \| 37.1 \| 57.1 \| 45.7 \| 25.7 \|
	\| join \| 82.9 \| 77.1 \| 74.3 \| 71.4 \| 65.7 \| 62.9 \| 57.1 \|
	\| where \| 80 \| 77.1 \| 74.3 \| 74.3 \| 62.9 \| 60 \| 54.3 \|

	## Using SQLCoder
	You can use SQLCoder via the `transformers` library by downloading our model weights from the Hugging Face repo. We have added sample code for [inference](./inference.py) on a [sample database schema](./metadata.sql).
	```bash
	python inference.py -q "Question about the sample database goes here"

	# Sample question:
	# Do we get more revenue from customers in New York compared to customers in San Francisco? Give me the total revenue for each city, and the difference between the two.
	```

	You can also use a demo on our website [here](https://defog.ai/sqlcoder-demo), or run SQLCoder in Colab [here](https://colab.research.google.com/drive/13BIKsqHnPOBcQ-ba2p77L5saiepTIwu0#scrollTo=ZpbVgVHMkJvC)

	## Hardware Requirements
	SQLCoder has been tested on an A100 40GB GPU with `bfloat16` weights. You can also load an 8-bit and 4-bit quantized version of the model on consumer GPUs with 20GB or more of memory – like RTX 4090, RTX 3090, and Apple M2 Pro, M2 Max, or M2 Ultra Chips with 20GB or more of memory.

	## Todo

	- [x] Open-source the v1 model weights
	- [x] Train the model on more data, with higher data variance
	- [ ] Tune the model further with Reward Modelling and RLHF
	- [ ] Pretrain a model from scratch that specializes in SQL analysis