Llama.cpp can perform mixed inference in CPU & GPU environments. Here is a record of the process of running Qwen-1.8B on the Windows 10 platform with a 1660Ti graphics card.
Prepare the Model#
- Install conda
- Tun Mode (Administrator privileges)
conda create -n llamaConvert python=3.10 git -c conda-forge
conda activate llamaConvert
cd D:\llama
git clone --depth=1 https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
python -m pip install -r requirements.txt
pip install tiktoken
python -c "from huggingface_hub import snapshot_download; snapshot_download(repo_id='Qwen/Qwen-1_8B-Chat', local_dir='D:\qwen', ignore_patterns=['*.h5', '*.ot', '*.msgpack', '*.safetensors'])"
cd D:\qwen
D:\aria2\aria2c.exe --all-proxy='http://127.0.0.1:7890' -o 'model-00001-of-00002.safetensors' "https://huggingface.co/Qwen/Qwen-1_8B-Chat/resolve/main/model-00001-of-00002.safetensors?download=true"
D:\aria2\aria2c.exe --all-proxy='http://127.0.0.1:7890' -o 'model-00002-of-00002.safetensors' "https://huggingface.co/Qwen/Qwen-1_8B-Chat/resolve/main/model-00002-of-00002.safetensors?download=true"
cd D:\llama\llama.cpp
python convert-hf-to-gguf.py D:\qwen
# Model successfully exported to 'D:\qwen\ggml-model-f16.gguf'
Run the Model#
- Download llama-b1732-bin-win-cublas-cu12.2.0-x64.zip
- Extract the files to
D:\llama
conda create -n llamaCpp libcublas cuda-toolkit git -c nvidia -c conda-forge
conda activate llamaCpp
cd D:\llama | .\main.exe ## Check if it runs correctly
cd D:\llama | .\quantize.exe --help ## Decide on the quantization method yourself
.\quantize.exe D:\qwen\ggml-model-f16.gguf .\qwen-1_8-f16.gguf COPY
.\server.exe -m .\qwen-1_8-f16.gguf -c 4096 --n-gpu-layers 50 ## Adjust n-gpu-layers to balance CPU & GPU
- Access
http://127.0.0.1:8080and selectCompletionfor testing.