In the previous two articles, llama.cpp only ran as a CPU baseline. That conclusion was incomplete: CPU execution proves that the model format and program path work, but a fair comparison with Ollama requires a CUDA build.

This article records a complete deployment process. The environment is a Windows host plus WSL Ubuntu 26.04 LTS, with an RTX 4060 Laptop GPU and 8GB VRAM. The goal is clear: do not modify system directories, do not depend on a global CUDA Toolkit installation, build the llama.cpp CUDA backend in an isolated user-space environment, and benchmark the same Qwen3 4B GGUF file.

Final Result First

The final binaries are under:

~/opt/llama.cpp/build-cuda/bin/

Key programs include:

llama-cli
llama-server
llama-bench
libggml-cuda.so

llama-bench can detect the CUDA device:

ggml_cuda_init: found 1 CUDA devices (Total VRAM: 8187 MiB):
  Device 0: NVIDIA GeForce RTX 4060 Laptop GPU, compute capability 8.9, VMM: yes, VRAM: 8187 MiB

Benchmarking the same Qwen3 4B GGUF gives:

Backend	Prompt Processing	Generation
CPU llama.cpp	about 4.5 tok/s	about 12.4 tok/s
CUDA llama.cpp	about 1653.6 tok/s	about 78.7 tok/s

On this machine, the CUDA build is not merely “a bit faster.” It turns llama.cpp from a compatibility-check tool into a genuinely usable local inference backend.

Why Not Use the System CUDA Toolkit

The first error was direct:

Unable to find cudart library.
CUDA Toolkit not found

WSL could see the GPU, and Ollama could use it, so the driver path was working. But compiling the CUDA backend of llama.cpp requires the development toolchain: nvcc, libcudart, cuBLAS, CUDA headers, and a host compiler supported by CUDA.

Ubuntu 26.04 repositories provide nvidia-cuda-toolkit 12.4, but that route modifies the system environment and needs root privileges. To keep the experiment inside the current user directory, I used micromamba to create an isolated environment:

~/.local/envs/llama-cuda

This has several benefits:

• It does not pollute the default system compiler.
• It does not affect the already working Ollama setup.
• CUDA, GCC, and cuBLAS versions can be fixed.
• If I need to start over, deleting the directory is enough.

Install micromamba

Put micromamba in the current user’s directory:

mkdir -p ~/.local/bin ~/.local/share/micromamba ~/.cache/micromamba

curl -L https://micro.mamba.pm/api/micromamba/linux-64/latest \
  -o /tmp/micromamba.tar.bz2

tar -xjf /tmp/micromamba.tar.bz2 -C /tmp bin/micromamba
install -m 0755 /tmp/bin/micromamba ~/.local/bin/micromamba

~/.local/bin/micromamba --version

The installed version was:

2.8.1

Create the CUDA Build Environment

My first attempt installed CUDA, GCC, CMake, Ninja, and related dependencies in one shot. Dependency solving was fine, but download and transaction time were too long. I switched to a smaller set and installed only the essential CUDA components first:

MAMBA_ROOT_PREFIX=$HOME/.local/share/micromamba \
~/.local/bin/micromamba create -y \
  -p $HOME/.local/envs/llama-cuda \
  -c nvidia \
  cuda-nvcc=12.4 \
  cuda-cudart-dev=12.4 \
  libcublas-dev=12.4

Check nvcc:

~/.local/envs/llama-cuda/bin/nvcc --version

The output showed:

Cuda compilation tools, release 12.4, V12.4.131

At this point, nvcc, libcudart.so, and libcublas.so were already in the user directory. But it was not enough; the later build exposed several version issues.

First Issue: GCC 15 Is Too New

The system compiler is GCC 15:

g++ (Ubuntu 15.2.0-16ubuntu1) 15.2.0

CUDA 12.4 does not support GCC that new. Even with --allow-unsupported-compiler, it failed while parsing C++ standard-library headers. Typical errors included:

unsupported GNU version! gcc versions later than 13 are not supported

and then a chain of C++ header parsing errors.

The solution was to install GCC/G++ 13 inside the same micromamba environment and explicitly tell CMake to use it:

MAMBA_ROOT_PREFIX=$HOME/.local/share/micromamba \
~/.local/bin/micromamba install -y \
  -p $HOME/.local/envs/llama-cuda \
  -c conda-forge \
  gcc_linux-64=13 \
  gxx_linux-64=13

Check the version:

~/.local/envs/llama-cuda/bin/x86_64-conda-linux-gnu-g++ --version

Result:

x86_64-conda-linux-gnu-g++ (conda-forge gcc 13.4.0-19) 13.4.0

Second Issue: Missing Static CUDA Runtime

After switching to GCC 13, CUDA compiler detection progressed, but the link stage failed:

cannot find -lcudadevrt
cannot find -lcudart_static

That meant static runtime libraries were missing. Two small packages fixed it:

MAMBA_ROOT_PREFIX=$HOME/.local/share/micromamba \
~/.local/bin/micromamba install -y \
  -p $HOME/.local/envs/llama-cuda \
  -c nvidia \
  cuda-cudart-static=12.4 \
  cuda-driver-dev=12.4

This step was quick, with roughly 1MB of downloads.

Third Issue: Mixed CUDA Component Versions

Next, CUDA source files began compiling, but cuda_fp16.h reported:

fatal error: nv/target: No such file or directory

The problem was mixed CUDA component versions. cuda-nvcc was 12.4, but cuda-cccl had resolved to 12.9. Pinning CCCL to 12.4 added the missing nv/target headers:

MAMBA_ROOT_PREFIX=$HOME/.local/share/micromamba \
~/.local/bin/micromamba install -y \
  -p $HOME/.local/envs/llama-cuda \
  -c nvidia \
  cuda-cccl=12.4

Confirm:

find ~/.local/envs/llama-cuda/include -path '*nv/target*' -print

Output:

/home/alex/.local/envs/llama-cuda/include/nv/target

After this, the CUDA build of llama.cpp had the complete conditions it needed.

Configure llama.cpp

The configuration command is below. I deliberately used a clean PATH to avoid WSL mixing in too many host-side paths and slowing down CMake detection.

CUDA_ENV=$HOME/.local/envs/llama-cuda

rm -rf ~/opt/llama.cpp/build-cuda

PATH="$CUDA_ENV/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" \
LD_LIBRARY_PATH="$CUDA_ENV/lib:${LD_LIBRARY_PATH:-}" \
cmake -S ~/opt/llama.cpp -B ~/opt/llama.cpp/build-cuda \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON \
  -DCUDAToolkit_ROOT="$CUDA_ENV" \
  -DCMAKE_C_COMPILER="$CUDA_ENV/bin/x86_64-conda-linux-gnu-gcc" \
  -DCMAKE_CXX_COMPILER="$CUDA_ENV/bin/x86_64-conda-linux-gnu-g++" \
  -DCMAKE_CUDA_HOST_COMPILER="$CUDA_ENV/bin/x86_64-conda-linux-gnu-g++" \
  -DLLAMA_BUILD_SERVER=ON \
  -DLLAMA_BUILD_EXAMPLES=ON \
  -DLLAMA_BUILD_TESTS=OFF

Successful configuration showed:

-- The CUDA compiler identification is NVIDIA 12.4.131 with host compiler GNU 13.4.0
-- Using CMAKE_CUDA_ARCHITECTURES=89-real CMAKE_CUDA_ARCHITECTURES_NATIVE=89-real
-- CUDA host compiler is GNU 13.4.0
-- Including CUDA backend
-- Configuring done
-- Build files have been written to: /home/alex/opt/llama.cpp/build-cuda

89-real corresponds to the Ada architecture of the RTX 4060 Laptop GPU, which means CMake detected the correct target architecture.

Build the Key Targets

I built only the three targets I use most:

CUDA_ENV=$HOME/.local/envs/llama-cuda

PATH="$CUDA_ENV/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" \
LD_LIBRARY_PATH="$CUDA_ENV/lib:${LD_LIBRARY_PATH:-}" \
cmake --build ~/opt/llama.cpp/build-cuda \
  --config Release \
  -j 8 \
  --target llama-cli llama-server llama-bench

The slowest part was CUDA template instantiation, especially attention and quantized matrix-multiplication files inside ggml-cuda. Successful output included:

[ 41%] Linking CUDA shared library ../../../bin/libggml-cuda.so
[ 41%] Built target ggml-cuda
[100%] Built target llama-cli
[100%] Built target llama-server
[100%] Built target llama-bench

Final environment and build directory sizes:

1.7G  ~/.local/envs/llama-cuda
1.1G  ~/.local/share/micromamba/pkgs
1.1G  ~/opt/llama.cpp/build-cuda
824M  ~/opt/llama.cpp/build-cpu

If disk space is tight, the micromamba package cache can be cleaned after the environment is confirmed stable. I kept it for now so later packages or rebuilds are easier.

Add a Wrapper

CUDA binaries need to find libggml-cuda.so, libcudart.so, libcublas.so, and related libraries at runtime. To avoid writing environment variables by hand every time, I added a wrapper:

cat > ~/.local/bin/llama-cuda-run <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

CUDA_ENV="${CUDA_ENV:-$HOME/.local/envs/llama-cuda}"
LLAMA_CUDA_BIN="${LLAMA_CUDA_BIN:-$HOME/opt/llama.cpp/build-cuda/bin}"

export PATH="$CUDA_ENV/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
export LD_LIBRARY_PATH="$LLAMA_CUDA_BIN:$CUDA_ENV/lib:${LD_LIBRARY_PATH:-}"

exec "$@"
EOF

chmod +x ~/.local/bin/llama-cuda-run

Then it can be used like this:

llama-cuda-run ~/opt/llama.cpp/build-cuda/bin/llama-bench --list-devices

Output:

ggml_cuda_init: found 1 CUDA devices (Total VRAM: 8187 MiB):
  Device 0: NVIDIA GeForce RTX 4060 Laptop GPU, compute capability 8.9, VMM: yes, VRAM: 8187 MiB
Available devices:
  CUDA0: NVIDIA GeForce RTX 4060 Laptop GPU (8187 MiB, 7099 MiB free)

Run the Benchmark

The model is the Qwen3 4B GGUF blob already downloaded by Ollama:

MODEL=$HOME/.ollama/models/blobs/sha256-3e4cb14174460404e7a233e531675303b2fbf7749c02f91864fe311ab6344e4f

llama-cuda-run ~/opt/llama.cpp/build-cuda/bin/llama-bench \
  -m "$MODEL" \
  -ngl 999 \
  -p 32 \
  -n 32

Key output:

ggml_cuda_init: found 1 CUDA devices (Total VRAM: 8187 MiB):
  Device 0: NVIDIA GeForce RTX 4060 Laptop GPU, compute capability 8.9, VMM: yes, VRAM: 8187 MiB

| model                  | size     | params | backend | ngl | test | t/s              |
| ---------------------- | -------: | -----: | ------- | --: | ---: | ---------------: |
| qwen3 4B Q4_K - Medium | 2.32 GiB | 4.02 B | CUDA    | 999 | pp32 | 1653.55 ± 323.14 |
| qwen3 4B Q4_K - Medium | 2.32 GiB | 4.02 B | CUDA    | 999 | tg32 | 78.67 ± 0.43     |

The previous CPU result was:

pp32  about 4.5 tok/s
tg32  about 12.4 tok/s

The improvement is obvious. Prompt processing jumped from single digits to the thousand-token-per-second range, and generation went from about 12 tok/s to nearly 79 tok/s. For local interaction, the latter is already daily-usable speed.

How It Fits with Ollama

Now that CUDA llama.cpp runs, I would not immediately replace Ollama. They are better used for different roles.

Ollama is better for daily use: installing models, starting a service, and calling an API are simpler. It is especially convenient for multimodal models and template handling.

llama.cpp is better for benchmarking and fine control. With the same GGUF and the same parameters, it is easier to observe how GPU offload, context length, batch size, and sampling parameters affect speed and memory.

Next, I will run llama-server with an OpenAI-compatible API and compare end-to-end latency against Ollama using the same prompt. For now, at least one thing is confirmed: on this WSL development machine, the llama.cpp CUDA route is open.

WeChat

Follow ZiCode on WeChat

If this post was useful, you can follow later updates on WeChat as well.

X / Twitter

Follow @ax2_zicode

Faster technical notes, short thoughts, and new-post alerts are posted on X.

X Open X Profile @ax2_zicode