# How do you fine-tune a model and then run it in Ollama?

> Ollama cannot train. Fine-tune on a GPU, merge the LoRA, convert to GGUF with llama.cpp, then ollama create with a Modelfile that matches the chat template.

Source: https://www.nodus-compute.ai/blog/fine-tune-model-for-ollama/
Build revision: efc370c9eafcb9c2aca8a9150365e219dcbf82aa

Ollama only runs models, so you fine-tune somewhere with a GPU, merge the LoRA weights into the base model, convert the result to a GGUF file with llama.cpp, and import that file with a Modelfile and `ollama create`. Nodus runs the GPU half as one capped job: a QLoRA fine-tune of an 8B model fits an NVIDIA L4 listed at $0.43/hr, and the job hands back the GGUF for `ollama run` on your own machine. The merge is no longer optional, because current Ollama releases reject LoRA adapters outright.

Most guides that rank for this question were written before that change. Everything below was checked against the current Ollama, llama.cpp and Unsloth docs on November 22, 2026, and the import steps were run by us on CPU with Ollama 0.40.17 and a 135M-parameter model.

## Why can’t Ollama fine-tune a model itself?

Ollama is an inference server. Its `ollama create` command packages weights you already have into a model it can serve, and that is all. Training needs gradients, an optimizer and a GPU with room for both, which is a different program.

The packaging side also got narrower in 2026. The [Ollama v0.34.1 release notes](https://github.com/ollama/ollama/releases/tag/v0.34.1) say GGUF model creation “now requires using llama.cpp tooling for safetensor conversion and quantization”. The same change removed `ADAPTER` from the [Modelfile reference](https://docs.ollama.com/modelfile), and Ollama’s [server code](https://github.com/ollama/ollama/blob/main/server/create.go) now returns “LoRA adapters are no longer supported”.

|Step in older guides|What current Ollama does|
|-|-|
|`ADAPTER ./lora.gguf` on top of a base model|Fails with “LoRA adapters are no longer supported”|
|Point `FROM` at a safetensors folder and let Ollama convert it to GGUF|Safetensors imports now go through Ollama’s MLX path; GGUF conversion is left to llama.cpp|
|Let Ollama quantize during import|The [import docs](https://docs.ollama.com/import) say Ollama does not quantize GGUF models; use `llama-quantize` first|

We hit the first row ourselves. A Modelfile with `FROM ./merged-Q8_0.gguf` imported fine. Adding `ADAPTER ./lora.gguf`, a valid adapter produced by llama.cpp’s `convert_lora_to_gguf.py`, made `ollama create` stop with `Error: LoRA adapters are no longer supported`.

## What does the full pipeline look like?

Five steps, and only the first needs a GPU:

1. Fine-tune with LoRA or QLoRA using Unsloth or Hugging Face TRL.
2. Merge the adapter into the base weights and save them in 16-bit.
3. Convert the merged model to GGUF and quantize it.
4. Write a Modelfile whose template matches the format you trained on.
5. Run `ollama create`, then `ollama run`.

Steps 2 and 3 can happen at the end of the GPU job or later on a CPU. Our llama.cpp conversion and quantization of the small test model both ran on CPU only.

## How do you run the fine-tune on a cloud GPU?

For an 8B model, Unsloth’s [requirements table](https://unsloth.ai/docs/get-started/fine-tuning-for-beginners/unsloth-requirements) lists a minimum of 6 GB of VRAM for 4-bit QLoRA and 22 GB for 16-bit LoRA. QLoRA therefore fits any 24 GB card. Our [LoRA vs QLoRA guide](https://www.nodus-compute.ai/blog/lora-vs-qlora/) covers when the 16-bit version is worth the bigger GPU, and our [Unsloth on a cloud GPU walkthrough](https://www.nodus-compute.ai/blog/fine-tune-with-unsloth-on-cloud-gpu/) has a full training script with checkpoint resume.

Keep the training script as it is and add the export at the end. These two calls are copied from Unsloth’s [saving to GGUF docs](https://unsloth.ai/docs/basics/inference-and-deployment/saving-to-gguf), with the output folders pointed at `/nodus/outputs`, the directory Nodus collects after a successful job:

```python
model.save_pretrained_merged("/nodus/outputs/merged_model", tokenizer, save_method = "merged_16bit",)
model.save_pretrained_gguf("/nodus/outputs/gguf", tokenizer, quantization_method = "q4_k_m")
```

Then launch it from the folder holding `train.py`. The flags follow the [Nodus quickstart](https://www.nodus-compute.ai/docs/getting-started/quickstart/); the cost cap and timeout are example choices:

Terminal window

```sh
nodus run --name llama8b-ollama --gpu L4 --image nodus/pytorch \
  --max-cost 5 --timeout 6h --no-warm -- python train.py
```

Every file under `/nodus/outputs` is collected and checksummed when the job succeeds, per the [outputs guide](https://www.nodus-compute.ai/docs/guides/outputs/). The exact file names appear in the job’s `status.outputs`, and `nodus cp job/llama8b-ollama:outputs/<path> ./model.gguf` downloads one after verifying its SHA-256.

## How do you convert the merged model to GGUF yourself?

Use llama.cpp directly when you want control over the quant, or when Unsloth’s export fails. Our runs used llama.cpp at commit `8d60873`. The two commands follow the [llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/tree/master/tools/quantize):

Terminal window

```sh
python llama.cpp/convert_hf_to_gguf.py merged_model --outfile model-f16.gguf --outtype f16
./llama.cpp/build/bin/llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M
```

Convert to F16 once and keep that file. Each extra quant is then one fast `llama-quantize` call instead of a fresh conversion.

## Which GGUF quant should you pick for 8, 16 or 24 GB?

Pick the largest quant whose file, plus the KV cache for your context length, fits in memory with room to spare. The sizes below are the real file sizes in the public [bartowski/Meta-Llama-3.1-8B-Instruct-GGUF](https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/tree/main) repository. Bits per weight is file bytes x 8 / 8.03B parameters, where 8.03B comes from the repo’s F32 file (32.13 GB / 4 bytes).

|Quant|File size|Bits per weight|File + 8K-token KV cache|
|-|-|-|-|
|F16|16.06 GB (computed: 8.03B x 2 bytes)|16|17.13 GB|
|Q8_0|8.54 GB|8.51|9.61 GB|
|Q6_K|6.60 GB|6.57|7.67 GB|
|Q5_K_M|5.73 GB|5.71|6.80 GB|
|Q4_K_M|4.92 GB|4.90|5.99 GB|
|Q3_K_M|4.02 GB|4.00|5.09 GB|
|Q2_K|3.18 GB|3.17|4.25 GB|

The KV cache column uses the model’s [config.json](https://huggingface.co/unsloth/Meta-Llama-3.1-8B-Instruct/blob/main/config.json): 2 (keys and values) x 32 layers x 8 KV heads x 128 dimensions x 2 bytes = 131,072 bytes per token. At 8,192 tokens that is 1.07 GB.

![GGUF file sizes for Llama 3.1 8B Instruct: F16 16.06 GB, Q8_0 8.54 GB, Q6_K 6.60 GB, Q5_K_M 5.73 GB, Q4_K_M 4.92 GB, Q3_K_M 4.02 GB and Q2_K 3.18 GB.](https://www.nodus-compute.ai/blog-charts/fine-tune-model-for-ollama/gguf-sizes.svg) *Q4_K_M is under a third of the F16 file, which is why it is the usual default for a fine-tuned 8B.*

Our rule for an 8B fine-tune: on 8 GB, use Q4_K_M or Q5_K_M and keep Q6_K’s 7.67 GB for short contexts only. On 16 GB, Q8_0 at 9.61 GB is the easy choice. On 24 GB, the unquantized F16 fits at 17.13 GB, which is useful for checking whether a quality drop came from your training or from quantization.

Real use adds runtime buffers on top, so treat these as floors. Our [8, 16 and 24 GB VRAM guide](https://www.nodus-compute.ai/blog/which-llms-fit-in-8gb-16gb-24gb-vram/) works through longer contexts.

## How do you write the Modelfile?

Start from the GGUF and spell out the chat template and stop tokens you trained with. For a model trained in ChatML format, this is the template from the [Modelfile reference](https://docs.ollama.com/modelfile) plus one stop parameter:

```plaintext
FROM ./model-Q4_K_M.gguf
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
"""
PARAMETER stop "<|im_end|>"
PARAMETER temperature 0.2
```

For a Llama 3 format fine-tune, copy the `TEMPLATE` and the four `PARAMETER stop` lines from the `ollama show --modelfile` example in the same reference. Then build and run it:

Terminal window

```sh
ollama create my-model -f ./Modelfile
ollama run my-model
```

## Why does a fine-tuned model ramble or never stop in Ollama?

Almost always because the prompt format at run time differs from the one used in training, or the end-of-turn token is not a stop sequence. Unsloth’s [GGUF troubleshooting section](https://unsloth.ai/docs/basics/inference-and-deployment/saving-to-gguf) names an incorrect chat template as “the most common cause” of gibberish, endless generations or repeated outputs, followed by a wrong EOS token.

Our import test shows how this happens silently. When the GGUF carried SmolLM2’s standard ChatML template, Ollama 0.40.17 recognized it and filled in a ChatML `TEMPLATE` with `<|im_start|>` and `<|im_end|>` as stop sequences. When we swapped in a custom Alpaca-style template, `ollama show --modelfile` printed `TEMPLATE {{ .Prompt }}` and no stop parameters at all. Ollama’s [template docs](https://docs.ollama.com/template) describe `{{ .Prompt }}` as the default, which sends your text to the model verbatim.

So the check takes ten seconds: run `ollama show --modelfile my-model` after every import and compare the template and stop lines with your training data. If they differ, write them into the Modelfile yourself and run `ollama create` again.

## What does the GPU time cost?

Cost is the listed hourly price times the hours your job takes. We have not timed an 8B fine-tune for this post, so the table uses an assumed 3 GPU-hours to show the math. Replace 3 with your own measured hours.

|GPU on Nodus|Listed price|x 3 hours|
|-|-|-|
|RTX 4090 (24 GB)|$0.29/hr|$0.87|
|L4 (24 GB)|$0.43/hr|$1.29|
|L40S (48 GB)|$0.793/hr|$2.38|
|H100 SXM (80 GB)|$2.60/hr|$7.80|

Prices checked November 22, 2026 on the [Nodus pricing page](https://www.nodus-compute.ai/pricing/).

![Three GPU-hours at Nodus list prices cost $0.87 on an RTX 4090, $1.29 on an L4, $2.38 on an L40S and $7.80 on an H100 SXM.](https://www.nodus-compute.ai/blog-charts/fine-tune-model-for-ollama/training-cost.svg) *An H100 only wins on cost if it finishes the same QLoRA job about 9x faster than an RTX 4090.*

That 9x is $2.60 / $0.29 = 8.97. For a QLoRA 8B job, a 24 GB card is the sensible first try. The merge and GGUF conversion are CPU work, so if your job spends long on them, consider running them on your own machine instead of the GPU clock.

## FAQ

### Can I run the LoRA adapter in Ollama without merging?

Not in current releases. Since v0.34.1 Ollama rejects LoRA adapters at `ollama create`, so merge with Unsloth’s `save_pretrained_merged` or PEFT’s `merge_and_unload` before converting.

### Should I merge into the 4-bit base I trained QLoRA on?

No. Merge into the 16-bit base weights, which is what Unsloth’s `merged_16bit` save method does, and quantize afterwards with llama.cpp. That gives you one clean F16 GGUF to make every quant from.

### Can I serve the fine-tuned model to many users instead of just my laptop?

Ollama is built for one machine and a few users. For shared traffic, the merged 16-bit folder works in vLLM directly; our [vLLM vs Ollama comparison](https://www.nodus-compute.ai/blog/vllm-vs-ollama/) covers when to switch and what each concurrent user costs.

### Does Ollama still accept safetensors directly?

Yes, through its MLX import path, which the v0.34.1 notes moved out of experimental status. If you want a GGUF that runs the same on Linux, Windows and macOS, convert with llama.cpp yourself.
