Run your own model

Everything so far ran in your browser. This lesson runs a real language model on your machine, using the open tilt platform. It’s the one lesson on this track that needs real hardware, so check what your device can run first.

You’ll switch on three tools the platform already knows how to deploy:

Tool	What it is	Switch in tilt-config.json
Ollama	serves open models over a simple HTTP API	flux_apps.ollama
Qdrant	an open-source vector search engine, written in Rust	crossplane_apps.qdrant
Langfuse	open-source tracing and evaluation for LLM apps	crossplane_apps.langfuse

Be honest about the hardware

The platform runs Ollama on the CPU by default (gpu.enabled: false in its Helm values), asking for 4 GiB of memory with a limit of 8 GiB. CPU inference works, but it’s slow, and model size decides everything:

• tinyllama, the model the platform pulls by default, has 1.1B parameters and a 638 MB download. It’s fast enough to play with and not very clever.
• llama3.2:1b (1.3 GB) and llama3.2:3b (2.0 GB) are more capable and noticeably slower on a CPU.
• 7B-parameter models and up want far more memory than this setup gives them, and on a laptop CPU they’re painfully slow. That’s why serving at scale happens on GPUs, and why vLLM’s memory tricks matter.

Step 1: a model of your own

The switches in tilt-config.json only take effect in the platform’s full profile. That profile also brings up Crossplane, which deploys Qdrant and Langfuse. So bring it up with:

PROFILE=full ./scripts/platform.sh up

Then set this in tilt-config.json:

{ "flux_apps": { "ollama": { "enabled": true } } }

Tilt notices and deploys it. The first start downloads tinyllama, so give it a few minutes. Then:

curl -s https://ollama.localhost/api/generate \
  -d '{"model":"tinyllama","prompt":"Why do pandas eat bamboo?","stream":false}'

The answer is in the response field. Run it twice. It differs, because the model is sampling, exactly like the temperature knob in One token at a time.

Step 2: retrieval, so it can answer from your documents

A model only knows what was in its training data. Retrieval-augmented generation (RAG) gives it a library. The 2020 paper that named it combined a model’s built-in knowledge with a searchable “dense vector index” of documents (Wikipedia, in their case). The recipe:

1. Turn each document into an embedding: a long list of numbers where similar meanings land close together.
2. Store the embeddings in a vector database. Here that’s Qdrant.
3. When a question arrives, embed it too, find the closest documents, and paste them into the prompt.
4. The model answers with the evidence in front of it.

Turn Qdrant on (crossplane_apps.qdrant), pull a small embedding model into Ollama, and embed a sentence. (The last line uses jq to count the numbers; without it, you’ll see the raw list.)

kubectl -n ollama exec deploy/ollama -- ollama pull nomic-embed-text
curl -s https://ollama.localhost/api/embed \
  -d '{"model":"nomic-embed-text","input":"Bamboo can grow very fast."}' \
  | jq '.embeddings[0] | length'

That number is the embedding’s size. Use it to create a Qdrant collection (Qdrant is at https://qdrant.localhost):

curl -s -X PUT https://qdrant.localhost/collections/pandas \
  -H 'Content-Type: application/json' \
  -d '{"vectors":{"size":<that number>,"distance":"Cosine"}}'

From here, storing documents and searching them are two more calls. A guided, graded version is planned as a tilt lab. Until then, the Qdrant API reference covers the rest.

Step 3: watch what it does

Once an app chains a retrieval, a prompt and a model call, “why did it say that?” gets hard to answer. Langfuse records each step as a trace, so you can see which documents were retrieved and what prompt the model actually received. Turn on crossplane_apps.langfuse and open https://langfuse.localhost.

Do it for real · your machine+50 XP bonus

Everything above ran in your browser. Now do it on a real cluster withthe tilt platform. You need Docker, kind and Tilt, about 15 GB of free disk and 8 GB of spare RAM. Setup guide · Can my machine run it?

You passed when api/generate returns an answer from tinyllama and the api/embed call prints an embedding length. The platform’s student setup guide covers bringing the platform up.

I passed the check on my machineSelf-reported. We can't see your machine, and we don't want to. This is a bonus, never required.