Ollama

1 / Local Model Serving

Ollama was the local LLM inference server that made running open-weight models practical during development. Instead of routing every prompt through cloud APIs — with their per-token costs, rate limits, and network latency — Ollama served quantized models from local hardware with an API-compatible interface.

2 / Where I Used It

In Conclave, Ollama instances served as debate participants running models like Llama 3 and Mistral locally. The multi-model debate format required high token throughput, and running locally eliminated per-token billing that would have made iterative development prohibitively expensive. Spring AI integrated with Ollama's OpenAI-compatible API, allowing the application to treat local and cloud models interchangeably.

3 / Trade-offs

Local inference trades API costs for hardware dependency. Quantized models running on consumer GPUs produce lower-quality output than full-precision cloud-hosted models. The throughput is also hardware-limited — a local Ollama instance cannot match the parallelism of a cloud inference cluster. For development and privacy-sensitive workloads, these trade-offs are acceptable. For production systems requiring consistent quality, cloud APIs with fallback routing are more reliable.

4 / Models I Have Worked With

Llama 3, Mistral, and Gemma through Ollama for conversational agent tasks. all-MiniLM-L6-v2 through SentenceTransformers for Vector Embeddings. gemini-1.5-flash through the Google Gemini API for multimodal tasks. The model choice has always been driven by the task constraints — size limits for local inference, quality requirements for production, and cost budgets for iterative development.