Running models on my own hardware
Home lab inference, fine-tuning experiments, llama.cpp optimization, and practical AI tooling. No cloud required — everything runs locally on home lab hardware.
Experiments & Setup
Notes on models, quantization, server configs, and tooling for local AI workflows.
llama.cpp Server
Running GGUF models via llama.cpp with CUDA offload. Config tuning for speed vs quality trade-offs.
AI-Assisted Development
VS Code integration with local models for code completion, refactoring, and explanation. Faster than waiting on cloud APIs.
Quantization Benchmarks
Testing different quant methods — Q4, Q5, Q6, Q8 — to find the sweet spot between quality and inference speed.
Mistral 7B
Primary coding and general-purpose model. Strong instruction following, fast inference, and great with local fine-tunes.
Fine-Tuning
Planning and testing LoRA fine-tunes on custom datasets. Domain-specific models for project notes, code review, and documentation.
Local AI Stack
llama.cpp, Ollama, text-generation-webui, and custom Python tooling for the local AI pipeline.
Setup Notes
Hardware and software details for running local AI on a home lab.
Hardware
Running on home lab hardware with dedicated GPU for CUDA inference. Proxmox host with GPU passthrough to the AI LXC container. Models stored on fast NVMe for quick loading.
Software Stack
Ubuntu LXC with llama.cpp, GGUF format models, and custom Python scripts for model management and inference. VS Code extensions for local AI-assisted coding workflows.