FR
OverviewHardwareDeploymentServicesData
← My Homelab

My Homelab - Services

AI Setup

As I shared in the hardware section, I have 4 different GPUs:

  • RTX 5090: The beast
  • RTX 5060 Ti 16GB: Good VRAM / Price ratio, low bandwidth but consumes little energy
  • RTX 3090: The beast, but two generations older
  • RTX 3060 Ti: Well, you need to connect monitors somewhere on the dev PC

All this represents a total of 80GB of VRAM. (32 + 16 + 24 + 8)

Which LLM Do I Run Locally?

The first question I asked myself was: "what size LLM can I run properly with these GPUs?" It's not an obvious question, the GPUs have different amounts of VRAM and are in different computers.

Doing Tensor Parallelism (TP) seems absolutely impractical. Splitting the weight matrices of a single model layer across multiple GPUs requires very low latency between GPUs, which is only feasible if they communicate directly via PCI-e. Now, the only two GPUs that are physically on the same motherboard are the 5090 and the 5060 Ti. It's theoretically doable, but the 5090's capabilities will be bottlenecked by the 5060 Ti's.

So the remaining option is Pipeline Parallelism (PP) with the 5090 and 3090 communicating over the network. The difference is that instead of running inference using both GPUs simultaneously, the network layers are split. The 5090 takes the first layers (the largest ones), and the 3090 takes the next ones. It works! With vLLM and Ray, I can run inference on 70B quantized models at a decent throughput.

Despite this success, I decided to drop multi-GPU inference for now, because the 5090 runs a 27B/31B model all by itself very well with 250k context at ~120 tokens / second. And for now, there aren't really interesting models in the 40-70B range that justify the response throughput loss.

"But why don't you just put the 5090 and 3090 in the same case to do tensor parallelism?"

I'd love to, it would allow using the two most powerful GPUs together. Except it's a thermal nightmare to fit two big GPUs in a consumer case. (Even my Lian Li O11 XL...) If I put the 3090 next to my 5090, they'll be pressed against each other and prevent each other from breathing. It would also require me to invest in a 1500-1600W PSU to properly power these two monsters.

Here, with the 5090 and 5060 Ti in the same case, GPU temperatures don't go beyond 70° when they're both running inference... And that, is great.

How Did I Set Up RAG?

Setting up a RAG is crucial for obtaining good quality generations. The documents provided can be of different types, here's what I have on my side:

  • Personal administrative documents: I can ask my LLM questions about... my electricity bills for example
  • Novel excerpts: I'm experimenting with LLM narrative creation inspired by book excerpts
  • Literary theory resources: This also helps with my narrative creation experiments

I don't use a custom RAG for documentation access, because I use Context7 for that purpose.

The pipeline:

A RAG always works with the same steps:

  1. Gather all the documents you want to provide access to
  2. Pre-format them, and ensure they're in a format usable for an LLM.
  3. Set up a system allowing the LLM to find the relevant documents for its request
  4. Provide the data to the LLM

[...]

How to Set Up RAG?

Running a good 27B LLM on your own infra is great. However, it's even better when the LLM reasons over your own data.

For this, the idea is simple: you create a search engine, and you allow the LLM to ask it questions:

  1. You take the list of relevant documents (the ones our LLM needs access to)
  2. You create embeddings, meaning vector representations of our documents
  3. You store these embeddings in a vector database
  4. When the LLM asks a question, you vectorize the question with the same model that created the embeddings and you find the documents whose embeddings are closest.

Which embedding model to choose?

For text, the best open-source embedding model currently available is Qwen Embedding 8B. For other formats (audio, video...) it's Jina embedding.

And which vector database?

The answer is fairly simple for a self-hosted homelab, regardless of its size => Qdrant, because it's mature, open-source, and honestly simple to use. These simple arguments are more than sufficient for a homelab, which regardless of its projects won't need to ask itself the same questions as a company that has to manage millions of embeddings.

For a company that wants to set up an embedding system capable of cataloging tens of millions of documents, other questions arise to control costs and performance. (hardware management, retrieval algorithm choice, backup and replication...)

Continuing the AI Section

My GitLab

I have little to say about it, it's a GitLab. It hosts my code, and manages my CI. I'm really happy to host it myself.

For the runner, I use another Docker container hosted on the same VM.

Media Services

I try to host as many multimedia services as possible at home. This mostly comes from a desire to control my data, and minimize my dependence on major services. (Netflix, Prime Video, Spotify, Deezer, Amazon ebooks...)

For once, open-source initiatives are very mature, I really appreciate the following solutions:

Jellyfin: A Netflix / Amazon Prime / Disney+ / HBO / ... equivalent but local, with our own videos.

Navidrome: The Jellyfin equivalent but for music, exposes an OpenSubsonic API, so it's possible to access your music with third-party applications.

Immich: I see no reason for my personal photos / videos to be in the Cloud. They need to be managed and organized!

Paperless-ngx: Incredible application, which hosts all my administrative documents, sorts them, and does OCR. I made an MCP that I connected to Qwen (local, so), which can therefore help me in a personalized way with this info...