Swarthmore has an RTX PRO 6000 Blackwell with 96GB of VRAM. For a liberal arts college that is a strange amount of compute to have sitting in a rack, and for most of the year it isn't doing much.
SwatGPT came out of wanting to use it for something. ChatGPT is better than anything I can host, but it doesn't know that Sharples is now Dining Center, or which CS courses have prerequisites that aren't listed on the registrar page, or what the shuttle schedule is. That gap is small enough to be fixable and specific enough to be useful.
picking a model
Qwen3-30B-A3B. It's mixture-of-experts, so 30B parameters total but only about 3B active per forward pass, which is why it's fast enough to serve several people at once on one card.
I started with llama.cpp because it took ten minutes to get running and I wanted to see the thing talk. That was the right call for a first day and the wrong thing to keep. Once more than one person was using it the responses got noticeably slower, because requests were being handled more or less one after another. Switching to vLLM fixed that. Its continuous batching packs concurrent requests into the same forward pass instead of queueing them, and with 96GB there's plenty of headroom for the KV cache that makes possible.
The quantization result surprised me. I expected INT4 to cost real quality and mostly it didn't, at least on the campus-question workload I was testing. What it bought was memory, and memory is throughput here, because it's what determines how many concurrent sequences fit in the cache. I tried Q8 in the middle and never found a reason to keep it.
retrieval
The base model knows nothing about Swarthmore, so everything useful comes from retrieval.
I wrote a crawler for swarthmore.edu: course listings, department pages, dining, housing, club directories, faculty research pages. It respects robots.txt and rate limits itself, which mostly meant it took two days instead of two hours. Cleaning the output was worse than crawling it. Institutional websites are full of navigation chrome, cookie banners, and the same footer on nine thousand pages, and if you don't strip that out you get a vector index where everything is 0.7 similar to everything else. After dedup I had a few hundred thousand chunks.
Embeddings run locally on the same GPU, into Qdrant. No API calls, which matters less for cost than for the fact that I can re-embed the whole corpus on a whim without asking anyone for a budget.
The re-ranker is where the answers actually got good. Vector search gives you documents that are about the same topic as the question, which is not the same as documents that answer it. Ask about registration deadlines and you get back every page that mentions registration. A cross-encoder reads the question and a candidate document together and scores them jointly, so it can tell "this page explains the deadline" from "this page mentions deadlines exist." I pull top-20 from Qdrant, re-rank, and put the top 3 in the prompt.
If you're building RAG and you skip re-ranking to save latency, I think you're trading the wrong thing.
docker, for real this time
I'd used containers before in the sense of running docker run on someone else's image. This was the first time I had to build the thing.
Five services: vLLM, Qdrant, the embedding service, the re-ranker, and a small layer that orchestrates the pipeline. Compose makes that one command, which sounds like a small win until you've done it by hand a few times.
The mistake I made repeatedly, over about a week: using localhost for inter-container calls. Inside a container localhost is that container. You want the service name from the compose file as the hostname. This is documented clearly, and I still had to hit connection-refused four or five times before it became automatic.
The other one was volumes. Qdrant's index lives on disk, and if you don't mount it the index disappears when the container restarts. I re-embedded the entire corpus twice before that lesson landed.
the frontend
I forked LibreChat rather than building a chat UI, which was obviously correct and I should have done it sooner instead of spending a weekend on my own. It's an open source ChatGPT-style interface that already handles streaming, conversation history, and custom endpoints.
I stripped the third-party provider integrations, pointed it at the local vLLM endpoint, and put Swarthmore branding on it. The fork is SwatChat-v2.
The codebase is large. I spent longer than I want to admit tracing how conversation state flows through it before I trusted myself to change anything.
what actually took the time
Almost none of it was the model. Getting Qwen running behind vLLM was an afternoon. The rest was crawling, cleaning HTML, chunking, wiring services together, and handling the case where one stage of the pipeline returns nothing and the next stage doesn't expect that.
Latency was the thing I underestimated. Embed, retrieve, re-rank, generate, and each step is fine on its own but they stack, and people notice a pipeline that takes four seconds before the first token even though every piece of it is "fast." I cache Qdrant results for the questions that get asked constantly, which helps the median and does nothing for the tail.
It's still running. The chunking is naive fixed-size splitting and it cuts through tables and course descriptions in ways that clearly hurt retrieval, so that's next. I'd also like some kind of thumbs-up signal so I can tell which answers are actually landing instead of guessing from the ones people complain about.