Swarthmore has an RTX PRO 6000 Blackwell with 96GB of VRAM. For a liberal arts college that is a strange amount of compute to have sitting in a rack, and for most of the year it wasn't doing much.
SwatGPT came out of wanting to use it for something. ChatGPT is better than anything I can host, but it doesn't know that Sharples is now Dining Center, or which CS courses have prerequisites the registrar page doesn't list, or how to get on the VPN. That gap is small enough to be fixable and specific enough to be useful. It's live at chat.sccs.swarthmore.edu behind an SCCS login. About 1,300 people have used it, around 200 in a given week, and it generates something like 88 million tokens a week. Nothing anyone types leaves campus.
This post has been rewritten once. The first version of SwatGPT was a spring project: my own crawler, OpenWebUI in front, its built-in retrieval in the middle. In August I rebuilt it from scratch on top of current LibreChat under the SCCS org, and most of what follows is about that version.
the model
Qwen3.6-35B-A3B in FP8. It's mixture-of-experts, 35B parameters total and about 3B active per token, which is why it can serve several people at once on one card. Thinking mode is off. It made answers slower and, for "when is the post office open," not better.
I started the spring version with llama.cpp because it took ten minutes to get running and I wanted to see the thing talk. That was the right call for a first day and the wrong thing to keep. Once more than one person was using it the responses got slower, because requests were handled more or less one after another. vLLM fixed that. Its continuous batching packs concurrent requests into the same forward pass, and with 96GB there's headroom for the KV cache that makes possible. The embedding and reranking models share the same GPU now, at 85% memory utilization for vLLM, which leaves about 13GB for them. We used to have a second card for that and then we didn't.
The GPU box is one machine, loon. Everything stateful, LibreChat, Mongo, Meilisearch, Qdrant, the MCP server and its Postgres, lives on a second VM, eagle. Rebuilding the app never touches the model server, and if Qdrant dies the chat keeps working without context, whereas if vLLM dies there's no chat at all. Those are different outages and it's nice that they're on different boxes.
the corpus
The base model knows nothing about Swarthmore, so everything useful comes from retrieval, and retrieval is only as good as what you feed it.
The spring version had a general crawler over swarthmore.edu. It respected robots.txt, took two days, and produced a few hundred thousand chunks where everything was 0.7 similar to everything else, because institutional websites are nine thousand pages of the same nav chrome and footer. I spent more time cleaning that output than crawling it.
This time there are three targeted scrapers. scrape_kb.py pulls every article from the ITS knowledge base through the MediaWiki API, which gives you clean wikitext instead of rendered HTML. scrape_www.py walks the sitemap with an allowlist of sections and department pages, runs the body through trafilatura, and drops anything under 300 characters. scrape_catalog.py is the ugly one: the course catalog lives in Acalog behind an AWS WAF that blocks normal crawling, so it uses the same ajax/preview_*.php endpoints the site's own JavaScript calls, at one request a second, with backoff that goes from 60 seconds to 480 when it gets throttled. About 200 individual printer pages got merged into one directory and one setup guide because nobody wants a chunk per printer.
Total: 3,528 markdown files. 487 knowledge base articles, 1,080 pages from the main site, 1,961 course entries.
ingestion
Chunking is heading-aware now. A markdown parser splits on sections first, then a sentence splitter caps chunks at 512 tokens with 64 of overlap, and every chunk gets embedded with its breadcrumb prepended: title › section, then the text. That breadcrumb also feeds the lexical index, so a course title is searchable even when the body of the chunk never repeats it. The spring version used fixed-size splitting and cut course descriptions in half, which I said was the next thing to fix in the first version of this post. It was.
Embeddings are gte-modernbert-base, 768 dimensions, served by a TEI container on the GPU box. The whole corpus, 3,528 documents into 11,302 chunks, embeds and indexes in 23 seconds.
Reindexing is zero-downtime because of a Qdrant alias. Every ingest writes to a fresh timestamped collection, then swaps the kb alias to it in one operation and deletes the old ones. Nobody talking to the bot notices. It's also a full rebuild every time, not incremental, and nothing checks that the new index is better than the old one before the swap. Atomic isn't the same as validated.
retrieval
The spring version's retrieval was whatever OpenWebUI did, which was a serial embed-then-search pass before every message that you couldn't see into or tune. Average response time was about 4.5 seconds. LibreChat ships with its own separate RAG service backed by pgvector, and I ripped that out too. Retrieval is now about 300 lines of TypeScript inside LibreChat's own packages/api, and it runs on every message.
It's hybrid. A dense query and a sparse query go to Qdrant in parallel, top 20 each. The sparse side is a bag of hashed tokens: lowercase, split on [a-z0-9]+, FNV-1a each token to an index, and let Qdrant apply IDF weighting. The same tokenizer exists twice, once in TypeScript for queries and once in Python for ingest, with a comment on each saying it must stay byte-identical to the other, because if the two ever disagree the sparse index silently returns garbage. Dense catches "how do I get my laptop on the network" when the article says Wi-Fi. Sparse catches "Cornell" when you mean the library and the embedding thinks you mean the university.
The two candidate lists get merged and deduplicated to 32, which is TEI's default batch limit for the reranker. A cross-encoder scores each chunk against the question, anything below 0.3 is dropped, and the top 8 go into the context. One rule on top of that: if the best lexical hit ranked in the top three on the sparse side and passed the 0.3 floor but didn't survive the rerank, it takes the eighth slot anyway. Entity questions ground reliably because of that rule and not because of anything smarter.
The whole thing has a budget of 1.5 seconds, wall clock, across embed, search, and rerank, on one abort signal. If it blows the budget or anything in the chain errors, the function returns undefined, logs one warning, and the chat proceeds without context. Slow Qdrant should never mean no answer. And it's kicked off at the top of message building, overlapped with attachments and tool setup, so most of that 1.5 seconds is hidden behind work that had to happen anyway. Average response time went from 4.5 seconds to 1.8. This one you mostly don't feel.
Nineteen unit tests cover it, mostly the failure modes: connection refused, Qdrant 500, reranker 503, timeout, each env var missing. Each one should produce exactly one warning and a normal chat. Accuracy I check by hand against the list of questions people actually ask, and 96% of them come back right. That's not an eval harness, and I know it.
live data
Retrieval answers "how do I connect to Wi-Fi." It can't answer "when is Sharples open today," because that changes daily and the knowledge base doesn't. Damian built an MCP server for that: twelve read-only tools over The Dash, the campus dashboard, for hours, dining menus, events, news, SEPTA departures, weather, alerts. It polls on a schedule into its own Postgres and serves from the snapshot, so if The Dash goes down, the bot degrades to yesterday's data rather than hanging. Dining has one extra path: if the meal or date you asked about isn't in the snapshot, it fetches that one thing upstream and saves it.
The part of this I had to fix was LibreChat's tool cache. It expires app-level MCP tools after twelve hours, and once expired every request failed with "configured to use MCP tools, but none are available" until someone restarted the container. Two days of that before I found it. The recovery path did refetch the tool list, but it published it without the revision number the cache writer insists on, so the write was silently refused and the app went on believing it had zero tools while the MCP server sat there perfectly healthy. The fix is one revision counter and a warning log for any refusal that's left. The deploy now also refuses to go green unless the app log shows the MCP server initialized with a nonzero tool count.
the frontend
It's LibreChat. The spring version was OpenWebUI, which was slow, hid its retrieval, and fought every attempt to make it look like anything other than OpenWebUI. LibreChat handles streaming, conversation history, and custom endpoints, and stays out of the way.
What's changed is how much of it is turned off. There's one model spec, enforced, so there is no model picker. Presets, bookmarks, memories, code execution, web search, the agent builder, the prompt marketplace: gated off in config, and I added an interface gate on the client so a disabled route redirects to a new chat instead of rendering an empty page. Login is Keycloak through OIDC, email registration is off, and the palettes are borrowed from two other SCCS projects rather than designed for this one.
deploying
Push to main. A self-hosted GitHub Actions runner on the app VM checks out the commit, rebuilds the app image and the MCP image, restarts only the changed containers, polls the app until it returns 200, checks the MCP container is healthy, and fails the deploy if the tool count is zero. About 90 seconds. Data volumes and config never get touched.
The lesson from the spring version is still the lesson: inside a container, localhost is that container, and an unmounted volume means the index disappears on restart. I re-embedded the whole corpus twice before that one landed. This time everything is a named volume and the GPU box and the app box are different machines with different compose files.
since then
September added the boring stuff that makes the numbers above checkable. Retrieval now returns a typed outcome, hit, empty, disabled, error, or timeout, with elapsed time and chunk count, and that feeds Prometheus instead of collapsing into one blank. Every response writes a row to a generation ledger: tokens, time to first token, duration, which tools ran, what retrieval did. There's an admin view over that with its own audit log, which means I should be precise about the privacy claim: inference is local and nothing leaves campus, but conversations are stored and an admin can look at them. That was always true of the database. Now there's a screen for it.
There's also a second structured-context path next to retrieval: a small directory lookup that catches housing questions, matches them against an imported directory snapshot, and hands the model a formatted answer instead of hoping the right chunk surfaces.
Things I still don't have: an eval set I'd defend, and a scheduler for the scrapers, which I run by hand. The retrieval also only looks at the latest message with no query rewriting, so "what about the other one" after a good answer gets a bad one. Those are the next things.
sources
The code is at swat-sccs/SwatGPT. The pieces this post talks about: the catalog scraper, ingest, hybrid search and retrieve with their tests, the Dash MCP server, the cache fix, and the deploy workflow. It's built on LibreChat; the campus parts are ours.