The answer was already in another thread, so I built ThreadSherpa

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

I got the idea from the MLH Discord.

People kept asking about account names, missing offer codes and student benefits. Different people, same questions. The useful answer was usually already there, buried in another thread, and a moderator had to go find it and type it out again.

ThreadSherpa is a Discord support bot that remembers the answer so the moderator doesn't have to. A moderator replies normally, right-clicks their message and picks Apps → Save this answer. That one click is the approval. The next time someone asks the same thing in different words, ThreadSherpa writes a fresh reply for that person from the saved facts and links back to the original.

The part I care about most: when the saved answers don't cover a question, ThreadSherpa says nothing. No "I'm not sure, maybe try…", no guessing. The question goes into a quiet moderator queue instead. A small open model is only as good as what you give it, so ThreadSherpa only gives it facts a human already approved, and stays out of the way otherwise.

Demo

Try it in your browser · Website and walkthrough · Setup guide

The /try/ sandbox is a fictional community where you can save a moderator's reply, ask a new question and watch it answered, all in your browser with no Discord server needed.

Below is a real run in my test server, with separate moderator and participant accounts, on a Minecraft support question: which version can join the server?

The moderator answers and saves the reply:

Later, a participant describes the same problem in completely different words, an "Outdated client" error. ThreadSherpa answers on its own, tells them which version to pick, and links to the moderator's original guidance:

Paper Instruments' Ultramarine wrote this reply and MongoDB Atlas stored the saved answer. The exact same flow also runs fully local on Gemma 4 through Ollama.

Code

Built solo, Apache-2.0 licensed. There's also a GitLab mirror.

How I Built It

TypeScript and discord.js. Saved answers, versions, support cases and evidence live in MongoDB Atlas. Models plug in through one provider interface, with two first-class options: Gemma 4 (gemma4:e4b-it-qat) running locally through Ollama, or Paper Instruments' Ultramarine through its API. EmbeddingGemma handles embeddings locally either way.

Moderator saves a reply
  → keep their exact wording, check for duplicates, store it in Atlas

New forum question + thread context
  → retrieve this server's current answers (exact, keyword and vector match)
  → give only that evidence to the model
  → check every citation and link in the draft
  → reply in the thread, or queue it quietly for a moderator

Retrieval combines exact matching, keyword search and embeddings, ranked with Atlas Vector Search. Every server has its own memory, so one community's answers never leak into another's.

Saving keeps the human's words; answering is a separate job

My first version stuffed titles, review details, versions and labels around a two-line answer, and pasted old messages back almost verbatim. It felt like a filing cabinet, not a helper.

So I split the two jobs. Saving stores the moderator's wording exactly as written. In the learning pipeline that's simply:

canonical_answer: options.adminMessage,

Generation happens later, with the new question and its thread in front of the model, so each reply fits the person asking. Saving the same answer twice reuses the existing entry.

Staying quiet is a feature

During testing the bot posted "I couldn't check this. A moderator can help." under a question a moderator had already answered. It added noise and helped nobody.

Now a queued result sends nothing. The Discord handler only posts when it has something useful:

if (result.policyResult.action !== 'queue') {
  await thread.send(await this.card(c.case_id));
}

A few other guardrails sit around that:

• Narrow answers ask first. If a saved answer only applies under certain conditions, ThreadSherpa asks a clarifying question instead of guessing.
• Account actions go to humans. Sensitive requests are routed to moderators, never answered by the model.
• No open web search. Besides saved answers, the bot can only read documentation URLs a moderator explicitly adds with /sources, fetched through an SSRF-safe fetcher.

It learns from how the conversation ends

Each reply comes with Solved and Need help buttons, and the bot also reads plain follow-ups. "That worked" closes the case. "That didn't help" hands it to a moderator and adds the failed advice to the context, so the bot doesn't keep suggesting the same fix.

Moderators stay in control

Three autonomy modes, picked in /setup:

• Quiet: record new posts, no replies. Good for a trial week.
• On request: reply only to /ask and follow-ups.
• Automatic: answer new support questions on its own.

And a set of moderator tools:

The /memory panel is visible only to the moderator using it:

Already have a FAQ? Import FAQs takes pasted Markdown, shows a preview, and saves it in one go, skipping duplicates:

## Can I submit by myself?
Yes, solo projects are fine.

## Do I need a video?
A video is optional.

Answers can have a source, conditions and an expiry date. Expired answers stay visible in the panel for review but stop being used in replies, so last month's deadline never gets quoted as this month's.

/queue is also where the lazy loop closes: if the same unanswered question shows up five times, the moderator sees it grouped together, answers once, saves it, and it's handled from then on.

Testing

• 140 automated tests cover routing, approval, duplicate saves, expiry, source checks, FAQ import and panel permissions, and run on every push through GitHub Actions.
• 12 support scenarios (paraphrases, a different event with a similar name, expired guidance and a correction) passed on both local Gemma and Ultramarine. Reproduce them with npm run eval:local or npm run eval:paper.
• Atlas was checked for persistence across processes, real vector queries and per-server isolation with npm run verify:atlas.
• npm run demo runs the whole flow on fixtures, with no Discord and no paid inference.

The Ultramarine eval made 10 generation calls (5,408 input and 1,109 output tokens) for about $0.0014 at the 23 October rates, roughly a hundredth of a cent per generated reply.

Tracing every answer with Sentry

Every answer runs inside a Sentry agent trace. Each support.answer agent span contains the tool.search_knowledge tool call that finds the saved answers, then the GroundedAnswerDraft model call and the HTTP request behind it. Embedding calls get their own spans too. Model spans record the model name and input and output token counts; message text and prompts are never sent, only numbers.

On a traced Ultramarine eval run, each reply used 506–588 input and 50–176 output tokens, and the model call took 1.1–3.2 seconds. Knowledge search took under a millisecond and local embeddings a few milliseconds, so almost all of the answer time is the model writing. The trace showed something else useful: some questions the bot shouldn't answer stop right after the search, without calling the model at all, so those declines cost no tokens.

Set SENTRY_DSN and both the bot and npm run eval send these traces.

Why Does Open Innovation Matter?

A community's support knowledge stays with the community. Moderator answers often touch account names, offer codes and other things members don't want shared. With Gemma running through Ollama, generation and embeddings happen on the machine the operator controls, and no third-party model provider ever sees them.

It's cheap enough for volunteers. Most Discord communities are run by volunteers, and a support bot with a big bill is a bot nobody keeps switched on. On local Gemma, one more answer costs nothing extra. On Ultramarine, the whole eval run cost about a seventh of a cent.

Moderators can see and change everything. The prompts, retrieval and policy rules are all in the repo. If a community wants a stricter bot, a friendlier tone or a different model, they change a config value or a prompt, not a vendor contract.

Open models make the choice the community's. Because both Gemma and Ultramarine are open models (Paper has announced Ultramarine's weights), a community isn't locked in. Run Gemma on a spare laptop for full privacy, use Ultramarine through the API for stronger writing at a tiny cost, or self-host Ultramarine once its weights are out. I swapped between the two without touching any of the saving, retrieval or Discord code. The community owns its knowledge and picks where its model runs.

The Lazy Payoff

Before: a moderator sees a question, remembers answering it last week, scrolls through old threads, finds it, and types it out again. Then does the same thing tomorrow.

After: the moderator answers once and clicks Save this answer. Every future version of that question, in whatever words, gets a tailored reply with a source, while the moderator is asleep. Questions the bot can't answer wait quietly in /queue until someone has time.

That's the sloth approach: do the work once, properly, and let it keep paying off.

Prize Categories

• Best Use of Gemma: Gemma 4 through Ollama is the default reply model and handles duplicate comparisons, with EmbeddingGemma for local retrieval embeddings. It passed the full scenario suite locally.
• Best Use of MongoDB Atlas: Atlas holds saved answers, versions, cases and evidence, scoped per server, with Atlas Vector Search ranking retrieval candidates. Persistence and real vector queries are verified by npm run verify:atlas.
• Best Use of Ultramarine: Ultramarine wrote the replies in the Discord demo above, working only from the moderator-approved evidence, and passed the full scenario suite. The eval run cost about $0.0014 for 10 generations, broken down above.
• Best Use of Render: Render hosts ThreadSherpa's front end at threadsherpa.onrender.com, including the interactive /try/ sandbox where anyone can run the save → ask → reply loop, deployed from the repo's render.yaml Blueprint. The repo also ships a background-worker Blueprint that runs the Discord bot itself on Render, preconfigured for Ultramarine or a Gemma endpoint.
• Best Use of Sentry Agent Tracing: every answer is a Sentry agent trace with tool, model and HTTP spans, plus token counts and latency per model call. The trace screenshot and the numbers are in the tracing section above.
• Best Use of GitHub Copilot: the Checks workflow in GitHub Actions type-checks, tests and builds the bot, verifies the fixture flow, and checks and builds the website on every push.

I used AI coding assistants while building ThreadSherpa and while writing this post.