Creating a Support Triage Agent (24 Hours)

HackerRank Orchestrate was a 24-hour hackathon with a deliberately unglamorous brief: build a terminal-based agent that triages real support tickets across three unrelated product ecosystems, HackerRank, Claude, and Visa, using nothing but the support corpus shipped in the repo. No live web calls, no outside knowledge. For each ticket the agent has to decide whether to reply or escalate, classify what kind of issue it is, and if it replies, ground that reply in an actual document instead of making something up. Out of 1,349 participants, this placed 27th.

The tickets themselves were the hard part. Some were plain FAQs. Others were prompt injections, jailbreaks in French, requests to delete files, or genuinely ambiguous one-line messages with no company and no context. A support agent that can't tell those apart isn't a support agent, it's a liability with good uptime.

The Short Version

It comes down to a pipeline that goes from raw ticket text to a structured, grounded decision: safety checks before anything else, retrieval scoped to the right company's corpus, a confidence gate that refuses to guess, and an LLM router that classifies and responds only from what it actually retrieved.

Tech Stack: Python, Milvus Lite, Voyage AI (embeddings + reranking), OpenRouter, langdetect.

The Shape of the Pipeline

Every ticket flows through the same driver, one class wiring together three collaborators:

main.py  ──►  Orchestrator
               ├── SafetyGuard      (pre + post LLM checks)
               ├── CorpusRetriever  ──► Milvus Lite + Voyage AI
               └── Router           ──► LLM Provider (OpenRouter)

The orchestrator's job is to decide, as early and as cheaply as possible, whether a ticket even needs an LLM call at all:

def process(self, issue: str, subject: str, company: str) -> TicketResult:
    safety_result = self._safety.check_input(issue, subject)
 
    if safety_result.is_safe and self._safety.is_social_phrase(issue):
        return TicketResult(status="replied", product_area="general_support", ...)
 
    if not safety_result.is_safe:
        chunks = []
    else:
        query  = f"{subject} {issue}".strip() if subject else issue
        chunks = self._retriever.retrieve(query=query, company=company, top_k=TOP_K)
 
    if norm_company in ("", "none") and (not chunks or all(c.score < CONFIDENCE_THRESHOLD for c in chunks)):
        return TicketResult(status="replied", product_area="conversation_management", ...)
 
    result = self._router.route(issue=issue, subject=subject, company=company,
                                 chunks=chunks, safety_result=safety_result, ...)
    return TicketResult(...)

"Thank you for helping me" doesn't need retrieval, a vector search, or a model call. It needs a canned reply. Sending it through the full pipeline anyway was an actual bug caught against the ground-truth set: without the social fast-path, an empty retrieval result for a message with no real question in it was falling through to the router, which escalated it. The fix isn't clever, it's just recognizing that not every input deserves the same amount of machinery.

Retrieval: Rerank, Then Filter for Confidence, Then Diversify

Retrieval is a four-step pipeline, not a single vector search:

ANN search (Milvus, voyage-4-large)
      ↓
cross-encoder rerank (rerank-2.5)
      ↓
confidence gate (best score < threshold?)
      ↓
MMR diversification

The first two steps are standard RAG: a fast approximate nearest-neighbor search over embeddings pulls a wide candidate pool, then a slower cross-encoder reranks those candidates by actual relevance to the query, since embedding similarity and true relevance are correlated but not the same thing.

The confidence gate is where this stops being generic. If the reranker's top score falls below a threshold, every candidate chunk gets flagged low_confidence in its metadata, and that flag rides along all the way to the router, which is instructed to escalate rather than answer from documentation it isn't actually confident matches the question.

The last step is Maximal Marginal Relevance, which trades a small amount of relevance for diversity:

def _mmr(query_embedding, chunks, embeddings, top_k, lambda_mult):
    selected_idx, remaining_idx = [], list(range(len(chunks)))
    for _ in range(top_k):
        best_idx, best_score = -1, -float("inf")
        for i in remaining_idx:
            relevance  = _cosine(query_embedding, embeddings[i])
            redundancy = max((_cosine(embeddings[i], embeddings[j]) for j in selected_idx), default=0.0)
            mmr_score = lambda_mult * relevance - (1 - lambda_mult) * redundancy
            if mmr_score > best_score:
                best_score, best_idx = mmr_score, i
        selected_idx.append(best_idx)
        remaining_idx.remove(best_idx)
    return [chunks[i] for i in selected_idx]

Without it, the top slots can fill up with five near-identical paragraphs from the same article, wasting context the LLM could have spent on a second, differently-worded document that actually answers a different part of the question. There's also a narrower rule worth calling out: once a company is known, retrieval never falls back to the full corpus if that company's own documents come up short. Visa docs bleeding into a Claude answer because the filtered search returned too few hits would produce a confidently wrong response, so a thin result set gets flagged low-confidence instead of quietly widened.

Guardrails Before the LLM Ever Runs

Every ticket passes through safety checks before retrieval even starts, in a fixed order: a multilingual jailbreak probe, then a regex check for malicious commands. English-language prompt injection and jailbreak attempts are handled separately, inside the router's own system prompt, which keeps regex from having to guess at intent in a language it's actually good at parsing.

The interesting layer is what happens to everything else:

def check_input(self, issue: str, subject: str = "") -> SafetyResult:
    combined = f"{subject} {issue}".strip()
 
    lang = _detect_language(combined)
    if lang != "en" and _semantic_safety_probe(combined):
        return SafetyResult(is_safe=False, threat_type=ThreatType.JAILBREAK, ...)
 
    if _matches(combined, _MALICIOUS_PATTERNS):
        return SafetyResult(is_safe=False, threat_type=ThreatType.MALICIOUS_COMMAND, ...)
 
    return SafetyResult(is_safe=True, threat_type=ThreatType.CLEAN)

English-only regex patterns are blind to a jailbreak phrased in French. Rather than trying to maintain parallel pattern lists in a dozen languages, non-English text gets a five-token LLM call instead: a single yes/no question asking whether the text is trying to extract internal rules or bypass safety. It costs about fifty input tokens and runs only when langdetect flags something as not-English, which in this ticket set was rare enough that the cost was negligible against the value of catching a jailbreak that regex would have waved through.

Knowing When to Escalate

The single design decision that shaped everything else is: when in doubt, don't answer. Confidence-gated retrieval feeds that principle, the social and off-topic fast-paths protect it from being short-circuited by cases that never needed a model call, and the router's system prompt spells it out directly: if the documentation doesn't contain enough to answer confidently, escalate and say why.

That preference for restraint over completeness is also why schemas.py exists as a single source of truth for every enum value the agent can output. Product areas, statuses, and request types all live in one file, and the JSON schema, the validation logic, and the system prompt's guidance text are all generated from it:

"""
schemas.py: single source of truth for all triage output enumerations.
 
To add, rename, or remove a product_area, status, or request_type:
  1. Edit ONLY this file.
  2. Re-run the agent. The JSON schema, validation frozenset, system prompt
     guidance, and product area examples are all derived from here automatically.
 
Nothing else needs to change.
"""

A model that hallucinates a product_area outside the allowed set is a silent failure mode if nothing catches it, so every parsed response gets clamped against a frozenset derived from this same file before it ever reaches the output CSV.

Results

The scoring rubric here was an LLM-as-judge run across five weighted dimensions: status, product area, response quality, justification, and request type. Six iterations moved the weighted score from a working baseline to something that held up under a held-out ground-truth check too:

A few individual tickets are worth calling out on their own:

Looking Back

None of the individual pieces here are novel on their own: RAG with reranking is well-trodden ground, and a confidence threshold isn't a new idea. What made this agent hold up under adversarial tickets wasn't any single component, it was refusing to let convenience erode the one rule that mattered: don't answer what you're not confident about, and don't let a fast-path or a missing guardrail quietly bypass that rule for the one ticket that needed it most.

The ground-truth bugs were the clearest evidence of that. A thank-you message escalating because it fell through an empty-retrieval path, an off-topic movie trivia question escalating because nothing in the pipeline recognized "no company, no relevant chunks" as its own case, these weren't model failures. They were gaps in the scaffolding around the model, and closing them mattered more than any embedding upgrade did.

The full source is on GitHub.