🎯 InterviewIQ

Concept

InterviewIQ is an AI-powered mock-interview coach engineered with a multi-turn ReAct (Reasoning + Acting) tool-calling architecture. Instead of relying on a single subjective prompt that risks arithmetic hallucinations and inconsistent grading, InterviewIQ embeds the LLM as a cognitive orchestrator inside an autonomous loop.

The agent dynamically decides which specialized diagnostic tools to invoke (filler-word density, STAR framework coverage, keyword relevance scoring), inspects the structured outputs, and synthesizes nuanced, actionable coaching advice. In parallel, an append-only session memory ledger tracks candidate performance across all turns, applying deterministic mathematical ranking to eliminate recency bias and power mid-session meta-coaching.

Theory & Concepts

1. Autonomous ReAct Agent Loop vs. Static Pipelines

Traditional LLM applications operate in a rigid, single-turn sequence: User Prompt → LLM → Output. This passive paradigm struggles in complex evaluation tasks because language models suffer from cognitive overload, hallucination during precise mathematical counting, and an inability to inspect intermediate diagnostic data.

InterviewIQ implements the ReAct (Reasoning + Acting) framework (Yao et al., 2022) via the OpenAI Function Calling protocol. The LLM operates in an autonomous loop:

• Tool Schema Contract: The application exposes a JSON schema (TOOLS_SCHEMA) defining available Python evaluation functions, their parameter specifications, and natural-language descriptions.
• Autonomous Tool Selection: Given a candidate's answer and question metadata, the LLM acts as an autonomous planner. It decides which tools are necessary (e.g., invoking detect_filler_words, check_star_structure, and score_relevance in parallel or sequentially) and extracts the appropriate arguments.
• Environment Execution & Feedback: The runtime dispatches tool calls to deterministic Python functions, captures the structured outputs, and appends them back to the message transcript under the tool role.
• Iterative Convergence: The agent loops (up to MAX_TOOL_ROUNDS) until all necessary observations are gathered, then switches from tool calling to generating its final synthesized coaching critique.

2. Separation of Concerns: Deterministic NLP vs. Generative Synthesis

A core principle in robust AI engineering is never asking an LLM to do what deterministic code does with 100% precision. Large Language Models are probabilistic next-token predictors; they are inherently unreliable at exact character counting, regex matching, and statistical calculations.

• Exact Filler-Word Density (detect_filler_words): Uses compiled regex word-boundary patterns (\b(um|uh|like|basically|actually|literally|you know)\b) to compute exact word frequencies and normalized density per 100 words ((fillers / word_count) * 100).
• STAR Structural Parsing (check_star_structure): Heuristically parses behavioral responses across the 4 pillars of the STAR framework (Situation, Task, Action, Result) using contextual phrases, temporal indicators, ownership verbs, and quantifiable impact metrics (e.g., \d+% or \$\d+).
• Inflectional Keyword Relevance (score_relevance): Generates morphological regex stems to match expected domain keywords across grammatical inflections (e.g., -tion, -ment, -e, -y), scoring coverage against a calibrated tiered curve (0–100).
• Empathetic Cognitive Synthesis: By delegating mathematical and structural analysis to deterministic tools, the LLM is freed from arithmetic. It uses its language capability solely to interpret raw metrics, contextualize trade-offs, and deliver encouraging, constructive feedback.

3. Structured Session Memory & Anti-Recency Bias

Standard conversational AI setups pass entire chat transcripts into the prompt context. This introduces severe recency bias and context dilution ("lost-in-the-middle" phenomenon), where the model overemphasizes recent turns and forgets early answers.

InterviewIQ overcomes this using an append-only structured session ledger (InterviewSessionMemory):

• Immutable Turn Ledgers: Every interview turn stores the question metadata, candidate answer, and raw structured dicts from all tool executions.
• Mathematical Composite Ranking: Weakest and strongest performance areas are determined globally across all turns using a deterministic composite sort key:
  rank_key(turn) = (relevance_score ↑, star_score ↑, -total_filler_count ↓)
  The globally weakest turn is computed as min(turns, key=rank_key), ensuring an objective assessment even if the candidate aced the most recent question.
• Dynamic Meta-Tool Invocation: The agent exposes generate_final_report as an executable tool. When a candidate asks a mid-session meta-question (e.g., "What is my weakest area so far?"), the LLM autonomously calls this tool, reads the aggregated memory state, and provides an evidence-based progress review.

Request flow

Code flow

Backend — ReAct Agent & Session Memory

class InterviewSessionMemory:
    """Append-only session store for interview turns.

    Each turn records the question, answer, and the structured dict results
    returned by each evaluation tool.
    """

    def __init__(self):
        self._turns: list[dict] = []

    def add_turn(
        self, question: str, answer: str, results: dict,
        category: str = "", question_id: int = 0,
        expected_keywords: list | None = None,
    ) -> None:
        self._turns.append({
            "turn_id": len(self._turns) + 1,
            "question_id": question_id,
            "question": question,
            "category": category,
            "expected_keywords": expected_keywords or [],
            "answer": answer,
            "results": results,
        })

    def get_weakest_area(self) -> dict | None:
        """Return the turn with the lowest composite score.

        Uses a 3-key sort (relevance ASC, star_score ASC, filler_count DESC)
        to explicitly avoid recency bias — the weakest area is the globally
        worst turn, not the most recent one.
        """
        if not self._turns:
            return None

        def sort_key(turn):
            r = turn["results"]
            rel = r.get("score_relevance", {}).get("score", 50)
            star = r.get("check_star_structure", {}).get("star_score", 50)
            fillers = r.get("detect_filler_words", {}).get("total_filler_count", 0)
            # Lower relevance, lower STAR, higher fillers = weaker.
            return (rel, star, -fillers)

        weakest = min(self._turns, key=sort_key)
        rel_data = weakest["results"].get("score_relevance", {})
        star_data = weakest["results"].get("check_star_structure", {})
        filler_data = weakest["results"].get("detect_filler_words", {})
        return {
            "turn_id": weakest["turn_id"],
            "question_id": weakest["question_id"],
            "question": weakest["question"],
            "category": weakest["category"],
            "relevance_score": rel_data.get("score", 0),
            "star_score": star_data.get("star_score", 0),
            "filler_count": filler_data.get("total_filler_count", 0),
            "unmatched_keywords": rel_data.get("unmatched_keywords", []),
        }


class EvaluatorAgent:
    """Tool-calling evaluator agent with session memory."""

    def _run_loop(self, messages: list, results_out: dict | None = None) -> str:
        """Shared tool-calling loop (the authentic ReAct pattern).

        Sends messages to the LLM, executes whichever tools the LLM requests,
        appends the results back, and loops until the model stops calling
        tools (up to MAX_TOOL_ROUNDS).
        """
        for _ in range(MAX_TOOL_ROUNDS):
            response = self._create_with_retry(
                model=self._model,
                messages=messages,
                tools=TOOLS_SCHEMA,
                tool_choice="auto",
            )
            msg = response.choices[0].message

            if not msg.tool_calls:
                return msg.content or ""

            messages.append(msg)
            for call in msg.tool_calls:
                args = (
                    json.loads(call.function.arguments)
                    if call.function.arguments
                    else {}
                )
                fn = self._tool_functions[call.function.name]
                result = fn(args)

                # Record tool results for session logging (except the report
                # itself, which is meta-data, not per-answer evaluation).
                if (
                    results_out is not None
                    and call.function.name != "generate_final_report"
                ):
                    results_out[call.function.name] = result

                messages.append({
                    "role": "tool",
                    "tool_call_id": call.id,
                    "content": (
                        result if isinstance(result, str) else json.dumps(result)
                    ),
                })

        # Ran out of rounds — ask for a final answer without offering tools.
        response = self._create_with_retry(
            model=self._model, messages=messages
        )
        return response.choices[0].message.content or ""

Deterministic Tools

def detect_filler_words(answer: str) -> dict[str, Any]:
    """Count filler words (um, like, basically, etc.) and compute density."""
    if not answer or not answer.strip():
        return {"detected_fillers": {}, "total_filler_count": 0, "filler_density_per_100_words": 0.0}

    text_lower = answer.lower()
    detected_fillers: dict[str, int] = {}
    total_count = 0

    for name, pattern in FILLER_PATTERNS:
        matches = re.findall(pattern, text_lower, flags=re.IGNORECASE)
        count = len(matches)
        if count > 0:
            detected_fillers[name] = count
            total_count += count

    words = re.findall(r"\b\w+\b", text_lower)
    word_count = len(words)
    density = round((total_count / max(word_count, 1)) * 100, 1)

    return {
        "detected_fillers": detected_fillers,
        "total_filler_count": total_count,
        "has_fillers": total_count > 0,
        "filler_density_per_100_words": density,
        "word_count": word_count,
    }


def check_star_structure(answer: str) -> dict[str, Any]:
    """Check whether a behavioral answer covers Situation, Task, Action, Result."""
    text_lower = answer.lower()
    covered: list[str] = []
    missing: list[str] = []
    components_status: dict[str, bool] = {}

    for component, patterns in STAR_PATTERNS.items():
        found = any(re.search(p, text_lower, flags=re.IGNORECASE) for p in patterns)
        components_status[component.lower()] = found
        if found:
            covered.append(component)
        else:
            missing.append(component)

    star_score = round((len(covered) / 4) * 100, 1)
    return {
        "covered_components": covered,
        "missing_components": missing,
        "star_score": star_score,
        "is_star_complete": len(covered) == 4,
    }


def score_relevance(answer: str, expected_keywords: list[str]) -> dict[str, Any]:
    """Score relevance against expected domain concepts using inflection matching."""
    text_lower = answer.lower()
    matched_keywords: list[str] = []
    unmatched_keywords: list[str] = []

    for kw in expected_keywords:
        pattern = _build_keyword_pattern(kw.lower().strip())
        if re.search(pattern, text_lower, flags=re.IGNORECASE):
            matched_keywords.append(kw)
        else:
            unmatched_keywords.append(kw)

    total_expected = len(expected_keywords)
    total_matched = len(matched_keywords)
    coverage_ratio = total_matched / max(total_expected, 1)

    # Calibrated tiered scoring curve
    if total_expected == 0:
        score = 100
    elif total_matched == 0:
        score = 0
    elif coverage_ratio >= 0.70:
        score = min(100, int(round(90 + ((coverage_ratio - 0.70) / 0.30) * 10)))
    elif coverage_ratio >= 0.45:
        score = int(round(75 + ((coverage_ratio - 0.45) / 0.25) * 14))
    elif coverage_ratio >= 0.25:
        score = int(round(50 + ((coverage_ratio - 0.25) / 0.20) * 24))
    else:
        score = max(5, int(round((coverage_ratio / 0.25) * 45)))

    return {
        "score": score,
        "matched_keywords": matched_keywords,
        "unmatched_keywords": unmatched_keywords,
        "keyword_coverage_ratio": round(coverage_ratio, 2),
    }

API route

@bp.route("/evaluate", methods=["POST"])
def evaluate():
    """Evaluate a candidate's answer to an interview question."""
    data = request.get_json(force=True)
    question_id = data.get("question_id")
    answer = (data.get("answer") or "").strip()

    if not answer:
        return jsonify({"error": "An answer is required."}), 400

    q = get_question_by_id(question_id)
    if not q:
        return jsonify({"error": f"Question ID {question_id} not found."}), 400

    try:
        result = _agent.evaluate_answer(q, answer)
        return jsonify(result)
    except Exception as e:
        return jsonify({"error": str(e)}), 500


@bp.route("/coach", methods=["POST"])
def coach():
    """Handle free-form candidate meta-questions via the ReAct agent."""
    data = request.get_json(force=True)
    message = (data.get("query") or data.get("message") or "").strip()

    if not message:
        return jsonify({"error": "A question is required."}), 400

    try:
        reply = _agent.ask_agent(message)
        return jsonify({"response": reply})
    except Exception as e:
        return jsonify({"error": str(e)}), 500