LLM Response Evaluation & Multi-Response Comparison
A structured comparative evaluation of two large language model outputs responding to a constrained technical explanation prompt, assessed against a Six-Dimension Evaluation Rubric (1–5 score per dimension).
Evaluation Context & Prompt Specification
Establishing explicit user intent, boundary conditions, and acceptance criteria prior to scoring.
Your response must:
1. Use exactly two real-world retail examples (one for each type).
2. Stay strictly under 180 words.
3. Conclude with a single-sentence rule of thumb for deciding when to use which approach.”
Business Analyst with zero programming background. Must avoid mathematical formulas and algorithmic jargon.
Must use exactly two distinct retail industry examples (one for supervised, one for unsupervised).
Strict limit of under 180 words. Length tolerance is evaluated under instruction adherence.
Must conclude with exactly one sentence functioning as an actionable decision rule of thumb.
Candidate Model Outputs
Two distinct AI-generated responses tested under identical prompt conditions.
In supervised learning, the computer learns from historical records where the outcome is already known and labeled. For example, in retail, you feed the system past transaction data labeled with customer churn to predict which current loyalty members are at risk of leaving next month.
In unsupervised learning, there are no pre-existing labels or correct answers provided. The algorithm explores raw data to discover organic groupings on its own. For instance, you can give the system purchasing baskets across all store locations, and it automatically clusters shoppers into distinct lifestyle personas based on buying habits you had not previously defined.
Rule of thumb: Choose supervised learning when you have labeled historical outcomes to predict a specific target, and choose unsupervised learning when you want to discover hidden patterns in raw data without preconceived categories.”
In contrast, unsupervised learning processes an unlabeled input matrix X without supervisory feedback, performing manifold estimation or spatial partitioning. In retail inventory management, k-means clustering algorithms partition store stock keeping units (SKUs) across multidimensional velocity vectors to isolate anomalous supply chain bottlenecks and latent demand correlations.
While supervised architectures yield deterministic point predictions suitable for revenue forecasting pipelines, unsupervised approaches are indispensable for exploratory data mining across non-stationary transaction streams.
For production deployment, engineering teams should assess whether annotated ground-truth telemetry exists prior to selecting model architectures, while continually monitoring inference latency.”
Hard Constraint Validation (Binary Pass/Fail Gates)
Before qualitative rubric scoring, candidate responses must pass objective boundary criteria.
| Constraint Gate | Target Acceptance Threshold | Response A Result | Response B Result |
|---|---|---|---|
| Length Boundary | Word count strictly < 180 words | PASS (156 words) | PASS (158 words) |
| Target Persona Clamping | Zero programming/math jargon | PASS (Zero code/formulas) | FAIL (Dense math & telemetry) |
| Domain Example Count | Exactly 2 retail examples | PASS (Churn + Basket clustering) | FAIL (Abstract SKU spatial math) |
| Output Structure | Single-sentence decision heuristic | PASS (Exactly 1 sentence) | FAIL (2 sentences engineering advice) |
| GATE STATUS SUMMARY | 4 Required Constraints | 4 / 4 PASSED (ELIGIBLE) | 1 / 4 PASSED (DISQUALIFIED) |
Six-Dimension Evaluation Rubric (1–5 Scale)
Scored on a 1 to 5 scale per dimension: 1 = Poor, 2 = Needs Improvement, 3 = Acceptable, 4 = Strong, 5 = Excellent. Equal 16.67% weighting per dimension (Composite Score out of 30).
| Evaluation Dimension | Response A | Response B |
|---|---|---|
| Factual Accuracy Correctness of underlying definitions |
5 / 5 — Excellent
Accurately characterizes supervised learning as learning from labeled historical data and unsupervised as discovering latent groupings without pre-existing labels. |
4 / 5 — Strong
Mathematically accurate descriptions of f(X) → Y and manifold estimation, but inappropriately frames unsupervised learning as "spatial partitioning" for supply chain anomaly detection. |
| Instruction Adherence Constraint fulfillment (length, persona, format) |
5 / 5 — Excellent
Strictly compliant with all 4 constraints: 156 words (<180 words), non-technical persona, exactly 2 retail examples, single-sentence decision rule. |
2 / 5 — Substandard
Met length cap (158 words), but failed audience persona constraint and failed the single-sentence rule-of-thumb requirement (offered 2 disjointed sentences of engineering advice). |
| Relevance Direct alignment to business analyst inquiry |
5 / 5 — Excellent
Laser-focused on the operational distinction between predicting known outcomes vs. discovering new clusters. |
3 / 5 — Acceptable
Digresses into technical implementation topics (gradient boosting, inference latency, telemetry) irrelevant to the analyst's core inquiry. |
| Clarity & Accessibility Ease of comprehension for target persona |
5 / 5 — Excellent
Uses intuitive business concepts (customer loyalty churn risk, basket persona clusters). Zero cognitive friction. |
1 / 5 — Critical Failure
Major audience calibration failure: uses dense academic terminology (f(X) → Y, manifold estimation, non-stationary streams) impenetrable to a non-programmer. |
| Logical Consistency Internal coherence and comparative symmetry |
5 / 5 — Excellent
Balanced, symmetrical structure: definition → retail use case for both paradigms, synthesized by a clear decision heuristic. |
4 / 5 — Strong
Internally cohesive from an academic computer science perspective, but logically asymmetrical in practical utility. |
| Completeness Coverage of essential distinction elements |
5 / 5 — Excellent
Covers input nature (labels vs. unlabeled), guidance process, concrete retail application, and strategic selection criteria. |
3 / 5 — Acceptable
Omits the fundamental operational concept of historical ground-truth training vs. unsupervised discovery. |
| TOTAL COMPOSITE SCORE | 30 / 30 (100% — Preferred) | 17 / 30 (56.7% — Rejected) |
Scoring rationale is deterministic for constraints (word count, sentence count, example count) and strictly grounded in target audience requirements for qualitative dimensions.
{
"benchmark_id": "LLM-EVAL-RETAIL-01",
"evaluator": "Fikri Irawan Abdurahman",
"rubric_name": "Six-Dimension Evaluation Rubric",
"scoring_scale": "1 to 5 per dimension",
"weights": {
"factual_accuracy": 0.20,
"instruction_adherence": 0.25,
"relevance": 0.15,
"clarity_accessibility": 0.20,
"logical_consistency": 0.10,
"completeness": 0.10
},
"hard_fail_criteria": [
"word_count > 180",
"retail_examples_count != 2",
"missing_single_sentence_rule_of_thumb",
"target_persona_incomprehension_jargon"
],
"candidate_scores": {
"response_a": {
"hard_gate": "PASS",
"scores": {"factual_accuracy": 5, "instruction_adherence": 5, "relevance": 5, "clarity_accessibility": 5, "logical_consistency": 5, "completeness": 5},
"composite_score": "30 / 30",
"weighted_score": 5.0,
"decision": "ACCEPTED_FOR_PRODUCTION"
},
"response_b": {
"hard_gate": "FAIL (Persona Mismatch & Missing Single-Sentence Rule)",
"scores": {"factual_accuracy": 4, "instruction_adherence": 2, "relevance": 3, "clarity_accessibility": 1, "logical_consistency": 4, "completeness": 3},
"composite_score": "17 / 30",
"weighted_score": 2.7,
"decision": "REJECTED_CRITICAL_DEFECT"
}
}
}Evaluator Reasoning & Defect Breakdown
Evidence-backed analysis derived from 15+ years of software QA and specification verification.
Why Response A Succeeded
- Persona Calibration: Calibrated cognitive load for a business analyst. Used customer churn prediction (supervised) and shopper basket clustering (unsupervised)—metrics standard in retail business analysis.
- Hard Constraint Adherence: Respects all boundary conditions: 156 words (under 180-word ceiling), exactly two retail examples, and exactly one concluding sentence functioning as a decision heuristic.
- Cognitive Ergonomics: Bolded terms guide rapid executive scanning; plain language eliminates confusion without sacrificing conceptual precision.
Why Response B Failed Acceptance
- The “Illusion of Competence” Defect: Illustrates a classic LLM failure mode: generating dense, highly academic vocabulary (f(X) → Y, manifold estimation) to sound authoritative while failing the user's declared persona constraint.
- Instruction Defect (Rule of Thumb Omission): Failed to provide a single-sentence decision heuristic. Instead produced two complex sentences on engineering telemetry and inference latency.
- Downstream Risk: A business analyst receiving Response B cannot use it in an executive briefing or product requirements document.
Final Assessment & Model Optimization Feedback
Translating quality evaluation into actionable model training and prompt-tuning guidance.
Primary Takeaway: In prompt evaluation and RLHF preference ranking, models frequently over-index on vocabulary density at the expense of user persona constraints. Response A demonstrated exemplary compliance, clarity, and conciseness.
Reinforce negative weighting during RLHF fine-tuning when the model generates mathematical notation (f(X) → Y) or postgraduate jargon in response to explicit “no programming background” directives.
Train model internal chain-of-thought verification to check: len(examples) == requested_count, sentence_count(conclusion) == 1, and word_count < requested_limit.
Reproducibility Evaluation Artifact (JSON Schema Spec)
{
"test_id": "EVAL-ML-001",
"task": "pairwise_comparison",
"hard_gates": {
"max_words": 180,
"required_examples": 2,
"conclusion_sentences": 1,
"target_audience": "non_technical"
},
"rubric_dimensions": [
"factual_accuracy", "instruction_adherence", "relevance",
"clarity", "logical_consistency", "completeness"
],
"verdict": {
"winner": "Response A",
"composite_scores": {"A": 30, "B": 17},
"evaluator_confidence": "HIGH"
}
}