Validation | HyperKit.ai

Definition

Validation checks that LLM output meets business logic and schema requirements before being used downstream. Even with JSON mode enabled, outputs may not satisfy all application-specific constraints.

Validation catches errors early, prevents bad data from propagating, and enables retry logic when outputs don't meet requirements.

Key Concepts

Schema validation: Checking structure matches expected format
Type validation: Ensuring values have correct data types
Business logic validation: Checking application-specific rules
Retry strategies: Re-prompting when validation fails

Examples

Schema Validation

JSON Structure Check

import jsonschema

schema = {
    "type": "object",
    "required": ["sentiment", "score"],
    "properties": {
        "sentiment": {"type": "string"},
        "score": {"type": "number", "minimum": 0, "maximum": 1}
    }
}

try:
    jsonschema.validate(llm_output, schema)
except jsonschema.ValidationError as e:
    # Handle invalid output
    retry_with_feedback(e.message)

Business Logic

Custom Rules

def validate_booking(output):
    errors = []

    # Date must be in future
    if output["date"] < datetime.now():
        errors.append("Date must be in the future")

    # Duration must be reasonable
    if not (15 <= output["duration"] <= 480):
        errors.append("Duration must be 15-480 minutes")

    # Attendees must exist
    for attendee in output["attendees"]:
        if not user_exists(attendee):
            errors.append(f"Unknown user: {attendee}")

    return errors

Interactive Exercise

✎

Design Validation Rules

What validation rules would you add for an LLM that extracts email addresses?

Output format: {"emails": ["email1", "email2"]}

Pro Tips

Always validate even when using JSON mode
Include validation error messages in retry prompts
Set max retries to avoid infinite loops
Log validation failures for prompt improvement

Definition

Key Concepts

Examples

Interactive Exercise

Related Terms