Generative AI Guide

TL;DR

• Generative AI creates new content — text, images, audio, video, code — rather than classifying inputs into fixed labels.

• Foundation models are large pretrained models adapted to many tasks via prompting, fine-tuning, or retrieval — you almost never train them from scratch.

• Text and code generation use autoregressive models that predict the next token sequentially. Image generation primarily uses diffusion models that denoise random noise into images.

• Generation is probabilistic — the same input can produce different outputs. Production systems require validation, guardrails, and evaluation.

• Distinguish model capability from application engineering — APIs expose generation; reliability comes from prompts, retrieval, structured outputs, and checks you build around the model.

On this page

• Why This Matters
• The Problem Generative AI Solves
• How We Got Here
• What Is Generative AI?
• How Generative AI Works
• Architecture
• Step-by-Step Flow
• Real Production Example
• Design Decisions
• Comparisons
• Common Mistakes
• Where It Breaks Down
• When NOT to Use Generative AI
• Running in Production
• Related Guides
• Interview Questions
• Key Takeaways
• FAQs
• References
• Further Reading

Why This Matters

Generative AI changed what software can ship. Before foundation models, writing emails, summarizing contracts, generating images from descriptions, or completing code required specialized models and pipelines per task. Today, one pretrained model handles dozens of generation tasks through different prompts and tools.

If you build software, you will integrate generative AI — through provider APIs, open-weight models, or vendor features embedded in existing products. The hard problem is not calling an API. It is building reliable systems on probabilistic outputs: managing cost, latency, hallucination, safety, and evaluation.

This guide is the entry point for the LLM Concepts cluster. Later guides deepen Large Language Models, tokens, prompting, fine-tuning, and hallucinations. Downstream clusters — RAG, AI Agents, Knowledge Graphs, AI System Architecture — assume you understand why generation is probabilistic and why application layers exist.

The Problem Generative AI Solves

Traditional software produces deterministic outputs from explicit logic. Classical ML classifies or regresses — it labels inputs or predicts numbers. Neither creates open-ended content at scale.

Generative AI addresses the content creation and task-unification bottleneck:

1. Text at scale — Drafts, summaries, translations, and explanations without a template engine or human writer for every variant.

2. Multimodal creation — Images, audio, and video from natural language without a specialized tool per asset type.

3. Code synthesis — Functions, tests, and boilerplate from specifications — still requiring tests before ship.

4. Personalization — Unique outputs per user context without combinatorial template libraries.

5. Task unification — One model interface for many generation tasks via prompting instead of per-task training pipelines.

The trade-off: you gain flexibility and lose determinism. Every generative output is a sample from a distribution, not a guarantee.

How We Got Here

Generative systems existed before ChatGPT — GANs, VAEs, and early language models — but foundation-scale training and chat interfaces made them product-default.

Diagram: Evolution of generative AI

timeline
    title From classical ML to foundation models
    2014-2017 : GANs and early seq2seq
              : Task-specific generators
    2017-2020 : Transformers + GPT-2/3
              : Scaling laws emerge
    2021-2022 : Diffusion images + Codex
              : Multimodal experiments
    2023-2026 : Chat UIs + APIs everywhere
              : RAG, agents, eval as defaults

Capability scaled first; production discipline (validation, retrieval, eval) followed once demos hit real traffic.

What Is Generative AI?

Generative AI refers to machine learning systems that produce new data samples resembling their training distribution. Discriminative models learn boundaries between classes (spam vs not spam). Generative models learn enough structure of the data that they can sample new plausible instances — paragraphs, images, or functions.

Intuition first

Think of the model as having compressed a large corpus into parameters (parametric knowledge). At inference, it samples continuations that are statistically consistent with that compression. It does not look up a database row unless you add retrieval or tools (retrieved knowledge).

That distinction matters for everything that follows:

Major modalities

Foundation models

A foundation model is a large model pretrained on broad data and adapted to downstream tasks. The "foundation" metaphor: one base capability supports many applications.

Adaptation methods (application engineering, not pretraining):

• Prompting — Describe the task in natural language. See Prompt Engineering.
• Fine-tuning — Continue training on domain examples to change behavior or style. See Fine-tuning.
• RAG / tools — Inject external knowledge or side effects at inference. See RAG and Function Calling.

Engineering Insight

Foundation models are generalists. Production value comes from adaptation layers — prompts, retrieval, fine-tuning, structured outputs, and validation — not from the raw model alone.

How Generative AI Works

Training vs inference

You almost always consume inference. Understanding training explains limitations (hallucination, stale knowledge, bias); your engineering leverage is at inference and in the systems around it.

Text generation: autoregressive prediction

Large language models generate text one token at a time. Given prior tokens, the model outputs a probability distribution over the vocabulary and selects (or samples) the next token.

Source: The Illustrated GPT-2 — Jay Alammar

Diagram: Autoregressive generation loop

flowchart LR
    P[Prompt tokens] --> M[Model forward pass]
    M --> D[Distribution over vocab]
    D --> S[Sample / argmax]
    S --> A[Append token]
    A --> M

Each new token requires another forward pass — output length dominates latency and cost.

Image generation: diffusion

Diffusion models start from noise and iteratively denoise toward a coherent image, guided by a text embedding. Each step is GPU-heavy — image generation is typically slower and more expensive per request than short text completions.

Code generation

Code uses the same autoregressive mechanism as text, trained heavily on repositories. Models learn syntax and APIs; they do not execute code. Generated code must be tested, typed-checked, and reviewed — treating it as a PR from a junior engineer is the right mental model.

Generation vs reasoning

Providers market "reasoning" modes that spend more tokens on intermediate steps. Mechanically, this is still generation — longer sampled trajectories, sometimes with tools. Application engineering still must verify claims, bound cost, and decide when a cheaper non-reasoning path is enough.

Architecture

A production generative application wraps the model with engineering layers. The model is one component; reliability lives in the rest.

Diagram: Application layers around generation

flowchart TB
    User[Client] --> App[Application / API]
    App --> Prompt[Prompt / template version]
    App --> Ret[Optional retrieval / tools]
    Prompt --> Gen[Model inference]
    Ret --> Gen
    Gen --> Val[Validate / guardrails]
    Val --> User
    Gen --> Obs[Logs / cost / eval]

Prompting and retrieval shape inputs; validation and observability make outputs safe to ship.

Important

Prompting changes behavior for a request. Fine-tuning changes weights. Retrieval changes available evidence. Mixing these up leads to the wrong fix for quality problems.

Step-by-Step Flow

Typical text-generation request in production:

1. Authorize and rate-limit the caller; attach a trace ID.
2. Build the prompt from a versioned template + user input (sanitized).
3. Optionally retrieve private documents or call tools for live data.
4. Call the model with pinned model ID, max_tokens, temperature, and optional JSON schema.
5. Validate the output (schema, business rules, safety filters).
6. Retry or fallback on validation failure (bounded attempts).
7. Log prompt version, model ID, tokens, latency, and outcome (redact PII).
8. Return only validated content to the client.

Diagram: Request lifecycle

sequenceDiagram
    participant C as Client
    participant A as App
    participant R as Retrieval
    participant M as Model API
    participant V as Validator

    C->>A: Request
    A->>R: Optional fetch evidence
    R-->>A: Context
    A->>M: Prompt + params
    M-->>A: Raw generation
    A->>V: Validate
    alt valid
        V-->>C: Safe response
    else invalid
        V-->>A: Retry / fallback
    end

Real Production Example

A marketing platform generates personalized email variants with schema validation and bounded retries:

from __future__ import annotations

import json
from dataclasses import dataclass

from openai import OpenAI
from pydantic import BaseModel, Field, ValidationError


class EmailDraft(BaseModel):
    subject: str = Field(max_length=60)
    body: str = Field(max_length=2000)
    cta: str = Field(max_length=30)


@dataclass(frozen=True)
class GenerationConfig:
    model: str = "gpt-4.1-mini"  # pin a concrete version in real configs
    temperature: float = 0.7
    max_tokens: int = 500


class EmailGenerator:
    SYSTEM_PROMPT = (
        "Generate a marketing email as JSON with keys subject, body, cta. "
        "Match the brand voice. Do not invent pricing or discounts."
    )

    def __init__(self, client: OpenAI, config: GenerationConfig):
        self.client = client
        self.config = config

    def generate(self, *, brand_voice: str, product: str, audience: str) -> EmailDraft:
        response = self.client.chat.completions.create(
            model=self.config.model,
            temperature=self.config.temperature,
            max_tokens=self.config.max_tokens,
            response_format={"type": "json_object"},
            messages=[
                {"role": "system", "content": self.SYSTEM_PROMPT},
                {
                    "role": "user",
                    "content": (
                        f"Brand voice: {brand_voice}\n"
                        f"Product: {product}\n"
                        f"Audience: {audience}"
                    ),
                },
            ],
        )
        raw = response.choices[0].message.content or ""
        try:
            return EmailDraft.model_validate(json.loads(raw))
        except (json.JSONDecodeError, ValidationError) as exc:
            raise ValueError(f"Invalid generation output: {exc}") from exc


def generate_with_retries(generator: EmailGenerator, **kwargs) -> EmailDraft:
    last_error: Exception | None = None
    for _ in range(3):
        try:
            return generator.generate(**kwargs)
        except ValueError as exc:
            last_error = exc
    assert last_error is not None
    raise last_error

JSON mode + Pydantic turns a probabilistic sample into a typed contract. For stricter guarantees, prefer provider structured outputs / constrained decoding when available.

Design Decisions

When should I use this?

Comparisons

Common Mistakes

1. Treating output as deterministic. Same prompt, different results. Design retries and validation, not assumptions.
2. No output validation. Generated JSON, code, and URLs can be malformed — validate before downstream use.
3. Ignoring cost. Long outputs and images are expensive. Set max_tokens, cache, monitor per feature.
4. Skipping safety filters. User-facing generation needs moderation for toxicity, PII, and policy.
5. Confusing capability with reliability. Models write code that fails tests and cite papers that do not exist. Verify.
6. Fine-tuning when RAG was needed. Weights do not stay current; documents do. See RAG.
7. Unpinned model versions. Behavior drifts across upgrades. Pin and regression-test.

Warning

Never execute generated code without sandboxing. Models can emit insecure or destructive commands.

Where It Breaks Down

• Factual accuracy — Plausible fiction is the default failure mode. Mitigate with retrieval, tools, and hallucination detection.
• Exact reproduction — Hashes, serial numbers, and precise quotes are unreliable. Use traditional software for deterministic values.
• Long-form coherence — Very long generations drift. Chunk with intermediate checks.
• Freshness — Parametric knowledge is frozen at training cutoffs unless you retrieve or tool-call.
• Copyright / licensing — Training data and outputs raise legal questions for commercial media — involve counsel for high-risk uses.

When NOT to Use Generative AI

Prefer non-generative systems when:

• Outputs must be bit-exact (IDs, totals, cryptographic values)
• A small classifier or rules engine already meets the SLA
• Latency budgets are sub-100ms end-to-end without a specialized stack
• You cannot afford validation, monitoring, or human review for the risk class

Generative AI is a poor default for core banking ledgers, access-control decisions, and anything where a wrong sample is catastrophic without a hard verifier.

Running in Production

Best Practice

Pin model versions, always set max_tokens, validate every machine-consumed output, and block deploys on golden-set regressions.

Decision Trade-off

Higher temperature improves variety but increases format errors and hallucination. Route creative and factual traffic differently.

Production checklist

• Model version pinned; upgrade process documented
• max_tokens set on all generation calls
• Output validation (schema, format, business rules)
• Temperature configured by task type
• Cost monitoring per feature with anomaly alerts
• Content moderation for user-facing surfaces
• Bounded retries on validation failure
• Prompt templates versioned in source control
• Logging with PII redaction
• Eval golden set for regression testing

Related Guides

Inside LLM Concepts:

• Large Language Models — how text generators work end to end
• Tokens · Context Windows — cost and capacity units
• Prompt Engineering · Structured Outputs · Function Calling
• Hallucinations · Fine-tuning · RLHF · DPO

What you will build next:

• RAG — grounded generation with private data
• AI Agents — generation inside tool loops
• Knowledge Graphs — structured knowledge vs parametric memory
• AI System Architecture — platform layers around models

Tools: ChatGPT · Claude · Gemini · LangChain · LlamaIndex

Interview Questions

1. How does generative AI differ from discriminative ML?
   Discriminative models score or label inputs; generative models sample new content from a learned distribution.

2. What is parametric vs retrieved knowledge?
   Parametric lives in weights; retrieved is fetched at inference from indexes, graphs, or APIs.

3. Why is generation non-deterministic?
   Decoding samples from a probability distribution (unless fully greedy and the stack is deterministic).

4. When do you fine-tune vs use RAG?
   Fine-tune for stable style/format; RAG for private or changing facts.

5. Why must generated code be tested?
   Models predict tokens, not execution outcomes — syntax-plausible code can still be wrong or unsafe.

6. What belongs in the application layer around a foundation model?
   Prompt versioning, retrieval/tools, validation, guardrails, cost controls, and evaluation.

Key Takeaways

• Generative AI samples new content from learned distributions — it is not deterministic software.
• Foundation models are adapted via prompting, retrieval/tools, and fine-tuning — not by retraining from scratch in product teams.
• Text/code are autoregressive; images are typically diffusion — different cost and latency profiles.
• Reliability is an application problem: validate, ground, monitor, and evaluate.
• This cluster’s mental model (training vs inference, parametric vs retrieved, capability vs engineering) is reused by RAG, agents, graphs, and architecture guides.

FAQs

What is the difference between generative AI and discriminative AI?

Discriminative models classify or predict labels (spam detection, image classification). Generative models create new samples (text, images) that resemble training data.

Is ChatGPT generative AI?

Yes. ChatGPT is a conversational product over generative LLMs that produce text token by token.

What is a foundation model?

A large pretrained model adaptable to many tasks via prompting, fine-tuning, or retrieval — as opposed to a small model trained for one narrow task.

How is image generation different from text generation?

Text uses autoregressive token prediction. Images primarily use diffusion — iterative denoising conditioned on text embeddings — and are usually more compute-heavy per request.

Do I need to train my own generative model?

Almost never. Use APIs or open-weight models. Train adapters (LoRA) or fine-tune for specific behavior. Full pretraining is a research-lab cost center.

Why does the same prompt give different answers?

Models sample from probability distributions. Temperature > 0 increases randomness. Temperature 0 is more stable but still not a substitute for validation.

Can generative AI replace developers?

It augments developers — boilerplate, tests, drafts. It does not replace architecture, debugging, security review, or system design. Generated code needs human review.

Is generative AI the same as AGI?

No. Generative AI produces content within learned distributions. AGI implies broad human-level competence across domains. Current systems are capable generalists with sharp limitations.

How do I control output format?

Use structured outputs, function calling, or constrained decoding — then validate with Pydantic or JSON Schema.

What are the main risks?

Hallucination, bias, copyright concerns, prompt injection, cost overruns, and harmful content. Mitigate with validation, grounding, guardrails, and monitoring.

References

• OpenAI API Documentation
• Anthropic Documentation
• Google AI for Developers
• Hugging Face Diffusers
• The Illustrated GPT-2 — Jay Alammar

Further Reading

• Attention Is All You Need
• LangChain Documentation
• LlamaIndex Documentation
• Stable Diffusion overview