Key Concepts & Terminology

Before diving deeper into Generative AI, let's establish a clear understanding of the essential terms and concepts you'll encounter throughout this course.

Core Terminology

Tokens

The fundamental units that LLMs process. Text is broken into tokens (subwords) before processing.

• "Hello world" → ["Hello", " world"] (2 tokens)
• "unbelievable" → ["un", "believ", "able"] (3 tokens)
• Rule of thumb: 1 token ≈ 4 characters in English

Prompt

The input text you send to an LLM. Prompts can be simple questions, complex instructions, or multi-turn conversations.

Simple prompt:    "What is photosynthesis?"
Complex prompt:   "You are a biology tutor. Explain photosynthesis
                   to a 10-year-old using an analogy. Keep it under
                   100 words."

Completion / Response

The text generated by the model in response to your prompt.

Context Window

The maximum number of tokens (input + output) a model can handle in a single interaction. Think of it as the model's "working memory."

Inference

The process of generating output from a trained model. When you send a prompt and get a response, that's inference.

Model Parameters

Temperature

Controls randomness. Lower values = more deterministic; higher values = more creative.

# Deterministic (factual tasks)
temperature = 0.0  # Always picks the most likely token

# Balanced (general use)
temperature = 0.7  # Good mix of accuracy and variety

# Creative (brainstorming)
temperature = 1.0  # More diverse and unexpected outputs

Max Tokens

The maximum number of tokens in the model's response. Controls output length.

Top-p (Nucleus Sampling)

An alternative to temperature. Considers tokens whose cumulative probability reaches p.

• top_p = 0.1: Only considers the most likely tokens (focused)
• top_p = 0.9: Considers a wider range (more diverse)

Stop Sequences

Tokens or strings that signal the model to stop generating. Useful for controlling output format.

Training Concepts

Pre-training

The initial training phase where a model learns language patterns from massive datasets. This produces a base model with general knowledge.

Fine-tuning

Additional training on specific data to customize a model for particular tasks or domains.

RLHF (Reinforcement Learning from Human Feedback)

A technique where human preferences are used to align model behavior — making it more helpful, safe, and accurate.

Foundation Model

A large pre-trained model (like GPT-4 or Claude) that serves as a base for various applications. Also called a "base model."

Architecture Concepts

Transformer

The neural network architecture powering modern LLMs. Uses self-attention to process all tokens in parallel.

Attention / Self-Attention

The mechanism that allows each token to consider relationships with every other token in the input.

Parameters

The learned weights in a neural network. More parameters generally means more capability:

• Small: ~7B parameters (Llama 3 8B)
• Medium: ~70B parameters (Llama 3 70B)
• Large: ~1.7T parameters (GPT-4, estimated)

Embeddings

Numerical vector representations of text. Words with similar meanings have similar embeddings.

"king"  → [0.52, -0.31, 0.89, ...]
"queen" → [0.51, -0.29, 0.87, ...]  ← Similar vectors!
"car"   → [-0.12, 0.76, 0.15, ...]  ← Very different

API & Integration Concepts

API (Application Programming Interface)

A way to programmatically interact with LLMs. You send requests and receive responses via HTTP.

System Prompt

A special instruction that sets the model's behavior, persona, or rules for the entire conversation.

system_prompt = "You are a helpful coding assistant. Always provide
                 code examples in Python. Be concise."

Messages / Chat Format

Modern LLMs use a structured conversation format:

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is Python?"},
    {"role": "assistant", "content": "Python is a programming language..."},
    {"role": "user", "content": "Show me a hello world example."}
]

Streaming

Receiving the model's response token by token in real-time, rather than waiting for the complete response.

Rate Limits

Restrictions on how many API requests you can make per minute/day. Varies by provider and plan.

Advanced Concepts

Hallucination

When a model generates plausible-sounding but factually incorrect information. A major challenge in GenAI applications.

Grounding

Connecting model outputs to verified information sources (documents, databases, APIs) to reduce hallucinations.

RAG (Retrieval-Augmented Generation)

A technique that retrieves relevant documents and includes them in the prompt, giving the model accurate context.

Fine-Tuning vs RAG

Function Calling / Tool Use

The ability for an LLM to invoke external functions or APIs — checking weather, querying databases, running code.

AI Agent

An AI system that can autonomously plan, reason, and take actions using tools to accomplish goals.

Prompt Injection

A security concern where malicious input tries to override the model's instructions. Important to defend against in production.

Guardrails

Safety measures that constrain model behavior — content filters, output validation, topic restrictions.

Cost & Performance Metrics

Tokens Per Second (TPS)

How fast a model generates output. Important for user experience.

Latency

Time between sending a request and receiving the first token (Time to First Token, or TTFT).

Cost Per Token

Pricing model for API usage. Typically measured in dollars per million tokens.

Example (approximate):
GPT-4:    $30 / 1M input tokens,  $60 / 1M output tokens
Claude:   $15 / 1M input tokens,  $75 / 1M output tokens
GPT-3.5:  $0.50 / 1M input tokens, $1.50 / 1M output tokens

Context Length vs Cost

Longer inputs = more tokens = higher cost. Efficient prompt design saves money.

Quick Reference Card

With these concepts in your toolkit, you're ready to start building with Generative AI. The next module covers the art and science of prompt engineering.