30 Generative AI Interview Questions and Answers (2026): LLMs, RAG and AI Agents

Generative AI interviews in 2026 have moved on from "what is ChatGPT?". Interviewers for AI engineer and GenAI developer roles now probe how you build, evaluate and secure LLM systems. Expect at least one system design question and several "what would you do when..." follow-ups.

Here are 30 questions we see often, with answers you can build on. Do not memorise them. Understand them well enough to explain in your own words, with an example from a project you built.

LLM fundamentals

1. What is a token, and why does it matter?

A token is a chunk of text the model processes, often a word piece. Models price, limit and process input in tokens, so token counts drive cost, latency and whether your input fits in the context window. Indian languages and code can use more tokens per word than English, which affects cost.

2. What is a context window?

The maximum number of tokens a model can consider in one request: instructions, history, retrieved documents and its own output. Larger windows allow more context but cost more per request and can still lose focus on details buried in the middle.

3. What does temperature control?

Randomness in token selection. Lower values make output more predictable; higher values make it more varied. Some newer reasoning models fix or restrict sampling parameters and are steered through instructions and other settings instead, so check the model's documentation.

4. Why do LLMs hallucinate?

They generate plausible text based on patterns, not by looking up facts. When the right information is missing from training data or context, the most plausible continuation can be wrong. Mitigations: grounding with RAG, asking the model to cite sources, allowing "I don't know", and verifying outputs in code where possible.

5. Hosted API models vs open-weight models: how do you choose?

Compare quality on your task, latency, cost at your volume, data privacy and residency requirements, and operational effort. Hosted APIs are fastest to start and usually strongest; open-weight models give more control and can run on your own infrastructure, at the cost of running it.

6. What is structured output, and why use it?

Constraining the model to return data matching a schema (such as JSON with specific fields). It turns free text into something your code can rely on, which is essential for extraction, tool calling and pipelines.

Prompting and context

7. What makes a good system prompt?

Clear role and goal, the rules that matter, the output format, examples of tricky cases, and what to do when information is missing. Good prompts explain why a rule exists, which helps the model generalise.

8. What is few-shot prompting?

Including a few input/output examples in the prompt to show the model the pattern you want. Useful for format and edge cases; keep examples varied so the model does not copy one too literally.

9. What is "context engineering"?

Deciding what information the model sees at each step: which instructions, which retrieved documents, how much history, which tool results. In agents, this often matters more than the wording of the prompt.

Embeddings and RAG

10. What are embeddings?

Numeric vectors that represent meaning. Texts with similar meanings have vectors that are close together, which enables semantic search.

11. Explain RAG end to end.

Index: load documents, chunk them, embed the chunks, store them in a vector database with metadata. Query: embed the question, retrieve the most similar chunks, put them in the prompt, and have the model answer with citations. (Full walkthrough: What is RAG?)

12. How do you choose a chunk size?

Small enough that a chunk is about one idea, large enough to keep context. Split on natural boundaries such as headings and paragraphs, add a small overlap, and store the section title with each chunk. Then measure retrieval recall with a test set rather than guessing.

13. What is hybrid search, and when do you need it?

Combining keyword search (like BM25) with vector search. You need it when queries include exact terms (product codes, error numbers, names) that embeddings handle poorly.

14. What is a reranker?

A model that re-scores retrieved candidates for relevance to the query. Retrieve 20 to 50 candidates cheaply, rerank, then pass the top few to the LLM. It often gives a large quality gain.

15. How do you evaluate a RAG system?

Separately measure retrieval (did the right passage appear in the top k?) and generation (is the answer correct, complete and supported by the cited sources?). Use a fixed test set of real questions, run it on every change, and look at failures by hand.

16. RAG vs fine-tuning: when would you use each?

RAG for knowledge that changes or needs citations. Fine-tuning for consistent behaviour, style or format that prompting cannot achieve. They can be combined.

Agents, tools and MCP

17. What is an AI agent?

A system where an LLM decides on actions, typically tool calls, observes the results, and continues until a goal is met. The key difference from a fixed workflow is that the model chooses the path.

18. When should you not build an agent?

When the steps are known in advance. A fixed workflow with one or two LLM calls is cheaper, faster, more predictable and easier to test. Use agents when the path genuinely depends on intermediate results.

19. How does tool calling work?

You send tool definitions (name, description, input schema) with the request. The model responds with a tool name and arguments instead of a final answer. Your code runs the tool, returns the result, and the model continues.

20. How do you design good tools?

Narrow, clearly described, typed and validated. Prefer get_order_status(order_id) to run_sql(query). Return helpful error messages so the model can recover. Enforce permissions and limits in code, not in the prompt.

21. What is MCP?

The Model Context Protocol, an open standard for connecting AI applications to tools and data. An MCP server exposes tools, resources and prompts once; any MCP-compatible client can use them. (Explained with code)

22. What does LangGraph add over a simple agent loop?

Explicit state, nodes and edges; checkpointing for memory and resumption; interrupts for human-in-the-loop; and better control and debugging for branching, multi-step workflows. (Tutorial)

23. How do you add a human in the loop?

Pause before risky actions (refunds, deletions, external emails), show the reviewer the proposed action, and resume with their decision. Put the gate where no path can bypass it, ideally inside the tool.

24. Single agent or multiple agents?

Start with one agent and good tools. Add multiple agents only when tasks are clearly separable and need different tools or context. Multi-agent systems add cost, latency and failure modes.

Evaluation, security and operations

25. How do you evaluate an LLM application?

Build a dataset of realistic inputs with expected outcomes, including edge cases and adversarial inputs. Grade with code checks where possible, LLM-as-judge where necessary (validated against human labels), and human review for a sample. Run evals on every prompt, model or code change.

26. What is prompt injection?

Instructions hidden in content the model reads (a user message, a document, an email, a web page) that try to override its intended behaviour. Indirect prompt injection, through retrieved or tool-returned content, is especially dangerous for agents.

27. How do you defend against prompt injection?

There is no single fix, so layer defences: least-privilege tools, human approval for risky actions, treating external content as data, validating outputs before acting on them, isolating untrusted content, and monitoring. Refer to the OWASP Top 10 for LLM Applications.

28. How do you control cost and latency?

Measure cost per completed task. Use prompt caching for repeated context, smaller or faster models for simple steps, shorter context, fewer agent steps, streaming for perceived speed, and limits on retries and steps.

29. What do you monitor in production?

Traces of every model and tool call, latency, cost, error and refusal rates, user feedback, and quality metrics sampled from real traffic. Alert on regressions and spikes.

System design

30. Design an AI assistant that answers employee questions from internal policies.

A strong answer covers:

• Ingestion: connectors to the document sources, chunking, embedding, scheduled re-indexing, deletion handling
• Access control: filter retrieval by the employee's permissions using document metadata
• Retrieval: hybrid search plus reranking
• Generation: grounded answers with citations; "I don't know" when sources are missing
• Safety: injection-resistant prompts, no tools with write access, personal-data handling under the DPDP Act
• Evaluation: a test set of real questions, measured before every release
• Operations: tracing, feedback buttons, cost and latency budgets
• Rollout: pilot with one department, measure, then expand

Talk through trade-offs as you go. Interviewers value reasoning more than buzzwords.

How to prepare

1. Build two projects: a RAG system with an evaluation report, and a tool-using agent with a human approval step
2. Be ready to discuss failures: what broke, how you found it, how you fixed it
3. Practise one system design out loud, timed at 30 minutes
4. Follow a structured path: our agentic AI roadmap covers every topic above