Production RAG systems need more than relevant retrieval. They must enforce data permissions, remove deleted or revoked content, detect quality regressions, keep embeddings and indexes current, and provide an audit trail that can explain how an answer was produced. These requirements become especially important in enterprise and regulated environments.
Explain prevention, detection, response, and evidence. Cover controls at ingestion and retrieval time, source-of-truth checks, measurable service-level objectives, incident triage, and reproducible audit records.
1. How would you prevent sensitive information from appearing in RAG responses?
Answer: Use defense in depth. Sensitive-data protection must happen before content is indexed, when retrieval is performed, and before an answer is returned. A prompt instruction alone is not a security control.
- Classify at ingestion: identify personal, confidential, regulated, and restricted fields using source metadata and data discovery. Apply approved masking, redaction, tokenization, or exclusion policies.
- Carry access metadata: store tenant, owner, classification, source ACLs, and document version with every indexed document or chunk.
- Authorize retrieval: authenticate the user and filter results using current permissions before retrieved text enters the model context. Enforce row/column-level security for structured data.
- Minimize context: retrieve only the passages and fields needed for the request. Avoid broad context dumps and prevent tools from returning unrestricted raw records.
- Validate output: use policy checks and sensitive-data detectors on generated responses where appropriate. Redact or block outputs that violate policy, while recognizing that output filtering cannot replace retrieval-time authorization.
- Protect logs: avoid storing secrets or full sensitive prompts unnecessarily; use access controls, retention limits, and redaction for telemetry.
- Test adversarially: evaluate direct requests, indirect prompt injection, cross-tenant access, and attempts to infer restricted information.
Example: If a user asks for another employee's salary, the retrieval layer should exclude records the user cannot access. The model should never receive those salary values and then be expected to hide them.
Apply least privilege and authorization before model context construction. Prompt rules and output filters are useful additional safeguards, not substitutes for access control.
2. How would you ensure deleted source documents are no longer retrievable?
Answer: Treat deletion as a lifecycle event that must propagate to every derived system: extracted text, chunks, lexical indexes, vector indexes, caches, and any persisted conversation or answer stores covered by the deletion policy. Removing the original file alone is not sufficient.
- Capture deletion events: use source webhooks, CDC, audit events, or periodic reconciliation to detect deleted documents and access revocations.
- Track lineage: maintain a mapping from source document ID and version to extracted artifacts, chunk IDs, embedding IDs, index entries, and cached results.
- Make deletion idempotent: represent a deletion as a durable event or tombstone and safely retry propagation across all dependent stores.
- Remove or invalidate derived data: delete chunks and vectors, remove lexical entries, invalidate caches, and prevent stale replicas or snapshots from serving the content.
- Block during propagation: where immediate physical deletion is not possible, maintain a denylist/tombstone checked at retrieval time so deleted content is not returned while cleanup completes.
- Verify completion: query each index by source ID and known content fragments, inspect cache invalidation, and record per-system acknowledgements.
- Reconcile periodically: compare source inventory against indexed inventory to discover missed deletion events and orphaned chunks.
Source deletion event
-> durable tombstone / deletion workflow
-> map source ID to all derived artifacts
-> remove lexical + vector entries
-> invalidate caches and dependent stores
-> verify absence and record acknowledgementsImportant nuance: define what “deleted” means for backups, immutable audit logs, retained legal records, and previously generated responses. Follow the applicable retention and legal policy. Do not claim global deletion is complete until all in-scope stores have confirmed it.
3. How would you investigate an AI application's sudden drop in answer quality?
Answer: Treat it as an incident and isolate the failing layer before making changes. Compare the quality drop with recent deployments, source-data changes, index updates, traffic shifts, and model/provider changes.
- Scope the incident: identify when the regression began, affected tenants and query types, severity, and whether safety or access-control failures are involved. Roll back or disable risky changes if needed.
- Check the data path: inspect source ingestion failures, schema changes, missing documents, stale indexes, embedding job errors, and source-to-index lag.
- Evaluate retrieval: sample failed queries and inspect top-k results, Recall@k, ranking, filters, duplicates, and whether relevant passages are present.
- Evaluate generation: inspect context order, truncation, prompt/model version, tool results, citations, output format, and groundedness.
- Check access and freshness: ensure ACL filters are correct and that the system is not retrieving revoked or outdated content.
- Run a fixed regression suite: compare the current version against the last known-good baseline using the same queries and source snapshot where possible.
- Fix and verify: address the root cause, rerun targeted and broad tests, canary the fix, and monitor quality, latency, and error rates.
| Symptom | Likely area to investigate |
|---|---|
| Relevant evidence missing | Ingestion, chunking, embedding model, filters, index freshness |
| Evidence is good, answer is poor | Prompt, model version, context order, truncation, generation settings |
| Only one business domain is affected | Domain source changes, permissions, metadata, or content quality |
| Latency and quality both degrade | Provider throttling, timeouts, fallbacks, retrieval load, context limits |
Interview tip: do not immediately switch embedding models or rewrite the prompt. First reproduce the regression and determine whether the failure is in source data, retrieval, generation, policy, or infrastructure.
4. What metrics would you monitor for an embedding pipeline?
Answer: Monitor pipeline reliability, data quality, embedding correctness, index publication, freshness, cost, and downstream retrieval outcomes. A successful job status alone does not prove the resulting vectors are usable.
| Category | Metrics to track | Why it matters |
|---|---|---|
| Throughput & reliability | Documents/chunks processed, job success rate, retries, dead-letter count | Detect stalled or failing ingestion |
| Freshness | Source-to-embedding lag, oldest unprocessed update, index publish lag | Ensure changed content becomes searchable on time |
| Data quality | Empty text rate, parse failures, chunk count distribution, duplicate rate | Catch extraction or chunking defects |
| Vector integrity | Dimension mismatch, null/NaN vectors, norm distribution, model/version coverage | Detect invalid or inconsistent embeddings |
| Index health | Upsert failures, indexed-vs-expected count, query latency, replica health | Ensure generated vectors are actually searchable |
| Cost & performance | Tokens or characters embedded, cost per document, batch latency, throttling | Control operating cost and meet throughput targets |
| Retrieval outcomes | Recall@k, no-result rate, relevance, duplicate-result rate | Connect pipeline health to user-visible quality |
Break metrics down by source, tenant where appropriate, language, document type, embedding model version, and pipeline stage. Set thresholds based on service-level objectives and historical baselines. Alert on meaningful changes rather than normal daily variation.
5. How would you detect stale vector indexes?
Answer: Compare index contents and timestamps against authoritative source metadata and embedding-job state. A vector database may be available and return results while still containing old, missing, or superseded vectors.
- Track timestamps: store source-modified time, extraction time, embedding time, and index-upsert time for each document or chunk.
- Measure lag: monitor the age of the oldest pending update and the distribution of source-to-index delay, not just the average.
- Reconcile inventories: compare expected active source versions with indexed versions and identify missing, orphaned, or outdated vectors.
- Use content hashes: compare a source checksum with the indexed record's checksum to detect changes that were not re-embedded.
- Check model versions: ensure vectors use the expected embedding model, preprocessing, and dimensionality. Model changes may require a new index rather than an in-place mixed index.
- Run canary queries: test known documents and recently updated content to verify that expected results are retrievable.
- Test deletions: periodically verify that deleted or access-revoked documents no longer appear.
Recovery strategy: retry failed incremental updates, reprocess affected partitions, or build a clean index from a source snapshot if corruption is widespread. For a full rebuild, use a parallel index, validate it, switch traffic safely, and keep the previous index for rollback until the new one is confirmed healthy.
Track source version expected versus source version indexed. Index size alone cannot tell you whether the right content is present.
6. How would you design an auditable data pipeline for a regulated AI application?
Answer: Design for traceability, controlled access, reproducibility, and evidence retention from ingestion through final response. The exact controls depend on the regulation and risk classification, so involve security, privacy, legal, and compliance stakeholders early.
- Define governance: document data owners, classifications, lawful purpose, retention rules, access policies, and approved model/use-case boundaries.
- Preserve lineage: track source system, source record/document ID, source version, ingestion time, transformations, chunk IDs, embedding model/version, index version, and downstream consumers.
- Version code and configuration: retain pipeline code commits, schema versions, prompt/model versions, feature or retrieval configuration, and deployment history.
- Log decisions and actions: record request/trace ID, authenticated principal, policy outcomes, tools called, query IDs, evidence references, model version, and action status.
- Protect audit records: use access-controlled, tamper-evident or immutable storage where required, with explicit retention and deletion rules. Avoid logging unnecessary sensitive payloads or secrets.
- Make runs reproducible: preserve enough metadata to reconstruct the relevant data and configuration state, subject to privacy and retention requirements.
- Monitor controls: track failed access checks, policy overrides, deletion completion, lineage gaps, pipeline failures, and quality or freshness SLOs.
- Test evidence retrieval: periodically select a response and trace it back to source records, transformation versions, permissions, and evaluation results.
Example audit chain: user request ID → authorization decision → SQL query ID or document/chunk IDs → source version → transformation/index version → model and prompt version → cited evidence → final response/action status.
Important trade-off: auditability does not mean retaining all raw prompts and sensitive content forever. Design privacy-preserving logs, access controls, retention schedules, and lawful deletion processes together.
Quick Revision Table
| Topic | Key takeaway |
|---|---|
| Sensitive information | Classify at ingestion, authorize before retrieval, minimize context, and validate outputs. |
| Document deletion | Propagate tombstones to chunks, vectors, lexical indexes, caches, and dependent stores; verify absence. |
| Quality incident | Scope, reproduce, isolate ingestion/retrieval/generation/policy layers, fix, and verify. |
| Embedding monitoring | Track reliability, freshness, input quality, vector integrity, index health, cost, and retrieval outcomes. |
| Stale vector index | Compare source versions/checksums with indexed versions and validate through canary queries. |
| Auditable pipeline | Preserve lineage, versioned configuration, authorization decisions, evidence references, and controlled logs. |
Scenario-Based Follow-up Questions
Practise explaining your incident response and control design aloud.
Scenario 1: A confidential document appears in an unauthorized user's answer. What do you do first?
Treat it as a security incident: stop or constrain the affected retrieval path, preserve appropriate audit evidence, identify the source and scope, and prevent further exposure. Review identity propagation, ACL filters, caches, and logs; notify the designated security/privacy response team and follow incident policy. Fix the control and test cross-tenant and permission-boundary cases before restoring service.
Scenario 2: The source document is deleted, but its text is still returned from search. How do you resolve it?
Use source-to-artifact lineage to identify chunks, vector IDs, lexical entries, and caches. Apply a retrieval-time tombstone or deny rule immediately, then propagate deletion to all stores and verify absence with targeted queries and reconciliation.
Scenario 3: Embedding jobs are green, but retrieval quality is falling. What would you inspect?
Check source-to-index lag, content extraction and chunk counts, vector dimensions/model versions, upsert failures, metadata filters, and known-query canaries. Then run the fixed retrieval evaluation set to determine whether the issue is stale data, changed content, ranking, or a model/configuration mismatch.
Scenario 4: Auditors ask you to reconstruct why an AI answer was produced six months ago. What must be available?
Retain the request/trace ID, principal and authorization decision, source/query/document versions, transformation and index versions, model and prompt versions, retrieved evidence references, deployment configuration, and response/action metadata. Ensure retention is consistent with applicable privacy and regulatory requirements.
Chapter Summary
Production RAG reliability depends on controlling what enters the model context, propagating updates and deletions through every derived store, monitoring both pipeline health and retrieval outcomes, and preserving enough provenance to investigate incidents. Strong systems make security, freshness, and auditability measurable—not merely documented.
Practice challenge: Design a regulated RAG pipeline that indexes confidential documents, supports immediate access revocation and deletion, and can explain the evidence behind every answer. Define your lineage model, security checks, embedding metrics, stale-index detection, incident workflow, and audit-retention policy.
Comments
Post a Comment