From RAG demo to a service you can operate: a Saudi architecture review

A retrieval-augmented generation demo usually proves one narrow fact: a selected model can answer selected questions from a selected document set under controlled conditions. That is useful, but it is not evidence that the service can be trusted with a changing corpus, real identities, mixed Arabic and English material, or an incident at 02:00. Curated success says little about whether a withdrawn procedure still appears in an answer, whether a user can retrieve a document they cannot open, or whether an operator can reconstruct the exact source fragments behind a response.

The architecture review should therefore begin with a harder thesis. Demo quality does not prove document authorization, withdrawal and version handling, Arabic-English retrieval, citation traceability, safe failure, or operational ownership. Each of those properties needs an explicit control, a test, an owner, and release evidence. A fluent answer is only the visible end of that chain.

Saudi public material provides reasons to ask disciplined questions about data management, responsible use, Arabic-language technology, privacy review, and data cybersecurity. It does not select a vector database, prescribe a hosting topology, or approve a RAG design. The SDAIA About page, National Strategy for Data & AI, NDMO page, Data Protection overview, Laws and Regulations index, and NCA Data Cybersecurity Controls page are bounded review inputs. Qualified customer legal, privacy, cybersecurity, records, and sector reviewers must decide which current requirements apply to the actual organization and workload.

Draw the service boundary before discussing models

Start with the request path and the failure path, not a vendor diagram. Name the identity provider, source repositories, ingestion workers, extraction services, indexes, policy decision point, retrieval API, generation endpoint, citation store, logs, evaluation jobs, operator console, and escalation channel. Mark which components process source text, derived chunks, prompts, responses, user identity, authorization attributes, and feedback. Then record where each category is stored, transmitted, cached, backed up, and deleted.

This boundary exposes questions that a demo hides. Can an ingestion worker read a repository more broadly than the end user? Does the vector index preserve document-level entitlements, or only text and embeddings? Can an operator remove one source version without rebuilding unrelated material? Are prompts or responses retained by a model endpoint? Does the citation record survive long enough to support an investigation? What happens when the identity service, policy service, index, or source repository is unavailable?

Treat every source as a versioned, withdrawable record

The ingestion contract needs more than a connector and a schedule. For each source, record a stable source identifier, repository, owner, classification supplied by the customer, approved purpose, language, authoritative status, effective date, version, checksum, entitlement reference, retention instruction, and ingestion state. Keep the original location and extraction result linked. A chunk without provenance is an orphaned claim waiting to surface in an answer.

Changes require explicit state transitions. New versions should not silently coexist with superseded text unless the use case requires historical comparison. A withdrawal must invalidate affected chunks, retrieval caches, answer caches, and any precomputed summaries. Deletion from the source repository should enter a reconciliation queue rather than depend on the next full crawl. Failed withdrawal propagation should page an owner because stale authoritative content is a service fault, not housekeeping.

Test this lifecycle with a controlled canary document. Publish version one, retrieve it, replace it with version two, verify version one no longer answers current questions, withdraw the source, and prove that neither direct retrieval nor a cached response exposes it. Record timestamps for each stage, but do not publish a universal timing target; the acceptable window belongs to the customer’s risk and operating requirements.

Separate extraction quality from language behavior

Arabic and English are not a single “multilingual” checkbox. Document structure can fail before retrieval begins: scanned pages, embedded images, right-to-left ordering, tables, headers, footnotes, Arabic-Indic digits, diacritics, ligatures, mixed scripts, transliterated names, and bilingual page layouts all change what reaches the index. OCR confidence alone does not show that a table row stayed attached to its heading or that a footnote remained with the relevant clause.

Build an extraction review set by document family and layout, using material the customer has approved for this purpose. Compare the extracted text and structure with the source page. Flag missing regions, reordered blocks, broken tables, uncertain fields, and language detection changes. Route low-confidence or structurally ambiguous pages to a defined review path instead of indexing convenient fragments.

Retrieval needs a separate test matrix. Include Arabic queries against Arabic sources, English against English, Arabic against English, English against Arabic, and mixed-language queries against mixed corpora where the workflow expects those combinations. Test terminology variants, abbreviations, spelling variation, transliteration, named entities, numerals, and questions whose answer should be “not found.” Segment results by language direction and document type; an aggregate score can conceal a broken Arabic path behind a larger English set.

Arabic, English, and mixed-language behavior must be measured on approved customer material with appropriate native-language review. No public strategy page or generic benchmark substantiates a model-quality claim for the customer’s documents, terminology, or tasks. Thresholds should be set against the consequences of a miss, not copied from a demonstration dataset.

Enforce entitlement before generation

Permission filtering after an answer is generated is too late. The service should resolve the authenticated subject, relevant roles or attributes, document entitlement, version state, and purpose constraint before eligible chunks enter retrieval. The generator must receive only material authorized for that request. A response filter cannot reliably remove information already combined into generated text.

Define fail-closed behavior for missing identity claims, stale policy data, policy-service timeouts, unclassified documents, and unknown entitlement mappings. The safe result may be no answer, a request to re-authenticate, or escalation to an authorized operator. It should not be an unfiltered search “for availability.” Keep the denial reason in controlled telemetry without exposing sensitive repository details to the requester.

Authorization tests should use contrasting identities and documents, not only a privileged test account. Verify allowed access, denied access, group changes, revoked users, document reclassification, superseded versions, and cross-session cache isolation. Include negative tests that ask the model to reveal restricted material indirectly, summarize another user’s result, or follow instructions embedded in a document. The policy decision and the retrieved document identifiers need a shared request trace so an investigator can reconstruct what was permitted.

Make citations an evidence chain

A citation link beside generated prose is not enough. Preserve a mapping from each supported answer span to the retrieved chunk, source identifier, source version, page or section locator, extraction version, retrieval configuration, and request trace. The user-facing citation should open an authorized source view or provide a locator the user can verify. If the source cannot be shown to that user, the service should not use it to compose the answer.

Prefer a bounded refusal when evidence is absent, conflicting, withdrawn, or outside the user’s authorization. The refusal should distinguish “no approved source found” from a system fault where appropriate, while avoiding disclosure about restricted sources. Human escalation needs the request, authorized evidence set, failure reason, and accountable queue—not just a transcript pasted into a chat channel.

Evaluate releases as changes to a service

Keep a versioned evaluation set tied to approved tasks, document families, language paths, entitlement cases, citation cases, and expected refusals. Separate extraction, retrieval, citation, generation, authorization, latency, and operator-workflow results. This allows a team to identify whether a regression came from OCR, chunking, metadata, embedding, ranking, prompting, policy data, or a model change.

Run the relevant slices when any dependent component changes: source parser, OCR service, chunking rules, metadata schema, embedding model, index settings, reranker, prompt, model endpoint, policy integration, or citation renderer. Record configuration and dataset versions with the result. A release decision should show failures and exclusions, not only a headline average.

Monitor live service signals that correspond to reviewable failure modes: ingestion lag, quarantined documents, withdrawal propagation, policy failures, zero-result rate, citation resolution failures, refusal categories, escalation backlog, configuration changes, and evaluation regressions. Usage volume alone does not show that answers remain authorized or supportable.

Assign incidents, rollback, and operating ownership

Before release, write the incident paths for unauthorized retrieval, stale or withdrawn content, incorrect citation, extraction corruption, harmful instruction in a source, unavailable dependencies, and suspected data exposure. Each path needs a triage owner, severity decision, containment action, evidence location, customer notification owner, reviewer involvement, and recovery authority. Legal or regulatory notification decisions are not engineering defaults; route them to qualified customer reviewers.

Rollback must cover more than the model version. Operators may need to disable a source, freeze ingestion, revert a parser, restore an index snapshot, invalidate caches, roll back policy data, disable generation while preserving authorized search, or shut the service boundary. Exercise those controls in a non-production environment and keep access to them narrow and audited.

Ownership should survive team changes and supplier handoffs. Name who approves sources, owns entitlement data, reviews Arabic, reviews English, accepts evaluation results, operates on call, investigates citation failures, authorizes rollback, and decides when service can resume. If one unnamed “AI team” owns every row, the operating model is unfinished.

Release-gate table

The table is a decision record, not a guarantee. Evidence should link to customer-controlled artifacts and named approvals.

On narrow screens, scroll horizontally to inspect every column.

Meeting-ready review checklist

Bring this list to the architecture and release meeting. A “not applicable” answer still needs a named reviewer and rationale.

• The system boundary identifies every source, store, model endpoint, policy decision, cache, log, evaluation job, and human queue.
• The source register records owner, authority status, version, effective date, entitlement reference, classification supplied by the customer, and withdrawal state.
• Version replacement and withdrawal have been exercised through retrieval, generation, citations, and caches.
• Quarantined or failed documents cannot enter the searchable corpus.
• Extraction tests cover the approved page layouts, scans, tables, right-to-left structures, and bilingual documents relevant to the workflow.
• Arabic, English, cross-language, and mixed-language retrieval slices use approved customer material and named native review where required.
• Entitlement is resolved before retrieval results reach generation, with tested fail-closed behavior for missing or stale policy data.
• Contrasting identities prove allow, deny, revocation, reclassification, and cache-isolation cases.
• Answer claims map to authorized source versions and locators that the requester can inspect.
• Expected refusals, conflicting evidence, missing evidence, and dependency failures have distinct tested paths.
• Evaluation results identify component versions, dataset versions, failed cases, exclusions, and accepted exceptions.
• Operators can disable a source, invalidate affected caches, restore an index, restrict generation, and invoke human escalation.
• Incident ownership, rollback authority, resumption authority, and evidence retention are assigned by role and person.
• Qualified customer legal, privacy, cybersecurity, records, and sector reviewers have decided applicability and recorded their decisions.

A service is ready for release only when the people accepting these risks can inspect the evidence and operate the controls. The model may be the least difficult part. Source authority, identity, language evidence, traceability, and ownership determine whether the answer can be used—and whether the team can recover when it should not have been produced.