---
title: SharePoint to Bedrock without leaking a single document: a reference ingestion architecture for a claims AI assistant
seoTitle: SharePoint to AWS Bedrock: permission-aware RAG ingestion
subtitle: We designed this as a proof of concept for one of Australia's largest general insurers. It did not go ahead, so here it is: half a million documents, five processing lanes on AWS in Sydney, and one rule above all others, that nobody gets an answer from a document they are not allowed to open. Then we ask executives whether they would fund it, and architects where it is wrong.
description: A proof-of-concept design we proposed to one of Australia's largest general insurers, published as a reference architecture. It keeps a permission-aware AI search index in step with about 500,000 SharePoint documents on AWS in Sydney, with permissions in their own fast lane and a nightly reconciliation that proves there is no drift. We ask CXOs whether they would fund it and architects where it is wrong. Comments are open.
searchDescription: A reference architecture for permission-aware RAG ingestion from SharePoint Online to AWS Bedrock in Sydney: five lanes, fail-closed access and zero drift.
author: Nick Forshteyn
authorType: Person
published: 2026-11-01
tags: rag, retrieval augmented generation, sharepoint, microsoft graph, aws, amazon bedrock, opensearch, aurora postgresql, ai security, access control, insurance, cps 234
image: /intel/analysis-sharepoint-bedrock-ingestion-og.jpg
imageSize: 1280x720
video: xa6BTQpXKdw
videoTitle: Claims Knowledge Assistant: Ingestion Architecture on AWS (Animated Walkthrough)
---

Most AI assistant demos skip the hard part. Pointing a model at a folder of PDFs is easy. Keeping half a million documents, and the permissions on every one of them, in step with the system people actually work in is not. Get it wrong and the assistant quotes an endorsement that was withdrawn last year, or worse, answers a question from an assessor report the person asking was never allowed to see.

We designed this as a proof-of-concept proposal for one of Australia's largest general insurers, and talked it through with their data science team. It did not go ahead. The problem has not gone away, though, for them or for any insurer of that size, so we are publishing the design as an example of how it could be done.

The scale is our estimate, not a figure the insurer gave us: about 500,000 documents for an insurer of that size, once you count every policy wording, PDS and endorsement across its brands, plus the claims procedures, repair guidance, assessor reports and regulatory documents. Claims consultants ask questions in plain English and get cited answers drawn from all of it. The content lives in SharePoint Online. The design is AWS-native and runs entirely in `ap-southeast-2` (Sydney).

The video above walks through it in a few minutes. Every diagram below opens full screen: press **Expand** or click the diagram. At the end we ask two sets of questions: one for executives deciding whether to fund something like this, and one for the architects who would build it. The comments are open.

## The short version

- **SharePoint stays the source of truth** for content and for permissions. Nothing is authored in the AI platform.
- **Changes are detected** through Microsoft Graph webhooks, with delta queries as the authoritative change feed and a 15-minute poll as a safety net.
- **Permissions and content travel in separate lanes.** A permission change is enforced in minutes, even when a content re-index would take hours.
- **Each document goes through a durable, idempotent workflow:** fetch, OCR or parse, normalise, chunk, embed and index. It resumes after a failure and never processes the same version twice.
- **The initial load of about 500,000 documents** runs as a controlled, rate-limited backfill that is safe to rerun.
- **A nightly reconciliation** compares SharePoint, the registry and the index, and repairs any drift.

Aurora PostgreSQL holds the document registry, the access lists, the relationships between documents and all workflow state. OpenSearch is a derived index that can be thrown away and rebuilt from Aurora and S3 at any time. There is no DynamoDB and no Databricks.

![Architecture diagram: Microsoft 365 on the left feeds an AWS account in Sydney. Five lanes run left to right: library sync, the permission fast lane, per-document ingest, the 500k backfill and nightly reconciliation. Data stores on the right: S3 raw and parsed, Aurora PostgreSQL, OpenSearch and a phase-2 Neo4j projection. Observability runs along the bottom.](/intel/analysis-ingestion-architecture.png)
The whole ingestion platform. The red line is the permission fast lane; it never waits behind content.

## The principles that drive every choice

| Principle | What it means in this design |
|---|---|
| SharePoint is the source of truth | Every indexed item traces back to a SharePoint item, version and permission set. |
| Aurora is the source of truth for derived state | Registry, access lists, relationships, sync cursors and job state live in one relational store. OpenSearch and Neo4j are projections that can be rebuilt. |
| Access control lives in the data, not the model | Every chunk carries its access principals and business attributes, so the query path filters before anything reaches the LLM. |
| Fail closed | Unresolvable permissions, "Anyone" sharing links, missing business metadata and low OCR confidence all end in restricted or excluded content, never open content. |
| Permissions are faster than content | Permission changes have their own queue and their own lane, and never wait behind content processing. |
| Idempotent everywhere | Every document version is keyed by `drive_item_id + content_hash`. Retries, duplicate webhooks and backfill reruns cannot create duplicates. |
| Durable execution | A failure at step 6 resumes at step 6. It does not repeat OCR or embedding. |
| Evidence by default | Every ingest, permission update and reconciliation run leaves a queryable record for audit, AFCA disputes and CPS 234 evidence. |

The fourth row is the one we would defend hardest. An assistant that is sometimes wrong is a quality problem. An assistant that is sometimes wrong about who may see what is a privacy breach.

## Where the content comes from

**SharePoint Online** holds the whole knowledge estate across claims sites and libraries. Library columns and content types carry the business metadata the assistant depends on: brand, product, jurisdiction, document type, version, effective dates and status. Content owners keep working in the tool they already use, with no second authoring system and no export process.

Every artefact downstream, from the raw file to the parsed text, the chunk, the vector and the graph edge, is keyed back to the SharePoint `drive_item_id` and version. Citations deep-link to the SharePoint item, so a consultant can always open the original.

Two prerequisites decide whether this works in practice:

- **Metadata has to be tagged consistently.** Retrieval is scoped by brand, product, jurisdiction and effective date, so a wrongly tagged PDS gives a confident answer from the wrong product. Untagged content is classified automatically but not served until the classification is confident or a person has reviewed it.
- **Permission hygiene has to be good enough.** Broken inheritance and broad sharing links are common in SharePoint. Because of the fail-closed rules they show up as restricted content, and content owners have to fix them.

**Microsoft Graph** is the only interface the platform uses to read SharePoint. It uses four capabilities:

| Capability | Endpoint | Used for |
|---|---|---|
| Delta query | `/drives/{drive-id}/root/delta` | The authoritative list of what changed since the last saved delta token |
| Permissions | `/drives/{drive-id}/items/{item-id}/permissions` | Direct and inherited permissions per item |
| Change notifications | `/subscriptions` | A push signal that something in a library changed |
| Content download | `/drives/{drive-id}/items/{item-id}/content` | Streaming the file into S3 |

Delta queries carry the header `Prefer: deltashowsharingchanges, hierarchicalsharing`, so sharing and permission changes arrive in the same feed as content changes. A notification only says that a library changed, not what changed, so the platform always follows it with a delta query. Correctness never depends on a webhook arriving.

**Entra ID** authenticates the platform with the `Sites.Selected` application permission, granted read access to the claims sites only, not the whole tenant. The credential is a certificate in AWS Secrets Manager, not a client secret. If the ingestion platform were compromised, the attacker would get read access to the claims sites and nothing else.

Chunks store **principal IDs** (Entra groups, SharePoint groups, and individual users only where unavoidable), not expanded lists of users. The query path resolves the consultant's groups at request time, so removing someone from a group revokes their access straight away, with no re-indexing.

## The front door

**API Gateway with AWS WAF** is the only internet-facing component. It receives Graph change notifications and the subscription validation handshake, with WAF restricting sources and rate-limiting before any compute runs.

**EventBridge Scheduler** drives everything time-based:

| Schedule | Target | Purpose |
|---|---|---|
| Every 15 minutes | Library sync, per library | Safety poll in case notifications were missed |
| Daily | Subscription renewal | Renew Graph subscriptions well before they expire |
| Nightly, 02:00 AEST | Reconciliation | Full drift detection and repair |
| One-off or manual | Backfill | Initial load and planned re-indexes |

Webhooks bring latency down. Schedules guarantee correctness, so the platform converges on the right state even if every webhook is lost.

## Lane 1: library sync

The webhook receiver is a thin Lambda. It answers the Graph validation handshake, checks `clientState`, maps the subscription to a library, starts the library sync asynchronously and returns within 3 seconds. Graph expects a fast answer, and a slow one leads to retry storms and disabled subscriptions.

The library sync does the real work:

- It takes a per-library advisory lock in Aurora, so only one sync per library runs at a time.
- It reads the saved delta token, calls the Graph delta query and pages through the results.
- It classifies each change: new or changed content goes to the `content-changes` queue, a sharing or permission change goes to the `acl-changes` queue, a deletion becomes a delete instruction in the outbox, and a metadata-only change updates fields without re-parsing.
- It saves the new delta token **only after** every message is queued and committed, so a crash cannot lose a change.

This is the fork in the road. Getting the classification right here is what lets a permission change overtake a content backlog.

A separate Lambda renews each Graph subscription well before it expires, and recreates any that are missing and flags the library for an immediate sync. If renewal fails completely, the 15-minute poll and nightly reconciliation still keep the index correct. Only latency suffers.

## Lane 2: the permission fast lane

The target is a SharePoint permission change enforced in the index in **under 15 minutes**, whatever else is happening.

Permission changes have their own SQS queue, their own consumers and their own dead-letter queue. A 500,000-document backfill or an OCR backlog cannot delay an access revocation, and the age of the oldest message in that queue is a direct, alarmable measure of how far enforcement is behind.

The ACL updater fetches the item's current permissions from Graph, resolves them into principal IDs, and applies the fail-closed rules:

| Condition | Outcome |
|---|---|
| "Anyone" (anonymous) sharing link | Restricted to the admin group, flagged to the content owner |
| Organisation-wide sharing link | Restricted by default, flagged |
| Inheritance cannot be resolved | Restricted, flagged |
| Permissions unreadable, or Graph errors after retries | Restricted, flagged |

Then, in **one Aurora transaction**, it updates the access list and writes an outbox row in the `acl` lane. That transactional outbox is what stops the registry and the index from drifting apart silently: if the transaction commits, the index update is guaranteed to follow. A permission change never needs a re-parse or a re-embed. The outbox worker patches the access fields on every affected chunk in OpenSearch.

## Lane 3: ingesting one document

Each message on the content queue starts one `DocumentIngest` run, written as a Lambda durable function. Every step is checkpointed, so a completed step is never run again on replay, and each step has its own retry policy.

| Step | What happens |
|---|---|
| 1. Claim | Insert a job keyed on `(drive_item_id, content_hash)`. If that version is already indexed, stop. |
| 2. Fetch | Stream the file from Graph into the S3 raw bucket, encrypted with KMS, and read its metadata. |
| 3. Route pages | PyMuPDF checks each page for a real text layer. Mixed PDFs are split by page. |
| 4. OCR or parse | Scanned pages go to Textract. The function suspends until Textract finishes and costs nothing while it waits. Digital pages go to Docling on Fargate. |
| 5. Normalise | Merge both outputs into one schema (headings, paragraphs, tables, key-value pairs, page, bounding box, OCR confidence) and cache it in S3 by hash. |
| 6. Chunk | Split by clause for wordings and PDSs, by step for procedures and by task for repair guidance. Add a header (title, section path, clause), business metadata and access principals. Extract relationship edges. |
| 7. Embed | Embed only chunks whose hash changed, with Bedrock, and record the model version. |
| 8. Commit | One Aurora transaction: registry, chunks, relationships and an outbox row in the `content` lane. |

![Flowchart of the DocumentIngest durable function: a message arrives, deletes branch off to the outbox, versions already indexed are skipped, documents with missing metadata are classified by Bedrock, pages are routed to Textract or Docling, the output is normalised, low-confidence OCR is flagged, then chunk, embed, commit in one Aurora transaction, and the outbox worker upserts to OpenSearch.](/intel/analysis-documentingest-detail.png)
One document version, start to finish. Every red box is a way the workflow refuses to serve content rather than serve it badly.

Three quality gates sit inside the workflow:

- **Missing business metadata.** Bedrock suggests brand, product, jurisdiction and document type. Below the confidence threshold the document is stored as `needs_review` and kept out of retrieval until a content owner confirms it.
- **Low OCR confidence.** Pages or tables below the threshold are flagged, and their chunks are excluded or marked, so the assistant never cites badly read text.
- **Retries exhausted.** The job is marked failed with its error, the message goes to the dead-letter queue and an alarm fires. After the fix it can be replayed, and the completed steps are skipped.

Clause boundaries, page and bounding box for citations, effective dates for date-of-loss filtering, access principals for pre-filtering and edges for GraphRAG are all produced here. If this path is wrong, nothing the query side does can recover.

**Textract** is used only for pages that need it, because it is billed per page, and its output is cached by content hash so re-chunking or changing the embedding model never pays for OCR again. **Docling** (or Unstructured) on Fargate handles born-digital PDFs and Word files at a fraction of the cost, with pinned versions. **Bedrock** embeds only changed chunks, and records the model version on every chunk, so a model change is a controlled re-embed rather than a silent mix of vector spaces.

Step Functions Standard workflows could implement the same steps with a visual execution history. Durable functions were preferred for the code-first model, plain Python that is easy to test and review, subject to confirming their availability and limits in Sydney.

## Where everything is stored

| Store | Holds | Why |
|---|---|---|
| S3 raw | Every ingested version, byte for byte, versioned and KMS-encrypted | Rebuild the index without SharePoint, and prove what the assistant saw on a given date |
| S3 parsed | Normalised parser output per version | Change the chunking or the embedding model without re-parsing 500,000 documents |
| Aurora PostgreSQL | Registry, chunks, access principals, relationships, sync cursors, subscriptions, jobs, outbox, entitlements | One relational store for everything derived, with joins, transactions and simple SQL for audit |
| OpenSearch | Hybrid index: BM25 keyword fields plus k-NN vectors, and the filter fields | Keyword for clause numbers and defined terms, vectors for meaning, filters applied before ranking |
| Neo4j (phase 2) | A projection of the relationship edges | Multi-hop expansion: the endorsement that amends a clause, the definition that governs it |

Aurora does the heavy lifting with plain relational patterns: primary keys and `ON CONFLICT DO NOTHING` for idempotency, advisory locks for one writer per library, a transactional outbox, and `SELECT ... FOR UPDATE SKIP LOCKED` for work queues. Lambdas reach it through RDS Proxy or the Data API so a burst of parallel work cannot exhaust its connections.

A small **outbox worker** on Fargate is the only component that writes to OpenSearch. It claims outbox rows, always taking the `acl` lane first, applies them as bulk upserts, partial access updates or deletes, and marks them done. That gives one ordered, retryable path into the index, and an exact record of when each change was enforced.

OpenSearch filters on `acl_principals`, `brand`, `product`, `jurisdiction`, `claim_id`, `doc_type`, `status`, `effective_from` and `effective_to`. Filtering during the vector search, not after it, is what stops a forbidden document from taking a top-ten slot and then being dropped, leaving the answer short.

## Lane 4: the 500,000-document backfill

The initial load reads a JSONL manifest of every in-scope item from S3 and uses a Step Functions Distributed Map to start one `DocumentIngest` run per item. Concurrency is capped at the lowest of the Graph throttling limit, the Textract quota and the Bedrock quota, with batching, a tolerated failure percentage and results written to S3.

There is no separate backfill code: it reuses the exact per-document workflow, and because that workflow is idempotent, a partial or repeated backfill is always safe. The same mechanism handles planned re-indexes. Budget days, not hours, for the first load. Graph throttling and OCR throughput set the pace.

## Lane 5: nightly reconciliation

Every night at 02:00 a Step Functions workflow lists every in-scope item and its permissions from Graph, diffs that against Aurora, diffs Aurora against OpenSearch, and repairs what it finds: orphans deleted, access lists corrected through the outbox, drifted documents re-queued. It reports drift to CloudWatch and writes an audit row per run.

This is the safety net for dropped webhooks, expired subscriptions, partial failures and manual changes. Drift becomes a measured number rather than an assumption, and the repairs go through the same queues and outbox as normal processing, never straight into the index.

## Security, operations and the numbers we hold ourselves to

| Control | Implementation |
|---|---|
| Network isolation | Lambdas, Fargate tasks, Aurora and OpenSearch in private subnets. No public endpoint except the webhook. |
| Private AWS access | VPC endpoints for S3, SQS, Textract, Bedrock, Secrets Manager, KMS, CloudWatch Logs and STS |
| Egress | Outbound to Microsoft Graph only, through a domain allowlist |
| Least privilege | `Sites.Selected` read-only to the claims sites. One IAM role per function. Only the outbox worker writes to OpenSearch, and only the query service reads it. |
| Encryption | KMS customer-managed keys for S3, Aurora, OpenSearch, SQS and logs. TLS everywhere. |
| Data residency | Everything in `ap-southeast-2`. Bedrock in-region or through an Australian cross-region inference profile. |

CloudWatch tracks permission lag, queue age per lane, throughput, failure rate, OCR pages, drift and subscription headroom. X-Ray traces each document end to end. Dead-letter queues isolate poison documents per lane, with a replay runbook. CloudTrail records every API call.

| Service level objective | Target |
|---|---|
| Permission change in SharePoint to enforced in the index | Under 15 minutes (p95) |
| Content change to searchable | Under 60 minutes (p95), excluding large OCR jobs |
| Nightly drift after repair | Zero access mismatches; content drift under an agreed threshold |
| Subscription expiry headroom | No subscription within 48 hours of expiry |

## How ingestion serves the question

Ingestion is half the system. When a consultant asks a question, the query service validates their Entra token and loads the claim's context: policy, product, brand, state and date of loss. An entitlement service turns the consultant's groups, business attributes and claim assignment into a filter. OpenSearch runs a hybrid search with that filter applied first, restricted to the versions in force on the date of loss. Results are re-checked against the access lists in Aurora, reranked, optionally expanded through Neo4j, and Claude on Bedrock writes a cited answer. Each citation is checked against its source before the answer goes back.

| What the query needs | What ingestion provides |
|---|---|
| Users see only what they are allowed to | Access principals on every chunk, the fast permission lane, fail-closed rules and nightly reconciliation |
| The right policy version | Version and effective dates on every chunk |
| Precise, checkable citations | Clause-level chunks with page and bounding box, and deep links to SharePoint |
| Endorsements and definitions are not missed | Relationship edges extracted at ingest |
| Answers that can be reproduced in a dispute | Raw and parsed copies in S3, job and outbox history, and model versions |

## The decisions, and what we turned down

| Decision | Chosen | Alternatives considered | Reason |
|---|---|---|---|
| Durable execution | Lambda durable functions and Step Functions | Temporal; SQS with hand-rolled state | AWS-native, simpler security approval, code-first per-document workflow |
| Workflow state | Aurora PostgreSQL | DynamoDB | One relational store for registry, access lists, edges and state, with transactions and the outbox pattern |
| Retrieval index | OpenSearch Service | pgvector, Qdrant, S3 Vectors, Amazon Q Business | Hybrid search and in-search filtering at 10 to 25 million chunks, inside the VPC |
| OCR | Textract, routed per page | OCR everything; open-source OCR | Accuracy on tables and forms, confidence scores, and cost control |
| Digital parsing | Docling or Unstructured on Fargate | Textract for everything | Better structure for born-digital files, at lower cost |
| Permission propagation | A separate queue and outbox lane | One queue for everything | A revocation must not wait behind a content backlog |
| Relationship graph | Edges in Aurora now, Neo4j in phase 2 | Neo4j from day one | Ship the MVP sooner, add the graph when evaluation shows missing context |

Some points are still open: durable function availability and limits in Sydney, in-region Bedrock availability for the chosen models, current Graph subscription limits for drive resources, the policy for organisation-wide sharing links, the content-owner process for items in review, and retention periods for S3 and the audit tables.

## For CXOs, and CISOs in particular: would you fund this?

Our proposal stopped at the first conversation. We would like to know why projects like this stall, from the people who sign them off. Sign in and leave a comment at the bottom of the page.

- **Is the business case there?** A consultant who finds the right clause in seconds rather than minutes, across a few hundred consultants and thousands of claims a week, is real money. Is that enough to fund a platform like this, or does it have to show up as fewer complaints and faster claims before anyone signs?
- **Build, buy or wait?** Microsoft Copilot and Amazon Q Business both promise answers over SharePoint with permissions included. Would you trust an off-the-shelf product with your policy wordings and assessor reports, or does a regulated insurer need to own the pipeline?
- **Who owns the risk?** If the assistant gives a consultant the wrong excess, or an answer from a document they should not have seen, is that the claims executive's problem, the CIO's or the CISO's? Who would you want accountable before you said yes?
- **What actually stops it?** Is it budget, data quality, the regulator, the board's appetite for AI, or a pilot that never gets out of the data science team? Where did your last AI project stall?

## For architects: where we think the design could be wrong

We would rather hear the objections now than in production, especially from anyone who has run something like this.

- **Durable functions or Step Functions?** Lambda durable functions give a code-first workflow in plain Python. Step Functions give a visual history every operator can read. For a regulated insurer, which would you put in front of an auditor?
- **Is OpenSearch still the right index?** At 10 to 25 million chunks we chose OpenSearch over pgvector in the Aurora database we already run. Would you keep it all in Postgres and drop a whole service?
- **Is failing closed on organisation-wide links too strict?** Treating every "everyone in the company" link as restricted will hide a lot of genuinely shared content on day one. Would you map those links to a tenant-wide group instead, and accept the risk?
- **Is 15 minutes fast enough for a revocation?** If a consultant moves teams at 9am and can still get answers from their old team's assessor reports at 9:14, is that acceptable under CPS 234, or does the query path need to check live permissions as well?

**Related on CISO AI:** [Who is accountable for an AI agent?](/analysis/who-is-accountable-for-an-ai-agent), [least privilege for AI agents](/analysis/least-privilege-ai-agents-quantum), and our [APRA CPS 230 and CPS 234 guide for AI](/AI-APRA-CPS-230-234).