The Complete RAG Ingestion Pipeline for ASP.NET Core Developers

The Complete RAG Ingestion Pipeline for ASP.NET Core Developers

From an ordinary business document to trustworthy, permission-aware searchable knowledge

Embeddings allow an application to search by related meaning. That sounds impressive, but it leaves an important engineering question unanswered:

How does an ordinary PDF, Word document, SQL record or spreadsheet become clean, chunked, embedded and searchable without losing its origin, permissions or business meaning?

That transformation is the RAG ingestion pipeline.

Ingestion is not just “upload a file and call an embedding API.” A production-minded pipeline must extract useful information, remove noise without destroying evidence, choose sensible chunk boundaries, attach authoritative metadata, generate vectors, publish a complete index version, react to updates and deletions, and make failures visible.

This guide explains that journey for ASP.NET Core developers. It uses BuildEstate Pro as a realistic property-development example. The RAG functionality described here is a proposed teaching extension of that public case-study repository, not a claim that the repository already contains a production AI feature.

The central lesson is:

RAG quality begins long before a user asks a question. Poor extraction, chunking, metadata, permissions or version control gives even the best model unreliable evidence.

---

1. The source remains the truth

A company may already keep useful information in SQL Server, Azure Blob Storage, SharePoint, PDFs, Word documents, spreadsheets, APIs and document-management systems.

A RAG solution should not quietly replace those systems. They remain authoritative. Instead, ingestion creates a derived, search-optimised projection of selected information.

This is similar to a read model in CQRS:

Operational system                       Retrieval projection
------------------                       --------------------
Owns authoritative business data   ───►  Stores searchable chunks
Enforces business transactions           Optimised for retrieval
Maintains original documents             Contains text + vectors + metadata
Controls access and lifecycle             Can always be rebuilt

The distinction matters. If a planning report is corrected, withdrawn or made confidential, the retrieval projection must follow. The vector index is valuable, but it is not an independent source of truth.

A healthy design can answer these questions for every indexed chunk:

• Which source record and document version produced it?
• Which extraction and chunking rules were used?
• When was it indexed?
• Who is permitted to retrieve it?
• Can it be reproduced or removed?

If those answers are unavailable, the index is already accumulating operational debt.

---

2. Separate ingestion time from query time

RAG has two related but different pipelines.

Ingestion time: prepare the knowledge

Before a user asks anything, the application:

1. Detects new, changed or deleted content.
2. Extracts its text and structure.
3. Cleans and normalises the result.
4. Classifies the source and selects a strategy.
5. Divides the content into meaningful chunks.
6. Adds provenance, business and security metadata.
7. Generates an embedding for each chunk.
8. Publishes the complete records to a search index.

Query time: retrieve authorised evidence

When a user asks a question, the application:

1. Establishes the user's tenant and permissions.
2. Converts the question into an embedding.
3. Runs keyword, vector or hybrid search with security filters.
4. Ranks and selects the most useful chunks.
5. Builds a bounded prompt containing evidence and citations.
6. Asks the model to produce a grounded answer.

This guide concentrates on ingestion. The preceding embeddings and semantic search guide explains the retrieval side in detail.

---

3. The complete production-minded flow

The content transformation is easy to draw:

Source → extract → normalise → classify → chunk → enrich → embed → index

The operational loop is equally important:

Detect change
    ↓
Create durable job
    ↓
Process into a new generation
    ↓
Validate completeness and quality
    ↓
Publish new generation
    ↓
Retire superseded chunks
    ↓
Observe, retry, reprocess or delete

The first line creates vectors. The second makes the feature dependable.

---

4. Begin with a source registry

Suppose a user uploads Planning-Officer-Report.pdf to BuildEstate Pro. Blob Storage owns the binary file. SQL Server holds the business record: tenant, project, document type, version, approval state and access policy.

Represent that identity explicitly:

public sealed record SourceDocument(
    Guid TenantId,
    Guid ProjectId,
    Guid DocumentId,
    int Version,
    Uri BlobUri,
    string FileName,
    string MediaType,
    string SecurityScope,
    string ContentHash,
    DateTimeOffset UpdatedAtUtc);

The URI locates the original. The IDs provide stable business identity. The version and hash tell the pipeline whether the content changed. The security scope travels with the document instead of being invented later.

Do not use a filename as identity. Files can be renamed without changing, and replaced without being renamed.

---

5. Trigger ingestion without holding open the upload request

Extraction, OCR and embedding can take seconds or minutes. They should not normally run inside the user's upload HTTP request.

A better flow is:

POST document
  ├─ save source file
  ├─ commit document record
  ├─ publish IngestDocument command
  └─ return 202 Accepted with status URL

The trigger may be an Event Grid event, Service Bus message, scheduled indexer, database change marker, manual re-index command or full rebuild. The important property is durability: once the source operation succeeds, the work must not disappear because one web process restarts.

public sealed record IngestDocumentCommand(
    Guid TenantId,
    Guid DocumentId,
    int SourceVersion,
    string CorrelationId);

[HttpPost("{projectId:guid}/documents")]
public async Task<IActionResult> Upload(
    Guid projectId,
    IFormFile file,
    CancellationToken cancellationToken)
{
    var document = await documentService.StoreAsync(
        projectId, file, cancellationToken);

    await ingestionQueue.EnqueueAsync(
        new IngestDocumentCommand(
            document.TenantId,
            document.Id,
            document.Version,
            HttpContext.TraceIdentifier),
        cancellationToken);

    return AcceptedAtAction(
        nameof(GetIndexingStatus),
        new { projectId, documentId = document.Id },
        new { document.Id, status = "Pending" });
}

In Azure, an outbox pattern can close the gap between committing the SQL document and publishing the message. The API gives the user a truthful response: the source was accepted; indexing is still in progress.

---

6. Model ingestion as a state machine

“Processing” is too vague for support and operations. Store the current stage.

public enum DocumentIndexingStatus
{
    Pending,
    Extracting,
    Normalising,
    Chunking,
    Embedding,
    Publishing,
    Indexed,
    Failed,
    Deleting
}

An ingestion job should also record the source version, pipeline version, attempt count, correlation ID, timestamps, failure category and last safe checkpoint.

A background worker coordinates the stages:

public sealed class RagIngestionWorker(
    IIngestionQueue queue,
    IRagIngestionPipeline pipeline,
    ILogger<RagIngestionWorker> logger) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await foreach (var command in queue.ReadAllAsync(stoppingToken))
        {
            try
            {
                await pipeline.IngestAsync(command, stoppingToken);
                await queue.CompleteAsync(command, stoppingToken);
            }
            catch (Exception exception)
            {
                logger.LogError(exception,
                    "RAG ingestion failed for {DocumentId}, version {Version}",
                    command.DocumentId, command.SourceVersion);

                await queue.AbandonOrDeadLetterAsync(
                    command, exception, stoppingToken);
            }
        }
    }
}

The worker is intentionally boring. Domain-specific work belongs in testable pipeline services, not one enormous ExecuteAsync method.

---

7. Extraction is a document-understanding problem

A PDF is a container, not guaranteed clean prose. It may contain scanned pages, two-column layouts, tables, diagrams, handwriting, repeated headers and text positioned in a visually meaningful order.

A simple parser may be sufficient for a digitally generated document. Complex or scanned material may need OCR and layout analysis. Azure Document Intelligence can extract paragraphs, headings, tables, page structure and layout, and can provide Markdown that preserves useful structure for later chunking.

Use an intermediate contract rather than passing a raw string through the entire application:

public sealed record ExtractedBlock(
    string Kind,
    string Text,
    int PageNumber,
    string? Heading,
    IReadOnlyDictionary<string, string> Attributes);

public sealed record ExtractedDocument(
    Guid DocumentId,
    string ExtractorName,
    string ExtractorVersion,
    IReadOnlyList<ExtractedBlock> Blocks,
    IReadOnlyList<string> Warnings);

Extraction warnings matter. OCR can confuse 0 and O, lose minus signs or misread names and dates. A low-confidence financial value or planning reference may deserve review instead of silent publication.

Keep the original page number and structural signals. They support citations, diagnostics and better chunk boundaries.

---

8. Normalise noise without rewriting the evidence

Normalisation can remove repeated headers and footers, repair broken line endings, standardise whitespace, discard empty pages and convert tables into a consistent representation.

Its purpose is not to make a document prettier. It is to make retrieval more reliable while preserving meaning.

For example, removing Planning Committee Report — page 12 from every page may reduce noise. Removing every occurrence of Confidential could destroy security meaning.

A useful pipeline stores:

• original source version;
• extraction output or its reproducible location;
• normaliser name and version;
• warnings and transformations applied;
• hash of the normalised result.

When a rule changes, the pipeline can identify which documents need reprocessing.

---

9. Classify first, then choose a strategy

One generic splitter should not treat a planning report, contract, spreadsheet and application log identically.

public interface IChunkingStrategy
{
    bool CanHandle(DocumentClassification classification);

    IAsyncEnumerable<RagChunkDraft> CreateChunksAsync(
        NormalisedDocument document,
        CancellationToken cancellationToken);
}

Reasonable strategies include:

Classification can begin with trusted business metadata and file type. A model-based classifier may help for ambiguous content, but it should not override authoritative labels without governance.

---

10. A chunk must make sense when retrieved alone

An embedding for a 200-page report is too broad. A chunk containing half a sentence is too weak.

Poor chunk:

must be approved before work begins.

Better chunk:

Drainage Strategy — The detailed drainage strategy must be approved by the local authority before construction work begins.

The better version carries its subject and obligation. Useful chunking may combine headings, paragraphs, sentences, token limits and small overlaps.

public sealed record RagChunkDraft(
    int Sequence,
    string Text,
    int? PageFrom,
    int? PageTo,
    string? SectionPath,
    int ApproximateTokens);

Overlap can preserve meaning across a boundary, but too much overlap creates duplicates, increases embedding and storage cost, and crowds search results with near-identical passages.

Choose chunking empirically. Build a set of realistic questions, note the passages that should answer them, and measure whether those passages are retrieved. “Five hundred tokens with ten percent overlap” is a starting hypothesis, not an architecture principle.

---

11. Metadata carries facts that a vector cannot

The vector captures semantic relationships. Metadata carries deterministic identity, provenance and policy.

public sealed record RagChunk(
    string ChunkId,
    Guid TenantId,
    Guid ProjectId,
    Guid DocumentId,
    int DocumentVersion,
    string GenerationId,
    int Sequence,
    string Text,
    int? PageNumber,
    string? SectionPath,
    string SourceUri,
    IReadOnlyCollection<string> PermittedGroupIds,
    string ContentHash,
    string ChunkingVersion,
    string EmbeddingModel,
    DateTimeOffset IndexedAtUtc);

Metadata enables:

• tenant and permission filtering;
• citations back to the original source;
• project, date and document-type filters;
• deletion of every chunk derived from one document;
• investigation of a suspicious answer;
• targeted reprocessing after a pipeline change;
• exclusion of obsolete versions.

Do not put everything into one JSON string if the search engine must filter or facet on it. Model important fields explicitly and mark them appropriately in the index schema.

---

12. Stable IDs, hashes and idempotency stop duplication

Retries are normal. A message may be delivered twice, a worker may restart, or an operator may request a rebuild. The result should still be one correct projection.

A stable chunk key might derive from:

tenant ID + document ID + source version + chunking version + logical sequence

The same input then addresses the same derived record. A content hash provides another guard:

var normalisedHash = Convert.ToHexString(
    SHA256.HashData(Encoding.UTF8.GetBytes(normalisedText)));

if (await ingestionStore.IsCurrentAsync(
        document.Id, document.Version, normalisedHash, pipelineVersion,
        cancellationToken))
{
    return IngestionOutcome.AlreadyCurrent;
}

Idempotency does not mean ignoring change. Include source and pipeline versions deliberately so a genuine update creates a new generation while an identical retry does not create duplicate chunks.

---

13. Generate embeddings in controlled batches

Each final chunk is sent to an embedding model. Microsoft.Extensions.AI gives .NET applications the provider-neutral IEmbeddingGenerator abstraction.

using Microsoft.Extensions.AI;

public sealed class ChunkEmbeddingService(
    IEmbeddingGenerator<string, Embedding<float>> generator)
{
    public async Task<IReadOnlyList<EmbeddedChunk>> EmbedAsync(
        IReadOnlyList<RagChunk> chunks,
        CancellationToken cancellationToken)
    {
        var results = new List<EmbeddedChunk>(chunks.Count);

        foreach (var batch in chunks.Chunk(32))
        {
            var texts = batch.Select(chunk => chunk.Text).ToArray();
            var embeddings = await generator.GenerateAsync(
                texts,
                cancellationToken: cancellationToken);

            results.AddRange(batch.Zip(
                embeddings,
                (chunk, embedding) => new EmbeddedChunk(
                    chunk,
                    embedding.Vector)));
        }

        return results;
    }
}

The example illustrates responsibility boundaries, not a universal batch size. Respect the provider's token, request and quota limits. Add bounded retries with backoff for transient failures, but do not retry invalid input forever.

Record the model, deployment and vector dimensions. Query-time embeddings must be compatible with the indexed vectors. Changing model or dimensions normally requires re-embedding the collection or writing into a new index generation.

---

14. Store vectors and readable evidence together

A search document usually contains:

• a unique chunk key;
• human-readable chunk text;
• the vector field;
• source and parent-document IDs;
• tenant and ACL fields;
• section and page provenance;
• version and publication fields.

The vector helps locate a passage; it cannot reconstruct the original passage. The readable text is what the RAG application places into the prompt and cites to the user.

public sealed class SearchChunkDocument
{
    public required string Id { get; init; }
    public required string TenantId { get; init; }
    public required string DocumentId { get; init; }
    public required string GenerationId { get; init; }
    public required string Content { get; init; }
    public required ReadOnlyMemory<float> ContentVector { get; init; }
    public string? SectionPath { get; init; }
    public int? PageNumber { get; init; }
    public required string[] PermittedGroupIds { get; init; }
    public required bool IsPublished { get; init; }
}

Index in batches and inspect every item result. An HTTP success for a batch does not necessarily mean every record was accepted. Persist enough information to retry only failed items safely.

---

15. Never expose a half-published document

Imagine a document should create fifty chunks, but a failure occurs after twenty are written. A user could receive an incomplete answer without knowing the evidence is partial.

A safer generation flow is:

1. Allocate a new GenerationId.
2. Extract, chunk, embed and write all chunks as unpublished.
3. Validate expected counts and required metadata.
4. Mark the generation active or switch an active-generation pointer.
5. Retire the previous generation.

Document v3 (active generation G17) ─── searchable
Document v4 (building G18)          ─── hidden

after successful validation

Document v3 (G17)                   ─── retired
Document v4 (active G18)            ─── searchable

A staging index with an alias can offer a similar pattern for a complete collection rebuild. The exact mechanism depends on the search platform, but the invariant should remain: query-time code can tell which generation is complete and current.

---

16. Updates and deletions are part of ingestion

Production information changes continuously.

New document

Process and publish its first generation.

Updated document

Compare source version and content hash. For many business documents, reprocessing the whole document is simpler and safer than guessing which chunks changed. Keep an older version only when the business requires historical search.

Deleted or withdrawn document

Remove or deactivate every derived chunk using tenant ID and document ID. Deleting only the blob leaves orphaned knowledge that may continue appearing in answers.

Permission change

Update the ACL metadata or rebuild the affected projection before the old audience can retrieve it again. Permission changes are data changes.

Pipeline or embedding-model change

Create a controlled reprocessing campaign. Track progress, cost and failures, then switch generations when validation passes.

This lifecycle is why parent IDs, generation IDs and pipeline versions are not optional decoration.

---

17. Security starts before the first vector is written

A vector index contains derived copies of business information. It belongs inside the organisation's security and data-governance boundary.

At ingestion time, capture tenant, owner, department, permitted groups, classification, retention policy and geographical constraints where relevant. At query time, derive filters from authenticated server-side identity.

var filter = SearchFilter.Create(
    $"TenantId eq {currentUser.TenantId} " +
    $"and IsPublished eq true " +
    $"and PermittedGroupIds/any(g: search.in(g, {currentUser.GroupIdsCsv}))");

Never accept a tenant ID or permission list from an untrusted prompt and treat it as authority. The model does not decide access. The ASP.NET Core application and search filter do.

Also protect the ingestion path itself:

• validate file type and size;
• malware-scan uploads where required;
• isolate tenant storage paths;
• use managed identities and least privilege;
• encrypt in transit and at rest;
• avoid logging sensitive chunk text;
• apply retention and deletion requirements to derived data.

Content can also contain prompt-injection instructions. During retrieval, source text must be treated as untrusted evidence, not as higher-priority application instructions.

---

18. Failures must be retryable, isolated and explainable

The PDF may be corrupt. OCR may fail. A chunk may exceed the model limit. The embedding endpoint may throttle. An index field may reject a value. A worker may stop halfway through.

Classify failures:

After bounded attempts, move the command to a dead-letter queue or failure store. Record document ID, stage, source and pipeline versions, retry count, correlation ID and a sanitised diagnostic.

One failed document should not block the rest of the queue. Operators need actions to retry after correction, skip with a recorded reason or request deliberate reprocessing.

---

19. Add quality gates before publication

Successful API calls do not guarantee useful knowledge. Validate the output.

Useful ingestion checks include:

• extracted text is non-empty and plausible for the source size;
• expected pages were processed;
• no chunk exceeds the chosen embedding limit;
• tiny or duplicate chunks stay below an agreed threshold;
• every chunk has tenant, document, version and citation metadata;
• vector dimensions match the index schema;
• every expected chunk was accepted by the index;
• required ACL fields are present;
• sample retrieval questions return known passages.

For important document classes, keep a small evaluation dataset:

Question: Which access route is prohibited for construction vehicles?
Expected source: Planning Officer Report v4
Expected section: Highway Conditions
Expected page: 18

Run those checks when extraction, chunking, embedding or ranking configuration changes. Ingestion quality becomes measurable instead of anecdotal.

---

20. Observe cost, throughput and freshness

A production dashboard should reveal:

• queue depth and oldest-message age;
• documents and chunks processed per hour;
• time spent in each stage;
• extraction warnings and failure rate;
• retry and dead-letter counts;
• embedding tokens, requests and estimated cost;
• index write failures and throttling;
• average chunks per document;
• time from source change to searchable publication;
• documents running an old pipeline or embedding version.

Use correlation IDs across the upload, queue, worker, extraction call, embedding batches and index writes. OpenTelemetry traces, structured logs and metrics should identify the document and stage without leaking sensitive content.

Cost control begins with avoiding needless work: hashes prevent identical reprocessing, batching reduces request overhead, sensible chunks avoid vector bloat, and controlled re-index campaigns protect provider quotas.

---

21. Managed Azure AI Search ingestion or custom C#?

There are two broad approaches, and many solutions combine them.

Azure AI Search integrated vectorisation

Azure AI Search can coordinate supported data sources, indexers, skillsets, chunking, embedding skills, indexes and query-time vectorisers. This reduces custom plumbing and can be a strong choice when the source and transformations fit the supported model.

Custom ASP.NET Core or worker pipeline

A custom pipeline might combine Event Grid, Service Bus, Azure Functions or a Worker Service, Document Intelligence, domain-specific chunking, IEmbeddingGenerator, SQL processing records and the Azure AI Search SDK.

It is useful when the solution needs complex business routing, unusual formats, specialised permissions, precise publication generations, multiple destinations, provider portability or detailed workflow state.

Choose from actual constraints, not fashion. A custom pipeline is not automatically more professional, and managed ingestion is not automatically simplistic.

---

22. BuildEstate Pro: the complete journey

Now join the parts together.

1. A user authorised for project 152 uploads Planning-Officer-Report.pdf.
2. The API validates and stores the original in Blob Storage.
3. SQL Server creates document version 4 with status Pending, tenant and ACL metadata.
4. The outbox publishes IngestDocumentCommand.
5. A worker verifies that version 4 remains current and calculates the content hash.
6. Document Intelligence extracts structured Markdown, pages and tables.
7. The normaliser removes repeated footers but preserves headings, conditions and references.
8. The classifier selects the planning-report chunker.
9. Chunks receive project, tenant, document, version, page, section, permissions and stable IDs.
10. The embedding service processes chunks in controlled batches.
11. Text, vectors and filterable metadata are written under generation G18 as unpublished.
12. Quality checks confirm extraction, chunk counts, vector dimensions, ACLs and index results.
13. G18 becomes active; the previous generation is retired.
14. The SQL document status changes to Indexed and the UI reports completion.

At query time, an authenticated project user asks:

Are there restrictions on construction vehicle access?

ASP.NET Core builds tenant, project and group filters from the user's identity. Hybrid search finds the relevant highway-condition chunks. The context builder includes the text, document title, section, page and source link. The model produces an answer grounded in those passages and cites the report.

If version 4 is later withdrawn, its active generation is removed or deactivated. The application cannot continue quoting a document that the source system no longer treats as current.

That lifecycle—not the isolated vector call—is the real RAG ingestion feature.

---

23. Test the pipeline at several levels

Unit tests

Test normalisation, stable IDs, hash decisions, strategy selection, chunk boundaries, metadata mapping and state transitions with deterministic inputs.

Contract tests

Verify extraction, embedding and index adapters against recorded or test-service responses. Confirm vector dimensions and index field names.

Integration tests

Process representative PDFs, spreadsheets and SQL projections through a test index. Assert chunk counts, ACL filters, citations and deletion behaviour.

Failure tests

Simulate throttling, partial batch rejection, corrupt documents, worker restarts and duplicate messages. Prove that retries do not duplicate data and unpublished generations remain hidden.

Retrieval evaluation

Use known questions and expected passages to measure recall and ranking after pipeline changes. A technically successful ingestion run that makes the right evidence harder to find is a regression.

---

24. A sensible delivery sequence

Do not begin by connecting every company repository.

1. Choose one valuable, authorised document type.
2. Define source ownership, identity, retention and ACL behaviour.
3. Build extraction and inspect the results manually.
4. Create a small retrieval evaluation set.
5. Implement one chunking strategy and explicit metadata contract.
6. Add batching, stable IDs and content hashes.
7. Publish through complete generations.
8. Implement update, permission-change and deletion paths.
9. Add retries, dead-letter handling and observability.
10. Measure retrieval quality, freshness and cost.
11. Only then expand to more sources and formats.

This sequence creates a narrow end-to-end capability that can be tested and governed. It is far more useful than a wide collection of poorly understood connectors.

---

25. The senior engineering perspective

The interesting part of RAG ingestion is not calling an AI service. It is applying familiar software-engineering judgement to a new kind of read model:

• authoritative versus derived state;
• asynchronous workflows and idempotency;
• document parsing and data quality;
• stable contracts and version migrations;
• authentication, authorisation and data governance;
• atomic publication and deletion consistency;
• retries, telemetry, cost and operational support;
• measurable acceptance criteria.

My current work in Microsoft Foundry, Azure AI Search and RAG builds on those established .NET, SQL Server, Azure and enterprise workflow skills. I am documenting practical study and prototyping honestly; I do not present it as decades of production AI experience.

That honesty is useful. Organisations need engineers who can distinguish a demonstration from an operable feature—and can explain the work required to cross that gap.

---

Final mental model

Keep this sequence in mind:

Authoritative business source
        ↓
Durable change event
        ↓
Extraction with structure and provenance
        ↓
Careful normalisation
        ↓
Source-aware chunking
        ↓
Identity + metadata + ACLs + versions
        ↓
Batched compatible embeddings
        ↓
Validated, complete index generation
        ↓
Permission-filtered retrieval with citations

The search index is a rebuildable projection, not the source of truth. Embeddings make meaning comparable; they do not supply provenance or permissions. Metadata makes chunks governable. Publication generations prevent partial knowledge. Update and deletion paths keep answers current.

When those responsibilities are designed together, RAG stops being a clever demo and begins to resemble dependable business software.

Continue with the AI Engineering learning journey, revisit context engineering for trusted ASP.NET Core applications, or see the broader business data to production-minded RAG guide.

If this article helped you, or you would like to discuss ASP.NET Core, Azure, SQL Server, Microsoft Foundry or RAG engineering, contact me at dotnetdeveloper20xx@hotmail.com.

---

References and further reading

• Azure AI Search integrated vectorisation
• Azure Document Intelligence guidance for RAG
• Azure AI Search change and deletion detection for Blob Storage
• Chunking content in a RAG solution
• Microsoft.Extensions.AI embedding generation