The Agentic Leap — Part 10: Building a Production-Ready Agentic System
We have reached the capstone. The goal is not to add every agent feature we have studied. The goal is to assemble the smallest complete system that can produce a useful, authorized and recoverable business outcome.
Production readiness is the ability to understand, control and recover the whole system—not simply obtain an impressive answer.We will connect Microsoft Foundry, ASP.NET Core, Angular, Azure AI Search, durable state, controlled tools and human approval around one BuildEstate Pro planning scenario.
The capstone assembles the series into one bounded BuildEstate Pro capability: a project-support agent that reviews planning position, retrieves authorised evidence, identifies gaps, prepares a task proposal and waits for a qualified person before any business change.
“Production-ready” here means production-minded architecture and verifiable controls. It does not claim that this educational case study has operated a real planning process at scale.
1. Define the outcome and exclusions
The system may:
- explain current planning status with citations;
- compare approved records and policy evidence;
- identify missing or conflicting information;
- prepare a structured project-task proposal;
- execute that proposal only after valid approval.
Success is not “the model replied.” It is a correct, authorised and traceable outcome with an appropriate stop.
2. Reference architecture
Angular review UI
|
ASP.NET Core agent API and control plane
|-- Identity + project authorisation
|-- Goal/step state machine and budgets
|-- Model adapter with structured decisions
|-- Tool registry + policy gateway
|-- Approval and command handlers
|-- OpenTelemetry + evaluation events
|
|-- SQL Server authoritative data + execution tables
|-- Azure AI Search / RAG evidence with security filters
|-- Optional MCP capability adapters
`-- Optional bounded agent service / A2A delegationThe model provider and frameworks sit behind interfaces. ASP.NET Core remains the authority boundary.
3. API contracts
public sealed record StartPlanningReviewRequest(Guid ProjectId, string Objective);
public sealed record AgentExecutionView(
Guid ExecutionId,
string Status,
int CurrentStep,
BudgetView RemainingBudget,
IReadOnlyList<FindingView> Findings,
ProposalView? PendingProposal,
IReadOnlyList<CitationView> Citations,
string? RequiredUserAction);POST /api/projects/{projectId}/agent-reviews starts a run using authenticated identity. GET /api/agent-reviews/{id} returns a redacted view. Approval endpoints accept the execution ID, proposal hash and decision—not arbitrary replacement command arguments.
Use antiforgery or appropriate same-site/API protections, rate limits and resource-based authorisation.
4. Structured model decision
The model returns one of a discriminated set: call a permitted read/proposal tool, ask the user, or finish.
[JsonPolymorphic(TypeDiscriminatorPropertyName = "kind")]
[JsonDerivedType(typeof(CallToolDecision), "call_tool")]
[JsonDerivedType(typeof(AskUserDecision), "ask_user")]
[JsonDerivedType(typeof(FinishDecision), "finish")]
public abstract record AgentDecision;Provider-native strict schemas improve parsing. The application rejects unknown kinds, unknown tools, excessive argument sizes and identifiers outside the execution scope.
5. Retrieval with security and citations
Use SQL for current structured facts and RAG for policy/documents. Azure AI Search queries must apply tenant, project and classification filters before retrieval. Do not retrieve broadly and remove unauthorised chunks afterward.
Every evidence item carries source ID, title, version/effective date, location, access scope and retrieval score. The final response cites only evidence actually returned. If evidence conflicts or is insufficient, the correct result is escalation or a question.
6. Tool and policy pipeline
public async Task<ToolObservation> InvokeAsync(
AgentExecutionContext context,
ProposedToolCall call,
CancellationToken ct)
{
var tool = registry.Resolve(call.Name);
var args = tool.BindAndValidate(call.Arguments);
var decision = await policy.AuthoriseAsync(context, tool.Descriptor, args, ct);
if (!decision.Allowed)
return ToolObservation.Denied(decision.Code);
using var timeout = CancellationTokenSource.CreateLinkedTokenSource(ct);
timeout.CancelAfter(tool.Descriptor.Timeout);
return await tool.ExecuteAsync(context, args, timeout.Token);
}Tool results are sanitised and size-limited. Commands are absent until a valid approval transition makes one exact command available.
7. SQL execution and approval records
Persist execution, steps, observations and approvals. Encrypt or avoid sensitive prompt content. Use row-version concurrency, idempotency and an outbox.
An approval record includes initiator, approver, action, canonical arguments hash, evidence-set hash, policy version, created/expiry time and decision. Immediately before execution, reload the current project and repeat policy checks.
8. Angular human-review experience
The UI should show:
- objective and current status;
- what the agent found, with clickable citations;
- missing or conflicting evidence;
- exact proposed business change;
- consequence, reversibility and expiry;
- approve/reject controls available only to eligible users;
- a timeline of observable steps.
9. Security threat model
Important threats include prompt injection in documents, cross-tenant retrieval, forged identifiers, excessive agency, tool argument smuggling, approval tampering, SSRF through integrations, secret leakage, denial-of-wallet loops and poisoned memory.
Controls include least-privilege capabilities, pre-retrieval filters, schema validation, policy rechecks, approval hashes, egress allow-lists, secret isolation, budgets, content handling and redacted traces. Assume model instructions can fail; enforce important boundaries outside the model.
10. Reliability and degraded modes
Propagate cancellation and one execution deadline. Retry only classified transient reads. Reconcile unknown command outcomes by idempotency key. If the model provider is unavailable, allow users to inspect existing runs and project records. If command controls are unhealthy, switch the agent to read-only mode.
Use health checks for dependencies without treating a successful TCP connection as end-to-end readiness.
11. Observability
Trace HTTP request, execution, model calls, retrieval, tools, policy, approval and command. Record model/version, token usage, latency, decision type, policy result and evidence IDs. Keep prompts and personal data out of routine metrics.
Dashboards should cover task success, unsupported completion, policy denials, approval turnaround, repeated calls, budget stops, retrieval quality, cost and tool failures. Alerts need an owner and runbook.
12. Evaluation and release gates
Build a versioned evaluation set from realistic, synthetic or safely anonymised cases:
- clear evidence and correct explanation;
- missing and contradictory evidence;
- unauthorised project;
- malicious document instructions;
- ambiguous goal requiring a question;
- duplicate proposal and approval replay;
- revoked permission before execution;
- tool timeout and unknown downstream outcome.
Promote model, prompt, tool or retrieval changes through development, evaluation and controlled release. Keep a rollback path.
13. Deployment and operations
Configuration must define model deployment, tool allow-lists, budgets and feature flags per environment. Secrets belong in managed secret storage. Apply database migrations independently and compatibly. Use canary exposure or an internal user cohort before wider availability.
The runbook should cover disabling a tool, stopping new executions, resuming safe runs, reconciling commands, revoking an MCP/A2A integration, rotating credentials and deleting retained agent data.
Extended implementation review
14. API surface and trust boundaries
Expose endpoints for starting a bounded run, reading its status, supplying requested input, reviewing a proposal and cancelling. Every endpoint authenticates the caller and authorises the specific project and execution. Return display models, not internal checkpoints or provider payloads.
Use commands such as StartPlanningReview, ApproveTaskProposal and CancelAgentExecution. Each handler loads authoritative state, validates the transition and writes through the domain layer. The model adapter is never injected into the controller as a shortcut around orchestration.
Apply request limits, anti-forgery controls where cookies are used, rate limiting and idempotency for submissions. Streaming progress may use server-sent events or polling, but events contain status and evidence references rather than private reasoning.
15. Data model and transactional guarantees
Core tables can include executions, steps, observations, proposals, approvals, outbox messages and tool-call records. Use tenant keys on every relevant row, optimistic concurrency on execution state and unique constraints on idempotency keys. Store large evidence separately and protect it with the same resource policy.
Commit a state transition and its outbox event atomically. Commands to external services record intent before dispatch and reconcile uncertain outcomes. Completed executions are immutable except for governed retention metadata. Database backups, indexes and deletion jobs are part of the design.
16. Retrieval and evidence pipeline
Index only approved sources with document identity, version, tenant, project, classification and validity dates. Apply access filters before semantic or keyword ranking. Hybrid retrieval can improve recall, but relevance never overrides permission.
The evidence service returns bounded passages with stable citations and retrieval time. Treat document text as untrusted. Separate it visibly from system instructions, scan uploads and prevent documents from registering tools or changing policy. The final validator checks that material claims cite evidence actually supplied to the run.
17. Model decision contract
Ask the model for one typed decision at a time: answer, call an allowed read tool, request input, prepare a proposal or stop. Validate the schema and then validate semantics against current state and policy. The model does not select arbitrary URLs, identities, budgets or command approval.
Keep provider integration behind IAgentDecisionModel so tests use a fake and operational changes are contained. Record deployment and instruction versions. Set timeouts and cancellation, handle rate limits deliberately and reserve enough budget for a safe final explanation.
18. Tool and command implementation
Register tools from owned descriptors containing risk, schema, version and handler. Construct the runtime allow-list after authenticating the user and loading project policy. Reads return minimum data; calculations use deterministic versioned rules; proposals are durable drafts; commands require exact approval.
Every command uses an idempotency key derived from execution and proposal. If a timeout leaves the outcome unknown, query by that key or enter reconciliation. Never ask the model whether the command probably succeeded.
19. Angular review experience
The UI should show objective, current status, progress stages, evidence, limitations and costs in language appropriate to the audience. A proposal review displays the exact consequence, affected resource, supporting evidence and expiry. Approve and reject are distinct actions; editing creates a revised proposal.
Meet keyboard, focus, contrast and screen-reader requirements. Announce asynchronous status changes without flooding assistive technology. Do not imply certainty through animation or anthropomorphic copy. When the system lacks evidence or is degraded to read-only mode, say so plainly.
20. Security and privacy review
Threat-model prompt injection, cross-tenant access, excessive agency, compromised tool descriptions, SSRF, credential leakage, denial of wallet, approval tampering and poisoned memory. Apply layered controls: least privilege, resource checks, egress rules, content isolation, budgets, approval and audit.
Document each data category sent to the model, tool server or remote agent. Configure retention, regional processing and deletion. Redact telemetry and restrict support access. Secrets live in managed storage and rotate without editing prompts or source.
21. Evaluation programme
Create a versioned scenario dataset with normal, ambiguous, unauthorised, adversarial and recovery cases. Score tool selection, argument grounding, citation correctness, policy enforcement, terminal state, latency and cost. Critical access and command-safety cases require perfect results before release.
Use deterministic checks first and calibrated human or model review for subjective qualities. Run the suite for model, prompt, schema, retrieval, framework and policy changes. Shadow and canary new versions, compare against the current baseline and preserve immediate rollback.
22. Observability and operational readiness
Correlate HTTP request, execution, model call, retrieval, tool, policy, approval, checkpoint and downstream command using OpenTelemetry. Metrics include outcome, refusal, step count, repeated decisions, latency percentiles, cost, policy denials, queue age and reconciliation backlog.
Provide kill switches per command, tool, integration and model. Runbooks cover provider outage, duplicate effects, poisoned evidence, permission incident, runaway cost and stuck approvals. Test restoration from backup and resume of an older checkpoint version.
23. Delivery plan
Release in stages: offline evaluation; authenticated read-only pilot; cited recommendations; durable proposals; approval-bound single command; then carefully justified additional autonomy. Each stage has exit criteria and user research. Keep the deterministic or read-only fallback available.
Production readiness is maintained, not achieved once. Assign owners for tools, policy, evaluation, data, security and operations. Review metrics and incidents, retire unused capabilities and reduce autonomy when evidence does not justify it.
Definition of done
- Scope and non-goals are published.
- Every route applies authentication and resource authorisation.
- Retrieval filters access before content reaches the model.
- Model decisions and final results use validated contracts.
- Consequential actions require exact, expiring approval.
- State is durable, concurrent and replay-safe.
- Side effects are idempotent and reconcilable.
- Threats, data retention and operational ownership are documented.
- Evaluation covers success, refusal, attack and failure cases.
- Build, accessibility, SEO and observability checks pass.
- The site distinguishes educational design from deployed production evidence.
24. The completed learning path
The series progressed from definition to tools, architecture, state, MCP, A2A, orchestration and frameworks before assembling the system. That order matters: frameworks cannot compensate for an undefined goal, excessive permission or missing operational ownership.
Return to Part 1, or read the shorter Building Agent-Powered Applications overview. The next responsible step is not more autonomy by default; it is measuring this bounded capability and expanding only where evidence justifies it.
The end-to-end user story
A planning manager signs in and asks the system to assess Plot 14. ASP.NET Core establishes the user and project boundary, creates an execution with a budget and returns an ID. A worker retrieves approved evidence through security-filtered Azure AI Search, then asks the Foundry model for one typed next decision.
The model identifies missing ecology evidence and requests a task proposal. Policy permits proposals but not immediate task creation. The application stores the exact proposal and Angular displays its consequence, citations and expiry. A qualified manager approves it.
On resume, ASP.NET Core rechecks identity, authorization, evidence versions and the proposal hash. It writes command intent and an outbox record in one transaction. A dispatcher creates the task with an idempotency key. If the network response is lost, reconciliation finds the task rather than creating another one. The execution completes with a cited assessment and the confirmed task identifier.
This story is valuable because every important claim maps to a control we can implement and test.
A clear ASP.NET Core solution structure
BuildEstate.AgentApi
Endpoints/ authenticated start, status, input, review, cancel
Application/ goals, decisions, policies and orchestration
Domain/ lifecycle, proposals, approvals and invariants
Infrastructure/ Foundry, Search, SQL, Service Bus and telemetry
Contracts/ versioned API and tool schemas
Evaluation/ datasets, graders and regression reportsKeep controllers or minimal API handlers thin. A start endpoint sends an application command:
app.MapPost("/projects/{projectId:guid}/agent-executions",
async (Guid projectId, StartAgentRequest request,
ClaimsPrincipal user, IMediator mediator,
CancellationToken ct) =>
{
var result = await mediator.Send(
new StartPlanningReview(projectId, request.PlotId,
request.Objective, user), ct);
return Results.Accepted(
$"/agent-executions/{result.ExecutionId}", result);
})
.RequireAuthorization("CanUsePlanningAgent");The handler authorizes the specific project, creates bounded state and queues work. It does not hold an HTTP connection while an agent loops or waits for approval.
Azure deployment blueprint
- Microsoft Foundry supplies the governed model/agent platform and evaluation or observability capabilities.
- ASP.NET Core remains the trusted control plane and public domain boundary.
- Microsoft Entra ID and managed identity establish user and workload identity.
- Azure AI Search retrieves only authorized, versioned evidence.
- Azure SQL stores executions, steps, proposals, approvals and outbox records.
- Azure Service Bus delivers durable work with duplicate-safe processing.
- Blob Storage holds protected large artifacts when required.
- Key Vault stores secrets and certificates.
- Application Insights and OpenTelemetry correlate redacted operational traces.
- Angular provides accessible progress, evidence and approval experiences.
Release the capability in safe stages
First run offline evaluations. Then release read-only assessment to an internal cohort. Add cited recommendations after retrieval quality is stable. Introduce durable proposals with human review next. Finally enable one approval-bound command after idempotency and reconciliation tests pass.
At each stage measure accepted outcomes, unsupported claims, policy denials, user corrections, p95 latency, cost, repeated steps and recovery. Keep feature flags that disable the model, individual tools, remote integrations or all commands. A read-only fallback can remain useful during an incident.
Final mentoring perspective
The model is an important component, but it is not the architecture. The professional work lies in the boundaries around it: clear goals, typed decisions, least-privilege tools, durable state, exact approval, observable outcomes and practised recovery.
That should feel reassuring. You do not have to abandon fifteen years of software engineering to build agents. You apply those skills to a component that is flexible, probabilistic and sometimes wrong. Strong ASP.NET Core and Azure foundations are an advantage.