The Agentic Leap — Part 7: Agent Orchestration and Production Control
Orchestration is the calm adult in the room. The model may suggest the next step, tools may return uncertain results and users may take hours to approve a proposal. The orchestrator keeps track of what is true, what is allowed and what must happen next.
Prompts guide behaviour; orchestration enforces behaviour.For an experienced ASP.NET Core developer, this is a state machine, policy layer and durable worker system—not magic.
Orchestration is the trusted control plane around model decisions. It determines what state may advance, which capability may run, how much work is allowed, when approval is required and how the system recovers. A prompt saying “be careful” is not orchestration.
1. Model the lifecycle before writing the loop
Created -> Running -> InputRequired
-> ApprovalRequired -> Running
-> Completed
-> Failed
-> Cancelled
-> BudgetExceededTransitions should be explicit and validated. A completed execution cannot return to running. An expired approval cannot authorise a command. A cancelled run may need reconciliation if a tool outcome is unknown.
public static bool CanTransition(AgentExecutionStatus from, AgentExecutionStatus to) =>
(from, to) switch
{
(Created, Running) => true,
(Running, InputRequired or ApprovalRequired or Completed or Failed
or Cancelled or BudgetExceeded) => true,
(InputRequired, Running or Cancelled) => true,
(ApprovalRequired, Running or Cancelled) => true,
_ => false
};2. Treat budgets as policy
Every execution should have limits for model calls, tool calls, elapsed time, input/output tokens, retries, delegation depth and money. Some limits are global; others depend on user, tenant, goal and environment.
public sealed record ExecutionBudget(
int MaxModelCalls,
int MaxToolCalls,
int MaxRetries,
TimeSpan Deadline,
decimal MaxEstimatedCost,
int MaxDelegationDepth);Reserve budget before starting an operation so parallel workers cannot all spend the remaining allowance. A budget breach is a normal terminal outcome with an explanation, not an exception to hide.
3. Separate decision, policy and execution
The model decision adapter should not execute tools. The policy engine should not call the model. The executor should receive an already-bound request and repeat critical authorisation immediately before a side effect.
DecisionModel -> proposed action
PolicyEngine -> allowed / approval required / denied
Executor -> observation
StateMachine -> durable next stateThis separation makes model providers and orchestration frameworks replaceable without moving business policy into prompts.
4. Approval is a durable workflow
When approval is required:
- Freeze the canonical command arguments.
- Record evidence, policy version and expected consequence.
- Persist state as
ApprovalRequired. - Notify an eligible approver without exposing secrets.
- Record approve, edit or reject with identity and time.
- Revalidate freshness, permission and command hash.
- Execute once using idempotency.
5. Retries belong to failure classes
Use retries for transient faults with a reasonable chance of success. Do not retry policy denials, invalid business transitions or malformed requests unchanged. Coordinate model-level correction with HTTP/database retry policies so attempts do not multiply.
Use exponential backoff with jitter for remote reads and obey server guidance. Apply circuit breakers to unhealthy dependencies. The execution deadline remains authoritative even when a library wants another retry.
6. Parallelism with limits
Independent read-only evidence checks can run concurrently. Commands and state transitions usually require ordering.
await parallelismGate.WaitAsync(ct);
try
{
return await evidenceTool.ExecuteAsync(request, ct);
}
finally
{
parallelismGate.Release();
}Bound fan-out and aggregate partial results explicitly. “Three workers succeeded and one timed out” is more informative than silently continuing with incomplete evidence.
7. Durable execution and the outbox
Persist orchestration state and outgoing messages atomically when possible. An outbox lets a background dispatcher deliver notifications or remote work without losing intent between database commit and broker publication.
For external commands where atomicity is impossible, use idempotency, reconciliation and compensating business operations. Compensation is not database rollback; it is a new auditable action with its own consequences.
8. Trace decisions without recording private reasoning
Create spans for model calls, policy decisions, tool calls, checkpoints, approvals and delegated tasks. Useful attributes include execution ID, action name, outcome, latency, retry count, evidence IDs, model/provider version and token use.
Do not require or store private chain-of-thought. Record concise decision summaries and observable evidence. Apply redaction before telemetry leaves the process.
9. Stopping conditions
An agent must stop when:
- the goal’s completion predicate is satisfied;
- required information is missing;
- permission or approval is denied;
- progress repeats without new evidence;
- budgets or deadlines are reached;
- the user cancels;
- an unexpected failure makes the outcome uncertain.
10. Operational ownership
Provide dashboards for success, refusal, input-required, approval time, failures, costs and tool health. Alerts should identify actionable system conditions rather than every model imperfection. Support staff need a safe timeline, current state, evidence references and recovery action.
Use feature flags and kill switches at agent, model, server and tool level. A read-only degradation mode can preserve useful explanations while commands are disabled.
Extended implementation review
11. Build the control plane as ordinary application code
Separate four responsibilities. The decision adapter asks a model for a typed next step. Policy decides whether that step is permitted. A handler executes one capability. The state machine commits the resulting transition. This allows each part to be tested without a live model and prevents persuasive text from bypassing business rules.
The orchestrator should own the loop counter, deadline and cancellation token. It loads state with a version, assembles minimum context, obtains a decision, validates it, executes or pauses, then saves the next state. A model never writes execution status directly.
Use a queue when work may outlive an HTTP request. Claim runs with leases, renew leases during long operations and rely on optimistic concurrency when committing. Duplicate delivery must be harmless.
12. Budget enforcement before and after work
Estimate whether a proposed step fits remaining time, tokens, money and call count before starting it. Record actual usage afterwards. Reserve capacity for producing a final safe response; otherwise the agent can spend its entire budget gathering evidence and fail without explaining what happened.
Budgets may vary by tenant, user, environment and capability risk. A delegated task needs a sub-budget deducted from the parent. Parallel branches share a total allowance rather than each receiving the full limit. When a limit is reached, persist BudgetExceeded with completed evidence and a user-actionable explanation.
13. Approval workflow details
Approval records should include proposal digest, command type, resource, consequence, evidence versions, policy version, requester, eligible approver, expiry and decision. Use separation of duties where the proposer cannot approve sensitive work. Enforce it in code, not merely in the screen copy.
An edit creates a new proposal and normally a new policy check. Expired approvals fail closed. A second browser submission is idempotent. Recheck resource permission and command preconditions immediately before execution because business state may have changed while waiting.
14. Side effects and the transactional outbox
For a local database command, write the domain change and outbox event in one transaction. A dispatcher publishes the event later and records delivery. Consumers deduplicate by message ID. For an external API, send an idempotency key and persist the request before calling.
If the external outcome is unknown, move to ReconciliationRequired. A reconciliation worker queries the provider or presents the case to operations. Never translate a timeout into “failed” when the action may actually have succeeded.
15. Parallelism and backpressure
Parallel reads are useful when independent, but cap fan-out and connection usage. Preserve which branch produced each observation. Define whether partial results are acceptable and cancel remaining branches when the goal is already satisfied.
Apply queue limits, per-tenant quotas and downstream bulkheads. Backpressure should reject or defer new runs predictably rather than letting latency expand without bound. A dashboard needs queue age and saturation, not only request count.
16. Observability without leaking reasoning
Trace execution, model call, tool call, policy decision, checkpoint and approval as correlated spans. Record model/deployment, schema version, token usage, latency, outcome and evidence identifiers. Do not store hidden reasoning or raw sensitive documents in span attributes.
Metrics should cover task success, safe refusal, input required, approval conversion, step count, repeated decisions, policy denials, cost and recovery. Logs provide redacted operational detail. Audits provide immutable business accountability. These are related but not interchangeable.
17. Incident response and safe degradation
Prepare controls to disable a model, tool, MCP server, A2A partner or all commands independently. Read-only operation can often continue safely. Drain or pause queued runs and show users an honest status.
Runbooks should cover credential compromise, cross-tenant exposure, runaway cost, duplicate commands, poisoned evidence and provider outage. Identify how to find affected runs, revoke access, preserve evidence, reconcile consequences and notify owners. Exercise these paths before an incident.
18. Release and evaluation gates
Use recorded scenarios and shadow traffic before enabling a new model or prompt. Compare tool selection, groundedness, policy outcomes, latency and cost against the current release. Canary by tenant or internal cohort and keep instant rollback.
Production samples should feed a governed evaluation set after redaction and review. A change passes only if critical safety cases remain perfect and overall quality improves within agreed cost and latency. Average quality must never hide a severe permission failure.
19. Worked command timeline
At step five, the model proposes a project task. Policy allows proposal creation but requires a planning manager for execution. The orchestrator stores the proposal body, hash, evidence versions and expiry, then enters ApprovalRequired. No command tool is exposed while the run waits.
The manager approves through an authenticated screen. A queue delivers the resume message twice. The first worker claims the run, verifies the approval and project version, writes command intent and an outbox message, then advances state with optimistic concurrency. The second delivery sees the later state and exits safely. The dispatcher sends the command with an idempotency key.
The downstream response times out. The run becomes ReconciliationRequired, not failed. A worker queries by the same key and finds the created task. It records the external identifier and completes the run. The trace connects proposal, approval, outbox, request and reconciliation without relying on model narration.
20. Control-plane review questions
- Can the model change its own limits or permission set?
- Is every transition validated against durable current state?
- Can one tenant consume another tenant’s worker capacity?
- Are parallel branches bounded and their partial results labelled?
- Is an unknown command outcome distinct from failure?
- Can operators disable commands while retaining safe reads?
- Does every alert point to a documented recovery action?
- Can a new model be canaried and rolled back independently?
21. Capacity and cost planning
Model-call latency is variable, and long-running tools can occupy workers far beyond normal API requests. Size queues and concurrency from measured service times. Apply separate pools for inexpensive reads and consequential commands so a burst of research cannot delay approved work indefinitely.
Forecast worst-case spend using enforced limits, not average prompt size. Alert on cost per accepted outcome, unusual tenant consumption and repeated no-progress steps. Budgets should fail predictably before provider account limits do. Preserve enough telemetry to attribute spend without retaining sensitive prompts.
Test overload deliberately. Confirm that new executions receive a clear deferred or unavailable response, existing approvals do not expire unnoticed and cancellation still works. Recovery after throttling should avoid a simultaneous retry storm. Capacity control is part of safety because an unavailable policy service or saturated database can tempt unsafe fallbacks.
Never bypass an unavailable authorisation or audit dependency to improve availability. Fail closed for commands and explain the degraded state.
Production checklist
- Lifecycle states and transitions are explicit.
- Budgets cover time, calls, cost, retries and delegation.
- Decision, policy, execution and persistence are separate.
- Approval binds to exact, fresh command intent.
- Parallel work is bounded and partial failure is visible.
- Outbox/idempotency/reconciliation protect side effects.
- Traces record observable decisions, not private reasoning.
- Loop detection and hard stopping conditions exist.
- Operators have kill switches and recovery guidance.
22. Continue the series
The control plane can be implemented directly or combined with a framework. Part 8 examines current LangChain agents, tools and middleware while keeping ordinary engineering visible.
A small C# orchestration loop
Keep the loop explicit enough to test:
public async Task AdvanceAsync(Guid executionId, CancellationToken ct)
{
var state = await store.LoadAsync(executionId, ct);
budget.EnsureStepIsAllowed(state);
var context = await contextFactory.CreateAsync(state, ct);
var decision = await model.DecideAsync(context, ct);
var policyResult = await policy.EvaluateAsync(state, decision, ct);
if (!policyResult.Allowed)
{
await store.RecordDenialAsync(state, policyResult, ct);
return;
}
var observation = await handlers.ExecuteAsync(decision, state, ct);
await store.TryCommitAsync(state, observation, ct);
}The real implementation handles typed terminal states, optimistic concurrency, unknown command outcomes and cancellation. The value of this example is the separation: model decision, policy, execution and persistence remain independently testable.
Azure production shape
An HTTP request creates the execution and returns quickly. Azure Service Bus delivers an execution ID to workers. Azure SQL stores lifecycle, budgets, proposals and approvals. Workers use managed identity and claim work with leases plus optimistic concurrency. Key Vault holds secrets, while App Configuration or feature management supplies tool and model kill switches.
OpenTelemetry spans connect the ASP.NET Core request, Foundry model call, Azure AI Search retrieval, policy decision, tool execution and checkpoint. Application Insights dashboards show success, refusal, approval time, repeated steps, cost and reconciliation backlog. Business audits remain separate from diagnostic telemetry.
If a command times out after dispatch, the state becomes ReconciliationRequired. A worker queries by idempotency key and records the known result. It never asks the model to guess. If authorization or audit dependencies are unavailable, commands fail closed while safe read-only experiences may continue.
A mentor's operational test
Take one execution and place a failure between every pair of steps. Ask what the next worker observes and how it continues without duplicating consequences. Then test two workers, a repeated queue message, an expired approval and a permission change.
If the answer depends on an in-memory variable, a prompt remembering correctly or an operator manually editing status, the control plane is not yet production-ready.