Distributed .NET Systems: Every Boundary Has a Cost

A distributed system is one business process carried out by multiple independent pieces of software that communicate with each other. This can provide independent deployment, scaling and ownership, but it also turns ordinary method calls into network calls.

1. What changes when a system becomes distributed?

Order service
      ↓ network
Payment service
      ↓ network
Stock and delivery services

The caller must now consider timeouts, unavailable services, lost responses, duplicate messages, partial completion, inconsistent views and end-to-end tracing. Distribution does not automatically improve performance; it can introduce latency, serialisation, infrastructure and more difficult debugging.

Use distribution when scaling requirements, business boundaries, team ownership, deployment independence or resilience needs justify those costs.

2. Partial failure is the defining problem

Create order        ✓
Take payment        ✓
Reserve stock       ✗
Send confirmation   ?

A distributed operation may be only partly complete. The design needs explicit behaviour: retry a transient failure, compensate for completed work, wait for recovery, mark the order for manual review or reject it before irreversible work begins.

Code-review question

What should the business do when only part of the process succeeds?

3. CQRS separates the intent to change from the intent to read

public sealed record UpdateOrderTotalCommand(
    int OrderId,
    decimal Total);

public sealed record GetOrderQuery(
    int OrderId);

A command asks the system to change state. A query asks for information without changing business state. A command may still return an identifier, status or validation result. CQRS can begin inside one application and one database; it does not require microservices, queues and separate storage.

4. When can CQRS help?

Read-heavy and write-heavy paths may have different needs. The write side can prioritise business rules, validation and consistency, while the read side uses denormalised data, caching, search indexes or UI-shaped DTOs.

COMMAND → transactional model → SQL Server

QUERY → read-optimised model or index → UI result

Separate read models add synchronisation, storage and possible delay before a write appears in a query. Use CQRS when the workloads genuinely benefit from separation, not because the diagram looks impressive.

5. Event sourcing is not the same as CQRS

AccountOpened     £0
MoneyDeposited    £1,000
MoneyWithdrawn    £200
MoneyWithdrawn    £100
                    ↓
Current balance   £700

public sealed record MoneyDeposited(
    Guid AccountId,
    decimal Amount,
    DateTimeOffset OccurredAt);

CQRS separates commands from queries. Event sourcing stores the events that produced current state. A command says please do this; an event says this happened. Events are facts and are normally immutable. The two patterns are often combined, but either can exist without the other.

Event sourcing provides business history, state rebuilding, new projections and temporal investigation. It also brings event evolution, projection management, eventual consistency, replay and duplicate-handling concerns. Use it where history has real business value, not as a default for ordinary CRUD.

6. Messages can be delivered more than once

if (await processedMessages.ExistsAsync(
        message.PaymentId,
        cancellationToken))
{
    return;
}

await ApplyPaymentAsync(message, cancellationToken);
await processedMessages.AddAsync(
    message.PaymentId,
    cancellationToken);

Reliable messaging commonly permits redelivery. Consumers should be idempotent where practical, so processing the same message again does not create another business effect. In production, the business change and processed-message record require an appropriate atomic design.

7. Avoid losing events between database and broker

If an order is saved but event publication fails, other services never learn about it. Publishing first creates the opposite risk. The transactional outbox commits local business data and an outgoing event together.

One database transaction
   ├── Save order
   └── Save event to outbox

Background publisher
   ├── Read unpublished events
   ├── Publish to broker
   └── Mark as published

The outbox closes the gap between committing data and publishing a message. Consumers still need duplicate handling, monitoring and recovery.

8. Azure Functions run code in response to events

[Function("ProcessOrder")]
public async Task RunAsync(
    [ServiceBusTrigger("orders")] string message,
    CancellationToken cancellationToken)
{
    await _orderService.ProcessAsync(
        message,
        cancellationToken);
}

Azure Functions provide event-driven compute for queues, timers, integrations, bursty workloads and suitable HTTP endpoints. For current C# development, use the .NET isolated worker model. The older in-process model reaches end of support on 10 November 2026; platform dates should still be checked when planning a migration.

9. Durable Functions coordinate long-running workflows

Receive application
        ↓
Check identity and credit
        ↓
Wait for human approval
        ↓
Create agreement and notify

Durable Functions add stateful workflow orchestration. Orchestrators coordinate, activities perform work, entities manage small durable state and clients start or manage workflow instances. The runtime checkpoints progress so the original process does not stay alive for hours or days.

Orchestrators have determinism and replay rules. External I/O and non-deterministic work normally belong in activities rather than being copied directly into orchestration code.

10. Containers and serverless solve different problems

Containers suit long-running APIs, workers, custom runtimes and consistently busy workloads. Azure Functions suit queue processing, timers, event handlers, integrations and work that naturally starts when an event occurs.

Code-review question

Is this a continuously running application, or work that naturally begins when an event occurs?

Flex Consumption is Microsoft's recommended serverless Azure Functions plan where its Linux-based capabilities fit. A real decision should still account for cold starts, regional availability, networking, duration, quotas, cost and supported features.

11. Infrastructure as code makes the environment repeatable

var resourceGroup =
    new ResourceGroup(
        "orders-rg",
        new ResourceGroupArgs
        {
            Location = "uksouth"
        });

Pulumi lets teams define Azure resources with languages including C#. Infrastructure can then participate in version control, review, automated deployment and repeatable environments. For new Pulumi Azure infrastructure projects, Pulumi currently recommends the Azure Native provider.

12. Observability must cross service boundaries

API → Order service → Message broker → Payment worker → Provider

One service log cannot explain the whole journey. Distributed systems need correlation identifiers, structured logs, traces, metrics, queue depth and age, retries, dead-letter counts, dependency timing and business-operation status. Observability is part of the communication design, not an optional dashboard added later.

A practical distributed-systems review checklist

• ✓Business boundary: Is the responsibility genuinely independent?
• ✓Scaling: Does this component need to scale differently?
• ✓Ownership: Can a team own and deploy it independently?
• ✓Latency: What does the network add to the critical path?
• ✓Failure: What happens when the dependency is unavailable?
• ✓Retry: Is retrying safe, bounded and useful?
• ✓Duplication: Can a message be processed more than once safely?
• ✓Partial completion: What happens if only half the workflow succeeds?
• ✓Consistency: How quickly must readers see a change?
• ✓Publishing: Could a database update succeed while event publication fails?
• ✓Recovery: Can failed messages be inspected and replayed?
• ✓Observability: Can one operation be traced across every component?
• ✓Operations: Who deploys, monitors and supports the infrastructure?
• ✓Justification: Is the benefit worth the complexity?

The distributed-systems model to remember

CQRS → separate write and read intent
Event sourcing → store events that produced state
Idempotent consumer → tolerate duplicate delivery
Transactional outbox → commit data and outgoing event together
Azure Functions → run code because an event occurred
Durable Functions → remember workflow progress
Containers → package longer-lived applications
Infrastructure as code → make cloud resources repeatable

The design is not complete when services communicate successfully. It is complete when the system can time out, retry safely, tolerate duplicates, recover from partial failure and explain what happened.