Revisiting Microservices Design Patterns in .NET: Trade-offs and Practice Notes

Let’s sit down and treat this as a serious architecture mentoring session.

I am assuming you already understand C#, ASP.NET Core, EF Core, SQL and APIs. The next step is learning how those skills change when one application becomes a distributed system: boundaries matter more, networks fail, data is no longer updated in one transaction and operational visibility becomes part of the design.

I want to walk you through the decisions I consider when designing microservices in .NET—from monoliths and Domain-Driven Design through REST, gRPC, messaging, CQRS, sagas, resilience, gateways, security, containers and observability.

Because microservices are not “many APIs.”

That is the first thing I want you to remember.

Breaking an application into several APIs does not automatically create a good microservices architecture. I want you to ask: Have we separated genuine business capabilities? Does each service own its data? What happens when a dependency is unavailable? How is consistency maintained? Can we trace a request across the system? Can services be deployed independently? How is every hop secured?

That is the real microservices conversation.

---

1. Monolith, modular monolith, and microservices

Let’s begin with the big picture.

A monolith is a single application where the UI, business logic, data access and database are usually deployed together. That is not automatically bad. In fact, many successful systems start as monoliths because they are comparatively simple to build, deploy and debug.

The problem starts when it grows badly.

One change affects many modules. One deployment redeploys the whole system. One database becomes shared by everything. One slow module can affect the whole app. Different teams begin stepping on each other’s code.

Then comes the modular monolith: one deployable application organised internally around business modules such as appointments, patients, billing, documents and reporting. This is often the best middle ground because each module can have clear logic and boundaries without taking on distributed-system costs.

This is what I usually recommend before microservices.

Do not jump from messy monolith to messy microservices. First learn to build a clean modular monolith.

A microservices architecture goes further. Each major business capability becomes an independent service. The key idea is that services are autonomous units, not merely folders in a solution.

A healthcare system example could become:

Patient Service
Appointment Service
Doctor Service
Billing Service
Document Service
Notification Service
Reporting Service

Each service should have a business reason to exist. Do not create a microservice because the table name looks important. Create a microservice because the business capability has its own lifecycle, rules, data ownership, team ownership, scaling needs, and deployment needs.

My rule of thumb:

Start with a modular monolith unless the business, team, deployment, scale, or domain complexity genuinely earns microservices.

---

2. Domain-Driven Design: the map before the code

Microservices need boundaries. DDD helps us find those boundaries.

DDD means we design software around the business domain, not around technical tables. Its core ideas include entities, domain models, ubiquitous language, bounded contexts, value objects, aggregates and aggregate roots. The same concept, such as Patient, can appear in different bounded contexts with different meanings.

This is very important.

In Patient Management, a patient may have full medical history, demographic data, NHS number, insurance, emergency contacts and consent documents.

In Appointment Management, maybe we only need:

public record PatientReference(Guid PatientId, string FullName);

That is enough. We do not drag the whole patient record into every service.

A bounded context is a boundary where a model has a specific meaning. “Patient” in document management is not necessarily the same model as “Patient” in billing.

Now look at entities and value objects.

An entity has identity. Its ID matters.

public abstract class Entity<TId>
{
    public TId Id { get; protected set; }
}

public class Patient : Entity<Guid>
{
    public string FullName { get; private set; }
    public DateOnly DateOfBirth { get; private set; }

    private Patient() { } // EF Core

    public Patient(Guid id, string fullName, DateOnly dateOfBirth)
    {
        if (string.IsNullOrWhiteSpace(fullName))
            throw new ArgumentException("Patient name is required.");

        Id = id;
        FullName = fullName;
        DateOfBirth = dateOfBirth;
    }

    public void Rename(string newName)
    {
        if (string.IsNullOrWhiteSpace(newName))
            throw new ArgumentException("Patient name is required.");

        FullName = newName;
    }
}

This is not just a data class. It protects business rules.

A value object has no identity; its values define it. Entities are identity-based and independently persisted, while value objects use structural equality and are usually immutable.

Example:

public readonly record struct TimeSlot(DateTime Start, DateTime End)
{
    public TimeSlot
    {
        if (End <= Start)
            throw new ArgumentException("End time must be after start time.");
    }

    public bool Overlaps(TimeSlot other)
    {
        return Start < other.End && other.Start < End;
    }
}

Two TimeSlot objects with the same start and end are equal. We do not care about an ID.

An aggregate is a consistency boundary. It groups related entities and value objects, and all changes go through the aggregate root.

public class Appointment : Entity<Guid>
{
    public Guid PatientId { get; private set; }
    public Guid DoctorId { get; private set; }
    public TimeSlot Slot { get; private set; }
    public string Purpose { get; private set; }
    public bool Confirmed { get; private set; }

    private Appointment() { }

    public Appointment(Guid id, Guid patientId, Guid doctorId, TimeSlot slot, string purpose)
    {
        Id = id;
        PatientId = patientId;
        DoctorId = doctorId;
        Slot = slot;
        Purpose = purpose;
    }

    public void Reschedule(TimeSlot newSlot)
    {
        if (newSlot.Start < DateTime.UtcNow)
            throw new InvalidOperationException("Cannot reschedule to the past.");

        Slot = newSlot;
        Confirmed = false;
    }

    public void Confirm()
    {
        Confirmed = true;
    }
}

The aggregate root protects the rules. External code should not directly manipulate internal objects in a way that breaks consistency.

My rule of thumb:

Microservices begin with boundaries. DDD helps you discover those boundaries before you start creating APIs.

---

3. Synchronous communication: REST, HTTP and gRPC

Sooner or later, one service needs information from another service.

Synchronous communication means Service A calls Service B and waits for a response. REST and gRPC are the important modern choices, but both bring latency, coupling and cascading-failure risks.

Example: Appointment Service needs doctor details.

public sealed class DoctorsApiClient
{
    private readonly HttpClient _httpClient;

    public DoctorsApiClient(HttpClient httpClient)
    {
        _httpClient = httpClient;
    }

    public async Task<DoctorDto?> GetDoctorAsync(Guid doctorId, CancellationToken ct)
    {
        var response = await _httpClient.GetAsync($"/api/doctors/{doctorId}", ct);

        if (response.StatusCode == HttpStatusCode.NotFound)
            return null;

        response.EnsureSuccessStatusCode();

        return await response.Content.ReadFromJsonAsync<DoctorDto>(cancellationToken: ct);
    }
}

Register it properly:

builder.Services.AddHttpClient<DoctorsApiClient>(client =>
{
    client.BaseAddress = new Uri(builder.Configuration["Services:DoctorsApi"]!);
});

Do not create new HttpClient() randomly everywhere. Use IHttpClientFactory through typed clients.

REST is simple and works very well for resource-based operations:

GET /api/appointments/123
POST /api/appointments
PUT /api/appointments/123
DELETE /api/appointments/123

gRPC is more contract-driven and high-performance. It uses Protocol Buffers and HTTP/2: the client calls through a generated stub, serialises with Protobuf and sends the request for the server to deserialise and process.

A simplified .proto contract:

syntax = "proto3";

service DocumentService {
  rpc GetPatientDocuments (PatientDocumentRequest) returns (DocumentList);
}

message PatientDocumentRequest {
  string patientId = 1;
}

message DocumentDto {
  string id = 1;
  string name = 2;
  string documentType = 3;
}

message DocumentList {
  repeated DocumentDto documents = 1;
}

Use REST when you want broad compatibility, simple debugging, browser/mobile friendliness and standard HTTP semantics.

Use gRPC when you need high-performance service-to-service communication, strong contracts, streaming, smaller payloads and controlled internal communication.

My rule of thumb:

Synchronous communication is easy to understand but dangerous when overused. Every extra network hop is another chance for latency and failure.

---

4. Asynchronous communication: queues, events and eventual consistency

Asynchronous communication means one service sends a message and does not wait for the final work to complete.

This is where microservices start becoming real.

Asynchronous communication introduces message queues, message buses, publish-subscribe, eventual consistency, message ordering and duplicate delivery. A queue normally distributes work to consumers, while publish-subscribe makes one event available to several interested consumers.

Think about booking an appointment. The critical operation is creating the appointment. But sending email, creating calendar entry, notifying billing and updating reporting do not all need to block the user.

So after booking:

public record AppointmentBookedIntegrationEvent(
    Guid AppointmentId,
    Guid PatientId,
    Guid DoctorId,
    DateTime Start,
    DateTime End,
    string CorrelationId);

Publish the event:

public async Task<Guid> BookAppointmentAsync(BookAppointmentCommand command)
{
    var appointment = new Appointment(
        Guid.NewGuid(),
        command.PatientId,
        command.DoctorId,
        new TimeSlot(command.Start, command.End),
        command.Purpose);

    _db.Appointments.Add(appointment);

    await _db.SaveChangesAsync();

    await _eventBus.PublishAsync(new AppointmentBookedIntegrationEvent(
        appointment.Id,
        appointment.PatientId,
        appointment.DoctorId,
        appointment.Slot.Start,
        appointment.Slot.End,
        command.CorrelationId));

    return appointment.Id;
}

Then Notification Service subscribes:

public sealed class AppointmentBookedConsumer
{
    private readonly IEmailSender _emailSender;

    public AppointmentBookedConsumer(IEmailSender emailSender)
    {
        _emailSender = emailSender;
    }

    public async Task HandleAsync(AppointmentBookedIntegrationEvent message)
    {
        await _emailSender.SendAsync(
            message.PatientId,
            "Your appointment is booked",
            $"Appointment time: {message.Start}");
    }
}

Now the Appointment Service does not need to know how email works.

But asynchronous messaging introduces a new reality: eventual consistency.

The appointment may be created now. The email may be sent in 10 seconds. The reporting read model may update in 30 seconds. The calendar service may retry if it is temporarily down.

That is acceptable if the business understands it.

My rule of thumb:

Use async messaging for side effects, integration events, workflows and decoupling. But design idempotency, retries, dead-letter queues and monitoring from day one.

Idempotency means processing the same message twice should not cause duplicate damage.

public async Task HandleAsync(AppointmentBookedIntegrationEvent message)
{
    bool alreadyProcessed = await _db.ProcessedMessages
        .AnyAsync(x => x.MessageId == message.AppointmentId);

    if (alreadyProcessed)
        return;

    await _calendar.CreateEntryAsync(message);

    _db.ProcessedMessages.Add(new ProcessedMessage(message.AppointmentId));
    await _db.SaveChangesAsync();
}

This protects you from duplicate messages.

---

5. Aggregator pattern: protecting the UI from service chaos

Without an aggregator, the frontend may need to call:

GET /appointments/123
GET /patients/456
GET /doctors/789
GET /documents?patientId=456
GET /billing?appointmentId=123

That makes the UI chatty and tightly coupled to backend service structure.

The Aggregator pattern creates a backend service that composes data from multiple services and gives the UI one clean response. I use it to serve the UI efficiently and keep cross-service knowledge out of the frontend.

[ApiController]
[Route("api/appointment-details")]
public class AppointmentDetailsController : ControllerBase
{
    private readonly AppointmentsApiClient _appointments;
    private readonly PatientsApiClient _patients;
    private readonly DoctorsApiClient _doctors;

    public AppointmentDetailsController(
        AppointmentsApiClient appointments,
        PatientsApiClient patients,
        DoctorsApiClient doctors)
    {
        _appointments = appointments;
        _patients = patients;
        _doctors = doctors;
    }

    [HttpGet("{appointmentId:guid}")]
    public async Task<ActionResult<AppointmentDetailsDto>> Get(Guid appointmentId)
    {
        var appointment = await _appointments.GetAsync(appointmentId);

        if (appointment is null)
            return NotFound();

        var patientTask = _patients.GetAsync(appointment.PatientId);
        var doctorTask = _doctors.GetAsync(appointment.DoctorId);

        await Task.WhenAll(patientTask, doctorTask);

        return Ok(new AppointmentDetailsDto(
            appointment.Id,
            patientTask.Result!,
            doctorTask.Result!,
            appointment.Start,
            appointment.End,
            appointment.Purpose));
    }
}

This pattern is powerful, but be careful. Aggregator can become a mini-monolith if it starts owning business logic. It should compose, not secretly become the brain of the system.

Best practices:

Cache reference data. Use parallel calls where safe. Handle partial failures gracefully. Compress large responses. Add logging and correlation IDs. Avoid putting domain rules inside aggregator.

---

6. CQRS: commands change state, queries return data

CQRS stands for Command Query Responsibility Segregation. It separates reads and writes into different paths so each can be understood and optimised independently.

A command changes state:

public record BookAppointmentCommand(
    Guid PatientId,
    Guid DoctorId,
    DateTime Start,
    DateTime End,
    string Purpose) : IRequest<Guid>;

Handler:

public sealed class BookAppointmentHandler
    : IRequestHandler<BookAppointmentCommand, Guid>
{
    private readonly AppDbContext _db;

    public BookAppointmentHandler(AppDbContext db)
    {
        _db = db;
    }

    public async Task<Guid> Handle(BookAppointmentCommand request, CancellationToken ct)
    {
        var slot = new TimeSlot(request.Start, request.End);

        bool doctorBusy = await _db.Appointments.AnyAsync(a =>
            a.DoctorId == request.DoctorId &&
            a.Slot.Start < request.End &&
            request.Start < a.Slot.End,
            ct);

        if (doctorBusy)
            throw new InvalidOperationException("Doctor is not available.");

        var appointment = new Appointment(
            Guid.NewGuid(),
            request.PatientId,
            request.DoctorId,
            slot,
            request.Purpose);

        _db.Appointments.Add(appointment);
        await _db.SaveChangesAsync(ct);

        return appointment.Id;
    }
}

A query returns data:

public record GetDoctorAppointmentsQuery(Guid DoctorId, DateOnly Date)
    : IRequest<IReadOnlyList<AppointmentListItemDto>>;

Handler:

public sealed class GetDoctorAppointmentsHandler
    : IRequestHandler<GetDoctorAppointmentsQuery, IReadOnlyList<AppointmentListItemDto>>
{
    private readonly AppDbContext _db;

    public GetDoctorAppointmentsHandler(AppDbContext db)
    {
        _db = db;
    }

    public async Task<IReadOnlyList<AppointmentListItemDto>> Handle(
        GetDoctorAppointmentsQuery request,
        CancellationToken ct)
    {
        var from = request.Date.ToDateTime(TimeOnly.MinValue);
        var to = request.Date.ToDateTime(TimeOnly.MaxValue);

        return await _db.Appointments
            .AsNoTracking()
            .Where(a => a.DoctorId == request.DoctorId &&
                        a.Slot.Start >= from &&
                        a.Slot.Start <= to)
            .Select(a => new AppointmentListItemDto(
                a.Id,
                a.PatientId,
                a.Slot.Start,
                a.Slot.End,
                a.Purpose))
            .ToListAsync(ct);
    }
}

The beauty is clarity.

Commands are behaviour. Queries are read models. Commands care about business invariants. Queries care about speed and shape.

My rule of thumb:

CQRS is not always separate databases. At minimum, it is separate mental paths for reads and writes.

---

7. Event sourcing: store what happened, not just current state

Normal systems store current state:

Appointment.Status = Confirmed

Event sourcing stores the history of changes:

AppointmentBooked
AppointmentRescheduled
AppointmentConfirmed

Then current state is rebuilt by replaying events.

public abstract record AppointmentEvent(Guid AppointmentId, DateTime OccurredAt);

public record AppointmentBooked(
    Guid AppointmentId,
    Guid PatientId,
    Guid DoctorId,
    DateTime Start,
    DateTime End,
    DateTime OccurredAt) : AppointmentEvent(AppointmentId, OccurredAt);

public record AppointmentConfirmed(
    Guid AppointmentId,
    DateTime OccurredAt) : AppointmentEvent(AppointmentId, OccurredAt);

Aggregate applying events:

public class AppointmentState
{
    public Guid Id { get; private set; }
    public bool Confirmed { get; private set; }
    public TimeSlot Slot { get; private set; }

    public void Apply(AppointmentEvent evt)
    {
        switch (evt)
        {
            case AppointmentBooked booked:
                Id = booked.AppointmentId;
                Slot = new TimeSlot(booked.Start, booked.End);
                break;

            case AppointmentConfirmed:
                Confirmed = true;
                break;
        }
    }
}

Event sourcing is excellent when audit/history matters, such as healthcare, banking, payments, legal and compliance. But it adds complexity. You need event versioning, snapshots, projections, replay, event store design and careful schema evolution.

My rule of thumb:

Event sourcing is powerful when history is the business. Do not use it just because it sounds architectural.

---

8. Database per service, polyglot persistence and consistency

One of the biggest microservices rules is: each service should own its data.

That does not always mean every service must use a different database technology. It means no other service should directly own or modify its tables.

Bad:

Billing Service directly queries AppointmentDb.Appointments

Better:

Billing Service receives AppointmentBooked event
Billing Service stores its own billing-relevant appointment reference

The wider data discussion includes database-per-service, shared databases, relational and non-relational stores, ORM choices, polyglot persistence, sharding and eventual consistency.

A service may use SQL Server for transactional data. Another may use MongoDB for documents. Another may use Redis for cache. Another may use Elasticsearch/Azure AI Search for search. Another may use blob storage for files.

That is polyglot persistence: choose the data store based on the service’s need.

My rule of thumb:

A database is not just storage. It is part of the service boundary.

---

9. Saga pattern: business transactions across services

In a monolith, a transaction can update multiple tables.

In microservices, each service has its own database. You cannot rely on one giant database transaction across everything.

The Saga pattern coordinates a long-running business workflow.

Example appointment booking workflow:

Create appointment
Reserve doctor slot
Generate invoice
Take payment
Confirm appointment
Send notification

If payment fails, we may need compensating actions:

Cancel invoice
Release doctor slot
Cancel appointment
Notify user

Choreography means services react to events:

AppointmentCreated → Billing creates invoice
InvoiceCreated → Payment takes payment
PaymentSucceeded → Appointment confirms
PaymentFailed → Appointment cancels

Orchestration means one coordinator controls the workflow:

BookingSaga orchestrator tells each service what to do next

Choreography is loosely coupled but can become hard to understand. Orchestration is easier to follow but introduces a central coordinator.

My rule of thumb:

In microservices, rollback is not always “undo SQL transaction.” It is often “run compensating business action.”

---

10. Resilience: design for failure from day one

Modern applications must expect failure. That makes timeouts, careful retries, circuit breakers, caching, asynchronous failure handling, broker resilience and the outbox pattern part of the architecture rather than later enhancements.

This is where many microservice systems fail in real life.

You call Doctor Service. It is slow. Then Appointment Service hangs. Then UI waits. Then thread pool gets exhausted. Then users retry. Then traffic doubles. Then everything collapses.

Use timeout:

builder.Services.AddHttpClient<DoctorsApiClient>()
    .ConfigureHttpClient(client =>
    {
        client.Timeout = TimeSpan.FromSeconds(3);
    });

Use retry carefully. Retry only transient failures, not validation errors.

Use circuit breaker so repeated failures stop hammering the broken service.

Use cache for reference data:

public async Task<DoctorDto?> GetDoctorCachedAsync(Guid id)
{
    string cacheKey = $"doctor:{id}";

    var cached = await _cache.GetStringAsync(cacheKey);

    if (cached is not null)
        return JsonSerializer.Deserialize<DoctorDto>(cached);

    var doctor = await _doctorsApi.GetDoctorAsync(id);

    if (doctor is not null)
    {
        await _cache.SetStringAsync(
            cacheKey,
            JsonSerializer.Serialize(doctor),
            new DistributedCacheEntryOptions
            {
                AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(10)
            });
    }

    return doctor;
}

The outbox pattern is especially important.

Problem:

Save appointment to DB succeeds
Publish event fails

Now the system is inconsistent.

Outbox solution:

Save the appointment and outgoing event in the same database transaction. A background worker later publishes unpublished events.

public class OutboxMessage
{
    public Guid Id { get; set; }
    public string Type { get; set; } = "";
    public string Payload { get; set; } = "";
    public DateTime CreatedAt { get; set; }
    public DateTime? PublishedAt { get; set; }
}

My rule of thumb:

Resilience is not try/catch. Resilience is timeouts, retries, circuit breakers, idempotency, outbox, queues, health checks, observability and graceful degradation.

---

11. API Gateway and BFF

An API Gateway is the front door to your microservices.

It can handle routing, authentication, rate limiting, header forwarding, request shaping, SSL termination, logging and policy enforcement.

In .NET, the gateway discussion commonly includes reverse proxies, YARP, Ocelot, BFFs, gateway security and rate limiting.

YARP-style thinking:

{
  "ReverseProxy": {
    "Routes": {
      "appointments-route": {
        "ClusterId": "appointments-cluster",
        "Match": {
          "Path": "/appointments/{**catch-all}"
        }
      }
    },
    "Clusters": {
      "appointments-cluster": {
        "Destinations": {
          "appointments-api": {
            "Address": "https://localhost:7001/"
          }
        }
      }
    }
  }
}

BFF means Backend for Frontend.

You may have:

Web BFF
Mobile BFF
Admin Portal BFF

Why? Because mobile and web may need different response shapes. Do not force one generic API to satisfy every client badly.

My rule of thumb:

API Gateway protects and routes. BFF shapes experiences for specific frontends.

---

12. Micro frontends

Micro frontends apply microservice thinking to the UI.

Instead of one giant frontend owned by one team, different domain teams own parts of the UI.

Example:

Appointments UI
Billing UI
Documents UI
Patient Profile UI
Admin UI

Micro-frontend design brings DDD alignment, framework choices, gateway support, lazy loading, AOT compilation, resilience, state sharing and session state into the conversation.

But be careful. Micro frontends can create duplicated styling, inconsistent UX, shared dependency problems and complicated deployments.

My rule of thumb:

Use micro frontends when team independence and domain ownership matter enough to justify frontend composition complexity.

---

13. Security: JWT, OAuth, OpenID Connect and service trust

Security becomes harder in microservices because there are more boundaries.

In a monolith, the request enters one app.

In microservices:

User → Gateway → BFF → Service A → Service B → Message Broker → Service C

Every hop matters.

The security model includes bearer tokens, JWTs, OAuth, OpenID Connect, identity providers, token authentication and gateway security.

JWT contains claims:

{
  "sub": "user-123",
  "role": "Admin",
  "scope": "appointments.read appointments.write"
}

In ASP.NET Core:

builder.Services
    .AddAuthentication("Bearer")
    .AddJwtBearer("Bearer", options =>
    {
        options.Authority = builder.Configuration["Identity:Authority"];
        options.Audience = "appointments-api";
    });

builder.Services.AddAuthorization(options =>
{
    options.AddPolicy("CanBookAppointments", policy =>
    {
        policy.RequireClaim("scope", "appointments.write");
    });
});

Endpoint:

[Authorize(Policy = "CanBookAppointments")]
[HttpPost]
public async Task<IActionResult> Book(BookAppointmentCommand command)
{
    var id = await _mediator.Send(command);
    return CreatedAtAction(nameof(GetById), new { id }, null);
}

My rule of thumb:

Authentication proves who you are. Authorization decides what you can do. In microservices, both must be enforced consistently, not only at the UI.

---

14. Containers, Docker, Docker Compose and Kubernetes

Containers make services portable and repeatable.

Cloud deployment brings Docker, container images, Dockerfiles, native .NET container support, registries, Docker Compose and Kubernetes into the architecture.

A simple Dockerfile:

FROM mcr.microsoft.com/dotnet/aspnet:10.0 AS runtime
WORKDIR /app

FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
WORKDIR /src

COPY ["Appointments.Api/Appointments.Api.csproj", "Appointments.Api/"]
RUN dotnet restore "Appointments.Api/Appointments.Api.csproj"

COPY . .
WORKDIR "/src/Appointments.Api"
RUN dotnet publish "Appointments.Api.csproj" -c Release -o /app/publish

FROM runtime AS final
WORKDIR /app
COPY --from=build /app/publish .
ENTRYPOINT ["dotnet", "Appointments.Api.dll"]

Docker Compose lets you run multiple services locally:

services:
  appointments-api:
    build: ./Appointments.Api
    ports:
      - "7001:8080"
    depends_on:
      - rabbitmq
      - redis

  rabbitmq:
    image: rabbitmq:4.0-management
    ports:
      - "5672:5672"
      - "15672:15672"

  redis:
    image: redis:latest
    ports:
      - "6379:6379"

Kubernetes handles orchestration at scale: scheduling, replicas, service discovery, health checks, rolling deployments and self-healing.

My rule of thumb:

Docker packages the service. Kubernetes operates many services.

---

15. Serverless microservices

Serverless means you focus on functions and events while the cloud provider manages much of the infrastructure.

In .NET, serverless work commonly involves Azure Functions, triggers, queues, durable orchestration, dependency injection, observability and security.

Example Azure Function idea:

public class AppointmentEmailFunction
{
    private readonly IEmailSender _emailSender;

    public AppointmentEmailFunction(IEmailSender emailSender)
    {
        _emailSender = emailSender;
    }

    [Function("SendAppointmentEmail")]
    public async Task Run(
        [QueueTrigger("appointment-booked")] AppointmentBookedEvent message,
        FunctionContext context)
    {
        await _emailSender.SendAsync(
            message.PatientId,
            "Appointment booked",
            $"Your appointment is booked for {message.Start}");
    }
}

Use serverless for event-driven workloads, background tasks, lightweight APIs, scheduled jobs and integration glue.

Do not use it blindly for everything. Cold starts, execution limits, local debugging, distributed tracing and vendor lock-in matter.

My rule of thumb:

Serverless is excellent for event-driven pieces. Containers are better when you need full runtime control.

---

16. Observability: logs, metrics, traces and Aspire

In a monolith, you can often debug locally.

In microservices, the bug may cross six services, one queue, one database and one cache.

That is why observability matters.

The .NET observability toolbox includes structured logging, the logging API, Serilog, Seq, distributed tracing, OpenTelemetry, sidecars, service meshes, Dapr and .NET Aspire.

Structured logging:

_logger.LogInformation(
    "Booking appointment {AppointmentId} for patient {PatientId} with doctor {DoctorId}",
    appointment.Id,
    appointment.PatientId,
    appointment.DoctorId);

This is better than:

_logger.LogInformation("Booking appointment");

Because now logs can be searched by appointment ID, patient ID and doctor ID.

Correlation ID matters:

app.Use(async (context, next) =>
{
    var correlationId =
        context.Request.Headers["X-Correlation-ID"].FirstOrDefault()
        ?? Guid.NewGuid().ToString();

    context.Response.Headers["X-Correlation-ID"] = correlationId;

    using (_logger.BeginScope(new Dictionary<string, object>
    {
        ["CorrelationId"] = correlationId
    }))
    {
        await next();
    }
});

OpenTelemetry allows traces to travel across services. You can see:

Gateway → Appointment API → Doctor API → SQL → RabbitMQ → Notification Service

This is how you find bottlenecks and failures.

Dapr sidecars can help with service invocation, pub-sub, state management and bindings, while .NET Aspire improves the local development story by helping developers run and observe distributed .NET applications.

My rule of thumb:

If you cannot observe it, you cannot operate it. Logs tell you what happened. Metrics tell you how often. Traces tell you where the request travelled.

---

17. The mentoring case study: should we extract Loan Decisions?

Let us turn the patterns into one architectural decision.

We maintain a modular lending application. Applications arrive through a web portal. Staff verify identity and affordability. A rules engine proposes a decision. An authorised underwriter can approve, decline or refer the case. Notifications and reporting follow.

The team says the decision area is slow to change because every release coordinates with the portal and documents work. Someone proposes a LoanDecisionService.

Junior: Microservices let us deploy independently, so should we extract it now?

>

Senior: Independent deployment is an outcome we must earn. First prove the boundary, ownership and operational need. A network call between two projects does not make either one independent.

Start with evidence:

• Which changes are currently coupled?
• Do decision rules have a distinct language and owner?
• Must this capability scale or deploy independently?
• Can one team own its code, data, support and on-call responsibility?
• What availability does the wider application require when decisions are unavailable?
• Which workflows cross the proposed boundary?
• Can the organisation operate another deployable unit safely?

Suppose interviews reveal that the Lending Decisions team owns eligibility policy and releases weekly, while the customer portal releases monthly. Decision calculation has bursty demand and needs an audit trail with a different retention policy. That is credible extraction pressure.

Now list the costs: another deployment pipeline, database, secrets, alerts, dashboards, runbook, dependency contract, failure modes, security boundary and local-development story. The decision is not “modern versus old.” It is whether the benefits exceed these recurring costs.

Write a service responsibility statement

A useful statement is narrow and testable:

Loan Decisions owns the rules and lifecycle for evaluating and recording a lending decision. It accepts an immutable decision request, owns the resulting decision and evidence, and publishes outcome facts. It does not own applicant identity, uploaded documents, customer communication or the portal workflow.

That statement prevents the service becoming “everything related to loans.” It also exposes dependencies. The decision needs selected applicant and affordability facts, but it must not reach into the Applications database. Data ownership means other services cannot treat a database schema as a shared internal API.

Define terms with the domain experts. Is “referred” a decision, a pending state or a work-queue outcome? Can a declined application be reconsidered? Who may override an automated recommendation? Ambiguous language becomes ambiguous contracts.

18. Start with a seam inside the modular monolith

Before extraction, make the boundary real in-process. Give the Decisions module an application API and private persistence schema. Prevent other modules from using its EF Core entities. Communicate through commands, queries and in-process events whose shapes could survive a process boundary.

public interface ILoanDecisionModule
{
    Task<DecisionReceipt> RequestDecisionAsync(
        RequestLoanDecision command,
        CancellationToken cancellationToken);

    Task<DecisionView?> GetDecisionAsync(
        Guid decisionId,
        CancellationToken cancellationToken);
}

public sealed record RequestLoanDecision(
    Guid RequestId,
    Guid ApplicationId,
    decimal RequestedAmount,
    decimal VerifiedAnnualIncome,
    int CreditBand,
    long ApplicationVersion);

The contract contains facts needed for a decision. It does not contain the Application entity or a callback into another module. RequestId gives the logical operation a stable identity. ApplicationVersion tells us which snapshot was evaluated.

Enforce the seam with project references, architecture tests and database permissions where practical. Run load and failure tests while everything is still in one process. If the boundary changes every week, extraction would convert cheap method refactoring into expensive distributed contract migration.

Junior: Is a modular monolith only a temporary step?

>

Senior: No. It can be the correct long-term architecture. The seam gives us clarity whether or not we ever add a network.

Use branch by abstraction when extraction becomes justified. Existing callers use the module interface. A new adapter can call the remote service for a controlled cohort. This supports comparison, rollback and gradual migration without rewriting every consumer at once.

19. Define the contract as a product

Once a boundary crosses a process, the contract needs compatibility, ownership and support rules.

For a synchronous request, a JSON HTTP contract might be:

POST /v1/decisions
Idempotency-Key: 8dbd40ce-9f23-4cbd-b850-b33ff681018d
Content-Type: application/json

{
  "applicationId": "7ef7c7bf-b1dc-4a91-a9a1-937706611f63",
  "applicationVersion": 18,
  "requestedAmount": 240000,
  "verifiedAnnualIncome": 72000,
  "creditBand": 4
}

A successful asynchronous acceptance can return 202 Accepted with an operation location. If evaluation completes inside the request, 201 Created with the decision resource may be appropriate. Choose semantics that describe reality; do not return 200 OK for every outcome.

Use RFC-compatible problem details for errors and define machine-readable codes:

{
  "type": "https://dotnetdeveloper.co.uk/problems/stale-application",
  "title": "The application snapshot is stale",
  "status": 409,
  "code": "application_version_conflict",
  "expectedVersion": 19,
  "traceId": "00-…"
}

Consumers should branch on status and stable codes, not English messages. Do not expose stack traces or internal type names.

Compatibility rules

Adding an optional response field is usually easier than renaming a field or changing its meaning. Consumers may reject unknown enum values, so plan for evolution. Never reuse an event name for a semantically different fact. Keep contract tests and consumer examples in source control.

Version only when compatibility cannot be maintained. Supporting /v1 and /v2 indefinitely is not a strategy; define adoption telemetry, a sunset date and responsible consumers. For events, parallel publication or an upcaster may help, but both increase operational work.

Avoid a “common models” package shared by every service. It creates compile-time convenience and organisational coupling: one team's internal type change forces coordinated upgrades. Share narrow infrastructure abstractions when valuable, but let each consumer own its interpretation of an external contract.

20. Synchronous calls: budgets, not hope

The portal calls Applications, which calls Decisions, which calls Risk. If each service has a ten-second timeout, the user does not have a ten-second experience; retries and queues can multiply delay and resource use.

Start with the end-to-end latency budget. Suppose the decision screen must respond within two seconds at the 95th percentile:

These are starting allocations, not promises. Measure them and revise. Propagate cancellation and deadlines so downstream work stops when the caller no longer needs it.

Use IHttpClientFactory and typed clients. Current .NET resilience support is provided by Microsoft.Extensions.Http.Resilience, built on Polly. The older Microsoft.Extensions.Http.Polly package is deprecated. A standard handler supplies a composed set of strategies, but defaults are not a substitute for understanding the operation.

builder.Services
    .AddHttpClient<DecisionClient>(client =>
    {
        client.BaseAddress = new Uri("https+http://decisions");
        client.Timeout = Timeout.InfiniteTimeSpan;
    })
    .AddStandardResilienceHandler(options =>
    {
        options.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(2);
        options.Retry.DisableForUnsafeHttpMethods();
    });

The exact values need production evidence. Disabling automatic retry for unsafe methods is a sound default when the operation may create side effects. If a POST is idempotent through a key and server-side record, you can deliberately configure retry for the known transient cases.

Junior: Why not retry every 500 three times?

>

Senior: Because the server may have committed before its response failed. A retry without idempotency can repeat the business action. It also triples load while a dependency is already unhealthy.

Use timeouts to bound waiting, circuit breakers to stop repeatedly calling an unhealthy dependency, concurrency limits to protect resources, and retry for brief faults where repetition is safe. A circuit breaker is not a retry. A timeout does not cancel work unless cancellation reaches and is honoured by the dependency.

Avoid long synchronous chains. Each dependency reduces total availability. If the caller does not need the downstream result now, publish work asynchronously or build a local read model.

21. gRPC or JSON HTTP?

gRPC uses an explicit Protocol Buffers contract, compact binary payloads and strong streaming support. JSON HTTP is widely interoperable, human-readable and directly browser-friendly. Neither is automatically “more microservice.”

Choose gRPC for controlled service-to-service environments where generated clients, streaming or high call volume justify it. Choose JSON HTTP for public or browser-facing APIs, broad ecosystem compatibility and straightforward diagnostics. Browser gRPC requires additional considerations such as gRPC-Web.

Whichever transport you choose, retain semantic boundaries. A fast gRPC call that exposes twenty CRUD methods over another service's database is still tight coupling. Measure payload, latency and CPU before claiming that serialisation format is the bottleneck.

Contracts need deadline and cancellation behaviour, status mapping, authentication, authorisation, tracing propagation and compatibility tests. Transport generation reduces typing; it does not choose these policies.

22. Asynchronous flow: request, fact and ownership

For our case, Applications may publish LoanDecisionRequested. Decisions consumes it, evaluates the snapshot and publishes LoanDecisionRecorded. Notifications and Reporting react independently.

Name events as facts or explicit requests:

public sealed record LoanDecisionRequestedV1(
    Guid MessageId,
    DateTimeOffset OccurredAt,
    Guid CorrelationId,
    Guid ApplicationId,
    long ApplicationVersion,
    decimal RequestedAmount,
    decimal VerifiedAnnualIncome,
    int CreditBand);

public sealed record LoanDecisionRecordedV1(
    Guid MessageId,
    DateTimeOffset OccurredAt,
    Guid CorrelationId,
    Guid DecisionId,
    Guid ApplicationId,
    long ApplicationVersion,
    string Outcome,
    string PolicyVersion);

The recorded event says what happened. Consumers do not command Decisions through it. Include enough immutable context to interpret the event, but do not turn every event into a copy of the database or include sensitive data “just in case.”

Messaging usually provides at-least-once delivery in practical failure scenarios. Design consumers for duplicates and out-of-order messages. “Exactly once” at a broker boundary does not automatically mean exactly one business effect across a database, email provider and audit system.

Transactional outbox

Without an outbox, this sequence loses information:

1. Save the decision.
2. Process crashes.
3. Publish the event.

The database contains the decision, but consumers never hear about it. Reversing the order can publish an event for a transaction that later fails.

Write the aggregate change and an outbox record in the same local database transaction:

await using var transaction = await db.Database.BeginTransactionAsync(ct);

decision.Record(outcome, policyVersion, clock.UtcNow);

db.OutboxMessages.Add(new OutboxMessage
{
    Id = eventId,
    OccurredAt = clock.UtcNow,
    Type = nameof(LoanDecisionRecordedV1),
    Payload = serializer.Serialize(integrationEvent)
});

await db.SaveChangesAsync(ct);
await transaction.CommitAsync(ct);

A background publisher reads undispatched rows, publishes and marks them. It may publish twice if it crashes between publish and mark, so consumers still need inbox/idempotency handling.

if (await db.ProcessedMessages.AnyAsync(x => x.Id == message.MessageId, ct))
    return;

await handler.ApplyAsync(message, ct);
db.ProcessedMessages.Add(new ProcessedMessage(message.MessageId, clock.UtcNow));
await db.SaveChangesAsync(ct);

The processed-message record and consumer state change should share a local transaction. Retention and indexing matter: an inbox table that grows forever becomes its own incident.

Ordering

Global ordering is expensive and usually unnecessary. Require ordering only for the same business key, and carry a version or sequence. A reporting consumer receiving application version 17 after version 18 should ignore or reconcile it. Partitioning by ApplicationId can preserve per-key broker order, but retries, replay and multiple sources still require explicit logic.

23. Eventual consistency is a user experience

If the portal submits a decision request and processing is asynchronous, show “Decision requested” with an operation ID. Do not claim “Approved” before the authoritative outcome exists.

The UI can poll a status resource, receive a server notification or query a local projection. Every approach needs a timeout and recovery story. If the outcome takes longer than expected, tell the user the request remains in progress and provide a safe way to return later.

Read-your-writes expectations must be explicit. Immediately after requesting a decision, the Applications read model may not yet contain the outcome. The command response can include the accepted operation and expected next state; the UI can temporarily combine that acknowledgement with the projection. Do not corrupt the projection to fake consistency.

Junior: How long does eventual consistency take?

>

Senior: That is an engineering requirement, not a philosophical answer. Set an objective, measure the distribution and alert when the delay harms users.

Track event age from occurrence to successful consumption. Queue depth alone is insufficient: ten old messages may matter more than a thousand fresh ones. Provide a replay process and decide how corrected events or poisoned messages are handled.

24. Saga design for the application workflow

A distributed workflow cannot rely on one ACID transaction across independently owned databases. A saga coordinates local transactions and compensating actions.

Our simplified workflow might be:

1. Applications verifies that the submission is ready.
2. Decisions records the evaluation.
3. Offers reserves a product offer for an approval.
4. Notifications sends the outcome.

Do not call “send another message” a complete saga. Model states, transitions, deadlines, duplicate events, invalid order, manual intervention and compensation.

public enum DecisionWorkflowState
{
    AwaitingDecision,
    AwaitingOfferReservation,
    ReadyToNotify,
    Completed,
    Compensating,
    RequiresManualReview
}

If offer reservation fails, can we undo the lending decision? Perhaps the correct business action is not reversal but referral to an underwriter. Compensation is a domain action, not a technical rollback. It may fail and require its own retry or manual queue.

Choose choreography when a few services react to clear facts and the flow remains understandable. Choose orchestration when central visibility, complex branching, timers and explicit control are valuable. An orchestrator must not steal every service's domain rules; it coordinates outcomes while each service protects its invariants.

Persist saga state and message handling atomically where possible. Give operations a screen showing stuck instances, last transition, correlation ID and safe recovery actions. A workflow nobody can inspect is not production-ready.

25. Data ownership without data isolation theatre

Database per service means one service owns writes and schema evolution for its data. It does not require a different database product or server on day one. Separate schemas and credentials can establish ownership while controlling cost, provided cross-service SQL access is prohibited.

Never join the Decisions tables directly from Reporting. Publish events or expose a query contract, then let Reporting build its own projection. Duplicated data is expected; duplicated authority is not. Decisions owns the official outcome while Reporting owns a disposable analytical representation.

For a new reporting field, ask whether the event should contain it, the reporting consumer should enrich it, or a dedicated export should supply it. Adding personal data to a widely consumed event expands its security and retention footprint.

Schema migration must support rolling deployment. Prefer expand-and-contract:

1. Add a compatible column or new event field.
2. Deploy code that can read old and new forms.
3. Backfill with observable, restartable work.
4. Switch writers and consumers.
5. Remove the old form after adoption is proven.

Avoid requiring all services to deploy at once. That recreates a distributed monolith.

26. Security across service boundaries

Authenticate the external user at the edge, but do not assume every internal call is trustworthy because it came from the network. Establish workload identity, encrypt transport and authorise sensitive service operations.

Forward only claims the downstream service needs. Do not pass a large, long-lived user token through an entire call chain by habit. Some operations act on behalf of the user; background event handlers act as workloads under recorded business context. Make that distinction auditable.

The Decisions service authorises both capability and resource. A caller permitted to request decisions may still be forbidden for another tenant. Tenant context must come from a trusted identity or mapping, not a body field alone.

Store secrets in an approved secret store, rotate them and avoid putting them in container images or source-controlled configuration. Use least-privilege database and broker credentials per service. Protect administration and replay endpoints separately from customer APIs.

Threat-model the new boundary:

• Can a caller enumerate application identifiers?
• Can a captured message be replayed?
• Can one tenant poison another tenant's projection?
• Does an event expose personal or financial information to every subscriber?
• Can a compromised service impersonate a user or another workload?
• Are logs and traces exporting confidential payloads?

Rate limiting protects capacity but is not authorisation. A gateway adds a control point but does not remove security responsibilities from services. Validate input and enforce invariants at the owner.

27. Observability designed around questions

Instrument the feature so a support engineer can answer:

• Was request R accepted?
• Which application version did it evaluate?
• Which policy version produced the result?
• Was the outcome persisted?
• Was its integration event published?
• Which consumers processed it?
• Where is the workflow now?

Use W3C trace context through HTTP and supported message propagation. A trace follows one execution; a business correlation ID can connect an asynchronous workflow lasting longer than one trace. A message ID identifies delivery and deduplication. Do not collapse all three concepts into one “correlation” string.

Structured logs should use stable properties:

logger.LogInformation(
    "Decision {DecisionId} recorded for application {ApplicationId} " +
    "at version {ApplicationVersion} using policy {PolicyVersion}",
    decision.Id,
    decision.ApplicationId,
    decision.ApplicationVersion,
    decision.PolicyVersion);

Avoid applicant names, incomes, access tokens and full event payloads. Sampling must retain errors and important workflow traces according to policy.

Metrics should describe service health and business flow: request rate, latency, error category, saturation, outbox age, consumer lag, duplicate count, dead-letter count, saga duration and manual-review rate. Keep labels low-cardinality; ApplicationId belongs in logs/traces, not a metric label.

.NET Aspire improves development orchestration and gives a useful local telemetry dashboard. Service Defaults can configure OpenTelemetry, health checks, service discovery and resilience conventions. It does not make a production architecture by itself. Export to durable production backends, set retention and alerts, and test that telemetry survives the real deployment topology.

28. Health checks, readiness and graceful shutdown

Liveness asks whether the process should be restarted. Readiness asks whether it should receive new traffic. Do not make liveness depend on every downstream service; a shared database outage could cause every instance to restart continuously.

Readiness may include dependencies essential for serving a request, but use bounded checks and understand their load. A service can remain ready for status queries while decision creation is degraded. One Boolean endpoint may be too coarse for operational decisions.

During shutdown:

1. Stop accepting new work.
2. Let in-flight HTTP requests finish within a deadline.
3. Stop fetching new messages.
4. Complete or abandon current handlers safely so the broker can redeliver.
5. Flush telemetry where supported.

Killing a consumer after it changes the database but before acknowledging the message is exactly why idempotency matters. Test termination during each critical step.

29. Testing the distributed contract

Keep most rule tests below the network. Domain tests should prove decisions and compensations quickly. Application integration tests should use a real database engine compatible with production semantics when concurrency, constraints or transactions matter.

Contract tests prove provider and consumer agree on request, response and event shapes. They do not prove the business workflow. Run compatibility checks in provider and consumer pipelines and retain examples for old supported versions.

Component/infrastructure tests can start the service with its database and broker dependency. Aspire can improve local composition, while test containers or dedicated ephemeral environments can supply realistic infrastructure. Do not mock the message broker and then claim redelivery behaviour is proven.

End-to-end tests should be few and valuable:

• request a decision and observe the final projection;
• deliver the request event twice and see one business effect;
• stop the publisher between broker send and outbox marking;
• process version 18 before version 17;
• make Offers unavailable until the saga deadline;
• rotate an instance during an active workflow;
• attempt cross-tenant access at HTTP and message boundaries.

Use fault injection in a controlled environment. Latency, timeout, refused connection, broker redelivery and partial dependency failure reveal more than another happy-path test.

Junior: Should CI start every service for every test?

>

Senior: No. Align scope with the claim. A domain rule needs no cluster. A broker redelivery claim needs a real enough broker. A customer journey needs the participating slice, not necessarily the entire company.

30. Deployment without creating a distributed monolith

Each service needs an independently executable pipeline, but independence also requires compatible contracts and data changes. A central release train for twenty services is evidence that boundaries or practices need work.

Build immutable images, generate a software bill of materials, scan dependencies, sign artifacts where required and promote the same artifact across environments. Separate deploy from release using a feature flag when business risk warrants it. Do not put irreversible database changes behind an application rollback.

Use canary or progressive delivery for high-impact services. Compare error rate, latency, saturation and business outcomes between versions. Roll back on defined evidence, not intuition.

Resource limits are part of reliability. Set requests and limits from measurements, then observe throttling, garbage collection and queue lag. Horizontal scaling helps stateless request handling, but consumers also need partition and ordering analysis. Ten consumers cannot accelerate one strictly ordered partition.

Maintain a service catalogue with owner, repository, dependencies, data classification, dashboards, alerts, runbook, SLO and escalation route. “Microservice owned by platform” is not enough when a lending decision is stuck at 02:00.

31. Service-level objectives and error budgets

Define reliability from the user's operation. For example:

• 99.9% of valid decision requests are durably accepted each month.
• 99% of accepted requests reach a terminal outcome within 30 seconds.
• 99.9% of decision status queries complete successfully within 500 ms.

Exclude invalid or unauthorised requests carefully and transparently. Measure at the service boundary and, where possible, from the consumer journey.

An error budget turns reliability into a prioritisation tool. If the workflow consumes its budget through timeouts and stuck sagas, pause risky feature rollout and improve the system. Do not turn SLOs into promises manipulated by redefining failures.

Dependencies need budgets too. If Decisions spends its entire latency allowance calling Risk, its own code has no room. Establish graceful degradation: perhaps automatic decisions pause while manual review remains available. Never silently substitute a weaker risk rule simply to keep a green availability graph.

32. Three production incidents to investigate

Incident one: duplicate approvals after a timeout

Applications sends a POST, times out and retries. Two decisions appear.

Trace both requests and compare idempotency keys. Check whether the resilience handler retried an unsafe method and whether the server recorded the key atomically with the decision. Fix the contract and persistence boundary, not only the client retry count. Add an uncertain-response integration test.

Incident two: queues are shallow but customers wait

Queue depth looks normal, yet some decisions take hours. Inspect message age by partition, dead-letter queues, repeated poison-message retries, consumer concurrency and saga state. An old message trapped behind retries may be hidden by an aggregate depth metric. Add age percentiles and a supported replay/manual-review procedure.

Incident three: deployment causes an event storm

A new consumer version crashes after applying its database change but before acknowledging. The broker redelivers repeatedly, and each attempt calls Notifications.

The consumer needs an inbox record in the same transaction as its state change. Notification intent should also cross a reliable boundary. Quarantine poison messages after a deliberate policy, alert with correlation, and ensure replay does not bypass deduplication.

For every incident, write a timeline from traces, logs and broker/database evidence. Record contributing conditions rather than blaming the last exception. Update tests, runbooks and architecture constraints.

33. Extraction playbook

Use this sequence when the evidence supports extraction:

1. Agree the bounded context and service responsibility statement.
2. Measure the current pain and define the desired outcome.
3. Establish an in-process module API and private data ownership.
4. Characterise current behaviour with tests and telemetry.
5. Define synchronous and asynchronous contracts with compatibility rules.
6. Build the new service and its operational baseline.
7. Backfill or migrate data with reconciliation counts.
8. Route a controlled cohort through the remote adapter.
9. Compare correctness, latency, reliability and team delivery metrics.
10. Expand traffic gradually with an explicit rollback path.
11. stop writes through the old path and verify no stragglers remain.
12. Remove old code and access only after evidence proves migration.

Dual writes are dangerous because one side can succeed. Prefer the outbox, change-data capture or a single authoritative write with event propagation. If temporary dual writing is unavoidable, define the source of truth and reconciliation before enabling it.

Data migration needs totals, hashes or domain reconciliation—not only “the script exited zero.” Test resumability and repeatability. Preserve audit history and legal retention. Ensure identifiers remain stable where consumers rely on them.

34. Mentoring review and exercises

Explain these aloud before implementation:

1. What business capability does Decisions own?
2. What evidence makes extraction worth its permanent cost?
3. Which workflow remains available when Decisions is down?
4. Which commands are safe to retry, and why?
5. How do persistence and event publication remain consistent?
6. How does a consumer handle duplicate and out-of-order delivery?
7. Who authorises a decision for a particular tenant?
8. How does support find one workflow across traces and messages?
9. What is the rollback plan after a schema change?
10. Which SLO represents the customer experience?

Exercise one: architecture decision record

Write an ADR comparing three choices: keep the current modular monolith, extract Decisions synchronously, or extract it with asynchronous requests. Include team ownership, latency, consistency, availability, migration, security, cost and reversibility. Recommend one and list the evidence that would change your mind.

Exercise two: failure-mode table

For every dependency, record timeout behaviour, safe retry policy, fallback, circuit-breaker effect, telemetry and user message. Include the database, broker, Risk service, identity provider and notification provider. Then test one failure from each category.

Exercise three: message laboratory

Implement a small outbox publisher and idempotent consumer. Force a crash after publish but before marking the outbox row, then restart. Force a crash after consumer state change but before acknowledgement. Prove the final business effect occurs once while delivery may occur more than once.

Exercise four: operational game day

Give a colleague only the dashboards and runbook. Delay one dependency, poison one message and terminate one consumer. Ask them to identify affected applications, stop further harm and recover safely. Improve the system wherever they need direct database guesswork.

Pull-request checklist

• Does the change respect data ownership?
• Is the contract backward compatible?
• Is sensitive data minimised?
• Is each retry demonstrably safe?
• Are timeouts bounded within the caller's budget?
• Are messages idempotent and version-aware?
• Does a state change and its outgoing intent share a reliable boundary?
• Are metric labels bounded?
• Can operators correlate and recover the workflow?
• Do rollout and rollback work with mixed versions?

35. Recognise and repair a distributed monolith

A distributed monolith has the deployment and failure costs of microservices while retaining monolithic coordination. Common symptoms are:

• one user request makes a long chain of synchronous internal calls;
• services share database tables or stored procedures;
• every release requires a coordinated version matrix;
• a shared domain-model package changes with every feature;
• local development requires the entire estate;
• one service cannot degrade without taking the main journey down;
• no team can explain end-to-end ownership;
• incidents are resolved by restarting everything.

Junior: If we deploy each API separately, how can it still be a distributed monolith?

>

Senior: Deployment units are physical boundaries. Independence comes from contracts, data ownership, failure isolation and team authority. Separate containers do not create those properties.

Repair begins by measuring coupling. For the last several features, record which repositories, databases and teams changed together. Trace critical request chains. Find shared writes and contracts that expose internal models. This gives a dependency map based on work and runtime evidence.

Then choose one seam. Move writes behind the owning service and remove direct database access. Replace chatty calls with a task-oriented contract or local projection. Define what the consumer does when the owner is unavailable. Split a shared package into stable contract schemas and consumer-owned models.

Do not respond by adding a message broker everywhere. Asynchronous coupling can be just as strong when consumers depend on undocumented event order or every service must react before a workflow succeeds. A broker changes temporal coupling; it does not fix an unclear domain.

Avoid the entity service trap

CustomerService, AddressService and OrderLineService often mirror database nouns rather than business capabilities. A single business operation then coordinates CRUD across all three. Prefer a boundary that owns a meaningful invariant and can complete useful work.

For lending, “Decisioning” is more coherent than separate IncomeService, CreditBandService and DecisionRuleService if those parts always change and operate together. Service size is not measured by lines of code. It is measured by cohesion, ownership and change patterns.

Avoid the nanoservice trap

A service with one trivial endpoint still needs identity, deployment, monitoring, patching, support and compatibility. Very small services can be appropriate for an independently owned or scaled capability, but smallness alone is not a virtue. If two services are always deployed, tested and operated together, consider merging them or restoring an in-process module boundary.

Architecture is allowed to consolidate. Microservice decomposition is not a one-way maturity ladder.

Avoid orchestration hidden in the frontend

If the browser calls six services and contains the recovery rules, it has become an untrusted saga coordinator. It also exposes internal topology and makes mobile or partner clients repeat the same logic. Use a BFF, gateway aggregation or workflow service where coordination belongs, while keeping domain decisions with their owners.

The BFF should return a screen-oriented model, not become a second database of record. Bound parallel calls, tolerate optional sections explicitly and avoid turning one page load into an unbounded fan-out.

36. Team topology is part of the architecture

A service needs a team capable of changing and operating it. If one central database team must approve every schema, one platform team must edit every pipeline and one security team manually provisions every identity, nominal service ownership will not produce flow.

The owning team should be able to:

• change rules and contracts within agreed governance;
• deploy and roll back safely;
• inspect production telemetry;
• respond to alerts and lead incidents;
• manage data migrations and retention;
• maintain dependency and runtime patches;
• communicate breaking-change plans to consumers.

Platform engineering can provide a paved road: repository templates, identity integration, standard telemetry, build pipelines, deployment policies, secret access and service cataloguing. A paved road reduces repeated work without forcing every domain into one shared runtime library.

Treat platform defaults as products. Document escape hatches and gather feedback. If every service copies 500 lines of pipeline YAML, the organisation has created distributed toil. If a platform abstraction prevents a necessary security or reliability control, it has become another bottleneck.

Junior: Should every microservice have a different team?

>

Senior: No. A team can own several cohesive services. The warning sign is a service whose change requires many teams, or a team owning so many unrelated services that nobody understands their behaviour.

Align on-call ownership with change authority. It is unfair and ineffective to make one operations group carry failures caused by teams that cannot see production. Shared incident response is healthy; abandoned operational responsibility is not.

37. Calculate the economic case

Architecture creates ongoing expenditure. Before extraction, estimate the annual cost in engineering time and infrastructure:

Now quantify benefits where possible: fewer coordinated releases, shorter lead time for decision-rule changes, isolated scaling, reduced blast radius or clearer regulatory ownership. “Teams feel more autonomous” is useful but should be paired with observable signals such as deployment frequency and cross-team wait time.

Establish a baseline before extraction. Compare six months later. If delivery is slower and incidents are harder while scale was unchanged, acknowledge it. The answer may be better tooling, boundary repair, consolidation or returning the capability to the modular monolith.

Cloud cost is not only average compute. Distributed systems create minimum instance counts, network transfer, telemetry volume, non-production environments and managed-service baselines. High-cardinality traces and verbose logs can cost more than the service itself. Set retention and sampling from diagnostic and compliance needs.

A reversible decision framework

Score, do not pretend to calculate certainty:

1. Boundary confidence: stable language, rules and data ownership.
2. Independent change pressure: real releases blocked by other areas.
3. Independent scale pressure: measured resource profile differs.
4. Reliability value: isolation produces a useful degraded mode.
5. Team readiness: clear owner with operational capability.
6. Migration safety: seam, reconciliation and rollback are credible.
7. Economic value: benefit plausibly exceeds recurring cost.

A low score in boundary confidence is a strong reason to remain modular, even if infrastructure is excellent. A high score does not require Kubernetes; choose the simplest hosting that meets the service's runtime needs.

Record what would trigger consolidation too. Decisions should be reviewable in both directions.

38. Operate an API gateway and BFF without hiding problems

A gateway can terminate external TLS, validate tokens, apply coarse rate limits, route versions and provide consistent edge telemetry. A BFF can adapt multiple service contracts into a user-journey model. They are useful boundaries, but they can also become choke points.

Keep business authorisation in the owning service. The gateway may reject a missing scope, but Decisions must still enforce tenant and resource rules. Otherwise an internal route or configuration mistake bypasses protection.

Do not put every response transformation, retry and fallback into gateway configuration. Complex workflow logic deserves tested application code. The gateway should not query service databases.

Protect fan-out. If a dashboard BFF calls ten dependencies concurrently, apply a total deadline and per-dependency budgets. Decide which panels are optional. Return partial data with explicit status only when the product can explain it; a silently missing risk warning is unsafe.

Cache only data whose security, freshness and invalidation rules are understood. Include tenant and relevant identity variation in keys. Never cache a personalised response under a shared URL without correct controls.

Observe gateway saturation and route-specific latency, but preserve trace context so the downstream cause is visible. A gateway 502 is a symptom, not a diagnosis.

39. A concise production readiness review

Before routing real decision traffic, gather the owning developers, platform engineer, security representative, product owner and support/on-call participant. Review evidence, not slides alone.

Ask the team to demonstrate:

1. a valid request and its trace through persistence and event publication;
2. an unauthorised and cross-tenant request being rejected;
3. a duplicate command and duplicate event causing one business effect;
4. a stale application version producing a recoverable conflict;
5. dependency latency ending within the caller's budget;
6. broker outage accumulating visible, recoverable outbox work;
7. a poison message reaching a controlled intervention path;
8. an instance termination during message handling;
9. mixed old/new versions during rollout;
10. restoration from backup with reconciliation of events and projections.

Review dashboards, alerts and runbooks from the perspective of a person who did not build the feature. Confirm the service catalogue identifies an owner. Confirm data retention and deletion processes include primary data, outbox/inbox rows, backups, logs, traces and downstream projections.

List accepted risks with owners and dates. “We will add idempotency later” is not acceptable for a retried financial action. A cosmetic dashboard improvement may be deferred. Use impact and recoverability to distinguish them.

Production readiness is not a one-time gate. Revisit it after major traffic change, a new consumer, a contract version, a persistence migration or a significant incident.

When I would pause the extraction

As the senior reviewer, I would stop or narrow the work if the team cannot name one data owner, if domain language is still changing weekly, or if the proposed service requires direct writes to another module's tables. I would also pause when there is no safe migration and reconciliation plan, no team accepting operational ownership, or no user-visible behaviour when the dependency fails.

I would reject claims that a broker guarantees one business effect, that internal traffic needs no authorisation, or that retries make an unsafe command reliable. Those are correctness gaps, not optional refinements. Similarly, a production launch without correlation, dependency timeouts, an alertable outbox and a duplicate-delivery test leaves the most likely failures invisible.

Pausing does not mean abandoning architectural improvement. Strengthen the module seam, clarify language, add telemetry, remove shared writes and measure delivery pain inside the monolith. Those changes create value immediately and make a later extraction safer.

Proceed when the team can state the boundary in one paragraph, demonstrate a complete vertical slice, recover an uncertain operation, deploy mixed compatible versions, and explain the economic reason for paying the distributed-systems cost. The burden of proof belongs to extraction because it adds permanent failure modes.

That is responsible senior engineering: not resisting change, and not approving complexity for status, but matching the architecture to evidence, ownership and the business workflow it must protect.

40. Continue the learning path

Microservices amplify fundamentals rather than replacing them. Read HTTP and the Web for protocol semantics, Web Security for trust boundaries, Clean Architecture for dependency direction, and Pragmatic TDD for evidence at the correct test layer. The Azure for .NET Developers and Azure Bicep Infrastructure as Code guides extend deployment, identity and infrastructure concerns. The Building Agent-Powered Applications guide shows why idempotency, tool boundaries and observability matter equally in AI workflows.

Primary version-aware references:

• Introduction to resilient app development in .NET
• Build resilient HTTP apps
• .NET observability with OpenTelemetry
• Add Aspire to an existing app
• Integration event communication between microservices
• Azure Architecture Center API design

Framework packages and platform defaults evolve. Confirm the documentation and package version used by the solution before copying configuration. The principles—explicit ownership, bounded waiting, safe repetition, compatible contracts and observable failure—remain the durable part.

---

What I want you to take away

Microservices are not automatically better than monoliths. They solve some problems and create new ones, and they demand operational maturity from the team.

They help when you need independent deployment, independent scaling, strong domain boundaries, separate teams, technology flexibility and resilience.

They hurt when the team lacks DevOps maturity, observability, automated testing, deployment pipelines, domain understanding and operational discipline.

The complete architectural journey is this:

Start by understanding the difference between monolith, modular monolith and microservices. Use DDD to find business boundaries. Use REST or gRPC for direct communication when immediate answers are needed. Use messaging, pub-sub and eventual consistency when work can happen asynchronously. Use aggregator and BFF patterns to protect the frontend. Use CQRS when read and write needs differ. Use event sourcing only when history matters. Use database-per-service to preserve autonomy. Use sagas for long-running business workflows. Use resilience patterns because failure is normal. Use gateways and security because every boundary must be protected. Use containers, Kubernetes or serverless depending on runtime needs. Use observability because distributed systems without telemetry are blind.

Here is the sentence I want a developer to be able to say in an interview:

“A microservices architecture is a distributed system built around business capabilities, where each service owns its data and communicates through well-defined contracts. The design challenge is not creating many APIs; it is managing boundaries, consistency, resilience, security, deployment and observability.”

That is the mature answer.

And here is the practical rule I return to:

Build a modular monolith first. Extract microservices only when the business boundary, team structure, scale, deployment independence or data ownership makes the cost worth paying.