Revisiting Multithreading, Async and Parallel C#: Practical Notes and Experiments

Let’s build a clear model of what happens when C# code runs concurrently.

This is one of those subjects where it is easy to use the vocabulary without understanding the mechanics. When I mentor a developer on concurrency, I want them to distinguish threading, asynchronous I/O and CPU parallelism before choosing an API.

They say:

“I used async.”

“I used Task.”

“I used a thread.”

“I used parallel.”

But in a serious interview, or during a production incident, that is not enough.

I want you to be able to explain the difference between:

Thread
Thread pool
Task
Parallel
Async
Await
Lock
Semaphore
CancellationToken
Concurrent collection
Asynchronous I/O
CPU-bound work
I/O-bound work

Because these words are not decoration. They decide whether your application is responsive, scalable, safe, or quietly destroying itself under load.

Let’s build the mental model properly.

---

1. Why multithreading exists

A long time ago, most computers had one CPU core. Programs appeared to run together because the operating system switched between them very quickly. That switching created the illusion of multitasking.

Then processors became multi-core, allowing a machine to genuinely execute more than one stream of work at the same time. Modern software can use those cores for performance while using asynchronous I/O to remain responsive and scalable.

Here is the simple mental model:

Process = running application
Thread  = execution path inside that application
CPU core = physical worker that can execute instructions

When you run a console app, ASP.NET Core app, Windows service, or desktop app, your application has at least one main thread.

A thread is like a worker in an office.

One worker can do one thing at a time. If you give that worker a long report to print, he cannot answer the phone at the same moment. If you hire more workers, more things can happen at once. But workers are not free. They need desks, coordination, rules, and management.

That is exactly how threads behave.

Threads are powerful, but expensive. Too many can make the operating system spend more time scheduling and switching than executing useful work.

My rule of thumb:

More threads do not automatically mean more performance. More threads mean more coordination cost.

---

2. Creating a thread: the lowest-level mental model

At the raw level, you can create a thread directly:

using System;
using System.Threading;

public class Program
{
    public static void Main()
    {
        // This creates a new dedicated thread.
        // The thread will run the PrintNumbers method.
        Thread worker = new Thread(PrintNumbers);

        // Start tells the operating system:
        // "Schedule this thread for execution."
        worker.Start();

        // Meanwhile, the main thread continues and also runs PrintNumbers.
        PrintNumbers();

        Console.WriteLine("Main method finished.");
    }

    private static void PrintNumbers()
    {
        Console.WriteLine($"Starting on thread {Thread.CurrentThread.ManagedThreadId}");

        for (int i = 1; i <= 5; i++)
        {
            Console.WriteLine($"Thread {Thread.CurrentThread.ManagedThreadId}: {i}");
        }
    }
}

When you run this, the output order is not guaranteed.

That is your first lesson.

Multithreaded code is not naturally ordered. Two threads can interleave in different ways each time the program runs.

It is tempting to expect:

Thread A finishes
Thread B finishes

The correct expectation is:

Thread A and Thread B may interleave unpredictably

That unpredictability is not a bug. It is the nature of concurrency.

If you need order, you must design for order.

---

3. Sleep, Join, foreground and background threads

Thread.Sleep pauses the current thread.

Thread.Sleep(TimeSpan.FromSeconds(2));

Important wording: it pauses the current thread. It does not pause the whole application.

If a worker thread sleeps, the main thread can still continue. While sleeping, the thread uses very little CPU, but it is still a thread sitting around.

Join means:

“Main thread, wait here until this other thread finishes.”

Thread worker = new Thread(PrintSlowNumbers);

worker.Start();

// Main thread blocks here until worker completes.
worker.Join();

Console.WriteLine("Worker completed.");

This is useful when you genuinely cannot continue until the other operation has finished.

But it blocks.

Blocking means the current thread cannot do anything else.

In UI apps, blocking the UI thread freezes the screen. In ASP.NET-style server apps, blocking request threads reduces scalability. In background services, blocking can be acceptable if it is intentional.

Foreground and background threads matter too.

A foreground thread keeps the process alive; a background thread does not. Once all foreground threads finish, remaining background threads do not keep the process running.

Thread background = new Thread(DoWork);
background.IsBackground = true;
background.Start();

My rule of thumb:

Direct threads are low-level tools. Use them when you truly need control, not because you want to “do async.”

---

4. Race conditions: the classic multithreading bug

Now we come to the dangerous part.

Imagine three threads updating the same counter.

public class UnsafeCounter
{
    public int Count { get; private set; }

    public void Increment()
    {
        Count++;
    }

    public void Decrement()
    {
        Count--;
    }
}

This looks innocent.

But Count++ is not one operation.

It is roughly:

Read Count
Add 1
Write Count

If two threads read the same value before either writes back, one update can be lost.

Example:

Count = 10

Thread A reads 10
Thread B reads 10
Thread A writes 11
Thread B writes 11

Expected: 12
Actual:   11

That is a race condition.

This counter scenario demonstrates why shared mutable state is dangerous and why locking or atomic operations are sometimes necessary.

A thread-safe version using lock:

public class SafeCounterWithLock
{
    private readonly object _syncRoot = new();
    public int Count { get; private set; }

    public void Increment()
    {
        lock (_syncRoot)
        {
            Count++;
        }
    }

    public void Decrement()
    {
        lock (_syncRoot)
        {
            Count--;
        }
    }
}

lock means:

“Only one thread at a time may enter this block for this lock object.”

This solves the race condition.

But locks are not free. They can block other threads. They can reduce throughput. Used badly, they can create deadlocks.

My rule of thumb:

Shared mutable state is where concurrency bugs are born. Avoid it where possible. Protect it carefully where unavoidable.

---

5. Deadlocks: when two threads wait forever

A deadlock happens when two or more threads wait for each other forever.

Classic example:

private static readonly object LockA = new();
private static readonly object LockB = new();

public static void MethodOne()
{
    lock (LockA)
    {
        Thread.Sleep(100);

        lock (LockB)
        {
            Console.WriteLine("MethodOne completed.");
        }
    }
}

public static void MethodTwo()
{
    lock (LockB)
    {
        Thread.Sleep(100);

        lock (LockA)
        {
            Console.WriteLine("MethodTwo completed.");
        }
    }
}

What can happen?

Thread 1 enters MethodOne and locks A
Thread 2 enters MethodTwo and locks B
Thread 1 waits for B
Thread 2 waits for A

Nobody can continue.

That is a deadlock.

Monitor.TryEnter can attempt to acquire a lock with a timeout instead of waiting indefinitely, giving the code an explicit failure path.

Example:

bool lockTaken = false;

try
{
    Monitor.TryEnter(LockA, TimeSpan.FromSeconds(2), ref lockTaken);

    if (!lockTaken)
    {
        Console.WriteLine("Could not acquire lock.");
        return;
    }

    // Protected work here
}
finally
{
    if (lockTaken)
    {
        Monitor.Exit(LockA);
    }
}

Best practices:

Always acquire locks in the same order. Keep lock blocks short. Never call unknown external code while holding a lock. Never perform long I/O inside a lock. Avoid nested locks unless absolutely necessary. Use higher-level concurrency constructs where possible.

My rule of thumb:

A lock fixes one class of bug and can create another. Use it deliberately.

---

6. Interlocked: atomic operations without locking

For simple numeric operations, you often do not need a full lock.

Use Interlocked.

public class AtomicCounter
{
    private int _count;

    public int Count => _count;

    public void Increment()
    {
        Interlocked.Increment(ref _count);
    }

    public void Decrement()
    {
        Interlocked.Decrement(ref _count);
    }
}

This performs the operation atomically. No other thread can interrupt halfway through the increment or decrement.

Interlocked.Increment, Interlocked.Decrement and related atomic operations can prevent simple races without taking a full lock.

Use Interlocked for simple counters, flags, and atomic swaps.

Do not try to build complex business workflows with Interlocked unless you really know what you are doing.

My rule of thumb:

For simple shared counters, prefer atomic operations over locks.

---

7. Mutex, SemaphoreSlim, events, barriers and reader-writer locks

Thread synchronization is not just lock.

.NET provides a broad synchronisation toolbox including Mutex, SemaphoreSlim, AutoResetEvent, ManualResetEventSlim, CountdownEvent, Barrier, ReaderWriterLockSlim and SpinWait.

Let’s translate these into real-life mental models.

A Mutex is like a named key that can protect a resource, even across processes.

Typical example: prevent two copies of the same desktop app from running.

using var mutex = new Mutex(false, "LoanManagementApp.SingleInstance");

if (!mutex.WaitOne(TimeSpan.FromSeconds(2)))
{
    Console.WriteLine("Another instance is already running.");
    return;
}

Console.WriteLine("Application running.");
Console.ReadLine();

mutex.ReleaseMutex();

A SemaphoreSlim limits how many operations can run at once.

Imagine a loan platform calling an external credit score API. You do not want 500 requests at the same time.

public class CreditScoreService
{
    private readonly SemaphoreSlim _semaphore = new(initialCount: 5);

    public async Task<int> GetScoreAsync(string applicantId)
    {
        // Only 5 callers can enter this section at the same time.
        await _semaphore.WaitAsync();

        try
        {
            // Simulate external HTTP call
            await Task.Delay(500);
            return 720;
        }
        finally
        {
            _semaphore.Release();
        }
    }
}

This is extremely practical.

It protects external APIs, databases, file processing, queue consumers, and expensive CPU resources.

AutoResetEvent is like a turnstile. One signal lets one waiting thread pass.

ManualResetEventSlim is like a gate. When open, all waiting threads may pass until you close it again.

CountdownEvent waits until a certain number of operations signal completion.

Barrier lets multiple threads meet at phases, useful in iterative parallel algorithms.

ReaderWriterLockSlim allows many readers at the same time, but only one writer.

Example:

public class ProductCache
{
    private readonly ReaderWriterLockSlim _lock = new();
    private readonly Dictionary<int, string> _products = new();

    public string? GetProductName(int id)
    {
        _lock.EnterReadLock();

        try
        {
            return _products.TryGetValue(id, out var name)
                ? name
                : null;
        }
        finally
        {
            _lock.ExitReadLock();
        }
    }

    public void UpdateProductName(int id, string name)
    {
        _lock.EnterWriteLock();

        try
        {
            _products[id] = name;
        }
        finally
        {
            _lock.ExitWriteLock();
        }
    }
}

My rule of thumb:

Choose synchronization based on the shape of the problem: exclusive access, limited access, signalling, phase coordination, or many-readers-one-writer.

---

8. Thread pool: do not create threads for everything

Creating raw threads repeatedly is expensive.

The thread pool solves this by maintaining reusable worker threads.

Instead of creating a new worker every time, you borrow one from a managed pool.

ThreadPool.QueueUserWorkItem(_ =>
{
    Console.WriteLine($"Running on thread {Thread.CurrentThread.ManagedThreadId}");
});

The thread pool is designed for reusable worker capacity, but long-running or blocking work can starve it—particularly in ASP.NET Core, where incoming requests depend on that shared capacity.

That point is golden.

In a web application, the thread pool is shared infrastructure. If you block all worker threads, your application becomes slow or unresponsive even if the database is fine.

Bad ASP.NET thinking:

public IActionResult GetReport()
{
    // Bad: blocks request thread waiting for async work
    var report = _reportService.GenerateAsync().Result;

    return Ok(report);
}

Better:

public async Task<IActionResult> GetReport()
{
    var report = await _reportService.GenerateAsync();

    return Ok(report);
}

await frees the thread while waiting for I/O.

My rule of thumb:

In server apps, blocking thread pool threads is one of the easiest ways to create scalability problems.

---

9. Task Parallel Library: the professional abstraction

The Task Parallel Library, or TPL, gives a higher-level abstraction over thread pool work.

A Task represents an operation that may complete in the future.

Task<int> calculateRiskTask = Task.Run(() =>
{
    // CPU-bound work
    return CalculateRiskScore();
});

int score = await calculateRiskTask;

But be careful.

Task.Run is not the same as true asynchronous I/O.

Task.Run says:

“Put this work onto a thread pool thread.”

That is useful for CPU-bound work.

It is not how you should normally make database calls, HTTP calls, or file calls asynchronous. Those should use real async APIs.

Bad:

public Task<Customer> GetCustomerAsync(int id)
{
    return Task.Run(() =>
    {
        // This still blocks a thread while the database works.
        return _db.Customers.Find(id);
    });
}

Better:

public async Task<Customer?> GetCustomerAsync(int id)
{
    // True async database I/O if using EF Core async provider.
    return await _db.Customers.FindAsync(id);
}

TPL provides Task.WhenAll, Task.WhenAny, continuations, cancellation, exception handling, schedulers and composition. It is a higher-level abstraction for representing and coordinating work.

Parallel independent operations:

public async Task<LoanDashboardDto> GetDashboardAsync()
{
    // Start all independent operations first.
    Task<int> submittedTask = GetSubmittedLoanCountAsync();
    Task<int> approvedTask = GetApprovedLoanCountAsync();
    Task<decimal> totalValueTask = GetTotalApprovedValueAsync();

    // Await all together.
    await Task.WhenAll(submittedTask, approvedTask, totalValueTask);

    return new LoanDashboardDto
    {
        SubmittedLoans = await submittedTask,
        ApprovedLoans = await approvedTask,
        TotalApprovedValue = await totalValueTask
    };
}

Important: Task.WhenAll is useful when operations are independent.

Do not parallelize operations that depend on each other.

Sequential:

var customer = await GetCustomerAsync(customerId);
var loans = await GetLoansForCustomerAsync(customer.Id);

Parallel:

var productsTask = GetLoanProductsAsync();
var ratesTask = GetCurrentRatesAsync();

await Task.WhenAll(productsTask, ratesTask);

My rule of thumb:

Start tasks together only when the operations are independent and the downstream systems can handle the load.

---

10. Async and await: not magic, not always parallel

This is the concept every .NET developer must master.

async and await do not automatically create a new thread.

They allow the current thread to stop waiting while an asynchronous operation is in progress.

Imagine a waiter in a restaurant.

Bad waiter:

Takes order. Stands outside kitchen doing nothing until food is ready. Then serves customer.

Good waiter:

Takes order. Gives order to kitchen. Serves other tables while kitchen works. Returns when food is ready.

That is async I/O.

Example:

public async Task<LoanApplicationDto> GetLoanAsync(Guid id)
{
    // Thread sends database request.
    // While database is working, thread can return to pool.
    var loan = await _dbContext.Loans.FindAsync(id);

    if (loan is null)
    {
        throw new InvalidOperationException("Loan not found.");
    }

    return new LoanApplicationDto
    {
        Id = loan.Id,
        ApplicantName = loan.ApplicantName,
        Amount = loan.Amount
    };
}

Two consecutive await calls do not automatically run in parallel; the second operation normally starts after the first completes.

Sequential:

var customer = await GetCustomerAsync(id);
var loans = await GetLoansAsync(customer.Id);

Parallel:

Task<Customer> customerTask = GetCustomerAsync(id);
Task<List<Product>> productsTask = GetProductsAsync();

await Task.WhenAll(customerTask, productsTask);

I avoid async void except for UI event handlers because callers cannot await it or observe completion and errors through the normal task model.

Bad:

public async void ProcessLoan()
{
    await SubmitLoanAsync();
}

Better:

public async Task ProcessLoanAsync()
{
    await SubmitLoanAsync();
}

My rule of thumb:

Async is mainly about freeing threads during waiting. Parallel is about doing work at the same time. They are related, but not the same.

---

11. CancellationToken: polite cancellation

Never use violent cancellation as your normal design.

Forcefully aborting threads is dangerous. Cooperative cancellation through CancellationToken is the normal modern approach.

Modern design:

public async Task ProcessDocumentsAsync(
    IEnumerable<Document> documents,
    CancellationToken cancellationToken)
{
    foreach (var document in documents)
    {
        cancellationToken.ThrowIfCancellationRequested();

        await ProcessSingleDocumentAsync(document, cancellationToken);
    }
}

Call site:

using var cts = new CancellationTokenSource();

cts.CancelAfter(TimeSpan.FromSeconds(30));

try
{
    await ProcessDocumentsAsync(documents, cts.Token);
}
catch (OperationCanceledException)
{
    Console.WriteLine("Document processing was cancelled.");
}

Cancellation is cooperative.

You are not killing the operation from the outside. You are asking it to stop at safe points.

My rule of thumb:

Cancellation should be designed into long-running work from the beginning.

---

12. Concurrent collections: do not lock everything manually

Normal collections are not automatically thread-safe.

This is unsafe:

private readonly Queue<LoanJob> _jobs = new();

public void Enqueue(LoanJob job)
{
    _jobs.Enqueue(job);
}

public LoanJob Dequeue()
{
    return _jobs.Dequeue();
}

If multiple threads call this together, trouble.

.NET provides concurrent collections including ConcurrentDictionary, ConcurrentQueue, ConcurrentStack, ConcurrentBag and BlockingCollection.

Use ConcurrentQueue for producer-consumer FIFO processing.

public class LoanJobQueue
{
    private readonly ConcurrentQueue<LoanJob> _queue = new();

    public void Enqueue(LoanJob job)
    {
        _queue.Enqueue(job);
    }

    public bool TryDequeue(out LoanJob? job)
    {
        return _queue.TryDequeue(out job);
    }
}

Use ConcurrentDictionary for thread-safe lookup/update.

private readonly ConcurrentDictionary<Guid, LoanStatus> _statusByLoanId = new();

public void SetStatus(Guid loanId, LoanStatus status)
{
    _statusByLoanId[loanId] = status;
}

public LoanStatus? GetStatus(Guid loanId)
{
    return _statusByLoanId.TryGetValue(loanId, out var status)
        ? status
        : null;
}

BlockingCollection supports producer-consumer pipelines. Workers can use GetConsumingEnumerable to wait for new items until adding is marked complete.

public class LoanProcessingPipeline
{
    private readonly BlockingCollection<LoanApplication> _queue = new();

    public void Add(LoanApplication loan)
    {
        _queue.Add(loan);
    }

    public void Complete()
    {
        _queue.CompleteAdding();
    }

    public async Task StartWorkerAsync(string workerName)
    {
        foreach (var loan in _queue.GetConsumingEnumerable())
        {
            Console.WriteLine($"{workerName} processing loan {loan.Id}");

            await ProcessLoanAsync(loan);
        }
    }

    private static Task ProcessLoanAsync(LoanApplication loan)
    {
        // Real processing would happen here.
        return Task.Delay(250);
    }
}

My rule of thumb:

Prefer proven concurrent collections over hand-rolled locking around normal collections.

---

13. PLINQ and Parallel: CPU-bound parallelism

PLINQ is Parallel LINQ.

It is useful when you have a collection and want to perform CPU-heavy work on each item.

The Parallel class and PLINQ support CPU-bound data parallelism, with controls for degree of parallelism, partitioning, exception handling and aggregation.

Example: calculate risk scores for many loans.

var scoredLoans = loans
    .AsParallel()
    .WithDegreeOfParallelism(Environment.ProcessorCount)
    .Select(loan => new LoanRiskResult
    {
        LoanId = loan.Id,
        RiskScore = CalculateRiskScore(loan)
    })
    .ToList();

This can help if CalculateRiskScore is CPU-heavy and independent per loan.

But do not blindly use PLINQ for database calls:

// Bad idea in many real systems:
loans.AsParallel().Select(async loan => await CallDatabaseAsync(loan.Id));

That mixes parallel CPU processing with async I/O in a messy way.

For I/O, use async APIs with controlled concurrency.

For CPU-bound independent calculations, use Parallel, PLINQ, or Task.Run carefully.

My rule of thumb:

PLINQ is for data parallelism. It is not a magic “make my LINQ faster” button.

---

14. Reactive Extensions: thinking in streams

Reactive Extensions, or Rx, is about asynchronous streams of values.

Reactive Extensions distinguishes pull-based sequences such as IEnumerable from push-based IObservable streams, where producers notify consumers as values arrive. Rx provides LINQ-style composition over those event sequences.

Normal pull model:

foreach (var item in items)
{
    Process(item);
}

You ask for the next item.

Reactive push model:

observable.Subscribe(item =>
{
    Process(item);
});

The producer tells you when the next item exists.

Where is this useful?

UI events. Live prices. Telemetry. Logs. Sensor data. Chat messages. Search suggestions. Stock updates. Real-time dashboards.

Example mental model:

// Pseudo-style example
IObservable<string> searchTextChanged = GetSearchTextChanges();

searchTextChanged
    .Where(text => text.Length >= 3)
    .Throttle(TimeSpan.FromMilliseconds(300))
    .DistinctUntilChanged()
    .Subscribe(async text =>
    {
        var results = await SearchLoansAsync(text);
        Display(results);
    });

That is a beautiful model for “things happening over time.”

My rule of thumb:

Use tasks for one future result. Use observables for streams of future results.

---

15. Asynchronous I/O: the server scalability topic

Asynchronous I/O is one of the most important topics for web developers.

Asynchronous I/O applies to files, HTTP clients and servers, databases, message brokers and other external resources.

I/O means waiting for something external:

Database. File system. Network. HTTP API. Message broker. Blob storage. Email server.

The key idea:

While the external thing is working, do not waste a thread.

Bad:

public byte[] DownloadStatement(string path)
{
    return File.ReadAllBytes(path);
}

Better:

public async Task<byte[]> DownloadStatementAsync(
    string path,
    CancellationToken cancellationToken)
{
    return await File.ReadAllBytesAsync(path, cancellationToken);
}

HTTP example:

public async Task<CreditScoreResponse> GetCreditScoreAsync(
    string applicantId,
    CancellationToken cancellationToken)
{
    using var response = await _httpClient.GetAsync(
        $"credit-score/{applicantId}",
        cancellationToken);

    response.EnsureSuccessStatusCode();

    return await response.Content.ReadFromJsonAsync<CreditScoreResponse>(
        cancellationToken: cancellationToken)
        ?? throw new InvalidOperationException("Empty credit score response.");
}

Database example:

public async Task<List<LoanSummaryDto>> GetSubmittedLoansAsync(
    CancellationToken cancellationToken)
{
    return await _dbContext.Loans
        .AsNoTracking()
        .Where(x => x.Status == LoanStatus.Submitted)
        .Select(x => new LoanSummaryDto
        {
            Id = x.Id,
            ApplicantName = x.ApplicantName,
            Amount = x.Amount
        })
        .ToListAsync(cancellationToken);
}

In ASP.NET Core, this matters because request handling depends on thread pool availability.

If 500 requests are waiting for SQL synchronously, 500 threads may be blocked.

If 500 requests are waiting asynchronously, those threads can return to the pool while SQL works.

My rule of thumb:

Async I/O does not make the database faster. It makes your application more scalable while waiting.

---

16. Parallel programming patterns: pipeline, lazy state, map/reduce

The final practical layer is not about individual APIs. It is about patterns.

The wider pattern toolbox includes lazy shared state, pipelines using BlockingCollection or TPL Dataflow and map/reduce-style processing with PLINQ.

Think of a loan document processing system.

A user uploads documents. The system must:

Extract text. Classify document type. Validate contents. Run fraud checks. Store results. Notify the broker.

This can be a pipeline.

Upload
  ↓
Extract Text
  ↓
Classify
  ↓
Validate
  ↓
Store
  ↓
Notify

Each stage can run independently. Different documents can be at different stages at the same time.

A simple conceptual pipeline with channels would be common today; the following example uses BlockingCollection to illustrate the same producer-consumer idea:

public class DocumentPipeline
{
    private readonly BlockingCollection<Document> _uploaded = new();
    private readonly BlockingCollection<ExtractedDocument> _extracted = new();

    public void Upload(Document document)
    {
        _uploaded.Add(document);
    }

    public void CompleteUploads()
    {
        _uploaded.CompleteAdding();
    }

    public Task StartExtractionWorkerAsync()
    {
        return Task.Run(async () =>
        {
            foreach (var document in _uploaded.GetConsumingEnumerable())
            {
                var extracted = await ExtractTextAsync(document);
                _extracted.Add(extracted);
            }

            _extracted.CompleteAdding();
        });
    }

    public Task StartValidationWorkerAsync()
    {
        return Task.Run(async () =>
        {
            foreach (var document in _extracted.GetConsumingEnumerable())
            {
                await ValidateAsync(document);
            }
        });
    }

    private static Task<ExtractedDocument> ExtractTextAsync(Document document)
    {
        return Task.FromResult(new ExtractedDocument(document.Id, "extracted text"));
    }

    private static Task ValidateAsync(ExtractedDocument document)
    {
        Console.WriteLine($"Validated {document.Id}");
        return Task.CompletedTask;
    }
}

public record Document(Guid Id);
public record ExtractedDocument(Guid Id, string Text);

This kind of architecture is useful when you have stages and backpressure.

Map/Reduce mental model:

Map: process each item independently. Reduce: combine results.

Example:

var totalRequestedAmount = loans
    .AsParallel()
    .Where(loan => loan.Status == LoanStatus.Submitted)
    .Select(loan => loan.Amount)
    .Sum();

My rule of thumb:

Patterns matter when single async calls are no longer enough and you need coordinated flows.

---

17. Mentoring checkpoint: concurrency is a resource policy

The pipeline sketch explains flow, but an unbounded pipeline can accept work faster than it completes. Memory grows, downstream services are flooded, timeouts create retries and the system becomes less reliable as load increases.

Junior: If asynchronous work does not block threads, why limit it?

>

Senior: Threads are only one resource. Every operation consumes memory, sockets, database connections, downstream quota, CPU later, and attention when it fails. Async makes waiting efficient; it does not create infinite capacity.

Before selecting an API, write the policy:

Input: uploaded document references, maximum 10 MB each
Extraction: I/O-bound call, at most 20 concurrent
Classification: CPU-bound, at most processor count
Validation: database/API I/O, at most 30 concurrent
Queue capacity: 200 documents
Per-document deadline: 60 seconds
Shutdown grace: 30 seconds
Delivery: at least once, idempotent by document/version

The figures are illustrative. Capacity comes from load tests, dependency limits and memory budgets. The important lesson is that concurrency is explicit and observable.

18. Rebuild the pipeline with bounded Channels

System.Threading.Channels provides asynchronous producer-consumer queues. A bounded channel can apply backpressure instead of growing forever.

using System.Threading.Channels;

public sealed record UploadedDocument(
    Guid DocumentId,
    int Version,
    Uri BlobUri,
    string TenantId);

public sealed record ExtractedDocument(
    Guid DocumentId,
    int Version,
    string TenantId,
    string Text);

var uploads = Channel.CreateBounded<UploadedDocument>(
    new BoundedChannelOptions(200)
    {
        FullMode = BoundedChannelFullMode.Wait,
        SingleWriter = false,
        SingleReader = false,
        AllowSynchronousContinuations = false,
    });

Wait means a producer asynchronously waits when capacity is full. Other full modes can drop newest, oldest or the incoming item; dropping is valid only when product semantics allow loss and metrics expose it. Loan documents should not disappear silently.

Producer:

public static async Task ProduceAsync(
    IAsyncEnumerable<UploadedDocument> source,
    ChannelWriter<UploadedDocument> writer,
    CancellationToken cancellationToken)
{
    Exception? failure = null;
    try
    {
        await foreach (var document in source.WithCancellation(cancellationToken))
            await writer.WriteAsync(document, cancellationToken);
    }
    catch (Exception exception)
    {
        failure = exception;
        throw;
    }
    finally
    {
        writer.TryComplete(failure);
    }
}

Completing the writer tells readers that no more items will arrive after queued items drain. Complete once, by the owner of production. Several producers need coordination—such as Task.WhenAll followed by completion—rather than each closing the shared channel independently.

Consumer worker:

public static async Task ExtractWorkerAsync(
    int workerId,
    ChannelReader<UploadedDocument> reader,
    ChannelWriter<ExtractedDocument> output,
    IDocumentExtractor extractor,
    CancellationToken cancellationToken)
{
    await foreach (var document in reader.ReadAllAsync(cancellationToken))
    {
        var extracted = await extractor.ExtractAsync(document, cancellationToken);
        await output.WriteAsync(extracted, cancellationToken);
    }
}

Start a fixed number of workers and await all of them before completing the next stage’s writer:

var workers = Enumerable.Range(0, extractionConcurrency)
    .Select(index => ExtractWorkerAsync(
        index, uploads.Reader, extracted.Writer, extractor, cancellationToken))
    .ToArray();

try
{
    await Task.WhenAll(workers);
    extracted.Writer.TryComplete();
}
catch (Exception exception)
{
    extracted.Writer.TryComplete(exception);
    throw;
}

Junior: Could we create one task per document and use Task.WhenAll?

>

Senior: For a small known collection, yes. For a long or unbounded stream, that schedules all work and retains all tasks. Fixed workers plus a bounded channel control memory and concurrency.

The channel is an in-process buffer, not durable messaging. Process crash loses queued items. If documents must survive restart, persist job state or use a durable broker and treat the channel as a local execution buffer.

19. Separate I/O concurrency from CPU parallelism

Extraction might call remote OCR and spend most time awaiting. Twenty in-flight calls can be efficient if the dependency and connection limits allow it. Classification may execute CPU-heavy code. Running twenty CPU classifiers on a four-core container can increase context switching and tail latency.

For CPU work over a finite collection, Parallel.ForEachAsync provides bounded parallel execution:

await Parallel.ForEachAsync(
    extractedBatch,
    new ParallelOptions
    {
        MaxDegreeOfParallelism = Environment.ProcessorCount,
        CancellationToken = cancellationToken,
    },
    async (document, token) =>
    {
        var classification = classifier.Classify(document.Text);
        await results.WriteAsync(classification, token);
    });

The delegate is async because writing to the next channel may wait. The classification itself is synchronous CPU work. Measure processor quota inside containers; Environment.ProcessorCount reflects runtime/environment decisions but is not always the final capacity policy.

Do not wrap every CPU method in Task.Run inside ASP.NET Core. That moves work onto another thread-pool thread without reducing CPU. For short CPU work, execute normally. For long work, use a bounded background queue or separate worker so request latency and server capacity are controlled.

PLINQ can efficiently parallelise pure transformations over in-memory collections. Ordering, exceptions, cancellation and merge cost matter. Avoid shared mutable writes inside the query; return values and reduce them.

20. Understand Task.WhenAll failure and cancellation

Task.WhenAll completes when every supplied task completes. If tasks fault, awaiting it throws an exception, while the returned task retains aggregated failure information. Do not assume the first caught exception is the only failure worth diagnosing.

var tasks = documents.Select(document => ProcessAsync(document, cancellationToken))
    .ToArray();

try
{
    await Task.WhenAll(tasks);
}
catch
{
    foreach (var task in tasks.Where(task => task.IsFaulted))
    {
        foreach (var exception in task.Exception!.Flatten().InnerExceptions)
            logger.LogError(exception, "Document task failed");
    }
    throw;
}

This pattern may log sensitive exception content; production logging still needs redaction and correlation. Prefer each operation to record a safe document identifier and stage.

Cancellation does not roll back completed work. Some tasks may succeed, some observe cancellation and some fault. Define whether partial results remain valid. A batch import may record each item independently; a transaction may require all-or-nothing within one database boundary.

Junior: If one task fails, does WhenAll cancel the others?

>

Senior: No. It observes all supplied tasks. If fail-fast is required, cancel a linked token source when a worker fails, while still awaiting every task so exceptions and cleanup are observed.

Do not dispose a CancellationTokenSource while operations may still use its token. Give lifetime ownership to the orchestration scope and await child tasks before disposal.

21. Structured concurrency with ownership

.NET tasks can outlive the method that created them if they are not awaited. That can be intentional for a hosted background service, but “fire and forget” inside a request loses exception, cancellation, service scope and shutdown ownership.

Bad:

app.MapPost("/documents", (UploadRequest request, DocumentProcessor processor) =>
{
    _ = processor.ProcessAsync(request.DocumentId, CancellationToken.None);
    return Results.Accepted();
});

The scoped processor may be disposed when the request ends. The work disappears on restart and exceptions may go unobserved.

Better: persist an accepted job and publish to a durable queue in the same consistency design, or enqueue into a bounded hosted service only if loss on restart is acceptable and documented.

public sealed class DocumentWorker(
    IDocumentJobStore jobs,
    IDocumentPipeline pipeline,
    ILogger<DocumentWorker> logger) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await foreach (var job in jobs.ReadReadyAsync(stoppingToken))
        {
            try
            {
                await pipeline.ProcessAsync(job, stoppingToken);
                await jobs.MarkCompletedAsync(job.Id, stoppingToken);
            }
            catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
            {
                break;
            }
            catch (Exception exception)
            {
                logger.LogError(exception, "Document job {JobId} failed", job.Id);
                await jobs.RecordFailureAsync(job.Id, exception, stoppingToken);
            }
        }
    }
}

The sample’s failure-record call uses the stopping token; during shutdown it may already be cancelled. A production design may use a short independent cleanup token, or rely on transactional leasing so an incomplete job becomes visible again. This is exactly why shutdown semantics need design, not copied snippets.

22. SemaphoreSlim: admission control, not a queueing strategy by itself

SemaphoreSlim can limit concurrent access:

public sealed class BoundedOcrClient(
    IOcrClient inner,
    int maximumConcurrency)
{
    private readonly SemaphoreSlim gate = new(maximumConcurrency);

    public async Task<OcrResult> ExtractAsync(
        Document document,
        CancellationToken cancellationToken)
    {
        await gate.WaitAsync(cancellationToken);
        try
        {
            return await inner.ExtractAsync(document, cancellationToken);
        }
        finally
        {
            gate.Release();
        }
    }
}

Release only after a successful wait. Use finally. Do not call synchronous Wait() from async request paths.

However, thousands of callers can still wait in memory behind the semaphore. A bounded channel provides explicit admission and queue capacity. A rate limiter can express permits, queue and rejection policy. Choose the abstraction that matches overload behaviour.

Semaphores do not guarantee the fairness a business workflow may require. Tenant A can occupy every permit. Per-tenant quotas or fair scheduling may be necessary. A single global limit may also underutilise independent downstream partitions.

23. Locks: protect invariants, not asynchronous workflows

The C# lock statement provides mutual exclusion for synchronous critical sections. Keep the section small and never await inside it. Do not lock on this, strings, types or externally visible objects because unrelated code can share them.

private readonly Lock cacheLock = new();
private readonly Dictionary<string, CacheEntry> cache = [];

public CacheEntry? Find(string key)
{
    lock (cacheLock)
        return cache.GetValueOrDefault(key);
}

Modern C# recognises System.Threading.Lock with dedicated semantics when used in lock; use the target framework/compiler combination supported by the project. A private object remains familiar for older targets.

Do not hold a lock across SQL or HTTP. Other callers then wait for a remote dependency while holding a local resource, increasing deadlock and latency risk. Snapshot required state under the lock, release it, perform I/O, then reacquire and validate that assumptions still hold—or use a higher-level asynchronous design.

Reader/writer locks help only when read concurrency and write rarity create measured benefit. They are easier to misuse and can suffer starvation depending on policy. Immutable snapshots or concurrent collections may be simpler.

Junior: Should every shared collection become ConcurrentDictionary?

>

Senior: It makes individual operations thread-safe, not multi-step business invariants. “Check then add then update another collection” can still race. Use atomic APIs or one owner for the state.

24. Atomic operations and memory visibility

Interlocked.Increment makes one increment atomic:

Interlocked.Increment(ref processedCount);

It does not make a larger workflow atomic. If completion requires incrementing a count, adding a result and setting a status consistently, a lock, channel owner or database transaction may be required.

volatile changes memory-access semantics for a field but does not make counter++ atomic and is not a general thread-safety switch. Prefer Interlocked, locks and concurrent abstractions whose contracts are clearer.

Immutable messages passed through channels minimise shared mutation. Each pipeline stage owns its local state and emits a new value. This actor-like ownership often removes the need to reason directly about memory barriers.

25. Async streams and backpressure at API boundaries

IAsyncEnumerable can stream results without materialising the entire collection:

public async IAsyncEnumerable<DocumentStatus> ReadStatusesAsync(
    TenantId tenant,
    [EnumeratorCancellation] CancellationToken cancellationToken = default)
{
    await foreach (var row in repository.StreamStatusesAsync(tenant, cancellationToken))
    {
        yield return DocumentStatus.From(row);
    }
}

The enumerator cancellation attribute links the consumer’s token to iteration. The database reader and context must remain alive until enumeration completes. Exceptions occur during enumeration, not necessarily when the method is called.

Streaming shifts rather than eliminates backpressure. A slow HTTP client can hold server/database resources. Buffer a bounded page or decouple through durable storage when long-lived streams threaten capacity. Configure timeouts and cancellation.

Do not return an async stream from inside a disposed using scope. The method body executes lazily, so resource lifetime must cover enumeration.

26. Deadlock beyond the textbook two-lock example

Classic lock-order deadlock is:

Task A holds customer lock, waits for loan lock
Task B holds loan lock, waits for customer lock

Prevent it with a global lock ordering, one combined owner or transactional redesign.

Sync-over-async can also deadlock in environments with a captured synchronization context:

var result = GetDataAsync().Result;

The caller blocks the context thread while the continuation waits to return to that context. ASP.NET Core does not install the classic request synchronization context, but blocking remains harmful because it consumes thread-pool capacity and can cause starvation. Libraries used in UI or older application environments still need context awareness.

Use await end to end. ConfigureAwait(false) is relevant in general-purpose libraries that need not resume on a captured context; it is not a repair for calling .Result at the top.

Database deadlocks are different: transactions lock resources in conflicting order and the database selects a victim. Keep transactions short, access resources consistently, index queries, and retry the whole transaction only when safe and idempotent.

27. Thread-pool starvation diagnosis

Symptoms include rising latency, low CPU, increasing thread count and work waiting while requests use synchronous blocking. Common causes are .Result, .Wait(), synchronous network/database APIs and long Task.Run jobs inside a server.

Collect runtime counters, traces and thread stacks. Look for many threads blocked on waits, locks or I/O. Correlate with endpoint and deployment version. Do not increase minimum threads as the first and only fix; that can hide blocking and increase downstream pressure.

Junior: CPU is only 30%, so why is the server slow?

>

Senior: CPU capacity is irrelevant when worker threads wait synchronously for I/O or locks. Measure queues, waits and dependencies, not only utilisation.

Replace synchronous APIs with true asynchronous versions through the entire call path. Bound background CPU work separately. Verify database connection pool and HTTP connection limits so async requests do not merely move the queue.

28. Race-condition clinic: stale cache population

Two requests miss a cache and load the same key. The older request finishes last and overwrites fresher data.

A reads source version 7 slowly
B reads source version 8 quickly, stores it
A completes and stores version 7

A ConcurrentDictionary prevents dictionary corruption, not stale overwrite. Include source version and update atomically:

cache.AddOrUpdate(
    key,
    _ => candidate,
    (_, current) => candidate.Version > current.Version ? candidate : current);

The update delegate can run more than once and should be side-effect free. If loading must be coalesced, a keyed async-lazy or single-flight abstraction may help, but failures and cancellation need careful cache eviction.

Avoid caching a faulted or cancelled task forever unless that is intentional. Do not let one caller’s cancellation cancel shared work needed by other callers without a defined ownership model.

29. Race-condition clinic: check-then-act approval

In memory:

if (loan.Status == LoanStatus.Submitted)
    loan.Status = LoanStatus.Approved;

Two server instances can both see Submitted. A local lock protects one process only. Correctness belongs in the database/domain concurrency design: optimistic version token, conditional update or transaction.

UPDATE LoanApplications
SET Status = 'Approved', Version = Version + 1
WHERE Id = @id AND Status = 'Submitted' AND Version = @expectedVersion;

If affected rows is zero, return a conflict and load current state. Do not retry a stale approval automatically. This illustrates the boundary between multithreading and distributed concurrency: no C# lock coordinates every process and database client.

30. Cancellation as a contract

Accept cancellation at public asynchronous boundaries, pass the token to cancellable calls and check it in long CPU loops at reasonable intervals.

for (var page = 0; page < pages.Count; page++)
{
    cancellationToken.ThrowIfCancellationRequested();
    ProcessPage(pages[page]);
}

Do not check on every trivial instruction; balance responsiveness and overhead. Library methods should normally propagate OperationCanceledException rather than translate it into failure.

After a point of no cancellation—such as once an irreversible commit begins—use a clear policy. You may finish the critical step, record outcome and let the caller query by idempotency key. Passing a cancelled token to essential cleanup can prevent cleanup; use a short bounded token owned by shutdown/recovery where appropriate.

CancellationToken is a notification, not a thread abort. Code must cooperate. External services may ignore connection cancellation after accepting work.

31. Graceful shutdown of a pipeline

Shutdown needs an order:

stop accepting new work
  -> complete input writer
  -> drain within grace period
  -> cancel remaining workers
  -> persist/release leases
  -> await tasks and record incomplete work

If work is durable, a worker lease with expiry allows another instance to recover. Mark completion only after output is durably committed. If work is not durable, tell operators and users what can be lost.

Use host ApplicationStopping/BackgroundService tokens for cooperative shutdown, but distinguish them from per-job deadlines. A linked token can represent either condition while telemetry records which source fired.

Kubernetes or another host may terminate after a configured grace period. Application shutdown time must fit it. Test by sending termination under load and verifying no duplicate or abandoned document remains hidden.

32. Observability for concurrent systems

Average duration hides queueing. Record:

• input queue depth and time waiting;
• in-flight operations per stage;
• processing duration separate from queue duration;
• throughput, success, retry and failure by stage;
• cancellations by caller, deadline and shutdown;
• worker utilisation and downstream throttling;
• thread-pool queue/thread counts and lock contention where available;
• oldest durable job and lease recovery;
• end-to-end document latency.

Use a correlation/job ID across stages. Logs from concurrent work interleave, so narrative text without identifiers is difficult to reconstruct. Do not log extracted document content or sensitive identifiers indiscriminately.

Tracing every item at full fidelity can be expensive. Sample normal work, retain error traces under privacy policy and use metrics for aggregate saturation. A health endpoint should distinguish process liveness from readiness to accept more work.

33. Testing concurrency without relying on sleeps

Tests using Thread.Sleep(100) are slow and nondeterministic. Coordinate exact moments with TaskCompletionSource, barriers or controllable fakes.

var firstStarted = new TaskCompletionSource(
    TaskCreationOptions.RunContinuationsAsynchronously);
var allowFirstToFinish = new TaskCompletionSource(
    TaskCreationOptions.RunContinuationsAsynchronously);

fakeSource.OnReadVersion(7, async () =>
{
    firstStarted.SetResult();
    await allowFirstToFinish.Task;
    return version7;
});

var first = cache.LoadAsync(key, cancellationToken);
await firstStarted.Task;
await cache.LoadAsync(key, cancellationToken); // stores newer version
allowFirstToFinish.SetResult();
await first;

cache.Get(key)!.Version.Should().Be(8);

RunContinuationsAsynchronously reduces surprising inline continuation execution in test coordination. Always put timeouts around coordination so a broken test fails rather than hangs the suite.

Stress tests repeat operations under concurrency to find rare races, but a passing stress test is not proof of absence. Combine them with design reasoning and deterministic schedules for known races. Run ThreadSanitizer-like expectations are not generally available for managed business logic; ownership and invariant tests matter.

Test cancellation before start, during queue wait, during I/O, after external acceptance and during shutdown. Test channel completion with success and failure. Assert every worker task is awaited.

34. Performance methodology

Concurrency can reduce latency or increase throughput only until a bottleneck saturates. Measure at degrees 1, 2, 4, 8 and beyond. Observe throughput, p50/p95/p99 latency, CPU, memory, GC, connections and dependency throttling.

Little’s Law connects average items in a stable system to throughput and time in system. Even without formal modelling, queue depth and wait time reveal whether arrival exceeds service rate. A continually rising bounded queue eventually applies backpressure; an unbounded queue eventually fails memory or latency objectives.

Avoid benchmarking with a fake zero-latency dependency when production waits 200 ms. Use representative distributions and failures. Warm up JIT and pools where appropriate, and include container CPU quotas.

Stop increasing concurrency when throughput plateaus or tail latency/errors rise. The optimum may differ per stage and tenant. Document the chosen values and make them configurable within safe bounds.

35. Security and tenant fairness under concurrency

Every queued item must retain trusted tenant and actor context without storing raw tokens. Re-authorise where a delayed job executes if policy or membership may change. A background service principal still needs domain checks for the represented operation.

Prevent one tenant from exhausting every channel slot or downstream permit. Use per-tenant quotas, weighted scheduling or partitioned queues according to service policy. Global FIFO may be simple but unfair under a noisy neighbour.

Bound document size before enqueue, validate content type, scan untrusted files and isolate parsers where risk demands it. A decompression bomb or pathological document can consume CPU and block workers. Apply per-item time and memory limits when possible.

Avoid including personal data in exception aggregation and traces. Concurrent failures can multiply log volume; rate-limit repetitive diagnostics while retaining counts and representative evidence.

36. Code-review checklist for concurrent C#

• Is the work CPU-bound, I/O-bound or mixed?
• What resource is bounded, and where is overload rejected or delayed?
• Who owns each mutable value?
• Are child tasks awaited and exceptions observed?
• Does cancellation propagate, and what happens after partial completion?
• Are writes idempotent across retry and process boundaries?
• Can stale results overwrite newer state?
• Is lock ordering defined, and are locks held across I/O?
• Does a concurrent collection actually protect the multi-step invariant?
• Can work survive process restart when required?
• Are queue wait and processing time observable separately?
• Does shutdown drain or recover in-flight work?
• Is capacity fair across tenants and safe for dependencies?
• Do deterministic tests exercise the dangerous schedules?

If a developer says “it is thread-safe,” ask which invariant and which scope: one object, one process, one database row or a distributed workflow. Precision prevents the phrase from hiding risk.

37. Exercises for the junior developer

Exercise one: bound a producer

Create a bounded channel of capacity five and a slow consumer. Produce twenty items. Measure how long each write waits. Change full mode to a dropping policy and record exactly what is lost; decide whether that is acceptable.

Exercise two: compare async and parallel

Run twenty delayed HTTP fakes sequentially, with unbounded WhenAll and with concurrency five. Then run a CPU calculation at different parallel degrees. Explain why the best policies differ.

Exercise three: force a stale completion

Coordinate two cache loads so the older result finishes last. Prove a naive ConcurrentDictionary update regresses the version, then fix it atomically.

Exercise four: test ambiguous cancellation

Make a fake provider commit then block its response. Cancel the caller and retry with the same operation ID. Prove one external operation and a recoverable result.

Exercise five: terminate under load

Fill the pipeline, request host shutdown and observe accepted, completed and recoverable work. Adjust grace and lease policy until no job is silently lost or applied twice.

Exercise six: diagnose starvation

Replace an async dependency call with .Result under load. Collect thread-pool and latency evidence, restore async end to end and compare. Do not “fix” it solely by raising thread counts.

38. Cross-links for continuing the mentoring path

Use C# Async/Await, Tasks and ASP.NET Core for deeper request-oriented asynchrony and C# Async/Await, Race Conditions and Locks for focused race analysis. High-Performance C# and .NET develops benchmarking and allocation discipline. Microservices with .NET extends idempotency, queues and partial failure across processes, while How to Investigate Slow Angular, ASP.NET Core and SQL Server Applications connects thread-pool and dependency symptoms across the stack.

The central boundary is worth repeating: thread synchronisation protects in-process memory; database concurrency protects durable rows; idempotency and messaging protocols protect distributed work. One mechanism cannot substitute for all three.

39. ValueTask is a measured API choice

Task is the normal return type for asynchronous operations. It is easy to cache, await more than once and compose. ValueTask can avoid a task allocation when an operation frequently completes synchronously, but it adds consumption rules and complexity.

public ValueTask<CacheEntry?> FindAsync(
    string key,
    CancellationToken cancellationToken)
{
    if (memory.TryGetValue(key, out var value))
        return ValueTask.FromResult<CacheEntry?>(value);

    return new ValueTask<CacheEntry?>(
        LoadFromDistributedCacheAsync(key, cancellationToken));
}

Unless documented otherwise, await a ValueTask once and do not store it for later, call Result prematurely or combine it repeatedly. Convert with AsTask() if a consuming API genuinely needs a Task, accepting the allocation.

Junior: Should every cache API return ValueTask because hits are synchronous?

>

Senior: Only after profiling shows task allocation matters on that hot path and the API’s callers can follow the contract. Task is the safer default.

An async method that always performs asynchronous I/O usually gains little from ValueTask. Public libraries must weigh a small allocation benefit against permanent API complexity.

40. Asynchronous disposal and pooled resources

Some resources need asynchronous cleanup, such as flushing or closing a network-backed operation. Use await using for IAsyncDisposable:

await using var lease = await jobLeases.AcquireAsync(jobId, cancellationToken);
await ProcessAsync(lease.Job, cancellationToken);
await lease.CompleteAsync(cancellationToken);

Disposal runs when leaving scope, including exceptions. Decide whether disposal should use caller cancellation. Essential release may need a short independent timeout; an unlimited cleanup can hang shutdown.

Do not return a lazy async sequence tied to a disposed context:

public IAsyncEnumerable<Row> Wrong()
{
    using var connection = OpenConnection();
    return QueryRowsAsync(connection); // connection closes before enumeration
}

Own the resource inside the async iterator so its lifetime covers enumeration, or materialise before disposal. Tests should consume the sequence, not merely construct it.

Pools require reset discipline. Returning an object with tenant data, transaction state or cancellation registration can contaminate the next borrower. Prefer platform pools and verify lifecycle hooks rather than creating a custom pool casually.

41. Rate limiting versus concurrency limiting

Concurrency limiting caps simultaneous work. Rate limiting caps starts or permits over time. A partner may allow ten concurrent calls but only one hundred requests per minute; those are separate constraints.

The .NET rate-limiting abstractions can implement concurrency, fixed-window, sliding-window or token-bucket policies. Whichever library/configuration is used, the application must define queue limit, processing order, rejection response and tenant fairness.

Conceptually:

using var lease = await limiter.AcquireAsync(permitCount: 1, cancellationToken);
if (!lease.IsAcquired)
    throw new CapacityRejectedException("OCR capacity is currently unavailable.");

return await inner.ExtractAsync(document, cancellationToken);

Inspect lease metadata if the limiter provides retry guidance. Do not claim a precise Retry-After when capacity cannot be predicted. Queueing inside the limiter plus queueing in a channel plus queueing in HttpClient can create several invisible wait layers; measure each or simplify.

Retries consume rate permits and can amplify an outage. Budget original attempts and retries together. Add jitter and a circuit-breaking or degradation policy where appropriate, while remembering that resilience libraries do not determine whether a write is safe to repeat.

42. Channels, TPL Dataflow and durable brokers

Choose based on needed semantics:

TPL Dataflow can express transform and action blocks with bounded capacity and parallelism. It may be clearer when a graph has several linked stages and propagation rules. Channels expose smaller primitives and may fit a custom worker loop. Do not rebuild durable messaging from either.

Junior: Can we put a Channel in a singleton and call it a message bus?

>

Senior: It is a useful in-process queue. A message bus implies durability, delivery and operational contracts that a process-local channel does not provide.

When moving from a channel to a broker, retain idempotent handlers and explicit message versions. Broker acknowledgement should follow durable local completion. Visibility timeouts or leases must exceed expected processing or be renewed safely.

43. Async coordination with TaskCompletionSource

TaskCompletionSource adapts callback/event completion into a task and coordinates tests. Always consider RunContinuationsAsynchronously so completing code does not unexpectedly run arbitrary consumer continuations inline while holding a lock.

private readonly TaskCompletionSource<ReadyState> ready =
    new(TaskCreationOptions.RunContinuationsAsynchronously);

public Task<ReadyState> Ready => ready.Task;

public bool TryMarkReady(ReadyState state) => ready.TrySetResult(state);
public bool TryFail(Exception error) => ready.TrySetException(error);

Prefer TrySet... when racing completion paths are possible. SetResult throws if another path already completed. Ensure cancellation registrations are disposed and cannot retain objects indefinitely.

Do not use a TaskCompletionSource as a general mutable event that completes repeatedly; a task completes once. Channels, observables or an async-reset event abstraction fit repeated signals.

44. Why lock-free code is rarely the first answer

Lock-free algorithms can improve progress under contention, but correctness depends on atomic compare/exchange, memory ordering and problems such as ABA. Ordinary business services should begin with immutable messages, one-owner state, locks or concurrent collections.

An optimistic compare/exchange loop looks simple:

while (true)
{
    var current = Volatile.Read(ref snapshot);
    var next = current.WithProcessedOneMore();
    if (ReferenceEquals(
        Interlocked.CompareExchange(ref snapshot, next, current),
        current))
        break;
}

It works because snapshots are immutable and replacement is atomic. Side effects must not occur inside the retry loop because the body may execute several times. Allocation under contention can erase benefits.

Junior: Is lock-free always faster?

>

Senior: No. A short uncontended lock can be simpler and faster. Measure the real workload and include tail latency, allocation and maintainability.

Reviewers should demand a written invariant, proof strategy, benchmark and stress tests for custom lock-free code. Prefer framework implementations maintained by specialists.

45. Exception policy per pipeline item

One malformed document should not necessarily stop all documents, while a corrupted configuration may require stopping the pipeline. Classify failures:

Permanent item failure:
  unsupported format, failed validation
  -> record outcome, continue

Transient item failure:
  throttling, temporary dependency outage
  -> bounded retry or durable reschedule

Systemic failure:
  invalid model/configuration, lost database authority
  -> stop accepting, alert, fail/drain according to policy

Cancellation/shutdown:
  -> preserve lease/checkpoint, do not report as business failure

Do not catch every exception inside a worker and continue. OutOfMemoryException, corrupted invariant or repeated configuration failure can make continued processing unsafe. Conversely, allowing one bad PDF to fault Task.WhenAll and abandon 199 good jobs may be the wrong batch contract.

Create a serialisable safe failure record with stage, code, retryability, operation ID and protected diagnostics reference. Never place full document content or exception dump on a dead-letter queue without data-governance approval.

Poison items need a maximum delivery count and visible quarantine. A manual replay must preserve idempotency and audit who changed or approved the item.

46. Full incident: throughput collapses after “more parallelism”

A deployment raises extraction concurrency from 20 to 100 to improve backlog. For five minutes throughput rises. Then OCR returns throttling, retries start, memory and queue wait grow, database updates time out and p99 document completion exceeds an hour.

Immediate response:

1. Stop or reduce new admission if the backlog threatens stability.
2. Roll concurrency to the last safe value through controlled configuration.
3. Disable or reduce retries that amplify throttling.
4. Preserve queue depth, in-flight, retry, dependency and pool evidence.
5. Confirm durable jobs remain recoverable before restarting workers.

Diagnosis should plot arrival, queue wait, stage throughput, downstream latency/status, retry volume, CPU, memory, connections and database waits on one timeline. The apparent improvement used concurrency beyond the dependency’s sustainable rate. Retries consumed more permits and prolonged occupancy.

Junior: Why did bounded concurrency not protect us?

>

Senior: A bound of 100 is still unsafe when the dependency sustains 20. Bounds must come from capacity evidence, and retry work must count toward them.

Permanent improvements include adaptive or explicitly configured rate policy, retry budgets, per-stage bulkheads, backpressure to upload admission, backlog alerts based on oldest age, and load tests with realistic throttling. Document why the chosen value exists.

Avoid an automatic controller that increases concurrency from short-term success without guardrails. Queuing systems have delayed feedback; by the time error rate rises, excessive work may already be in flight.

47. Full incident: graceful shutdown duplicates a notification

The worker stores a validation result, calls a notification provider, then receives termination before marking the job complete. Another instance recovers the lease and sends again.

This is not fixed by awaiting more carefully; process death can happen at any instruction. Make the notification idempotent using a stable key if the provider supports it. Otherwise persist a notification operation/outbox before sending and reconcile ambiguous delivery.

The job state machine could be:

Ready -> Leased -> ResultStored -> NotificationPending
      -> NotificationConfirmed -> Completed

Transitions use expected version and lease identity. Recovery examines durable state rather than replaying the whole pipeline blindly. If the provider outcome is uncertain, query it where possible or route to reconciliation.

Shutdown drains when time allows, but correctness cannot depend on receiving graceful notice. Machines crash and processes are killed. Durable state and idempotency make abrupt failure survivable.

48. Architecture decision: in-process or distributed pipeline

Keep processing in one service when volume fits one deployment, stages share ownership, latency benefits from local transfer and process restart can recover from durable job state. Split only for a reason: independent scaling, isolation of unsafe parsers, separate deployment ownership, specialised compute or a strong availability boundary.

Distribution introduces message contracts, eventual consistency, duplicate delivery, network latency, tracing and more operations. It can improve resilience only when those mechanisms are implemented and operated well.

Start with one worker service using bounded channels behind a durable job table or broker. Measure stage saturation. If classification CPU scales differently from OCR I/O, separate that stage later while retaining the message/idempotency contract.

Do not use distributed locks to preserve an in-memory design across services. Redesign ownership around durable commands and state transitions.

49. Mentoring kata: evolve sequential code safely

Begin with a clear sequential implementation and a test dataset. Record correctness, duration and resource use.

Step one: make genuine I/O asynchronous without parallelising. Confirm behaviour and cancellation.

Step two: start a small finite set of independent I/O tasks with a concurrency gate. Compare throughput and dependency pressure.

Step three: introduce a bounded channel for streaming input and record queue wait.

Step four: isolate CPU classification with measured parallel degree.

Step five: kill the process mid-item and add durable recovery/idempotency.

Step six: inject throttling, timeouts, malformed items and shutdown. Write runbooks from observed behaviour.

At every step, retain a simpler fallback and prove the new complexity creates value. The exercise teaches that concurrency architecture is an incremental response to capacity and correctness evidence.

50. Final production-readiness gate

Before release, demonstrate:

• every queue has a bound or durable capacity policy;
• each stage’s concurrency and rate limits are documented and measured;
• shared state has one owner or a proved synchronisation invariant;
• all tasks, async enumerables and disposables have explicit lifetime owners;
• cancellation, timeout and shutdown are distinguishable and tested;
• retries reuse idempotency identity and respect a total deadline/budget;
• database commands handle optimistic concurrency across instances;
• item and systemic failures follow different policies;
• durable work recovers after abrupt process death;
• metrics expose admission, queue wait, processing and downstream saturation;
• tenant fairness and untrusted-input resource limits are enforced;
• rollback can restore prior capacity configuration and compatible workers.

The gate is intentionally about behaviour rather than API count. A service using only await and a bounded worker can be more mature than one combining PLINQ, Dataflow, Rx and custom lock-free structures without a capacity model.

51. Choosing a synchronisation primitive by the problem

Use this as a conversation starter, not a mechanical lookup table:

Mutex can coordinate across processes on one machine, but it does not become a distributed lock across containers or hosts. Named operating-system primitives also bring platform, identity and abandonment semantics. Use them only when the deployment boundary truly is one machine.

Reset events and wait handles block threads and fit particular synchronous integration or low-level scenarios. In asynchronous application code, task-based signalling is usually more scalable. Do not wrap a blocking wait in Task.Run and assume it became naturally asynchronous; it still occupies a pool thread.

Junior: Why learn the lower-level primitives if channels and tasks are preferred?

>

Senior: They explain what higher-level tools protect and help diagnose libraries or legacy code. Understanding them lets you choose not to use them with confidence.

52. Review a misleading “parallel improvement” pull request

Suppose a pull request changes:

foreach (var loan in loans)
    await ValidateAsync(loan, cancellationToken);

to:

await Task.WhenAll(loans.Select(
    loan => ValidateAsync(loan, cancellationToken)));

The new code may be faster for ten independent items. For ten thousand items it immediately invokes ten thousand operations. A useful review asks:

• Is loans bounded and materialised?
• Does validation read or write shared state?
• What concurrency does the database/provider support?
• Is each write idempotent?
• What is the failure contract when item 500 fails?
• Does the caller need all results in memory?
• How does cancellation affect already accepted work?
• Which metric proves improvement?

The recommended change might be a fixed worker count, Parallel.ForEachAsync with an explicit maximum, or a channel. It might remain sequential if validation is a transactionally ordered workflow.

A review comment should state the risk and proof:

This starts one provider call per loan with no bound. The production batch can
contain 20,000 items while the provider permits 25 concurrent calls. Please
limit concurrency to the measured capacity and add a throttling load test that
asserts queue wait, retry volume and total completion remain within our budget.

That is stronger than “use SemaphoreSlim.” It preserves room for the author to choose the clearest mechanism.

53. The mentoring conclusion for concurrency

Concurrency expertise is the ability to state ownership, ordering and capacity precisely. Which work may overlap? Which result wins? What is shared? Where does waiting occur? What survives failure? Who cancels whom? How much pressure can every dependency sustain?

Junior: When should I make code concurrent?

>

Senior: After the sequential behaviour is correct and evidence shows overlap can improve an objective. Add the smallest concurrency policy, then test the schedules and failures it introduces.

Sometimes the best improvement is an index, a batch API, less data, caching, faster serial logic or removing an unnecessary remote call. Parallelism cannot compensate for the wrong algorithm or architecture. It can make the wrong work happen faster and fail more widely.

Keep one principle visible in every design review:

Concurrency is not free speed.
It is controlled overlap under explicit correctness and capacity rules.

When those rules are encoded, measured and observable, tasks and threads become useful implementation details rather than sources of mystery.

Finish by documenting the chosen limits beside the evidence that produced them. A number such as MaxDegreeOfParallelism = 8 without explanation becomes folklore and is eventually “optimised” upward. Link the load test, downstream quota, container capacity and rollback threshold. Revisit the decision when workload or infrastructure changes.

Likewise, always clearly document all intentional sequencing. Future developers should know when two awaits are consecutive because ordering protects a business invariant, not because nobody considered concurrency. Clear constraints prevent a well-meaning refactor from reintroducing an old race.

What I want you to take away

Multithreading in C# is not just about starting a Thread.

To use concurrency well, you need to understand the whole ladder.

At the bottom, a thread is an execution path inside a process. Threads allow work to happen concurrently, and on multi-core machines, sometimes truly in parallel. But threads are expensive, unordered, and dangerous when they share mutable state.

Race conditions happen when multiple threads read and write shared state without coordination. lock, Monitor, Interlocked, Mutex, SemaphoreSlim, reset events, barriers, countdown events and reader-writer locks exist to coordinate access, signal progress, and protect correctness. But synchronization has a cost, and bad locking can create deadlocks.

The thread pool exists because creating raw threads for every short operation is wasteful. It reuses worker threads, but it must not be abused with long-running or blocking operations, especially in server applications.

The Task Parallel Library raises the abstraction. A Task represents future work or a future result. It gives composition through Task.WhenAll, Task.WhenAny, continuations, cancellation, exceptions and schedulers.

async and await make asynchronous code readable, but they do not automatically create threads and they do not automatically run things in parallel. Consecutive awaits are sequential. Parallel async work requires starting independent tasks first and then awaiting them together. Avoid async void except for event handlers.

Concurrent collections give safer tools for shared data structures: ConcurrentDictionary, ConcurrentQueue, ConcurrentStack, ConcurrentBag, and BlockingCollection. PLINQ and Parallel help with CPU-bound data parallelism. Reactive Extensions help model asynchronous streams of values. Asynchronous I/O helps scalable applications avoid wasting threads while waiting for databases, files, networks, APIs or storage.

The serious interview answer is this:

“I separate CPU-bound parallelism from I/O-bound asynchrony. I use async/await for scalable I/O, Task.WhenAll for independent asynchronous operations, Parallel or PLINQ for CPU-bound work, concurrent collections for safe producer-consumer scenarios, and synchronization primitives only where shared mutable state cannot be avoided. I avoid blocking thread pool threads, design cancellation properly, handle exceptions explicitly, and measure before assuming concurrency improves performance.”

That is the difference between someone who has used Task.Run and someone who actually understands concurrency in C#.