The first two articles in Programmer Craft in the AI Era discussed programs and data structures. This one moves into another problem that AI-assisted programming can easily hide: who executes the code, where it runs, when it gives up the CPU, and when it blocks the entire service.

Many concurrency bugs are not caused by code that is too difficult. They happen because the execution model was never made explicit. Ask AI to “process a batch of tasks concurrently”, and it may write a thread pool. Ask it to “make it async”, and it may merely add async before a function. Ask it to “improve throughput”, and it may add more workers without noticing that the database connection pool is already exhausted.

The execution model is not low-level trivia. It determines whether a system can survive slow APIs, traffic spikes, CPU-heavy work, file I/O, and unstable external services.

Concurrency Is Not One Thing

“Run concurrently” is too vague.

It may mean multiple independent processes running at the same time. It may mean multiple threads inside one process competing for time slices. It may mean coroutines inside one thread switching while waiting for I/O. They all look like “processing multiple tasks at once”, but their costs, isolation, shared state, and failure modes are completely different.

Execution model	Common scenarios	Strengths	Risks
Multi-process	Web workers, batch workers, CPU-heavy tasks	Strong isolation, one crash has limited impact, uses multiple cores	Higher memory use, higher inter-process communication cost
Multi-thread	Blocking I/O concurrency, background tasks, GUI programs	Convenient shared memory, lower switch cost than processes	Races, deadlocks, thread-pool exhaustion
Coroutine / async	High-concurrency network I/O, API gateways, crawling, long connections	One thread can hold many connections; switching while waiting is cheap	One blocking call can stall the event loop
Event-driven	Message consumption, frontend UI, Node.js services, async frameworks	Good for many small events and I/O callbacks	Scattered call chains, harder error propagation and context tracing
Batch processing	Data sync, reports, offline jobs	High throughput, easier batch optimization	Higher latency, requires failure recovery and resumability

AI easily mixes these models. For example, it may generate an async def endpoint in FastAPI and then call a synchronous database driver, synchronous HTTP client, or local large-file read inside it. The syntax is async; the runtime behavior blocks the event loop.

So do not merely say “process asynchronously” in a prompt. Say this:

This is an async web endpoint running on a single-process event loop.
The request path must not perform blocking I/O.
Database access must use an async driver.
CPU-heavy computation must be offloaded to a process pool or independent worker.
External HTTP calls must have timeouts and concurrency limits.

These sentences are much more useful than “optimize performance”.

Processes: Expensive Isolation, Reliable Boundaries

A process is the isolation space the operating system gives a program. Each process has its own address space, file descriptors, environment variables, stack, heap, and runtime state. If one process crashes, it usually does not directly corrupt another process’s memory.

A common multi-process model for web services is a master process starting multiple workers:

master
  ├── worker-1
  ├── worker-2
  ├── worker-3
  └── worker-4

Each worker can accept requests. For runtimes like Python and Node.js, multi-process deployment is also a common way to bypass single-process CPU limits and use multiple cores.

But processes are not free.

If a model takes 2 GB of memory to load, starting 8 processes may not mean 2 GB total. It may be close to 16 GB, depending on copy-on-write, whether the model is modified, and how the runtime allocates memory. Database connections are similar. If each process has its own connection pool, 4 workers with 20 connections each means 80 total connections, not 20.

When AI configures web workers, it often gives advice like:

The CPU has 8 cores, so set workers to 8.

That is only valid for a narrow set of scenarios. A better estimate must consider:

• Resident memory per worker.
• Database connection-pool size per worker.
• Whether requests are CPU-heavy or I/O-heavy.
• Whether the machine also runs other services.
• Whether downstream databases, caches, and third-party APIs can handle the concurrency.
• Whether worker restarts trigger a cold-start surge.

A more useful prompt is:

Do not recommend worker count only by CPU cores.
First estimate:
1. Resident memory per worker.
2. DB connection-pool size per worker.
3. Peak request concurrency.
4. Downstream database maximum connections.
5. Share of CPU-heavy logic.
Then propose worker count, connection-pool size, and a load-test validation method.

The keyword for a process model is isolation. Isolation brings reliability and resource amplification at the same time. If AI only sees code and not deployment resources, it can easily turn “it runs” into “it exhausts the machine in production”.

Threads: Convenient Shared Memory, Hidden Damage

Threads inside the same process share memory. Shared memory makes communication easier and errors more subtle.

Consider a very common counter:

count = 0


def handle_request():
    global count
    count += 1

In a single-threaded program this is fine. In a multi-threaded program, count += 1 is not atomic. It includes at least read, increment, and write. If two threads execute it at the same time, increments can be lost.

AI easily generates code like this because it usually passes unit tests. Running the test 100 times may still pass. It fails only occasionally under production concurrency.

That is the hardest part of thread bugs: they are unstable, difficult to reproduce, and their logs often do not look wrong.

Common thread risks include:

• Multiple threads modifying the same dict, list, or object field.
• One thread holding a lock while calling slow I/O, making other threads wait.
• Multiple locks acquired in inconsistent order, causing deadlock.
• A thread pool submitting work and then waiting for the same pool, causing starvation.
• Request context, user identity, or trace ID stored in thread-local variables and then lost after the execution model changes.

Thread pools are especially easy for AI to misuse. For example, it may put synchronous external API calls into a thread pool:

from concurrent.futures import ThreadPoolExecutor

executor = ThreadPoolExecutor(max_workers=100)


def call_all(items):
    futures = [executor.submit(call_remote_api, item) for item in items]
    return [future.result() for future in futures]

This looks concurrent, but it misses several key constraints:

• If items has 100,000 elements, it submits 100,000 futures at once.
• Calls have no timeout, so threads may be occupied forever.
• There is no failure isolation; one slow downstream can block the whole batch.
• There is no global rate limit; multiple requests can exhaust the thread pool.
• There is no backpressure, so the caller does not know the system is already overloaded.

A steadier implementation controls batching, timeout, and concurrency:

from concurrent.futures import ThreadPoolExecutor, as_completed


def call_bounded(items, *, max_workers=20, batch_size=200, timeout=3):
    results = []
    with ThreadPoolExecutor(max_workers=max_workers) as executor:
        for start in range(0, len(items), batch_size):
            batch = items[start:start + batch_size]
            futures = [executor.submit(call_remote_api, item) for item in batch]
            for future in as_completed(futures, timeout=timeout * len(batch)):
                results.append(future.result(timeout=timeout))
    return results

This is still only a sketch. Production code also needs exception handling, cancellation, retry, metrics, and partial-failure behavior. But it has already changed the thread pool from “submit without limit” to “controlled concurrency”.

When asking AI to write threaded code, these words must appear in the prompt: shared state, lock ordering, thread-pool size, queue length, timeout, cancellation, rate limit, metrics.

Coroutines Matter Only When They Do Not Block

The value of coroutines is not magic. It is that they can voluntarily yield execution while waiting for I/O.

An async service roughly works like this:

event loop
  ├── request A waiting for database
  ├── request B waiting for HTTP response
  ├── request C writing socket
  └── request D ready to return

When A waits for the database, the event loop can process B, C, and D. This model can support many connections with a small number of threads.

The premise is that waiting must be yieldable.

The following code is async in syntax but can be disastrous in behavior:

import requests


async def fetch_user(user_id: str):
    response = requests.get(f"https://example.com/users/{user_id}")
    return response.json()

requests.get is synchronous and blocking. Until it returns, the event loop cannot process other requests. Once concurrency rises, the whole service’s latency is dragged up.

Use an async HTTP client instead:

import httpx


async def fetch_user(user_id: str):
    timeout = httpx.Timeout(3.0, connect=1.0)
    async with httpx.AsyncClient(timeout=timeout) as client:
        response = await client.get(f"https://example.com/users/{user_id}")
        response.raise_for_status()
        return response.json()

The same applies to databases. If an async def function uses synchronous ORM queries, it still blocks. File reads and writes, large JSON parsing, image processing, compression, encryption, and heavy regular expressions can also stall the event loop.

When reviewing async code, I first look for these danger points:

• requests, synchronous database drivers, synchronous SDKs.
• time.sleep() instead of await asyncio.sleep().
• Synchronous large-file reads and writes.
• CPU-heavy loops.
• External calls without timeouts.
• asyncio.gather() submitting too many tasks at once.
• Missing await, leaving coroutine objects unexecuted.

AI often treats gather as universal concurrency:

results = await asyncio.gather(*[fetch(item) for item in items])

If items has 50,000 elements, this creates 50,000 coroutines at once. Even if each coroutine only does network I/O, downstream services, DNS, connection pools, and local memory may fail.

A better version adds a concurrency limit:

import asyncio


async def gather_limited(items, limit: int = 100):
    semaphore = asyncio.Semaphore(limit)

    async def run_one(item):
        async with semaphore:
            return await fetch(item)

    return await asyncio.gather(*(run_one(item) for item in items))

If results can be processed as a stream, do not accumulate all results at once. Concurrency is not better because it is larger. It must be constrained by upper bounds, timeouts, and downstream capacity.

Think Separately About CPU-bound and I/O-bound Work

Many performance mistakes come from mixing CPU-bound and I/O-bound work together.

I/O-bound tasks spend most of their time waiting: database, network, disk, or third-party APIs. Coroutines and threads can both help; the key is not to waste waiting time as idle blocking.

CPU-bound tasks spend most of their time computing: compression, encryption, image processing, model inference, complex parsing, large sorting, and matrix computation. Coroutines do little for them because they keep occupying the CPU and do not naturally yield the event loop.

A simple diagnostic table:

Symptom	More likely cause	First things to consider
Low CPU, high latency	I/O wait, lock wait, connection-pool wait	Timeout, connection pool, async I/O, batching, downstream investigation
CPU saturated, queue growing	CPU-heavy computation	Process pool, independent worker, algorithm optimization, batch processing
DB connections exhausted	Excess concurrency or connection leak	Pool size, request limits, slow queries, transaction boundaries
Event-loop lag rising	Blocking call or CPU work in async path	Find synchronous calls, offload work, split workers
Many threads but no throughput gain	Downstream bottleneck or lock contention	Rate limiting, lock granularity, batching, degradation

Suppose an API uploads a CSV, parses 200,000 rows, and writes them to the database. AI may write everything inside the request:

async def upload(file):
    rows = parse_csv(file)
    await insert_rows(rows)
    return {"ok": True}

This puts all cost on the request path. Parsing may block the event loop, writes may hold a long transaction, and the user connection waits the whole time.

A more engineering-oriented design would be:

1. The request only authenticates, saves the file, and creates a task.
2. A background worker reads the task and parses in chunks.
3. Each batch is validated and written in bulk.
4. Task status is queryable, and failures can be located by batch.
5. Large files have limits on size, row count, and runtime.
6. Workers have concurrency limits; database writes have batch and transaction boundaries.

The execution model has changed from “one async request” to “request + queue + worker + batch processing”. This is not a code-style issue. It is a system-boundary issue.

Blocking Points Belong in the Design

The hardest thing for AI to guess from nothing is where the blocking points are.

Blocking points include, but are not limited to:

• Database queries and transactions.
• External HTTP/RPC calls.
• File-system reads and writes.
• Log writes.
• Message-queue production and consumption.
• Lock waits.
• CPU computation.
• Synchronous calls hidden inside third-party SDKs.
• DNS, TLS handshakes, and connection-pool waits.

If the prompt only says “implement this endpoint”, AI will put these things wherever they look convenient. You need to tell it explicitly which operations can run inside a request, which must be asynchronous, which must be rate-limited, and which must time out.

For a report-generation endpoint, describe it like this:

Implement report generation, but do not write code yet.

Execution-model requirements:
- The HTTP request only creates a report_job. It must not generate the whole report synchronously.
- Generation is executed by a background worker.
- One worker can generate at most 2 reports concurrently.
- Each database query reads at most 5,000 rows and must use pagination.
- External service calls must have a 10-second timeout and retry at most 2 times.
- Report states are pending/running/success/failed/cancelled.
- Users can query progress and cancel pending/running jobs.
- If a worker crashes, running jobs older than 30 minutes can be reclaimed.

This does not contain a single implementation line, but it fixes the execution model. Only then is AI less likely to put a five-minute job into an HTTP request.

Keep Shared State Small

Once an execution model involves concurrency, shared state becomes a risk source.

Shared state is not only a global variable. Database rows, cache keys, local files, message offsets, task states, user quotas, connection pools, and temporary directories can all be shared state.

A typical example is “the same task can run only once”. If code only checks in application logic:

job = get_job(job_id)
if job.status == "running":
    return
job.status = "running"
save(job)
run(job)

Two workers can read pending at the same time and both start executing.

The right core is not Python logic. It is atomic claiming:

update jobs
set status = 'running',
    locked_by = :worker_id,
    locked_at = now()
where id = :job_id
  and status = 'pending';

Then check the number of affected rows. Only the worker that updated the row can run the task.

This is where execution model and data structure meet. Concurrency control cannot rely on “I think it will not happen at the same time”. It must rely on database atomic operations, unique constraints, locks, version numbers, or idempotency keys.

When asking AI to write this kind of code, be direct:

Do not claim tasks with "check first, then update".
Use a single atomic UPDATE or SELECT ... FOR UPDATE SKIP LOCKED.
Write a test case where two workers try to claim the same task concurrently.

The last sentence matters. Only when AI writes a concurrency test is it more likely to realize this is not ordinary CRUD.

Event-loop Lag Is the Body Temperature of an Async Service

If an async service occasionally becomes slow, the database may not be slow, and the machine may not be underpowered. The event loop may be blocked.

Event-loop lag means something like this: a coroutine was supposed to be scheduled 10 milliseconds later, but it only ran 300 milliseconds later. That means the event loop was busy doing something else or was stuck on a synchronous call.

A simple background task can monitor it:

import asyncio
import time


async def monitor_event_loop_lag(interval: float = 0.1):
    while True:
        start = time.perf_counter()
        await asyncio.sleep(interval)
        lag = time.perf_counter() - start - interval
        if lag > 0.05:
            record_metric("event_loop_lag_seconds", lag)

This code does not solve the problem, but it makes the problem visible.

When event-loop lag rises, check:

• Whether a synchronous SDK was recently added.
• Whether large JSON serialization or deserialization is happening.
• Whether image processing, compression, encryption, or complex regex runs in the request path.
• Whether logs are synchronously written to slow storage.
• Whether too many tasks are submitted with gather.
• Whether large data is loaded into memory all at once before processing.

An AI-generated async service without event-loop lag, request latency, downstream latency, and connection-pool wait metrics is hard to diagnose when it becomes slow.

An Execution-model Prompt for AI

When asking AI to write code involving concurrency, async work, or background jobs, first make it answer these questions:

Do not write code yet.

Design the execution model for this feature:
1. Which logic runs inside the HTTP request, and which logic runs in background workers?
2. Which operations are I/O-bound, and which are CPU-bound?
3. Will the design use processes, threads, coroutines, message queues, or batch processing? Why?
4. List every call that may block the event loop or thread pool.
5. Provide concurrency limits, queue length, timeout, retry, cancellation, and degradation strategy.
6. List shared state and explain how each shared state remains concurrency-safe.
7. Estimate database connection pool, HTTP connection pool, and worker count.
8. Design at least 5 tests: concurrent claim, timeout, cancellation, slow downstream, worker crash recovery.
9. Only then provide the code implementation plan.

The goal is not to turn AI into an architect. The goal is to force it to expose assumptions first. Once assumptions are visible, a human can judge them.

If AI’s answer does not mention blocking points, shared state, timeouts, connection pools, concurrency limits, or failure recovery, it is probably still writing demo-level code.

A Final Acceptance Checklist

After the execution model is written, do not trust the code too quickly. Ask at least once:

• Does this endpoint contain synchronous I/O?
• Does any async function perform CPU-heavy computation?
• Do thread pools, connection pools, and task queues have upper bounds?
• Can multiple workers execute the same task?
• Do external calls have timeout, retry, and circuit-breaker boundaries?
• If the request is cancelled, will background work keep occupying resources?
• If a worker crashes, how is running state recovered?
• Can logs, metrics, and traces show where the system is stuck?
• During load testing, what happens to CPU, memory, DB connections, and event-loop lag?

These questions are not fancy, but they determine whether a system can move from “works locally” to “explainable in production”.

AI will become better and better at generating concurrent code. But the expensive part of concurrent code has never been writing correct syntax. It is knowing what should happen at the same time, what must never happen at the same time, what must yield when it becomes slow, and what must recover after failure.

That is the execution model programmers cannot let AI guess.

WeChat

Follow ZiCode on WeChat

If this post was useful, you can follow later updates on WeChat as well.

X / Twitter

Follow @ax2_zicode

Faster technical notes, short thoughts, and new-post alerts are posted on X.

X Open X Profile @ax2_zicode