The previous articles in Programmer Craft in the AI Era covered APIs, databases, caches, queues, and asynchronous jobs. They are all parts of high-concurrency systems. In this article, we raise the viewpoint a little: how does a service actually handle traffic?

AI easily answers “high concurrency” with familiar words: add cache, add machines, use async, add a queue, shard the database and tables. These words are not wrong, but they are not enough. Real high concurrency is not replacing every local point with a faster implementation. It is knowing where the whole call chain will fill up first, where to rate-limit, where to degrade, what must be protected, and where pressure must not be passed further downstream.

High concurrency is not just adding more machines. Machines are only part of capacity. Boundaries are the key.

Do the Math Before Optimizing

Many performance problems are not code problems at the beginning. They are accounting problems that were never done.

Suppose an API target is:

• Peak 2,000 QPS.
• P95 latency below 200ms.
• Each request queries the user, order, and inventory once.
• Each request also calls a recommendation service once.

If you directly ask AI to “optimize the API”, it may suggest cache, async processing, and indexes. A better first step is to calculate:

2000 QPS * 3 database queries per request = 6000 database queries/second
2000 QPS * 1 recommendation call per request = 2000 downstream calls/second

Then look at latency. If average request time is 200ms, according to Little’s Law, the approximate number of requests being processed concurrently is:

concurrency = throughput * average response time
2000 * 0.2 = 400

That is only the entry service’s concurrency. Downstream database connections, HTTP connections, thread pools, coroutine counts, and queue lengths must all be designed around this order of magnitude.

If the entry service has 8 instances, each instance averages 250 QPS. If each instance has a database pool size of 50, the number may look small, but the total connection count is:

8 * 50 = 400

If the database allows only 300 max connections, the service can exhaust database connections before it even reaches its target traffic.

The first step in high-concurrency work is not writing code. It is making QPS, latency, concurrency, connections, and downstream capacity explicit.

When asking AI to design for high concurrency, first ask it to calculate:

Do not give optimization suggestions yet.
Based on target QPS, P95 latency, downstream calls per request, instance count, and connection-pool size, estimate:
- Entry concurrency.
- Average QPS per instance.
- Total database connections.
- Downstream service call QPS.
- Which resource is most likely to become the first bottleneck.

If you cannot calculate capacity, optimization is just guessing.

Connection Pools Are Multiplication, Not Just Configuration

Connection pools are often treated as a setting that can be casually increased.

For example:

DB_POOL_SIZE=50
HTTP_POOL_SIZE=100
WORKER_COUNT=8

The problem is that these numbers multiply.

If every process has a database pool, with 4 machines, 4 processes per machine, and 50 connections per process, the total connection count is:

4 * 4 * 50 = 800

That does not include background workers, admin jobs, migration scripts, or temporary query tools. The database may not be able to tolerate it.

HTTP connection pools are the same. If the entry service calls a recommendation service, each instance opens 100 connections, and there are 20 instances, that is 2,000 connections. Can the recommendation service handle them? What about its own thread pool, connection pool, and downstream dependencies?

Connection-pool design should look at three things:

• How much concurrency upstream can generate.
• How many requests this service wants to let into the downstream system at most.
• What the downstream system can actually handle.

A connection pool is not “less likely to block if it is larger”. If the pool is too large, it passes pressure directly downstream. If it is too small, this service queues locally. Where to queue is a design choice.

A steadier approach is to make the connection pool a gate that protects downstream:

Each entry-service instance allows at most 20 concurrent database queries.
Requests exceeding the pool capacity queue locally; if waiting exceeds 50ms, fail fast or degrade.

That is more reliable than blindly increasing the pool size.

When asking AI to configure connection pools, require:

Do not only provide the per-instance pool size.
Calculate the global connection count based on instance count, process count, and worker count.
Explain the maximum connection count the downstream database or service can tolerate.
Provide connection wait timeout and degradation strategy.

If connection-pool configuration does not include global multiplication, it can easily become a production incident.

Batching Can Save a System, and It Can Also Break One

In high-concurrency systems, processing one item at a time is expensive. Batching can significantly reduce database round trips, network overhead, and transaction overhead.

For example, writing logs or history records one by one:

for record in records:
    await db.insert(record)

Batch writing:

for batch in chunks(records, 1000):
    await db.insert_many(batch)

Batching can turn 1,000 network round trips into 1. But bigger batches are not always better. If a batch is too large, one transaction becomes longer, locks are held longer, retry cost becomes higher, and memory usage increases.

Batch parameters are usually decided from several dimensions:

Parameter	Impact
batch_size	Throughput, transaction duration, single-failure cost
flush_interval	Balance between latency and throughput
queue_size	Memory usage and backpressure trigger point
max_retry	Recovery ability when downstream fluctuates
timeout	Prevents batch processing from being stuck for too long

A common short-term relief tactic is increasing batch settings from 100/1s to 1000-5000/5s. This reduces database write frequency, but increases visibility delay. Whether it is acceptable depends on whether the data is business state or observability detail.

Business state should not be delayed casually. Logs, telemetry, and history details can often tolerate a few seconds of delay.

When asking AI to optimize batch processing, be explicit:

Please distinguish business-state writes from observability-detail writes.
Business state must be persisted reliably and promptly.
Observability details can enter an in-memory queue and be written by a background task in batches of 1000-5000 rows or every 5 seconds.
When the queue is full, low-priority DEBUG records can be sampled or summarized and dropped.

The essence of batching is trading latency for throughput. That trade must be stated clearly.

Rate Limiting Is the System’s Fuse

A high-concurrency system cannot assume every request can be processed.

Rate limiting is not poor product experience. It is a basic way to protect the system. Without rate limiting, traffic spikes continue entering the service, filling threads, exhausting connections, overwhelming the database, and eventually making the system unavailable to everyone.

Common rate-limit dimensions:

• Global rate limit: maximum QPS of the whole service.
• User rate limit: maximum request frequency for one user.
• IP rate limit: protects against abnormal sources.
• API rate limit: controls expensive endpoints separately.
• Downstream rate limit: protects databases, third-party services, and model services.
• Job rate limit: limits pending/running jobs per user.

A simple token bucket can be understood like this:

class TokenBucket:
    def __init__(self, capacity: int, refill_per_second: int):
        self.capacity = capacity
        self.tokens = capacity
        self.refill_per_second = refill_per_second
        self.last_refill = time.monotonic()

    def allow(self) -> bool:
        now = time.monotonic()
        elapsed = now - self.last_refill
        self.tokens = min(
            self.capacity,
            self.tokens + elapsed * self.refill_per_second,
        )
        self.last_refill = now

        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False

This is a local version. Multiple instances need centralized counters, sharded counters, or gateway-level rate limiting. Precision, performance, and consistency have to be traded off.

The rate-limit response should also be a contract:

{
  "error": {
    "code": "RATE_LIMITED",
    "message": "Requests are too frequent. Please try again later.",
    "retryable": true
  },
  "retry_after_seconds": 3
}

When asking AI to write rate limiting, require:

Please provide rate-limit dimensions: global, user, API, and downstream.
Explain the tradeoff between local and distributed rate limiting.
429 responses must include retry_after_seconds.
Rate-limit hits must be recorded as metrics, not only logs.

Rate limiting is not a code snippet. It is part of the service contract.

Degradation Is Not Failure; It Protects the Core

The goal of degradation is not simply “do fewer features”. It is to protect the core path.

For example, a product detail page may contain:

• Basic product information.
• Price and inventory.
• Recommended products.
• Review summary.
• Browsing history.
• Advertising slot.

During traffic peaks or downstream failures, basic product information, price, and inventory are core. Recommendations, review summaries, browsing history, and ads can degrade.

Degradation strategies can be specific:

Module	Degradation method
Recommended products	Return an empty list or old cached results
Review summary	Return old cache and mark it as possibly delayed
Browsing history	Skip the write and move it to a low-priority queue
Advertising slot	Return default ad or empty result
Inventory	Must not degrade into fake data; only fail with a short timeout

When AI generates API code, it often puts every downstream call into one gather, so any failure makes the whole response fail. A better approach is to separate core and non-core dependencies:

product = await get_product(product_id)
price = await get_price(product_id)
inventory = await get_inventory(product_id)

recommendations = await best_effort(
    get_recommendations(product_id),
    fallback=[],
    timeout=0.05,
)

Failure in a best_effort call must not affect the main response, but it must record metrics. Degradation is not silently swallowing errors.

When asking AI to design degradation, require:

Please divide downstream dependencies into core dependencies and degradable dependencies.
If a core dependency fails, the API fails; if a degradable dependency fails, return fallback.
Every degradation must record metrics and traces. Silent swallowing is not allowed.

The key to degradation is deciding in advance what can be lost and what cannot.

Hot Spots Are More Dangerous Than Average Traffic

Average QPS is often misleading.

Site-wide 2,000 QPS may not be dangerous. But if 1,500 QPS hits the same product, the same user, the same cache key, or the same database row, that is a hot spot.

Common hot spots:

• Popular product detail pages.
• Influential user profile pages.
• Flash-sale inventory.
• A single configuration key.
• The same search keyword.
• A large batch of jobs from the same tenant.

Hot spots cannot be solved only by scaling out, because the pressure may concentrate on one key or one row.

Common approaches:

Hot-spot type	Handling
Hot reads	Multi-level cache, local cache, early refresh, randomized TTL
Hot writes	Sharded counters, asynchronous aggregation, queue serialization
Hot inventory	Atomic deduction, token pre-reservation, segmented inventory
Hot user	User-level rate limiting, isolated queue
Hot tenant	Tenant quota, independent worker pool

Take like counts as an example. If each like directly updates the same row:

update posts
set like_count = like_count + 1
where id = :post_id;

For a popular article, this row becomes a write hot spot. It can be changed to sharded counters:

create table post_like_counters (
  post_id uuid not null,
  shard_id integer not null,
  count bigint not null,
  primary key (post_id, shard_id)
);

Writes update a shard chosen by user or randomly:

update post_like_counters
set count = count + 1
where post_id = :post_id
  and shard_id = :shard_id;

Reads aggregate multiple shards, or a background job periodically summarizes the result into the main table.

When asking AI to design hot-spot handling, be explicit:

Do not analyze only by average QPS.
List possible hot keys, hot rows, hot users, and hot tenants.
Provide isolation strategies for hot reads and hot writes separately.

Hot spots are one of the easiest ways to punch through a high-concurrency system.

Cascading Failure Is Usually a Chain Reaction

A service avalanche rarely happens because one point suddenly breaks. More often, it is a chain reaction.

A typical chain:

Cache expires
  -> A large number of requests hit the database
  -> The database slows down
  -> Request threads pile up
  -> Connection pools are exhausted
  -> Upstream callers time out and retry
  -> Traffic doubles
  -> More services are dragged down

The truly dangerous parts are retry and queuing. When downstream is already slow, upstream retries add more traffic to the fire.

Preventing cascading failure requires several layers of protection:

• Random jitter on cache TTLs to avoid simultaneous expiry.
• Early refresh for hot keys.
• Short timeouts for downstream calls.
• Few retries, with backoff.
• Timeouts for connection-pool waits too.
• Rate limiting and circuit breakers to protect downstream.
• Degradation for non-core dependencies.
• Backpressure when queue length exceeds thresholds.

A circuit breaker means: if a downstream dependency fails repeatedly in a short time, stop calling it for a while and go directly to fallback or fast failure, giving it time to recover.

When asking AI to write retry logic, do not only say “retry on failure”:

Retries are only for temporary errors.
Retry at most 2 times, using exponential backoff and random jitter.
Total retry duration must not exceed the API timeout budget.
If downstream consecutive failure rate exceeds a threshold, open the circuit breaker and fail fast or degrade for a short time.

Retries must have a budget. Retry without a budget amplifies cascading failure.

Timeout Budget Must Be Allocated

Many APIs set only one total timeout, such as 3 seconds. But inside the API, it may call multiple downstream services, each with its own timeout. If every downstream gets 3 seconds, the total request time becomes uncontrollable.

A better approach is to allocate a timeout budget.

Suppose the target P95 is 200ms. It can be divided like this:

Stage	Budget
Authentication and parameter validation	10ms
Main database query	50ms
Inventory service	40ms
Recommendation service	30ms, degradable
Serialization and return	20ms
Reserve	50ms

If the recommendation service does not return within 30ms, degrade it. Do not wait until the full 200ms is spent.

In code, the timeout should be passed down:

async def get_detail(product_id: str, deadline: Deadline):
    product = await with_timeout(
        get_product(product_id),
        deadline.remaining(max_ms=50),
    )
    recommendations = await best_effort(
        get_recommendations(product_id),
        timeout_ms=min(30, deadline.remaining_ms()),
        fallback=[],
    )
    return build_response(product, recommendations)

This is illustrative code. The point is that every downstream call has a budget and is not allowed to wait forever.

When asking AI to write high-concurrency APIs, require:

Please provide a timeout budget.
Each downstream call must have an independent timeout.
Total time spent on all retries must not exceed the total budget.
Degradable dependencies should fallback directly after exceeding their budget.

Latency should not be considered only after measurement. It should be allocated during design.

Capacity Evaluation Needs a Load-Test Loop

High-concurrency design cannot stay on paper. It must be load-tested.

Load testing is not only about QPS. At minimum, observe:

• P50/P95/P99 latency.
• Error rate.
• CPU usage.
• Memory and GC.
• Database connection count.
• Database slow queries.
• Redis hit rate and latency.
• Queue length and consumer lag.
• Downstream call latency.
• Rate-limit, degradation, and circuit-breaker counts.

Load testing also should not only run the happy path. Test:

• Normal traffic.
• Peak traffic.
• Burst traffic.
• Hot keys.
• Slow downstream.
• Cache invalidation.
• Partial instance restarts.
• Queue backlog.

A load-test conclusion should look like this:

Target: 2000 QPS, P95 < 200ms, error rate < 0.1%
Result:
- At 1500 QPS, P95=130ms, DB CPU=55%
- At 2000 QPS, P95=210ms, DB connection waiting starts increasing
- In the hot-key scenario, Redis hit rate is 99%, local cache works
- When the recommendation service is slow at 500ms, detail API degradation rate is 18%, overall P95=170ms
Bottleneck: database connection waiting and order-query index
Action: reduce pool size from 50 to 30, add composite index for order query, degrade recommendation service after 50ms

That is an executable capacity conclusion.

When asking AI to help with a load-test plan, require:

Please provide a load-test plan, not only tool commands.
Include target, scenarios, traffic model, observed metrics, bottleneck judgment criteria, and rollback strategy.

Tools can generate traffic. The plan determines whether you can understand the result.

A High-Concurrency Prompt for AI

When asking AI to design a high-concurrency solution, start with this:

Do not write code yet.

Please design a high-concurrency plan for this service:
1. Provide target QPS, P95/P99 latency, error-rate target, and critical business paths.
2. Estimate entry concurrency, instance count, QPS per instance, total database connections, and downstream call QPS.
3. List all connection pools, thread pools, queues, and workers, and calculate global capacity.
4. Analyze which downstream dependencies are core and which can degrade.
5. Provide rate-limit dimensions: global, user, API, downstream, and job.
6. Analyze hot keys, hot rows, hot users, and hot tenants, and provide isolation strategies.
7. Design timeouts, retries, circuit breakers, and timeout budget.
8. Explain how the system is protected when cache expires, downstream is slow, queues back up, or some instances restart.
9. Provide load-test scenarios, metrics, bottleneck judgment, and capacity-conclusion format.
10. Only then provide code and configuration change suggestions.

This prompt forces AI to calculate capacity, draw boundaries, and find bottlenecks before writing implementation.

Closing Volume Two

Volume Two is about “turning features into services that can handle traffic”. API contracts, database constraints, cache and queue design, and high-concurrency design are all talking about the same thing: a system is not a single function. A system is a set of bounded promises.

Before writing code, ask yourself:

• What are the target QPS and latency?
• How many downstream calls does entry concurrency become?
• Have global connection-pool multipliers been calculated?
• Which requests must succeed, and which can degrade?
• Where are the hot keys and hot rows?
• Can retries amplify traffic?
• Who rejects traffic when the queue is full?
• Where does pressure stop when downstream is slow?
• Can load tests reproduce these scenarios?

AI can write fast local code, but high-concurrency systems do not rely on local cleverness. They rely on capacity, boundaries, and failure paths being considered in advance.

You can add machines. You cannot skip boundaries.

WeChat

Follow ZiCode on WeChat

If this post was useful, you can follow later updates on WeChat as well.

X / Twitter

Follow @ax2_zicode

Faster technical notes, short thoughts, and new-post alerts are posted on X.

X Open X Profile @ax2_zicode