A service replaces a fixed platform-thread pool with one virtual thread per request and immediately handles far more concurrent work. Throughput rises in a synthetic delay test, yet production database timeouts and tail latency get worse because the old thread pool had also been an accidental admission-control gate.
The reliable route is not the longest checklist; establish the failing boundary, keep one clean reproduction, and change one layer at a time until the evidence lets you adopt Java 21 virtual threads for blocking request flows while enforcing explicit downstream concurrency limits, deadlines, and observable overload behavior.
Read the failure at the right layer
- Virtual threads make waiting inexpensive but do not increase JDBC pool, database, CPU, or remote-service capacity.
- An unbounded request fan-out can occupy every downstream connection and leave no capacity for health checks or recovery work.
- Thread-local state, synchronized native calls, and observability assumptions can behave differently at very high concurrency.
The branches are ordered to protect the strongest evidence around this possibility: virtual threads make waiting inexpensive but do not increase JDBC pool, database, CPU, or remote-service capacity. The observed scope and logs—not a familiar-looking error screen—decide which one applies.
A Java latency incident must separate queueing, downstream saturation, allocation pressure, garbage collection, and transport time; concurrency features change where work waits; they do not create database connections, CPU, or remote capacity; in this guide, the practical goal is to adopt Java 21 virtual threads for blocking request flows while enforcing explicit downstream concurrency limits, deadlines, and observable overload behavior.
Build a clean diagnostic record
- Measure active requests, virtual-thread count, JDBC acquisition time, pool occupancy, downstream latency, and p95/p99 response time under the same arrival rate.
- Use JFR and the virtual-thread thread dump rather than interpreting a large thread count as failure by itself.
- Find each scarce downstream and calculate a concurrency budget from its tested throughput and latency, leaving recovery headroom.
- Run step-load and soak tests with realistic response bodies; verify that the load generator and client connection pools are not the limit.
The sequence moves from observation toward intervention. Preserve the result of the final check—run step-load and soak tests with realistic response bodies; verify that the load generator and client connection pools are not the limit—because it provides a useful comparison after the repair.
One practical branch through the failure
Treat “Virtual threads make waiting inexpensive but do not increase JDBC pool, database, CPU, or remote-service capacity” as a working hypothesis, not a conclusion. Establish a baseline first: measure active requests, virtual-thread count, JDBC acquisition time, pool occupancy, downstream latency, and p95/p99 response time under the same arrival rate. Record both the result you expected and the result you actually saw.
A supporting result justifies a staging test of the narrowest repair: create a virtual thread per independent request task, but guard each scarce downstream with a semaphore, bounded connection pool, or rate limiter. A result that contradicts “Virtual threads make waiting inexpensive but do not increase JDBC pool, database, CPU, or remote-service capacity” is useful too: it rules out one layer without disturbing production and gives the next operator a clean starting point.
Repair the cause—not the message
- Create a virtual thread per independent request task, but guard each scarce downstream with a semaphore, bounded connection pool, or rate limiter.
- Apply an end-to-end deadline and a shorter downstream timeout so abandoned work releases capacity before the caller’s budget expires.
- Return deliberate overload responses and metrics instead of allowing an unbounded queue to convert load into multi-second tail latency.
Before applying “Create a virtual thread per independent request task, but guard each scarce downstream with a semaphore, bounded connection pool, or rate limiter,” name its rollback point and the evidence that will count as success. Afterward, repeat the original request and specifically check whether you can compare throughput and p50/p95/p99 latency at each offered load, including the first point where demand exceeds capacity; a changed symptom at that point is new evidence, not permission to make several more changes at once.
Verification checklist
- Compare throughput and p50/p95/p99 latency at each offered load, including the first point where demand exceeds capacity.
- Fail one downstream and confirm permits, connections, and virtual threads return to baseline after timeouts.
- Inspect a virtual-thread dump and JFR recording to ensure the service remains diagnosable during peak concurrency.
One successful refresh is not closure. Keep the incident open until you can also fail one downstream and confirm permits, connections, and virtual threads return to baseline after timeouts, adjacent paths have not regressed, temporary diagnostics are gone, and another operator can explain what changed.
Prepare a useful escalation if the boundary is outside your control
Capture the JDK build, JVM flags, container limits, request rate, latency percentiles, error rate, thread or JFR evidence, downstream pool occupancy, and the smallest reproducible request; keep tokens, payload data, and internal hostnames out of shared traces; include the result of this first observation: measure active requests, virtual-thread count, JDBC acquisition time, pool occupancy, downstream latency, and p95/p99 response time under the same arrival rate.
State what was tested, including the result of “Measure active requests, virtual-thread count, JDBC acquisition time, pool occupancy, downstream latency, and p95/p99 response time under the same arrival rate,” and what changed between attempts; evidence tied to that observation is safer and more actionable than granting broad access or sending an unnecessary full database export.
Limit the scarce dependency, not the virtual threads
This Java 21 sketch gives each task a virtual thread but allows only 32 concurrent calls to one downstream. The timeout must be shorter than the caller’s remaining deadline.
private static final Semaphore DB_BUDGET = new Semaphore(32, true);
private static final ExecutorService TASKS =
Executors.newVirtualThreadPerTaskExecutor();
CompletableFuture<Result> submit(Query query) {
return CompletableFuture.supplyAsync(() -> {
try {
if (!DB_BUDGET.tryAcquire(40, TimeUnit.MILLISECONDS))
throw new RejectedExecutionException("database budget exhausted");
try { return repository.execute(query, Duration.ofMillis(180)); }
finally { DB_BUDGET.release(); }
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new CancellationException("interrupted");
}
}, TASKS);
}Interpretation and safety: Size 32 from a measured downstream budget, not from this example. Export acquisition wait and rejection metrics; close the executor during application shutdown.
Tempting moves to avoid
- Do not pool virtual threads; pool or limit the scarce resource they call.
- Do not declare success from a sleep-based benchmark that omits database and network limits.
Evidence log
Primary references
- OpenJDK JEP 444: Virtual Threads — official reference consulted for this guide.
- Java Semaphore API — official reference consulted for this guide.
Editorial note: The scenario above illustrates how to approach “Create a virtual thread per independent request task, but guard each scarce downstream with a semaphore, bounded connection pool, or rate limiter”; it is a documented example, not a claim about a reader’s server, so verify the cited documentation, take the appropriate backup, and follow the real environment’s access and change-control rules.