
Python asyncio coordinates work that spends time waiting for input and output. A service can wait for several network responses without assigning a thread to every request. The useful question is how much work to allow at once, who owns each task, and what should happen when a request fails.
This guide builds a small, bounded batch client using TaskGroup, a semaphore, and a timeout. It includes a local service so you can run the example without credentials or external endpoints. The code requires Python 3.11 or later and was tested on Python 3.12.14 on Windows. The results describe those local functional tests, not a performance benchmark.
An async def function defines a coroutine. Calling it creates a coroutine object; the call alone does not complete its work. An await expression waits for an awaitable result. When an operation suspends, the event loop can run other ready tasks. A task schedules a coroutine and tracks its eventual result, exception, or cancellation.
Within one event-loop thread, Python code still executes one task at a time. A long calculation or blocking network call can hold up the other tasks. Adding async to the function declaration does not change that. Python’s asyncio development guide explains the event-loop model and debugging tools.
Two consecutive await statements can also remain sequential. In the example below, the group creates all client tasks before waiting for them to finish, allowing their network waits to overlap. Keep task ownership visible: the batch caller should know when its work has finished or failed.
| Workload | Starting point | Question to answer |
|---|---|---|
| Several independent network operations | Async-capable clients with bounded concurrency | How many connections can the destination and your process support? |
| A synchronous library that blocks on I/O | A thread executor or an async-capable replacement | What happens to the underlying operation if its caller stops waiting? |
| Independent, CPU-heavy Python calculations | A process pool, after measuring a serial version | Does the computation justify process and data-transfer overhead? |
| Jobs that must survive a process restart | Persistent job storage and a worker system | How will you acknowledge, retry, and deduplicate work? |
Our Python multiprocessing guide covers the CPU-heavy case. Async tasks are held in the running application; creating one does not provide durable job delivery. For the application boundary, see FastAPI background tasks and when to use a worker queue.
Save this as batch_client.py. Each request sends one short name to a local TCP service and reads one reply. The semaphore admits at most limit client operations into the connection section at once. The timeout surrounds the semaphore too, so waiting for a slot consumes the request’s budget.
import asyncio
import re
async def fetch_one(name, port, slots, budget=1.0):
async with asyncio.timeout(budget):
async with slots:
reader, writer = await asyncio.open_connection('127.0.0.1', port)
try:
writer.write((name + '\n').encode())
await writer.drain()
reply = await reader.readline()
if not reply.startswith(b'OK:'):
raise ValueError('The local service rejected the request')
return reply[3:].decode().strip()
finally:
writer.close()
await writer.wait_closed()
async def fetch_many(names, port, limit=2, budget=1.0):
if type(limit) is not int or limit < 1:
raise ValueError('limit must be a positive integer')
if not 1 <= len(names) <= 100:
raise ValueError('Use a batch of 1 to 100 names')
if any(not isinstance(name, str) or not re.fullmatch(r'[a-z]{1,20}', name)
for name in names):
raise ValueError('Names must be 1 to 20 lowercase ASCII letters')
slots = asyncio.Semaphore(limit)
async with asyncio.TaskGroup() as group:
tasks = [group.create_task(fetch_one(name, port, slots, budget))
for name in names]
return [task.result() for task in tasks]
The input checks make this a finite example: one to 100 names, each containing one to 20 lowercase ASCII letters. The group creates one task per name. The semaphore limits active work inside its section; it does not reduce the number of task objects already created. With the default limit of two, a larger batch can use its budget waiting for earlier requests.
The returned list follows input order because the code reads results from the original task list. The finally block closes the client stream, including when a read fails or the task is canceled. Python’s stream documentation describes the connection, draining, and close operations used here.
This limit belongs to one batch. Two simultaneous calls to fetch_many create separate semaphores. If your application needs a shared connection budget, design that ownership explicitly. A semaphore also does not enforce requests per second: short operations can pass through a slot repeatedly. See the asyncio synchronization reference.
Save the following helper as local_service.py in the same directory. It listens only on 127.0.0.1, with an operating-system-assigned port. Its 30-millisecond delay makes overlapping work easy to observe; it represents controlled waiting, not measured service latency. The names slow and fail are reserved for the failure tests.
"""A small line-based loopback fixture; not an HTTP or production server."""
import asyncio
class LocalService:
def __init__(self):
self.active = self.peak = 0
self.tasks = set()
self.writers = set()
self.slow_started = asyncio.Event()
self.release_slow = asyncio.Event()
self.accepted = []
async def handle(self, reader, writer):
task = asyncio.current_task()
self.tasks.add(task)
self.writers.add(writer)
self.active += 1
self.peak = max(self.peak, self.active)
try:
name = (await reader.readline()).decode().strip()
self.accepted.append(name)
if name == 'slow':
self.slow_started.set()
await self.release_slow.wait()
elif name == 'fail':
await self.slow_started.wait()
writer.write(b'ERROR\n')
await writer.drain()
return
else:
await asyncio.sleep(0.03) # Controlled local I/O delay.
writer.write(('OK:' + name.upper() + '\n').encode())
await writer.drain()
except (ConnectionError, OSError):
pass
finally:
writer.close()
try:
await writer.wait_closed()
except (ConnectionError, OSError):
pass
self.active -= 1
self.tasks.discard(task)
self.writers.discard(writer)
async def __aenter__(self):
self.server = await asyncio.start_server(self.handle, '127.0.0.1', 0)
self.port = self.server.sockets[0].getsockname()[1]
return self
async def __aexit__(self, *exception):
self.server.close()
# Stop only the handlers owned by this fixture.
tasks = list(self.tasks)
for task in tasks:
task.cancel()
await asyncio.gather(*tasks, return_exceptions=True)
await self.server.wait_closed()
The service owns its handler tasks and stops them during teardown. That is separate from canceling client requests. Closing a client connection does not promise that a remote system has stopped processing a job.
Save this entry point as run_example.py:
import asyncio
import json
from batch_client import fetch_many
from local_service import LocalService
async def main():
async with LocalService() as service:
values = await fetch_many(['alpha', 'bravo', 'charlie', 'delta'],
service.port, limit=2)
print(json.dumps({'results': values,
'peak_service_handlers': service.peak}))
if __name__ == '__main__':
asyncio.run(main())
Run python run_example.py from that directory. No third-party packages are needed. The recorded output was {"results": ["ALPHA", "BRAVO", "CHARLIE", "DELTA"], "peak_service_handlers": 2}. This confirms the expected values and overlap in this fixture.
The script uses asyncio.run at its entry point. In a notebook or application that already runs an event loop in the current thread, use the environment’s supported async entry point instead of nesting another runner. Python documents that restriction in its runner reference.
For this batch, a rejected request makes the whole operation fail. In our test, one task waited while another received an error reply. TaskGroup canceled the unfinished client task and raised an ExceptionGroup containing the ValueError. After the group exited, no client task remained unfinished.
We ran the same pair through asyncio.gather with its default settings. The error reached the caller, but the other task kept waiting. We retained its reference, released the local service’s gate, and awaited its SLOW result. Choose this behavior deliberately if independent operations should continue after a sibling fails. The official tasks, groups, and timeout reference describes both APIs.
The timeout tests covered two different waits: a connected request with no reply and a request unable to acquire a semaphore slot. Both expired. A direct call to fetch_one raised TimeoutError outside its timeout context; the batch surfaced the error inside an exception group. Handle the shape returned by the layer you call, rather than assuming every timeout arrives as a standalone exception.
Cancellation also needs time to finish cleanup. A separate check used a 20-millisecond timeout and an 80-millisecond cleanup wait; cleanup completed before the timeout reached the caller. Treat the configured timeout as a cancellation deadline, not a promise that all code stops at that instant. A blocked event loop cannot respond promptly either.
The client does not suppress CancelledError. When its caller canceled the batch, cancellation propagated after cleanup. If you add exception handling, preserve that behavior. Automatic retries are another design decision: use a finite attempt budget and establish whether repeating the operation is safe.
The local validation passed 42 checks on Python 3.12.14. It exercised actual loopback TCP connections with controlled delays and failures. The following outcomes summarize the recorded checks.
| Scenario | Observed result |
|---|---|
| Four requests, two slots | Four expected values; peak of two service handlers |
| Three requests, one slot | Peak of one service handler |
| One request fails inside TaskGroup | Grouped ValueError; no unfinished client task |
| Default gather with a failing sibling | Error propagated; the waiting sibling completed later |
| Connection wait and occupied semaphore | TimeoutError in both cases |
| Caller cancellation and fixture teardown | Cancellation propagated; handlers, streams and listener closed |
The tests also rejected an empty batch, a zero concurrency limit, more than 100 names, and a name containing a newline before sending requests. Debug mode produced no recorded resource or unawaited-coroutine warnings. These checks cover this small protocol and its cleanup paths. They do not establish HTTP behavior, TLS configuration, production capacity, or application security.
For a large input stream, avoid creating a task for every item up front. A fixed worker count consuming an asyncio queue with a positive maximum size can make the producer wait when the buffer is full. The producer must await insertion for that pressure to work. This in-memory queue still needs a separate persistence design if work must survive restarts.
Start capacity planning with the application’s waiting time, connection count, memory use, and error rate. Raising concurrency can increase pressure on the service you call and the connections you hold. More CPU cores will not turn one event loop into multiple parallel executors. Use workload measurements to decide whether to add application processes or isolate CPU-heavy work.
Record request duration, time spent waiting for capacity, timeout counts, and failure categories. Include correlation identifiers without logging secrets or full payloads. The Python logging guide covers how to structure those records.
If you are choosing where to operate this kind of application, compare the responsibilities in our API hosting guide, then assess the CPU options in Compute with Hivenet. The customer operates the application, dependencies, process supervision, access controls, and monitoring. The local example is a starting point for testing task behavior; public networking, TLS, deployment, and recovery need their own validation.
Pick one AI, compute, or storage workload and see the difference for yourself. Spin it up in minutes, or let our team map your fastest path to production.