Lyniate Team

Scaling up with asynchronous I/O in Windows on the .NET Platform

October 20, 2015

Typical server workloads include lots of I/O operations as data is transferred across network connections and written to disk. Analyzing the thread profile of these workloads usually results in a chart that looks something like this (generally with one thread per CPU core):

the longer length of the red i/o segments relative to the green cpu segments is intended as a not-to-scale illustration of the fact that typical disk or network i/o operations are thousands of times slower than typical cpu and ram operations. without asynchronous i/o (aio) in most server operating systems and environments, a thread that invokes an i/o operation yields its cpu resource (usually a single cpu core) for the duration of the i/o operation and resumes cpu processing once the i/o operation has completed.

in a single-threaded environment, this means that cpu cores spend the majority of their time in an idle state since the typical duration of i/o operations is so much higher than cpu operations. multi-threading addresses that problem by allowing a single cpu core to service multiple threads in interleaved fashion. the thread profile of a multi-threaded server workload often looks much like this:

now each cpu core can pick up processing work from a new thread such as t2 once its current thread has invoked an i/o operation and yielded. of course, the new t2 thread is likely to invoke i/o operations of its own, too, so to keep cpu core utilization high, each cpu core must be able to pull from a large pool of threads with pending cpu work.

that is where the scaling problem arises because in today’s healthcare hardware environment with the proliferation of multi-core server processors, hundreds or thousands of threads could be required to maintain reasonable levels of cpu utilization. unfortunately, maintaining large numbers of threads is expensive and diverts computational resources from the user’s workload to the operating system’s internal “book-keeping” tasks for active threads.

aio has been around in various forms for a number of years in both windows and unix environments, but the competing aio implementations on unix platforms do not match the uniform and complete aio implementation that windows provides. the windows implementation of aio is one that is especially well-suited to addressing the problem of scaling up workloads with lots of i/o operations.

the key to solving the scaling problem is aio’s ability to sever the 1:1 connection between a thread and a work item. with synchronous i/o operations, any thread that initiated an i/o operation would be suspended for the duration of the i/o operation so that it could be awakened upon completion of the i/o operation to resume processing. with aio, any thread that invokes an i/o operation is immediately released as available for other work. once that i/o operation completes, the operating system acquires any thread that is available for work and schedules it to resume work with the results of the i/o operation.

as a specific example, in the illustration above, if arbitrarily-numbered thread t1 is processing work item w5 and initiates i/o operation i7, it is immediately marked as available for work and may pick up any other available work item such as w9. once i/o operation i7 completes, the operating system tasks the first available thread t3 with resuming work item w5 with the results of i/o operation i7. this approach essentially segments each work item into stages delineated by i/o operations and to allow for the processing of each stage of a work item on a different thread.

multiply the illustration by the number of cpu cores in a typical server and one begins to see that the net effect of that segmentation is a relatively small pool of threads is usually sufficient to handle the processing of a large number of work items while maintaining high cpu utilization. the scalability improvement on server workloads that are heavy on i/o with lots of cpu cores can be dramatic.

at corepoint health, we began introducing aio in corepoint integration engine in the v5.0 release and have continued to expand the range of i/o operations that are performed asynchronously through the 6.0 release. as a result of those technical changes, we have seen a steady and substantial increase in the engine’s scalability. we’ve seen existing servers handle increased workloads and the computational resources in larger servers are more efficiently utilized for message processing workloads because of our adoption of aio in the engine.

as an important aside – because of its potential for increasing scalability, aio has gotten lots of press in software engineering circles. developers who are constantly looking for ways to improve their products may wish to test the performance of aio on their particular workloads and in their particular environments. one pitfall to avoid, though, is the idea that aio improves single-threaded performance.

in general, aio operations perform comparably or perhaps slightly slower than their synchronous counterparts in single-threaded workloads. as illustrated above, the scalability improvements with aio come from reducing the number of threads that are required to process highly concurrent workloads, so make sure you do your performance testing in a concurrent environment if you want an accurate indicator of its scalability potential.

Related Blogs

Lyniate Team

Smooth sailing to the cloud with a hybrid cloud approach  

As you chart a course toward digital transformation, a hybrid approach can help you avoid rough waters. By using both cloud and on-prem capabilities you’ll have the tools you need to stay afloat and safely navigate your cloud journey.

Read more

Lyniate Team

How does an EMPI work to improve patient outcomes and increase revenue?

With an enterprise person master index (EMPI), healthcare organizations can significantly reduce duplicate patient records and correct errors in patient demographic information.

Read more

Chandan Padmanna

How migrating your healthcare integrations to the cloud modernizes your infrastructure 

To benefit from cloud infrastructure modernization, consider how to migrate your healthcare data integrations to the cloud.

Read more