The delay in the propagation of network states consequently impacted a network load balancer crucial for the stability of AWS services. Consequently, customers utilizing AWS encountered connection issues stemming from the US-East-1 region. The AWS network functions that were affected comprised the creation and modification of Redshift clusters, Lambda invocations, and launches of Fargate tasks, including Managed Workflows for Apache Airflow, operations related to Outposts lifecycle, and the AWS Support Center.
For now, Amazon has disabled the DynamoDB DNS Planner and the DNS Enactor automation globally while addressing the race condition and implementing safeguards to avert the application of erroneous DNS plans. Furthermore, engineers are adjusting EC2 and its network load balancer.
A cautionary tale
Ookla identified an overlooked contributing factor: a high concentration of customers routing their connectivity through the US-East-1 endpoint, coupled with the inability to reroute away from the region. Ookla elaborated:
The impacted US‑EAST‑1 is the oldest and most extensively utilized hub of AWS. The concentration within this region means that even global applications frequently anchor their identity, state, or metadata flows there. When regional dependencies fail, as seen in this occurrence, the consequences ripple globally since many “global” stacks traverse Virginia at some stage.
Contemporary applications link managed services like storage, queues, and serverless functions. If DNS cannot reliably resolve a vital endpoint (for instance, the DynamoDB API involved in this situation), errors can cascade through upstream APIs, leading to visible failures in applications that users do not associate with AWS. This is exactly what Downdetector recorded across platforms like Snapchat, Roblox, Signal, Ring, HMRC, and others.
This incident highlights a critical lesson for all cloud services: beyond simply preventing race conditions and similar issues, eliminating single points of failure in network architecture is essential.
“The path ahead,” noted Ookla, “is not about achieving zero failures but rather about managing contained failures, realized through multi-region architectures, diversity of dependencies, and a disciplined approach to incident preparedness, along with regulatory oversight that progresses towards recognizing the cloud as integral components of national and economic resilience.”
