This incident has been resolved. The backlog was able to be processed at a faster rate than originally anticipated. Everything should be back to normal.
Posted Jun 13, 2023 - 18:39 EDT
Monitoring
Update from AWS: Beginning at 11:49 AM PDT, customers began experiencing errors and latencies with multiple AWS services in the US-EAST-1 Region. Our engineering teams were immediately engaged and began investigating. We quickly narrowed down the root cause to be an issue with a subsystem responsible for capacity management for AWS Lambda, which caused errors directly for customers (including through API Gateway) and indirectly through the use by other AWS services.
Many AWS services are now fully recovered. However, Amazon is still working on processing the backlog of asynchronous Lambda invocations that accumulated during the event. This may take a few hours to fully complete. During this time, we expect to see continued delays in execution until the back log has been fully processed.
Posted Jun 13, 2023 - 17:38 EDT
Investigating
Amazon Web Services are reporting an outage affecting many of their services including one the system relies on for sending communication. We are monitoring the situation and will report on updates as we receive them.