Postmortem -
Read details
Sep 18, 14:23 UTC
Resolved -
SUMMARY
On September 16, 2026, between 01:50 AM UTC and 03:50 AM UTC (September 15, 6:50 PM – 8:50 PM PST), customers in the us-east-1 region experienced intermittent 502 errors while using Workflow Builder.
Timeline
September 16, 2026, 01:50 AM UTC | September 15, 6:50 PM PST
We are aware of an issue affecting Workflow Builder in the us-east-1 region, where API requests are intermittently returning 502 errors. Our engineering team is investigating and will provide an update shortly. Thank you for your patience.
September 16, 2026, 02:15 AM UTC | September 15, 7:15 PM PST
This has been classified as an incident. Our team has identified a sudden increase in MySQL database connections in the us-east-1 region, exceeding the configured maximum connection limit, which is causing the intermittent 502 errors. We are working on a fix.
September 16, 2026, 03:00 AM UTC | September 15, 8:00 PM PST
The maximum connection limit on the MySQL database has been increased to accommodate the higher connection demand. We are monitoring database connections and Workflow Builder API performance to confirm recovery.
September 16, 2026, 03:30 AM UTC | September 15, 8:30 PM PST
The Engineering team identified the root cause and implemented a hotfix to address the issue.
September 16, 2026, 03:50 AM UTC | September 15, 8:50 PM PST
We are pleased to confirm that Workflow Builder API performance has returned to normal and intermittent 502 errors have been resolved. Our team will continue monitoring to ensure service stability.
Root Cause Analysis
An event trigger configured in a table within the Workflow Builder database encountered an edge case that caused a table lock. Subsequent queries were left in a waiting state, leading to database connection thread exhaustion. This resulted in database connection failures and intermittent 502 errors on Workflow Builder API requests.
Resolution
The Engineering team identified the root cause and implemented a hotfix to address the issue. Additionally, the maximum connection limit on the MySQL database was increased to accommodate the elevated connection demand and help prevent similar issues.
Remediation Items
Review and adjust the MySQL maximum connection limit to better accommodate peak request volume in us-east-1.
Set up proactive alerting on MySQL active connection count to notify the team before connections approach exhaustion.
Sep 16, 01:50 UTC
