Summary

On October 6, 2026, from 12:59 PM to 1:39 PM PT, the Opal web app and API were largely unavailable. Pages did not load and most API calls failed. A database change in a routine deployment conflicted with a large background sync job and blocked normal database queries. Engineers cancelled the database change at 1:39 PM PT and the service recovered immediately. No data was lost or corrupted.

Impact

• Most pages on app.opal.dev did not load. Requests waited about 60 seconds and then failed with a server error.
• API and Terraform provider calls timed out after about 60 seconds and failed with HTTP 5xx errors.
• Actions taken during the window, such as submitting or approving access requests, may not have gone through. Any action that showed an error was not saved and can be retried.

Severity: Sev 1

We classified this as Sev 1 because the web app and API were largely unavailable for all cloud customers for about 40 minutes.

Root cause analysis

A routine deployment included a small database change that needed brief exclusive access to a core table. At the same moment, a background sync job was processing a large batch and had been holding that table for an extended period. While the database change waited on the sync job, normal web and API queries to that table queued behind it until they timed out.

Actions taken

• Rolled back the deployment so new instances would not retry the database change.
• Identified the waiting database change as the cause of the blocked queries.
• Cancelled the waiting database change, which immediately let queued requests complete.
• Confirmed that error rates and latency returned to normal.

Timeline

October 6, 2026 (all times PT)

• 12:56 PM: A background sync job begins processing a large batch.
• 12:59 PM: A deployment starts and its database change begins waiting for the sync job. Web and API requests start failing.
• 1:05 PM: Monitoring alerts on elevated errors and Opal team begins investigating.