Summary

From 3:01pm PDT on 05/27/25 to 6:35pm PDT on 05/29/25, Opal failed to sync updates for active access rules updates.

Impact

Opal cloud customers, as well as customers on self-hosted versions 1.984.0 to 1.986.0, did not receive access rule membership updates for any active access rule. After the hotfix to address this issue was deployed, all access rules were then brought back into their expected sync state.

Severity: Sev 2

This incident was classified by a SEV 2 in accordance with our SEV guidelines. The regression with access rules represents a key feature within the platform having a blocking failure.

Root cause analysis

The root cause was a code change introduced in version 1.984.0. In our system, certain parts of the sync process are intentionally skipped for the Opal app itself, since it's not considered an external integration.

As part of a change to improve performance — specifically, by breaking up a single sync task into individual tasks per app — some logic that processes access rules was inadvertently moved inside that conditional block.

As a result, access rule syncs for the Opal app were incorrectly skipped, causing them not to have memberships updates processed.

Actions taken

• Fixed the conditional logic to allow access membership refreshes to be processed
• Added new monitoring to alert Opal on the count of access rules updated.

Timeline

05/27/25

• 3:01 PM PDT - Version 1.984.0 was deployed with the bug causing access rules sync to be skipped.

05/29/25

• 5:12 PM PDT - Opal became aware of the access rules sync not running correctly and began investigating. Status page was updated to track the incident status.
• 5:15 PM PDT - The root cause code change was identified.
• 5:37 PM PDT - A fix was implemented and put up for review.