Summary

A breaking change to the Opal API endpoint /groups/{group_id}/users caused disruption to automated integrations. The endpoint was updated to include pagination with a default limit of 200 results, which was not backwards compatible with existing implementations. The issue was quickly identified and resolved through a rollback.

Severity

Sev2

Impact

• Automated scripts and integrations relying on the /groups/{group_id}/users endpoint failed
• Scripts were unable to retrieve complete group membership data due to unexpected pagination
• Potential access management disruptions for customers using this endpoint
• Attempted fixes were blocked as the Opal Python SDK had not yet been updated to handle pagination

Timeline (EDT)

• 11:00 - Release 1.1047 deployed to production, introducing pagination to the /groups/{group_id}/users endpoint
• 13:31 - Issue reported by affected customer
• 13:50 - Incident response initiated by Opal
• 14:02 - Release rolled back
• 14:05 - Affected integrations re-executed successfully
• 14:12 - Full recovery confirmed

Next Steps

Opal is implementing the following corrective actions:

1. API Compatibility: Implementing safeguards to ensure all public API changes maintain backwards compatibility
2. Regression Testing: Enhancing comprehensive regression test suites for all public API endpoints to validate backwards compatibility before release