Close sincerely apologizes for the interruption of our service. We take the stability of our platform very seriously. Below is an explanation of what happened and how we will prevent another such interruption from occurring.

Impact

Close systems were unavailable for 30 minutes from Sep 29, 17:40 UTC to Sep 29, 18:10 UTC.

Root Cause and Resolution

During deployment of a new application version, we ran a multi-phased migration that added a permission to user roles and assigned it to existing custom roles. The timing of the phased migration wasn’t fully thought out; many application servers were still running the previous version which didn’t recognize the new permission in the database. As a result, error rates increased. We restored service by removing the permission from existing roles.

To prevent similar incidents, we’re updating our testing and deployment pipeline to identify migrations that require a phased rollout.

Timeline

• 17:40 UTC - Migration was run and PR was merged.
• 17:41 UTC - Engineering team alerted of API failures and application unresponsiveness
• 17:43 UTC - Engineering team begins incident response, status page updated
• 17:44 UTC - Potential root cause identified (backfill migration was run before code finished deploying).
• 17:44 UTC - We thought we could fall forward, but realized that was going to take longer so we then decided to revert the migration. This added to the outage time.
• 18:10 UTC - Engineering team runs a script to drop the new permission from existing custom roles
• 18:10 UTC - Application loading again, error rates dropped. Customer impact ends.