Elevated Error Rate
Incident Report for Close
Postmortem

Close sincerely apologizes for the interruption of our service. We take the stability of our platform very seriously. Below is an explanation of what happened and how we will prevent another such interruption from occurring. 

Impact

Close users experienced elevated error rates from 16:00 until 16:16 UTC on September 5, 2023. During this time requests may have presented an error or timed out.

Root Cause and Resolution

At 16:00 UTC on September 5, 2023 a version of the Close app with a major performance issue was deployed to production. This resulted in degraded performance for all customers from 16:00 until 16:16 UTC. Close Engineering deployed a new version of the the Close app that did not have the performance issue at 16:16 UTC. Application performance then returned to normal.

Close Engineering is working on a new set of pre-deployment tests that will prevent similar performance issues from being introduced in the future.

Timeline

  • 2023-09-05 16:00 - A new version of the Close app is published.
  • 2023-09-05 16:03 - Close Engineering becomes aware of degraded performance of the Close application.
  • 2023-09-05 16:05 - Close Engineering reverts to the last working version of the Close application.
  • 2023-09-05 16:16 - The working version of the Close application is deployed. Performance recovers.
Posted Sep 05, 2023 - 14:39 PDT

Resolved
This incident has been resolved.
Posted Sep 05, 2023 - 10:04 PDT
Monitoring
The fix has been deployed. The Close application is now functioning normally. Close Engineering is continuing to monitor application performance.
Posted Sep 05, 2023 - 09:17 PDT
Identified
Customers may be experiencing elevated error rates. The root cause has been identified. A fix is being deployed.
Posted Sep 05, 2023 - 09:08 PDT
This incident affected: Application UI and API.