Degraded Performance
Incident Report for Close
Postmortem

Close sincerely apologizes for the interruption of our service. We take the stability of our platform very seriously. Below is an explanation of what happened and how we will prevent another such interruption from occurring. 

Impact

Beginning 22:00 UTC until 23:03 UTC on Tuesday February 11, 2025 users of the Close App experienced significantly degraded performance and an elevated error rate. This included slow page loads, long API response times, delayed email syncing, and delayed calendar syncing for all customers.

Root Cause and Resolution

At 22:00 UTC on Tuesday February 11, 2025 a change was deployed to the production environment of the Close system. This change introduced a poorly optimized database query. This caused our primary back end database to become overloaded and unstable. Close Engineering identified and removed the offending database query by 22:47 UTC. Performance began to gradually return to baseline, fully recovering by 23:03 UTC. 

Timeline

  • 22:00 UTC - A change is deployed to production introducing a poorly optimized database query
  • 22:01 UTC - Load on our primary back end database begins to climb
  • 22:06 UTC - Load on our primary back end database breaches critical thresholds alerting Close Engineering
  • 22:17 UTC - Close Engineering isolates the poorly optimized database query and begins to remove it from production
  • 22:47 UTC - Removal of the poorly optimized database query completes
  • 23:03 UTC - Close system performance returns to baseline
Posted Feb 14, 2025 - 11:25 PST

Resolved
This incident has been resolved.
Posted Feb 11, 2025 - 15:34 PST
Update
Our App UI and API have returned to normal performance. Our email and calendar syncing systems are operational but delayed. We are continuing to monitor for further issues.
Posted Feb 11, 2025 - 15:20 PST
Update
We are continuing to monitor for any further issues.
Posted Feb 11, 2025 - 14:58 PST
Monitoring
A fix has been implemented. We are monitoring the results.
Posted Feb 11, 2025 - 14:47 PST
Identified
The issue has been identified. A fix is being rolled out to our production system.
Posted Feb 11, 2025 - 14:40 PST
Investigating
We have become aware of degraded performance of the Close system. We are actively triaging the issue. Updates will be posted as they become available.
Posted Feb 11, 2025 - 14:31 PST
This incident affected: Application UI, API, Calendar - Syncing and Email (Syncing).