Close sincerely apologizes for the interruption of our service. We take the stability of a platform very seriously. Below is an explanation of what happened and how we will prevent another such interruption from occurring. 

Impact

Between 15:55 UTC and 16:05 UTC on August 1, 2023 Close users may have experienced degraded performance and an elevated error rate when using the Close application or API.

Root Cause and Resolution

This incident began after a routine deployment of changes to the Close backend. There was a subtle error in a query to the internal metrics service that caused our compute platform to intermittently and erroneously underestimate the true demand for resources after a deployment. 

This query to the metrics service has been corrected. Additionally Close has implemented automatic safeguards that will prevent our compute platform from deprovisioning too many resources during peak usage periods. 

Timeline

• 15:24 UTC - A routine deployment of changes to the Close application occurs. 
• 15:31 UTC - Our compute platform begins deprovisioning resources from the Close application.
• 15:55 UTC - The Close application becomes starved of resources. Customers begin to notice elevated error rates.
• 16:01 UTC - Another routine deployment of changes to the Close application occurs.
• 16:05 UTC - Our compute platform begins provisioning the correct amount of resource for the Close application. Performance recovers.