Custom Field Event Interruption

Close sincerely apologizes for the interruption of our service. We take the stability of our platform very seriously. Below is an explanation of what happened and how we will prevent another such interruption from occurring.

Impact

Between April 22nd 14:25 UTC and April 23rd 8:50 UTC our search functionality operated on outdated data for some users, which resulted in potentially incorrect search results. In addition, some webhooks with custom field updates were not delivered. No data was lost during the incident.

Root Cause and Resolution

A software update introduced a bug, where some events for custom field updates were malformed. As a result, the changes to custom fields weren’t reflected in search, and webhooks for these events were not sent. Close Engineering reverted the change, and reindexed all affected leads.

We are adding guardrails which will prevent us from sending such malformed events in future.

Timeline

• 22 April 14:25 UTC - A software update is deployed that sends malformed events for some Custom Field updates
• 22 April 21:20 UTC - Close Engineering is notified of leads not populating correctly in smart views
• 23 April 04:36 UTC - Root cause is identified
• 23 April 05:49 UTC - Measures applied to stop further impact
• 23 April 08:46 UTC - Completed reindexing of affected leads