Search Indexer Delays

Incident Report for Close

Postmortem

Close sincerely apologizes for the interruption of our service. We take the stability of our platform very seriously. Below is an explanation of what happened and how we will prevent another such interruption from occurring.

Impact

Changes to Lead data in Close were not reflected on search-based pages, including Leads, Inbox, and Reporting pages from 13:40 UTC to 15:05 UTC on Nov 18th 2024.

Root Cause and Resolution

Due to a bug in a data migration malformed events got sent into the indexing pipeline. After identifying the issue the indexer was modified to ignore these malformed events.

Timeline

13:40 UTC - malformed events get sent to the indexer pipeline
14:15 UTC - engineers identify the issue
15:05 UTC - rollout of the fix is complete

Posted Nov 18, 2024 - 09:26 PST

Resolved

This incident has been resolved.

Posted Nov 18, 2024 - 07:48 PST

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Nov 18, 2024 - 07:07 PST

Identified

We are investigating an issue where our Search Indexer's ingestion of new events is delayed. This may result in the latest changes not being reflected on many pages across the application – Leads Search, Contacts Search, Inbox, Opportunities, Reporting, etc.

Posted Nov 18, 2024 - 06:57 PST

This incident affected: Search (Indexing).