Search Indexer Delays

Incident Report for Close

Postmortem

Close sincerely apologizes for the interruption of our service. We take the stability of our platform very seriously. Below is an explanation of what happened and how we will prevent another such interruption from occurring.

Impact

For some clients changes performed in Close took longer than usual to be reflected in our search functionality on November 4th from 8:06 to 9:31 UTC.

Root Cause and Resolution

We are in the process of moving data to a new search infrastructure. Due to encountering a number of unexpectedly large objects during this process the indexing of new changes got delayed. We adjusted the indexing process to truncate the overly large objects to avoid encountering this in the future.

Posted Nov 07, 2024 - 07:08 PST

Resolved

The issue is resolved now.

Posted Nov 04, 2024 - 01:31 PST

Monitoring

The failover succeeded. The Indexer is still catching up with the backlog of events. We are monitoring the situation and estimate that everything will be back to normal within 3-5 minutes.

Posted Nov 04, 2024 - 01:29 PST

Identified

We have identified a problem with one of our search databases and have initiated a failover.

Posted Nov 04, 2024 - 01:20 PST

Investigating

We are investigating an issue where our Search Indexer's ingestion of new events is delayed. This may result in the latest changes not being reflected on many pages across the application – Leads Search, Contacts Search, Inbox, Opportunities, Reporting, etc.

Posted Nov 04, 2024 - 00:45 PST

This incident affected: Application UI, API and Search (Indexing, Querying).