Limit function not working for bulk actions
Incident Report for Close
Postmortem

Close sincerely apologizes for the interruption of our service. We take the stability of a platform very seriously. Below is an explanation of what happened and how we will prevent another such interruption from occurring. 

Impact

Bulk actions performed in Close between 2023-08-14 05:36 UTC and 2023-08-14 20:16 UTC were potentially delayed, executed on too many entities, or not executed at all.

If a bulk action was executed and was using the “limit” search option, said option was not observed and the bulk action was performed on all search results without limit applied.

Once we uncovered the issue we cancelled the execution of queued bulk actions to prevent action being taken on unintended entities such as leads or contacts.

Root Cause and Resolution

A bug was introduced in our API that caused the “limit” parameter to be discarded when bulk actions were being performed. Upon finding the bug we deployed an updated version of our API that solved the issue.

We have increased the number of test cases covering this functionality to prevent similar issues from happening again. Improvements were also put in place regarding our escalation procedures to ensure issues like these get resolved more promptly in the future.

Timeline

  • 2023-08-14 05:36 UTC - A bug is introduced in our API that discards the “limit” parameter when creating bulk actions
  • 2023-08-14 16:13 UTC - Close Engineering becomes aware of an issue with Bulk Actions
  • 2023-08-14 19:49 UTC - Close Engineering deploys a change fixing the issue with Bulk Actions
  • 2023-08-14 20:16 UTC - Bulk actions that weren’t executed yet were paused to avoid unintended action
Posted Aug 22, 2023 - 00:26 PDT

Resolved
This incident has been resolved.
Posted Aug 14, 2023 - 13:46 PDT
Identified
The issue has been identified and a fix is being implemented.
Posted Aug 14, 2023 - 12:45 PDT
Update
We are continuing to investigate this issue.
Posted Aug 14, 2023 - 11:50 PDT
Investigating
We are currently investigating an issue whereby the limit function in our search filters is not being applied to bulk actions.
Posted Aug 14, 2023 - 11:50 PDT
This incident affected: Search (Querying).