Incident: We experienced degraded performance in the Scheduler service starting at approximately 1:32 PM.
Impact: Some jobs using the Scheduler as a trigger or a Delay as a step experienced processing delays during the affected period. Scheduling was not completely stopped, and jobs continued to be processed, although at reduced performance. Journeys that did not rely on the Scheduler were not impacted and continued to execute successfully as usual.
Root Cause: A sudden increase in delay and scheduler jobs resulted in a higher number of items being locked concurrently than the Scheduler could efficiently handle. This increased database lock contention and caused the Scheduler to process jobs more slowly.
Detection & Investigation: The issue did not initially present through the usual error signals. There were no significant service-level errors or query timeouts, and MongoDB currentOp did not show any long-running queries that would indicate an immediate database performance issue. The Scheduler continued processing jobs, although at reduced performance, so there was no obvious error condition indicating the underlying database lock contention. The issue was identified by correlating the increase in job processing delays with the number of concurrently locked items.
Resolution: We identified the underlying issue and applied a fix to control the number of concurrently locked items. Scheduler performance returned to normal at approximately 5:12 PM. Previously delayed jobs were subsequently processed successfully.
We have also implemented the necessary controls to better handle sudden increases in Scheduler workload and avoid similar performance degradation in the future.