Skip to content
~/cynarski.me

Laravel schedules that randomly stop: withoutOverlapping() and its 24-hour lock

4 min read Architecture

A short note so I don't make this mistake again.

The symptom: a scheduled command that runs every few minutes just stops. No errors, no failed jobs, and schedule:run still prints it as Running ... DONE. A day later it starts working again on its own.

The cause

The culprit is ->withoutOverlapping() on the schedule definition.

withoutOverlapping() takes a cache lock before the command runs and releases it when the command finishes. Its signature (Laravel 13):

php
public function withoutOverlapping($expiresAt = 1440, $releaseOnTerminationSignals = true)

$expiresAt is in minutes, so the default lock lives for 1440 minutes, or 24 hours.

The lock is released in finish(), or on SIGTERM/SIGINT/SIGQUIT if pcntl is loaded. It is not released when the process dies in a way PHP can't catch:

  • SIGKILL: the OOM killer, or Kubernetes killing the pod after the termination grace period during a deploy or a node drain
  • a container or server that just goes away
  • a runInBackground() task killed before its trailing schedule:finish call runs (background tasks don't get the signal handlers at all)

When that happens, the lock stays in the cache (Redis in my case) until the TTL runs out. Every schedule:run in the meantime sees the lock, assumes the previous run is still going, and skips the task. A task that runs every five minutes misses 288 runs.

It's easy to miss because the skip is almost invisible. schedule:run still dispatches ScheduledTaskStarting and ScheduledTaskFinished and prints the task as done. The overlap check happens inside Event::run(), which just returns early.

If you suspected onOneServer() too, it's not the problem. Its lock key includes the minute the task was due (mutexName().Hi) and lives for an hour, so a stale one can't block the next run. It doesn't take a TTL argument at all.

I don't know about you, but I'd been adding ->withoutOverlapping() to longer-running tasks on autopilot, without thinking much about how it works or what can go wrong.

The fix

Always pass a TTL a bit longer than the longest run you expect, and shorter than the gap you're willing to lose:

php
// routes/console.php
Schedule::command('reports:sync')
    ->everyFiveMinutes()
    ->withoutOverlapping(10)   // minutes
    ->onOneServer();

Schedule::command('backups:run')
    ->dailyAt('03:00')
    ->withoutOverlapping(120)
    ->onOneServer();

If the process is killed, the task now comes back within minutes instead of a day.

That said, a daily task like backups:run probably doesn't need ->withoutOverlapping() at all. It would have to run for a full day to overlap with itself, and onOneServer() already stops two servers from running it at the same time.

A few more things that help:

  • Give the pod a terminationGracePeriodSeconds long enough for scheduled tasks to finish or exit cleanly after SIGTERM, before Kubernetes falls back to SIGKILL.
  • Make sure pcntl is installed in the scheduler image. Without it, even SIGTERM leaves the lock behind.
  • Clear stuck locks by hand with php artisan schedule:clear-cache. It's worth running as part of a deploy, after the old scheduler is gone.
  • Monitor that the task actually did something, such as a heartbeat ping at the end of the command or a "last run" timestamp, rather than relying on the scheduler's output.

A test to keep it that way

It's an easy argument to forget, so let a test catch it. This Pest test boots the app, walks every registered scheduled task, and fails if any of them uses withoutOverlapping() with the default 24-hour lock:

tests/Feature/ScheduleTest.php
<?php

use Illuminate\Console\Scheduling\Schedule;

it('gives every withoutOverlapping() task an explicit lock TTL', function () {
    $tasks = collect(app(Schedule::class)->events())
        ->filter(fn ($event) => $event->withoutOverlapping && $event->expiresAt === 1440)
        ->map(fn ($event) => $event->getSummaryForDisplay())
        ->values();

    expect($tasks)->toBeEmpty(
        'These tasks use withoutOverlapping() with the default 24-hour lock: '.$tasks->implode(', ')
    );
});

It inspects the schedule at runtime instead of grepping the source, so it also covers tasks registered in bootstrap/app.php or a service provider. It can't tell an empty withoutOverlapping() apart from an explicit withoutOverlapping(1440), but a 24-hour lock is what we want to avoid either way.