Job lifecycle logging
Log every job transition, and the queue failures Laravel handles without a word.
Skyline can log every transition a queued job makes (queued, reserved, released, migrated, retried,
timed out, completed, failed) to a log channel you choose, with each line tagged by job id so one job's whole history
is a single grep. It also logs the failures Laravel handles without saying anything: a dispatch discarded
over a held ShouldBeUnique lock, a unique lock left behind by a job that never ran, a job whose
reservation expired while it may still be running, and a job dropped by middleware as though it had completed.
Those silent cases are the reason the feature exists. A queued job passes through half a dozen states and Laravel logs almost none of them, and for several of the failures above the framework fires no event at all.
Enabling#
Point log_channel at the channel that should receive the lines:
// config/horizon.php
'log_channel' => env('HORIZON_LOG_CHANNEL'),
# .env
HORIZON_LOG_CHANNEL=queue
Leave it unset to use the application's default channel. If the configured channel name does not resolve, Skyline falls back to the default channel rather than throwing out of a queue listener — a logging misconfiguration should never take down job processing.
Log level is the volume dial#
Every event is emitted at a level chosen to match its operational severity, so you tune coverage purely by setting the level on the channel — no separate on/off switches.
error: terminal failures only.warning: the above, plus timeouts, expired reservations, exceptions caught during an attempt, and the silent drops and stranded locks listed below.info: the above, plus releases, migrations and retries.debug: everything, including a pending / reserved / completed line for every successful job.
At debug, a healthy queue writes three lines per job. On a queue doing 120,000 jobs a day that is
360,000 lines a day. Use debug to trace a problem, not as a steady state.
The events#
Lifecycle#
| Event | Level | When |
|---|---|---|
job.pending |
debug | Job was queued. Notes the delay, when delayed. |
job.reserved |
debug | A worker picked the job up and began processing. |
job.completed |
debug | Job finished successfully. |
job.migrated |
info | A delayed job became available and moved onto the ready queue. With reason=timeout, the retry of an attempt that timed out below retry_after came due. |
job.released |
info warning for exception and throttled_exception |
Job went back onto the queue, with the reason and the delay before it runs again. |
job.retried |
info | A failed job was retried, recording the new job's id. |
job.timed_out |
warning | Job exceeded its timeout, noting whether it will be retried or was marked failed. On Laravel 13 it also names the timeout and where it came from, and worker_killed says whether the worker exited or, with Worker::$killOnTimeout set to false (Laravel 13.33), the timeout was thrown into the job and the worker kept running. |
job.interrupted |
info | The worker received a stop signal while the job was running (Laravel 13.7+). The release an Interruptible job makes in response carries reason=interrupted. A timeout is not logged here, although Laravel 13.34 calls interrupted() for one too; it is logged as job.timed_out. |
job.failed |
error | Job failed terminally and will not be retried. A job picked up after its retryUntil() deadline carries reason=retry_until_expired. |
Failures Laravel doesn't report#
| Event | Level | When |
|---|---|---|
unique_lock.discarded |
warning | A dispatch was skipped because a ShouldBeUnique lock was already held. The job was never queued. |
unique_lock.released_after_failure |
warning | A ShouldBeUniqueUntilProcessing job failed before it started, and Skyline released the lock it left behind. |
unique_lock.stranded |
warning | The same failure, but the lock was not released: release is turned off, the job had no recorded owner, it could not be unserialized, or the release failed. detail says which. |
unique_lock.released_from_dashboard |
warning | Someone released a unique lock from the Locks & Limits screen. |
job.retry_until_before_available |
warning | Logged at dispatch: the job's delay alone carries it past its retryUntil() deadline, so it will fail without running. |
job.reservation_expired |
warning | A reserved job passed retry_after and was put back on the queue. Either its worker died, or it is still running and a second copy is about to start. The retry of a timeout below retry_after is logged as job.migrated instead, since the timeout was already reported. On Laravel 13.34, a job marked #[CountCrashesAsExceptions] is told that the lost attempt counts towards its maxExceptions. |
job.timeout_exceeds_retry_after |
warning | A job timed out after a timeout that is not below its connection's retry_after. Laravel 13 only, once per mismatch per process. |
supervisor.timeout_exceeds_retry_after |
warning | Logged when php artisan horizon starts, for each supervisor whose timeout is not below its connection's retry_after. The same warning is printed to the console. |
job.overlap_dropped |
warning | Job was dropped without releasing because a WithoutOverlapping lock was held and the middleware is set to dontRelease(). |
job.rate_limited_dropped |
warning | Job was dropped because its rate limiter was exhausted and RateLimited is set to dontRelease(). Laravel deletes it as though it completed, so a chain continues and a batch counts it as a success. |
job.throttled_exception_deleted |
warning | The job threw, and ThrottlesExceptions deleted it through deleteWhen(). It will not be retried. |
overlap_lock.released_from_dashboard |
warning | Someone released a WithoutOverlapping lock from the Locks & Limits screen. |
job.debounced |
info | A #[DebounceFor] job was deleted without running because a newer dispatch superseded it (Laravel 13.6+). Horizon used to record these as completed. |
job.debounce_max_wait_reached |
info | A debounced dispatch was queued to run straight away because the burst hit its maximum wait. |
queue.failed_over |
warning | A push to a Redis connection failed and Laravel's failover driver sent the job to the next connection, where Horizon may not see it. Logged by the dispatching process, once per outage rather than once per job. |
queues.paused, queues.resumed |
warning, info | queue:pause --all was switched on or off (Laravel 13.25+). |
The three middleware events and overlap_lock.released_from_dashboard need Skyline's
drop-in middleware. Laravel 13.25 added a UniqueJobSkipped event for a
skipped dispatch, but the framework still logs nothing for it. Skyline writes unique_lock.discarded from
that event where it exists and from its own cache wrapper on earlier versions.
Log lines are for reading after the fact. To be told while it is happening, turn on alerts.
Lines read like this, with the job id in the message so a plain text search finds it:
[job:8813] queued onto [default] (delayed 30s)
[job:8813] migrated to [default] and is ready to run.
[job:8813] reserved from [default] and started processing.
[job:8813] released back to [default] (reason=exception, delay=15s)
[job:8813] failed on [default] and will not be retried (RuntimeException).
[job:9120] failed on [default] without running — its retryUntil() deadline passed before a worker picked it up (reason=retry_until_expired, late_by=540s).
Structured context#
Every line also carries a structured context array, which is what you actually query once the lines are in a log
pipeline. Job events include event, job_id and job, and most include
connection and queue. Beyond that:
| Event | Additional context |
|---|---|
job.released |
reason, delay, plus what the middleware recorded: overlap_lock_key; limiter, max_attempts, decay_seconds, available_in; or exception, exception_message, exception_location |
job.timed_out |
reason, will_retry, worker_killed, and on Laravel 13 timeout, timeout_source |
job.failed |
exception, message. With reason=retry_until_expired also retry_until, retry_window_seconds, seconds_since_dispatch, late_by_seconds, attempt, never_ran |
job.retried |
retry_id |
unique_lock.discarded |
unique_id, unique_lock_key, job_hash |
unique_lock.stranded, unique_lock.released_after_failure |
reason (unique_lock_stranded), unique_lock_key, released, detail |
unique_lock.released_from_dashboard |
unique_lock_key |
job.retry_until_before_available |
reason, delay, retry_until, available_at |
job.reservation_expired |
reason, retry_after, attempt, and max_exceptions and counts_crashes for a job that counts crashes |
job.timeout_exceeds_retry_after |
timeout, retry_after |
supervisor.timeout_exceeds_retry_after |
supervisor, connection, timeout, retry_after (no job fields) |
job.overlap_dropped |
reason, overlap_lock_key |
job.rate_limited_dropped |
reason, limiter, and the limit that was hit |
job.throttled_exception_deleted |
reason, exception, exception_message |
Why a job was released#
"Released back to the queue" is the most ambiguous state in a Laravel queue. It can mean the job threw, that
middleware held it back, or that it called $job->release() itself. Skyline attributes it, so
job.released carries a reason, and the same text appears on the job's page under
Previous Attempts:
| Reason | Meaning |
|---|---|
exception | The job threw and has attempts left. |
without_overlapping | Another instance held the WithoutOverlapping lock, so this attempt was released without running. |
rate_limited | Its RateLimited limiter was exhausted, so this attempt was released without running. |
exceptions_throttled | The job had thrown too often recently, so ThrottlesExceptions released it without running. |
throttled_exception | The job threw, and ThrottlesExceptions caught the exception and released it. The exception is kept with the release. |
timeout | The job timed out and the worker survived (Worker::$killOnTimeout = false), so the job was released with its backoff. Logged at info, since job.timed_out has already warned. |
interrupted | The worker was told to stop during a deploy, and the Interruptible job released itself. The signal name is recorded with it. |
released | Anything Skyline could not attribute: a manual $job->release(), the framework's own middleware, Skip, or your own middleware. |
Every reason except exception and released needs Skyline's subclass in place of the
framework's. They are otherwise identical:
use Laravel\Horizon\Middleware\WithoutOverlapping;
use Laravel\Horizon\Middleware\RateLimited; // or RateLimitedWithRedis
use Laravel\Horizon\Middleware\ThrottlesExceptions; // or ThrottlesExceptionsWithRedis
With the framework's classes, those releases are reported as released, the drop events are never
emitted, and nothing reaches the Locks & Limits
screen.
Unique-lock discard caveat#
Discard logging works by instrumenting the lock the framework takes at dispatch, and it covers the
application's default cache store. A job that overrides uniqueVia() to lock on a
different store is not covered, and neither is a default cache repository replaced with a custom subclass.
For locks that are never released, and the paths that queue a unique job without a lock, see
Unique job locks. How ShouldBeUnique behaves
in the framework itself is covered in
ShouldBeUnique and WithoutOverlapping under
real traffic.