Time Is a Dependency: Virtual Clocks, Monotonic Time, and Durable Timers for Reliable Systems in 2025

If your system cares about “when,” then time is one of your most important dependencies. Treat it like one.

Time is not a simple global variable you read when you need it. It skewers logs, breaks caches, misfires scheduled jobs, and ruins SLAs—often at the worst possible time. In 2012 a leap second triggered widespread outages across JVMs and Linux systems. In 2017, a date parsing bug helped push an S3 region into a major outage. In 2022–2024, multiple cloud status events traced back in part to timing, retries, and backoff interactions. These are not rare; they are systemic.

The good news: you can design your systems to be predictably correct about time. In 2025, we have excellent primitives: monotonic clocks for measuring, well-behaved wall clocks for timestamping, virtual/fake clocks for deterministic tests, and durable timers that replace best-effort cron. This article lays out the patterns that keep production sane.

The two kinds of time you must distinguish

There are two conceptually different time sources:

Wall clock (aka "real" time): The human world’s time. It moves in UTC, jumps with leap seconds or smearing, changes offset with time zones and DST. Use it to timestamp events and represent schedules that humans care about.
Monotonic clock: A strictly non-decreasing counter used to measure intervals. It never jumps backward due to NTP adjustments. Use it to measure durations, timeouts, and latency.

If you only remember one rule: measure durations with a monotonic clock; timestamp events and schedule real-world things with wall time in UTC.

What your OS and runtime actually give you

Linux provides multiple clocks via clock_gettime():
- CLOCK_MONOTONIC: Monotonic since boot; affected by frequency slewing but not by wall-time jumps. Pauses during suspend.
- CLOCK_BOOTTIME: Like monotonic but includes time spent in suspend.
- CLOCK_MONOTONIC_RAW: Monotonic without NTP slewing (raw TSC-based).
- CLOCK_REALTIME: Wall clock (UTC), can jump when NTP steps the clock.
Windows provides QueryPerformanceCounter (monotonic) and GetSystemTimeAsFileTime (wall clock).
Go’s time.Now() returns a value that carries both wall and monotonic components. Durations computed from two time.Time values use the monotonic part, insulating you from NTP steps. When serialized, the monotonic part is stripped.
Rust’s std::time::Instant is monotonic; SystemTime is wall time. Tokio’s time driver supports test-time control.
Java’s System.nanoTime() is monotonic-like; Instant.now() uses wall time and is adjustable. Since Java 8, you can inject a java.time.Clock.
.NET (7/8) provides TimeProvider and monotonic Stopwatch.

Practical advice:

Use wall clock only when you must communicate with humans or persistent schedules. Always in UTC internally. Convert to local time at the UI boundary.
Use monotonic time for all deadlines, timeouts, exponential backoff, retry budgets, and latency metrics.
If your app runs on laptops/desktops that sleep, consider whether your timeout should count sleep time. If yes, use Linux CLOCK_BOOTTIME-based APIs when available.

Leap seconds, smearing, and DST

UTC includes leap seconds; POSIX time effectively ignores them (there is no 23:59:60). When a leap second occurs, different environments handle it differently: step, smear, or stall.
Google’s leap smear gradually distributes the leap second over a day to avoid a discontinuity. Many NTP deployments now smear by default.
DST and time zones are human conventions. They do not belong inside your storage or backends. Store UTC. Only present local time to humans.

References:

IETF RFC 3339 (timestamps) and RFC 5905 (NTPv4) and BCP 223/RFC 8633 (NTP best practices)
Linux clock_gettime(2) man page
Go time monotonic docs (Go 1.9+)

Time sync is not perfect: skew and drift in the real world

Even with NTP or PTP, clocks drift and skew. In multi-node systems this matters. A few implications:

Don’t compare wall-clock timestamps across hosts to infer causality. Use logical clocks, vector clocks, or monotonic counters when ordering matters.
Tolerate some skew in certificates, tokens, and TTLs. A 1–2 minute grace window can save you.
Measure and export clock_skew_seconds in your agents by comparing to a well-synchronized server.
On Linux, prefer chrony over ntpd for virtualized, intermittently connected instances.
In the cloud, consider PTP-enabled instances if you need sub-millisecond accuracy (trading cost/complexity for precision).

Design principle: inject time as a dependency

Treat time like a database or message bus: abstract it and inject it.

Define a Clock interface with wall and monotonic capabilities, e.g. now() and nowMonotonic()/instant().
Pass Clock into components, do not call System.currentTimeMillis(), Date.now(), or time.Now() directly inside business logic.
In tests, replace the real clock with a virtual/fake clock. Advance deterministically.
Make deadlines relative to a captured monotonic base: deadline = instant + duration.

Example in Go using benbjohnson/clock (battle-tested):

go
// go get github.com/benbjohnson/clock
package main

import (
    "context"
    "fmt"
    "time"
    benclock "github.com/benbjohnson/clock"
)

type Clock interface {
    Now() time.Time          // wall
    Since(t time.Time) time.Duration // uses monotonic part
    After(d time.Duration) <-chan time.Time
}

type realClock struct { c benclock.Clock }
func (r realClock) Now() time.Time { return r.c.Now() }
func (r realClock) Since(t time.Time) time.Duration { return r.c.Since(t) }
func (r realClock) After(d time.Duration) <-chan time.Time { return r.c.After(d) }

// With injected clock we can write deterministic timeouts.
func doWithTimeout(ctx context.Context, clk Clock, work func(context.Context) error, d time.Duration) error {
    ctx, cancel := context.WithCancel(ctx)
    defer cancel()

    done := make(chan error, 1)
    go func() { done <- work(ctx) }()

    select {
    case err := <-done:
        return err
    case <-clk.After(d):
        return context.DeadlineExceeded
    case <-ctx.Done():
        return ctx.Err()
    }
}

func main() {
    clk := realClock{c: benclock.New()}
    start := clk.Now()
    _ = start // use in code
    fmt.Println("monotonic duration is safe across NTP steps:", clk.Since(start))
}

In tests:

go
func TestDoWithTimeout(t *testing.T) {
    c := benclock.NewMock()
    called := false

    errc := make(chan error, 1)
    go func() {
        errc <- doWithTimeout(context.Background(), realClock{c}, func(ctx context.Context) error {
            // simulate work that never completes
            <-ctx.Done()
            called = true
            return ctx.Err()
        }, 5*time.Second)
    }()

    // Advance virtual time deterministically.
    c.Add(5 * time.Second)
    if err := <-errc; !errors.Is(err, context.DeadlineExceeded) {
        t.Fatal("expected deadline exceeded")
    }
    if !called { t.Fatal("work should have started") }
}

Similar ideas exist in:

Rust: tokio::time::pause(), advance() for virtual time.
Java: inject java.time.Clock and use ScheduledExecutorService for production; FixedClock/OffsetClock or libraries like ClockMock for tests.
Node/TypeScript: @sinonjs/fake-timers or Jest fake timers.
.NET: TimeProvider and VirtualTimeProvider.

Virtual/Deterministic schedulers in tests

Sleeping in tests is a code smell. Races, flakiness, and long runs follow. With a virtual clock:

Freeze time at t0. Schedule tasks against the virtual scheduler.
Drive the clock forward deterministically. Observe tasks firing in a predictable order.
Make the scheduler single-threaded in test harnesses to remove nondeterministic interleavings.

Rust example with Tokio:

rust
use tokio::time::{self, Duration, Instant};

#[tokio::test]
async fn test_retry_with_backoff() {
    time::pause(); // freeze the scheduler's clock

    let start = Instant::now();
    let mut attempts = 0;

    let task = tokio::spawn(async move {
        loop {
            attempts += 1;
            if attempts == 3 { return Ok::<_, ()>(()); }
            time::sleep(Duration::from_secs(10 * attempts)).await;
        }
    });

    // advance time deterministically
    time::advance(Duration::from_secs(10)).await; // attempt 2
    time::advance(Duration::from_secs(20)).await; // attempt 3 succeeds

    task.await.unwrap().unwrap();
    assert!(Instant::now() - start >= Duration::from_secs(30));
}

A few test-time patterns:

Avoid relying on system time directly; plumb a clock through.
Decouple schedule calculation (wall time) from waiting (monotonic) so you can assert the schedule without sleeping.
For property tests, randomize jitter deterministically by seeding your RNG.

Durable timers vs. cron: you probably need the former

Cron is per-node, best-effort fire-and-forget. It does not preserve intent if your node is down at the scheduled time, and its behavior across DST/zone changes is surprising. Kubernetes CronJob improves on this but still has misfire semantics, a single controlling controller per job, and limited durability guarantees.

Durable timers treat “fire at or after T” as data, persisted and replicated. The system remembers your intent and will eventually execute it, even across node failures and restarts. Durable timers have explicit semantics for misfires, idempotence, jitter, and retries.

You need durable timers when:

Missing a fire causes data loss, contractual breaches, or escalations.
You must catch up after downtime with backfills.
You need at-least-once execution with deduplication, or exactly-once effects in the target domain.
You schedule millions of timers or recurring jobs and need horizontal scale.

Examples of durable timer platforms:

Temporal/Cadence Workflows: timers and sleeps are recorded in the workflow history and replayed deterministically.
Quartz Scheduler (Java) with a JDBC job store: persists triggers and misfire instructions.
Kafka-based delay topics and outbox patterns.
Cloud services: AWS EventBridge Scheduler, Google Cloud Tasks (with ETA), SQS with per-message delay and dead-letter queues, Redis ZSET-based delay queues (with caveats).

Core properties of a durable timer service

Persistence: each timer is a record with a unique ID, fire time, payload/target, and policy.
Claim/ack protocol: workers atomically claim due timers to avoid duplicate processing. Use fencing tokens and idempotency keys.
Misfire/backfill semantics: explicit policy for late timers (execute immediately, skip, cap catch-up, or re-phase next occurrence).
Observability: measure schedule-to-fire latency (lag), execution duration, success rate, and dropped/skipped counts.
Sharding and fairness: balance across workers and tenants; add jitter to avoid herds.
Deadline enforcement: timers should not starve indefinitely; stuck claims must expire.

A minimal durable timer schema and worker loop

Schema (relational, simplified):

sql
CREATE TABLE timers (
  id BIGSERIAL PRIMARY KEY,
  key TEXT UNIQUE NOT NULL,          -- idempotency key
  due_at TIMESTAMPTZ NOT NULL,       -- in UTC
  payload JSONB NOT NULL,
  recurrence TEXT NULL,              -- cron or RRULE; null if one-shot
  next_due_at TIMESTAMPTZ NULL,      -- for recurring schedules
  status TEXT NOT NULL DEFAULT 'pending',
  claimed_by TEXT NULL,
  claimed_until TIMESTAMPTZ NULL,
  attempts INT NOT NULL DEFAULT 0,
  last_error TEXT NULL,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX timers_due_idx ON timers (status, due_at);

Worker loop (sketch in Go):

go
// Pseudocode: scan due timers, claim, execute, ack.
for {
    now := clk.Now().UTC()
    tx := db.Begin()
    // Select a batch of due timers using SKIP LOCKED to avoid contention.
    rows := tx.Query(`
      SELECT id, key, payload, due_at
        FROM timers
       WHERE status = 'pending' AND due_at <= $1
       ORDER BY due_at
       FOR UPDATE SKIP LOCKED
       LIMIT 100
    `, now)

    for rows.Next() {
        t := scan(rows)
        // Claim with an expiry (lease) to handle worker crashes.
        tx.Exec(`UPDATE timers SET status = 'claimed', claimed_by = $1, claimed_until = $2 WHERE id = $3`,
            workerID, now.Add(2*time.Minute), t.ID)
    }
    tx.Commit()

    // Process in parallel with idempotent target operations.
    // On success: mark complete or schedule next occurrence.
    // On failure: increment attempts, apply retry with backoff (bounded), and reschedule.

    // Sleep with monotonic clock until next due or a floor (e.g., 200ms), whichever is smaller.
    time.Sleep(nextPollInterval)
}

Key points:

Use UTC for all wall-clock fields. Do not store local time.
Use a unique idempotency key per logical timer to deduplicate accidental duplicates across retries or re-schedules.
Use a lease (claimed_until) so that a crashed worker does not stall the timer forever.
For recurring schedules, store both the definition (cron/rrule) and the next materialized occurrence. On fire, compute the next occurrence deterministically.

Cron expressions, time zones, and misfire policy

If a schedule is “every weekday at 08:00 New York time,” encode:

time zone: IANA name America/New_York (never abbreviations).
recurrence: a cron or RRULE.
materialization: compute the next fire time using a timezone-aware library (e.g., rrule libraries, Quartz cron parser), store it in UTC as next_due_at.

Misfire policies you should make explicit:

Fire immediately on startup if missed (catch-up) vs. skip if more than X behind.
Collapse multiple missed occurrences into one, or replay each one.
Apply a per-tenant backfill cap to avoid thundering herds after outages.

Kubernetes CronJobs offer levers like startingDeadlineSeconds, concurrencyPolicy, and timeZone. They are useful, but for strong guarantees or cross-cluster durability, implement or adopt a durable timer service.

Data structures and scale

Binary min-heaps are great in-memory for small queues; pair with persistence for durability.
Hierarchical timing wheels are efficient for millions of timers with coarse granularity.
Redis ZSETs can implement delay queues: score = epoch seconds. Poll with ZRANGEBYSCORE and ZREM. Beware of failover and losing claims.
Kafka delay topic: write to topic A, a router consumes and re-produces to B at due time, or use a header with due time and park in a priority queue; or use Kafka’s standard "+delay" pattern (not built in to Kafka itself).
Cloud-managed schedulers scale well but consider vendor lock-in and latency SLOs.

Precision: if you need <10ms jitter at 10M timers, you are building infra. Most business systems do fine with 100–500ms scheduling precision.

Timeouts, deadlines, and latency: always monotonic

When you start an operation, capture start = monotonic.now(); compute durations from monotonic.now() - start.
Deadlines should be stored as “monotonic instant” in memory when possible, not as wall time, to avoid NTP steps affecting timeouts.
Libraries: gRPC deadlines are relative durations; languages like Go use monotonic time under the hood to compute them safely.

If you must persist a deadline, store wall time UTC, but on read convert to a relative duration against “now” and use the monotonic clock once in memory.

TTLs and leases: correctness patterns

TTLs and leases are ubiquitous—caches, locks, ephemeral registrations. A few rules:

TTLs should be enforced by the system that stores the object, not by clients. Redis, etcd, ZooKeeper, Consul all support server-side TTL/lease expiration.
Leases expire; locks don’t. Favor leases + fencing tokens over “distributed locks.” A lease gives you a version number that monotonically increases on each acquisition; include that token in all writes so stale holders cannot clobber.
Use keepalives/heartbeats to renew leases. If the process pauses (GC stop-the-world, container throttling), the lease can expire—design for it.

Examples:

etcd: Create a lease, attach keys to it, and renew via KeepAlive. Consumers use revisions as fencing tokens.
ZooKeeper: Ephemeral nodes vanish when the session dies; use versions as fencing tokens.
Redis: SET key value NX EX ttl is a basic mutual exclusion primitive; beware cross-node failover semantics. “Redlock” remains controversial; use it only if you understand the failure model and can tolerate it.

Clock skew implications:

Server-enforced TTLs depend on the server’s clock, not the client’s. Good. But your clients should allow grace for skew.
For security tokens (JWT nbf/exp), allow small clock skew tolerance (e.g., 60s) when validating.

Cutovers and backfills without waking up on-call

You will eventually migrate schedulers, change time libraries, or move to a new region. Plan for cutovers with time-aware safety rails.

Dual-write, single-read: For a period, register timers in both the old and new systems. Serve executions from one to avoid duplication.
Dry-run mode: The new system computes the same next fire times as the old one and logs divergences.
Consistency checks: Sample 1% of schedules and assert next N occurrences match across systems for various time zones and DST transitions.
Backfill: Define a contract for missed jobs. For recurring jobs, decide whether to replay every missed instance, summarize, or re-phase.
Idempotency everywhere: The target of a timer (your handler) must tolerate duplicates. Use idempotency keys tied to the logical occurrence ID.

An idempotent handler contract:

The timer payload includes occurrence_id and timer_key.
The handler writes results to a table keyed by occurrence_id with upsert semantics. If retried, it becomes a no-op.
External side-effects (emails, payments) also use idempotency keys where available (e.g., Stripe’s idempotency-key header).

Observability: you can’t fix what you can’t see

Metrics you should export:

timer_schedule_lag_seconds: now - due_at when the timer starts executing (distribution/quantiles).
timer_execution_duration_seconds and success/failure counters.
clock_skew_seconds: difference between local wall clock and a reference.
ntp_sync_state: is the host in sync, stepped recently, leap second pending?
deadline_budget_seconds: remaining time at the start of a request (propagate budgets across RPCs).

Logs and tracing:

Include both wall time (UTC ISO 8601) and elapsed durations from a monotonic source for accurate performance analysis.
In distributed tracing, prefer span durations measured by the local monotonic clock. For inter-node ordering, rely on trace context, not wall time.

Dashboards:

Alert on sustained schedule lag beyond SLOs.
Visualize timer throughput, misfires, and catch-up backlog during/after outages.

Production patterns that keep systems sane

Always UTC in storage, APIs, and logs. Convert to local time at presentation boundaries only.
Use monotonic time for durations, backoff, and latencies. If you can’t access a monotonic time source in some language/platform, introduce an abstraction and centralize the unsafe bits.
Add jitter to periodic tasks and retries to avoid synchronized thundering herds. Use bounded jitter: base * uniform(0.5, 1.5) or decorrelated jitter.
Timeouts should be derived from budgets, not arbitrary constants. Propagate budgets across services (e.g., x-request-timeout headers, gRPC deadlines).
Don’t sleep in tests. Use virtual clocks and deterministic schedulers.
Make timer targets idempotent; treat timers as at-least-once.
Favor server-enforced TTLs and leases with fencing tokens over client-side timers.
Validate cron/rrule schedules across DST boundaries with golden tests for multiple time zones.
In containerized environments, understand CPU throttling: sleep and timers can be delayed when the process is not scheduled. Build slack into SLOs.
For laptops/mobile, decide whether timeouts should include sleep; use appropriate clock (BOOTTIME on Linux if you need wall-clock elapsed including suspend).

Concrete examples across ecosystems

Java: inject `Clock` and persist schedules in UTC

java
import java.time.*;
import java.util.concurrent.*;

public class ReminderService {
  private final Clock clock; // Injected
  private final ScheduledExecutorService ses;

  public ReminderService(Clock clock, ScheduledExecutorService ses) {
    this.clock = clock;
    this.ses = ses;
  }

  public ScheduledFuture<?> schedule(Duration delay, Runnable task) {
    // delay is relative; measure with a monotonic-like scheduler (SES uses system clock but JVM tracks delays)
    return ses.schedule(task, delay.toMillis(), TimeUnit.MILLISECONDS);
  }

  public Instant now() { return Instant.now(clock); }
}

// In prod: Clock.systemUTC(); in tests: Clock.fixed(Instant.parse("2025-01-01T00:00:00Z"), ZoneOffset.UTC)

For durable scheduling, use Quartz with a JDBC job store and set misfire instructions (MISFIRE_INSTRUCTION_FIRE_NOW, etc.). Always specify the time zone per trigger.

Node/TypeScript: fake timers

ts
import FakeTimers from '@sinonjs/fake-timers';

const clock = FakeTimers.install();
let count = 0;

setTimeout(() => { count++; }, 5000);
clock.tick(5000); // deterministically fire
console.assert(count === 1);

clock.uninstall();

Go: durable timers with Redis ZSET (sketch)

go
// Use ZADD key score member (score = epoch seconds). Poll due members reliably.
func pollDue(ctx context.Context, rdb *redis.Client, key string, batch int, now int64) ([]string, error) {
    // atomic claim: use LUA script to ZRANGEBYSCORE + ZREM within one roundtrip
    script := redis.NewScript(`
    local key = KEYS[1]
    local now = ARGV[1]
    local batch = tonumber(ARGV[2])
    local members = redis.call('ZRANGEBYSCORE', key, '-inf', now, 'LIMIT', 0, batch)
    for i, m in ipairs(members) do
      redis.call('ZREM', key, m)
    end
    return members
    `)
    res, err := script.Run(ctx, rdb, []string{key}, now, batch).Result()
    if err != nil { return nil, err }
    arr, _ := res.([]interface{})
    out := make([]string, 0, len(arr))
    for _, v := range arr { out = append(out, v.(string)) }
    return out, nil
}

Caveat: Redis failover can cause duplicates; ensure idempotent consumers. For stronger guarantees, use a DB with transactional claims or a workflow engine.

Determinism and replay: simulation pays off

For complex schedulers, invest in a simulation harness:

Model timer creation, advancement of a virtual clock, and worker crashes.
Run random workloads and fault injection (pause/resume, NTP step events, GC pauses) while asserting invariants: no lost timers, no duplicate side effects, bounded schedule lag, fairness across tenants.
Use property-based testing to explore edge cases around DST transitions and month-end recurrences.

A checklist you can paste into your design doc

Opinionated conclusions for 2025

Pretend time is unreliable and inconsistent—because it is. Design with that assumption and you’ll sleep better.
Inject a clock interface everywhere. It’s the cheapest high-leverage refactor you can make.
Cron is fine for best-effort background chores. For anything that matters, use durable timers with explicit misfire semantics.
Leases with fencing tokens beat locks. Do not roll your own distributed locks without a crystal-clear failure model.
Simulate time in tests. You’ll catch edge cases that only happen twice a year in production (DST transitions) and sleep-time anomalies.
Measure schedule lag as a first-class SLI. It’s your early warning for overload, skew, and bugs.

If you follow these patterns—monotonic for measurement, UTC wall for intent, durable timers for execution, and virtual time for tests—you’ll stop being surprised by time and start depending on it with confidence.

Selected references and further reading

RFC 3339: Date and Time on the Internet: Timestamps
RFC 5905: Network Time Protocol Version 4 (NTPv4)
RFC 8633 (BCP 223): Network Time Protocol Best Current Practices
Linux clock_gettime(2) man page
Go blog: Monotonic time in Go 1.9
Quartz Scheduler documentation (misfire instructions, time zones)
Kubernetes CronJob reference (timeZone, concurrencyPolicy, startingDeadlineSeconds)
Temporal.io documentation (durable timers and workflow replay)
Martin Fowler: Monotonic Clocks
AWS Builders’ Library: Timeouts, retries, and backoff