Linux Clocks, Hostnames, and Environment

Stage 2 : Linux and Operating System Internals
Subject area 2.1 : The Operating System Model
Article 6

The short version

Linux gives you three ways to describe a running system: time, name, and settings. Mixing them up causes quiet bugs. A clock tells you the time or how long a task took. A hostname is a machine’s local name. An environment variable tells a new process how to start.

Wall-clock time is the calendar time shown in logs. NTP (a service that syncs your clock over the network) may move it forward or backward, and an administrator can change it too. Monotonic time is different. It only moves forward. You use it to measure how long an operation took. A hostname is a machine’s local name. It is handy for prompts and logs, but it does not prove who the machine is. An environment variable is a string the parent passes to a child when it runs exec. The child gets a copy. If the parent changes its value later, the running child still sees the old one.

These problems start with wrong assumptions. A timeout built on wall-clock time may fire early or late when the clock jumps. A program that trusts a hostname to prove which service it reached can be fooled when a container restarts with a new name. A program that reads a secret from the environment without checking may miss it and use a risky default.

Where this article fits

The last article showed how /proc, /sys, and /dev show live kernel state as if they were files. This article covers the configuration side of that state. How does the system know the time, its own name, and what a new process should do?

You need this before CPU scheduling, because a timer going off is just a monotonic deadline the scheduler acts on. You will use these ideas later for network timeouts, SLO measurement with CLOCK_MONOTONIC, and to see why TLS checks a name in a certificate instead of a hostname string.

Time appears simple until you measure

A backend usually needs three kinds of time. One is for people, like the stamp on a log line. One is for measuring, like how long a request took. One is for deciding, like giving up after 500 milliseconds. Linux keeps these three apart for good reason.

Picture a request that must decide whether to give up. You can think of it as a choice between two clocks.

The diagram shows most of what you need. If you show a time to a person, use the wall clock. If you measure how long something took, use the monotonic clock. Picking the wrong one causes bugs that only show up when time is corrected.

Wall-clock time

Wall-clock time, called CLOCK_REALTIME on Linux, is the calendar time. It is the time you see with date or timedatectl. Because it should match the real world, it can be moved forward or backward. NTP (network time sync), a manual date -s, or a timezone change can all move it.

Use it for anything tied to a point in time. Log stamps, certificate validity, and dates users see should use it. Do not use it to measure a duration. The reason is simple. If you take two wall-clock readings and subtract them, the answer can be wrong if the clock moved between them.

The code below looks fine, but it has exactly that problem.

struct timespec a, b;
clock_gettime(CLOCK_REALTIME, &a);
do_work();
clock_gettime(CLOCK_REALTIME, &b);
long duration = b.tv_sec - a.tv_sec;

If NTP moves the clock back two seconds while do_work runs, the duration you get can be negative. If you used this for a 500 ms timeout, the timeout might take 2.5 seconds, or never fire. This is not just theory. It happens in production when a VM’s clock is fixed after a pause.

Monotonic time

A monotonic clock, CLOCK_MONOTONIC on Linux, is built for measuring. It always moves forward. On Linux, NTP may change its tick speed to stay close to real time, but it never makes it jump.

Use it whenever you care about how much time has passed. Request timeouts, deadlines, retry budgets, and latency histograms all belong here. In Go, time.Now and time.Since already use this clock internally. In C, you ask for it directly.

struct timespec start, now;
clock_gettime(CLOCK_MONOTONIC, &start);
// deadline is start + 500ms

Picking the wrong clock is a common slip, because both clocks look alike in a small test. The test runs when the wall clock is steady, so subtracting it works there. The failure only shows later, when time is corrected.

Timers

A timer asks the kernel to wake a thread after some time. It might be a timerfd, an epoll timeout, or a sleep in a language runtime. Two things can still go wrong.

First, when a timer expires, the event is ready. That does not mean your thread runs right away. The scheduler must still give it a CPU, and under load that can take extra time.

Second, a timer built on the wrong clock fires at the wrong moment. A timerfd made with CLOCK_REALTIME is hit by a clock step, while one made with CLOCK_MONOTONIC is not. The right way to set a deadline is to work out the remaining time from a monotonic clock.

deadline = monotonic_now() + 200ms
while work not done:
    remaining = deadline - monotonic_now()
    wait_for_event(remaining)

If you work out that remaining time with the wall clock, a step can make remaining suddenly large or negative. A caller that retries on timeout can then cause a storm.

Hostnames and identity

A hostname is the name a machine gives itself. You can see it with hostname, in /proc/sys/kernel/hostname, or in a container with its own network namespace. It helps with log lines, shell prompts, and as a hint for service discovery.

It is not proof of identity. A hostname can change after a reboot or when a container is moved. It may not be unique, since two clusters can both have a machine called web-1. It can resolve to different addresses depending on whether you check DNS or /etc/hosts. Inside a container, the hostname differs from the host’s hostname, because they live in different network namespaces.

Backends therefore keep several separate ideas of identity. The kernel hostname is for local display. A full DNS name is for routing. A machine ID or cloud instance ID is for inventory or autoscaling. A container hostname is local to that pod only. For authentication, you should check the service identity carried in a TLS certificate. If you log that you talked to host=web-3, that only proves which log line was written, not which service you checked. Use service discovery with certificate checks, not string matching on hostnames.

Environment variables and process configuration

An environment variable is a key and a value that a parent passes to a child when it calls exec. The child gets the values that exist at that moment. Changing the parent’s environment later does not change a child already running. You must restart the child to see the new value.

This inheritance is handy, but it has limits that cause real outages. An environment variable is just a string, so there is no type check. One program may read TIMEOUT=500 as milliseconds, while another reads it as 0.5s. If PORT is missing, code that silently defaults to 8080 in production but 3000 on a laptop can start on the wrong port. Because the environment is inherited, changing a unit file’s Environment= in systemd does nothing until you run daemon-reload and restart the service. The environment also shows in /proc/<pid>/environ and can land in crash dumps, so a raw DATABASE_URL there can leak a secret through ps e or an error reporter. For sensitive values, a file mount or secret manager is safer.

Good configuration code checks values at startup. It fails fast if a required value is missing, states the allowed range and default, and notes whether an old name is still supported.

Treat writing to /proc/sys/kernel/hostname or running timedatectl set-ntp as calling a kernel API with a fixed type. It changes live state at once, not like editing a file read later. A diagnostic command that writes there by mistake can affect the running machine.

A small code example

Go already does the right thing in its standard library. time.Since(start) subtracts monotonic time, while time.Unix is wall-clock time. In C you choose by hand. The function below waits for an event but uses the correct clock for the deadline and keeps wall-clock only for logging.

#include <time.h>

int wait_with_deadline_ms(long timeout_ms) {
    struct timespec start, now, deadline, wall;

    clock_gettime(CLOCK_MONOTONIC, &start);
    deadline = start;
    deadline.tv_sec  += timeout_ms / 1000;
    deadline.tv_nsec += (timeout_ms % 1000) * 1000000;
    if (deadline.tv_nsec >= 1000000000) {
        deadline.tv_sec++;
        deadline.tv_nsec -= 1000000000;
    }

    clock_gettime(CLOCK_REALTIME, &wall);
    // wall is only for a log line like "started at 2026-08-23T12:00:00Z"

    do {
        clock_gettime(CLOCK_MONOTONIC, &now);
        long remaining_ms = (deadline.tv_sec - now.tv_sec) * 1000
                          + (deadline.tv_nsec - now.tv_nsec) / 1000000;
        if (remaining_ms <= 0) return -1;
        // poll with remaining_ms
    } while (1);
}

The key line is the first clock_gettime with CLOCK_MONOTONIC. That clock drives the deadline. The wall-clock read is for display only. If you run this and step the wall clock in a VM, the timeout still fires after about 500 ms. If you swap CLOCK_MONOTONIC for CLOCK_REALTIME, the same step makes it fire early or late. The code does not handle scheduler delays, so in a real service you would add a small jitter and measure the actual wakeup time.

A realistic production example

A small Go service used time.Now().Add(200*time.Millisecond) for a downstream call. That part was correct, because Go’s time.Now includes monotonic time. After a deploy, a teammate added a new database host through an environment file but forgot to run systemctl daemon-reload. The old process kept the old DB_HOST value, which you could still see in /proc/<pid>/environ, so requests went to the wrong shard. The hostname in that setup, db-primary, also resolved to an old address because of a stale entry in /etc/hosts. At the same time, a dashboard worked out p99 latency by subtracting wall-clock timestamps. When NTP stepped the clock, the graph showed a two-second spike that never happened, and the alert fired.

The team fixed it in a few places. They made the program check for DB_HOST at startup and exit with a clear error if it was missing. They stopped using the hostname string for routing and used service discovery with a TLS certificate for identity. They kept time.Since for durations and used wall-clock only for log stamps. They changed the alert to use the monotonic-based histogram. Each change follows the earlier rule. For display, use wall-clock. For how long, use monotonic. For identity, check the service, not the name. For configuration, validate at start.

How experienced engineers handle this

They ask different questions for each interface. For time, they ask whether the value is for a person or for measuring, and they pick the clock before writing code. In tests they inject a fake clock so they can fake a step without touching the real machine. For identity, they ask which name the peer will actually check: a local hostname, a DNS entry, or a name inside a certificate, and they write down which one is the source of truth. For environment variables, they ask who sets the value, who inherits it, whether it is a secret, and what happens if it is missing. They check required keys at boot and restart the process to pick up changes.

You can see which path a program uses with ordinary tools. strace -e clock_gettime shows which clock is asked for. date and timedatectl status show wall-clock versus NTP discipline. hostname and hostname -f show the local name versus the DNS name, while cat /etc/machine-id shows a stabler identifier. tr '\0' '\n' < /proc/<pid>/environ shows exactly what a running service saw at start.

CLOCK_MONOTONIC_RAW and the older trap of wall-clock leases

NTP keeps CLOCK_MONOTONIC close to real time by speeding up or slowing down its tick, so the two agree without ever jumping. CLOCK_MONOTONIC_RAW is not adjusted by NTP at all. It reports the raw rate of the hardware oscillator, so it can slowly drift from true time, but it tells you exactly how fast the clock source runs. Use RAW when you want to measure the oscillator itself, for example to spot a crystal that runs fast, or to calibrate one clock against another.

The lease bug is the classic way this topic bites a backend. A distributed lock with a 30-second time to live is often worked out from wall-clock time on the holder. If the wall clock steps backward because an administrator ran date -s or a VM resumed from a pause, the holder thinks its lease is still valid long after it should have expired, while a second client takes the same lock thinking the first released it. The result is two writers at once. Work out lease expiry from monotonic time, and when you pass timestamps between hosts, use a clock both sides trust rather than local wall-clock time.

Leap seconds, TAI, and the minute that holds sixty-one seconds

Earth’s rotation is not perfectly even, so standards sometimes add a leap second to keep civil time aligned with the sun. On that day the final minute of UTC has sixty-one seconds, and you may see the stamp 23:59:60. POSIX systems handle this with a step or a smear, and a step can move the clock back a second at midnight, which again breaks any code that subtracts two wall-clock readings.

TAI is International Atomic Time, a steady count of seconds with no leap seconds ever inserted. UTC is TAI plus the accumulated leap seconds, so the two differ by a whole number of seconds that grows over the years. If you store an interval as “24 hours from this UTC stamp” and a leap second falls inside it, the real elapsed time is off by a second. For measuring intervals, use monotonic time. For scheduling across a leap boundary, prefer TAI or a time library that knows the leap-second table, rather than plain wall-clock arithmetic.

How NTP and chronyd correct drift, and the panic threshold

NTP and chronyd usually avoid jumping the clock. When the local clock is only slightly off, they slew it, meaning they nudge CLOCK_REALTIME faster or slower by a few milliseconds per second until it matches the server. Slewing keeps the order of events sensible and keeps monotonic behavior. When the error is large, they step the clock instead, which can move it forward or backward in one move.

For safety, chronyd applies a panic threshold, often 1000 seconds. If the observed offset is larger than the threshold, it refuses to correct the clock and logs an error instead of making a giant jump, because a gap that large usually means a broken hardware clock or a wrong timezone, not ordinary drift. Tune maxdistance and the threshold to your setup. In a virtual machine that was paused for a long time, expect a step on resume, and design your timeouts so a single step does not trigger a retry storm.

PTP and the need for sub-microsecond agreement between machines

NTP is enough for most backends, giving agreement in the millisecond to sub-millisecond range. When two machines must agree within hundreds of nanoseconds, for example to stamp market ticks or to order writes in a distributed store, you use PTP, the Precision Time Protocol defined in IEEE 1588. PTP uses hardware timestamping on the network interface to measure the exact delay on each link between a grandmaster clock and a slave, then steers the slave’s clock to that reference.

The accuracy depends on the hardware. With switches and cards that support hardware timestamping, PTP reaches sub-microsecond alignment. Without that support it falls back toward NTP-class results. The payoff is that two hosts can stamp the same external event within a fraction of a microsecond, so you can rebuild a global order of events without sending every write through one central sequencer.

Reading the environment safely and the hostname of a single container

Reading a variable with getenv is not free, and in many runtimes it is not safe to call at the same time as setenv. getenv walks the process-wide environment, so calling it on every request adds cost, and setenv can reallocate that environment, which makes concurrent reads unsafe. This is why a server reads its configuration once at startup into its own typed structures and never calls getenv in the request path.

Changing the hostname is scoped by the UTS namespace. A container can call sethostname and see its own name while the host keeps a different one, because only that namespace is changed. That is why a pod reports host=web-3 while the node shows something else, and it is another reason the hostname is a local convention rather than a fact you can trust across machines. Check configuration at startup, store it in your own structures, and treat the hostname as a hint that belongs to one namespace only.

Interview definitions

What is wall-clock time?

Wall-clock time is the calendar time that is adjusted to match the real world. It is the time you show in logs and use for certificate checks, but subtracting two wall-clock readings can give the wrong duration if the clock steps.

What is monotonic time?

Monotonic time always moves forward and is meant for measuring how long something took. NTP may adjust how fast it ticks, but it will not make it jump, so timeouts and SLO measurements should use it.

What is a hostname?

A hostname is the local name a machine calls itself. It is useful for logs and prompts, but it can change, it may not be unique, and it can resolve differently in DNS and /etc/hosts. It does not prove which service you talked to.

What is an environment variable?

An environment variable is a string passed from a parent to a child at exec. The child inherits it once, and a later change in the parent does not affect a running child. Because it is just a string and often visible in /proc, you should validate it at startup and avoid putting high-value secrets there.

Interview follow-up questions

Why is monotonic time better for timeouts?

Wall-clock time can jump because of NTP or a manual change. If you compute how much time is left with wall-clock time, a step can make the remaining time suddenly long or negative. Monotonic time does not jump, so the remaining budget stays correct.

Why can a hostname not be used as a secure identity?

The name can change when a machine or container restarts, two clusters can use the same name, and DNS and /etc/hosts can give different answers. A TLS certificate or a cloud instance identity is the right thing to verify.

Why does changing an environment variable not update a running service?

The variable is copied once when the child is created. A running process keeps the old copy. You need to restart or reload the service to make it see the new value.

Common misconceptions

“Changing an environment variable updates a running service.”

It does not. The value is inherited at start. An already running process keeps what it got. You need to restart the service.

“A hostname proves which service I talked to.”

It is only a hint. A local name can collide or be changed. Prove identity with a certificate or with the platform’s instance identity, not with a string comparison on the hostname.

“Wall-clock time is fine for timeouts.”

It can be stepped. If you subtract two wall-clock times, the result can be negative or very large. Use monotonic time for deadlines and keep wall-clock time for display.

Summary

Time, hostname, and environment look small, but they sit on every request. Use wall-clock time for when something happened and monotonic time for how long it took. Treat a hostname as a hint for humans, not as proof for authentication. Treat environment variables as input that is copied once, so validate it early, document defaults, and restart to pick up changes.

The habit to keep is to ask where each value comes from, whether it can change underneath you, and who the correct consumer is. Is the value for a person, for measuring, or for identity?

If you want to build this later

Extend the inspection tool from the previous article. Add a mode that prints both clocks every half second and a mode that simulates a deadline with each clock. Add another mode that reads /proc/<pid>/environ and prints the keys, redacting anything that looks like a secret, and that prints hostname, hostname -f, and machine-id side by side.

Run it in a VM where you can step the wall clock. The monotonic deadline should stay close to correct while the wall-clock deadline moves. Restart a child after changing the parent’s environment and show that the old child still sees the old value while a new child sees the new one. The exercise makes it clear which values are live observations and which are one-time inputs.