Linux Filesystem and System Interfaces

Stage 2 — Linux and Operating System Internals
Subject area 2.1 — The Operating System Model
Article 4

The short version

Linux exposes much of its live state through interfaces that look like files. These interfaces let programs and engineers inspect processes, memory, devices, hardware relationships, kernel settings, logs, clocks, and environment state.

The most important interfaces are:

• /proc for process and kernel runtime information
• /sys for devices, drivers, hardware relationships, and kernel objects
• /dev for device nodes and special kernel-managed resources
• Kernel logging interfaces for messages from the kernel and drivers
• Service logs for events reported by user-space services
• Clock and environment interfaces for time and process configuration

These are not all ordinary files stored on a disk. Many are virtual or pseudo-filesystem entries generated by the kernel or a user-space service when they are read. They provide a common way to inspect and control a live system.

The central idea is:

When a Linux system behaves unexpectedly, inspect the state that the kernel and services expose instead of guessing from the application alone.

Where this article fits

The previous article explained how Linux creates and supervises processes. This article explains where to inspect those processes and the rest of the operating system.

Later articles will use these interfaces when discussing memory, devices, filesystems, scheduling, resource limits, containers, security, and debugging. These interfaces are some of the first tools a systems engineer uses when investigating a machine.

Not every file is stored on disk

A regular file is persistent data stored through a filesystem. A pseudo-file is an interface that presents information through file-like operations without necessarily representing persistent disk data.

The file-like design is useful because programs already know how to open, read, write, and close file descriptors. The kernel can expose dynamic state through the same general mechanism.

flowchart LR
    Tool[Inspection tool] --> Open[open/read/write interface]
    Open --> Proc["/proc"]
    Open --> Sys["/sys"]
    Open --> Dev["/dev"]
    Proc --> Kernel[Kernel runtime state]
    Sys --> Kernel
    Dev --> Driver[Device driver or kernel resource]

Reading a pseudo-file may cause the kernel to format current state into text. Writing to one may change a kernel setting or send a request to a device driver. The meaning depends on the path and the interface contract.

This means ordinary filesystem assumptions do not always apply. A pseudo-file may change between reads, have no useful disk size, reject normal file operations, or require specific permissions.

/proc: process and kernel runtime information

/proc is a pseudo-filesystem that exposes process information and selected kernel state. It is usually mounted at /proc on Linux systems.

/proc
├── 1/
├── 2450/
├── self/
├── thread-self/
├── cpuinfo
├── meminfo
├── mounts
├── net/
├── sys/
└── uptime

Numeric directories usually represent process IDs. A process directory contains information about that process, such as its command line, memory mappings, file descriptors, status, and resource statistics.

Some entries describe the whole system rather than one process. cpuinfo describes processor information. meminfo describes memory statistics. uptime reports system uptime. mounts and related files describe mounted filesystems.

Inspecting a process through /proc

Suppose a process has PID 2450. Useful paths include:

/proc/2450/cmdline       Command-line arguments
/proc/2450/environ       Environment variables
/proc/2450/status        Human-readable process status
/proc/2450/stat          Process statistics in a compact format
/proc/2450/fd/           Symbolic links for open file descriptors
/proc/2450/maps          Current memory mappings
/proc/2450/smaps         Detailed mapping statistics
/proc/2450/limits        Resource limits
/proc/2450/cgroup        Control-group membership
/proc/2450/task/         Threads belonging to the process

The permissions on these paths depend on the user, system configuration, security settings, and kernel version. A process may be prevented from inspecting another process’s environment or memory details.

/proc/<pid>/status

The status file provides readable fields such as:

• Process name
• State
• Process ID and parent process ID
• Number of threads
• Virtual memory size
• Resident memory size
• User and group identifiers
• Capability information
• Signal masks

Resident memory means pages currently held in physical memory for the process. Virtual memory size includes address-space mappings that may not currently occupy physical memory. Confusing these numbers can lead to incorrect conclusions about memory pressure.

/proc/<pid>/fd

The fd directory contains one symbolic link for each file descriptor visible to the process.

/proc/2450/fd/0 → /dev/null
/proc/2450/fd/1 → /var/log/example.log
/proc/2450/fd/2 → /var/log/example.log
/proc/2450/fd/7 → socket:[123456]

Descriptors 0, 1, and 2 are conventionally standard input, standard output, and standard error. Other descriptors may refer to files, pipes, sockets, devices, event objects, or anonymous kernel objects.

This interface is useful when diagnosing “too many open files,” unexpected files that remain open, or a service that is holding a socket or pipe longer than expected.

/proc/<pid>/maps

The maps file shows the process’s virtual-memory regions. A mapping may represent:

• The executable
• A shared library
• The heap
• A thread stack
• An anonymous allocation
• A memory-mapped file
• A special kernel-provided region

An entry typically includes an address range, permissions, file offset, device and inode information, and an optional pathname.

address range        permissions  offset  object
55a00000-55a12000    r-xp         ...     /usr/bin/example
7f000000-7f020000    rw-p         ...     [heap]
7ffc0000-7ffc21000   rw-p         ...     [stack]

The permissions often appear as r for read, w for write, x for execute, and p or s for private or shared mapping. Memory maps are useful for understanding address-space layout, shared libraries, executable permissions, and memory growth.

/proc/self and /proc/thread-self

/proc/self is a convenient link to the /proc directory of the process performing the access. A program can read /proc/self/status without knowing its own PID.

/proc/thread-self refers to the current thread’s view where thread-specific information matters.

These paths are useful in tools and diagnostics because they avoid a race where a program discovers its PID and then accidentally inspects a different process after a PID reuse event.

/proc is a live view

The state exposed through /proc can change while a program reads it. A process can exit, create a thread, open a descriptor, close a descriptor, or change its memory mappings between two reads.

This creates an important rule:

A /proc observation is a snapshot or view of state at a particular point, not automatically a transactionally consistent picture of the entire system.

A monitoring tool that reads several files may observe values from slightly different moments. That is usually acceptable for diagnosis, but it matters when a program makes a safety-critical decision from the data.

/sys: the kernel’s device and object model

/sys, usually mounted as sysfs, exposes information about devices, drivers, buses, kernel objects, and relationships between them. It is different from /proc, although the two interfaces may contain related information.

Where /proc focuses heavily on processes and general kernel runtime state, /sys presents a structured view of devices and the kernel object hierarchy.

/sys
├── block/
├── bus/
├── class/
├── devices/
├── firmware/
├── fs/
├── kernel/
├── module/
└── power/

The exact entries depend on the hardware, drivers, kernel configuration, and system state.

/sys/devices

This hierarchy represents devices as they exist in the system’s device tree. It may show relationships between a device, its bus, its parent controller, and its driver.

The relationship matters when investigating a device problem. A network interface may depend on a PCI device, a driver, firmware, power-management state, and a physical link. /sys helps connect those pieces.

/sys/class

The class hierarchy groups devices by their functional role rather than only their physical location. Examples include:

/sys/class/net/      Network interfaces
/sys/class/block/    Block devices
/sys/class/tty/      Terminal devices
/sys/class/power_supply/

This provides a convenient way to discover devices by function. A network tool can inspect /sys/class/net without knowing where each network interface appears in the hardware tree.

/sys/block

Block devices provide storage in addressable blocks. /sys/block can expose device names, queue information, sizes, partitions, and relationships to underlying devices.

Storage performance and behavior may depend on queue depth, scheduler settings, rotational characteristics, device state, and layered devices such as RAID or device-mapper targets.

Reading and writing /sys

Some sysfs files are read-only observations. Others are writable configuration interfaces. Writing a value does not mean writing persistent data to a disk file; it may change kernel behavior immediately.

A setting may apply only until reboot, require a specific unit, or have safety restrictions. Writing to a sysfs entry without understanding its contract can change device behavior or system performance.

This is why /sys should be treated as a typed kernel interface represented through files, not as an ordinary directory that can be edited casually.

/dev: device nodes and special resources

/dev contains device nodes and other special entries through which programs access devices or kernel-managed resources.

Common examples include:

/dev/null       Discards writes and returns end-of-file on reads
/dev/zero       Produces zero bytes
/dev/random     Kernel-provided random-data interface
/dev/tty        Controlling terminal
/dev/sda        A block-device node, when present
/dev/console    System console interface

The entries are not regular files containing the device’s complete data. They are names associated with device drivers or kernel subsystems. Opening and using one invokes behavior defined by that device interface.

Character and block devices

A character device generally represents a stream of bytes or operations that are not addressed as fixed storage blocks. Terminals and many sensors are examples.

A block device provides block-oriented storage operations. Disks and virtual block devices are examples.

The distinction is useful but not enough to predict all behavior. A device’s driver defines details such as blocking, buffering, supported operations, errors, and synchronization.

Device permissions

Device nodes have ownership and permission rules. Access to /dev can expose hardware, sensitive random data, input devices, storage, or kernel functionality. Giving a process broad device access can weaken isolation even if the process has no permission to ordinary files.

Containers and service managers often use device policies to control which devices a process can see.

Kernel messages

The kernel and device drivers need a way to report events such as hardware detection, driver initialization, memory pressure, device errors, and security decisions. Linux maintains kernel logging facilities that user-space tools can read.

dmesg commonly displays messages from the kernel ring buffer:

dmesg --level=err,warn

The ring buffer is finite. Older messages may be overwritten by newer ones. Access may also be restricted because kernel messages can contain sensitive information.

Kernel messages are useful during boot, device discovery, driver failures, filesystem errors, and hardware problems. They are not a complete application log and should not replace service-level logging.

Service logs and the system journal

User-space services report events through standard output, standard error, log libraries, files, or a logging service. On systems using systemd, the journal can collect service output and metadata such as the unit name, PID, user, boot identifier, and timestamp.

Useful commands include:

journalctl -u example-worker
journalctl -u example-worker --since "10 minutes ago"
journalctl -b
journalctl -k

The first command filters logs for a service unit. The second limits the time range. -b selects the current boot, and -k focuses on kernel messages stored in the journal.

The journal is a user-space logging system with its own storage, filtering, rotation, and access policies. It is different from the kernel ring buffer even when it contains forwarded kernel messages.

Good service logs explain events with useful context:

• What operation was attempted
• Which resource or request was involved
• What failed
• What decision the service made
• Whether a retry or recovery occurred
• A request, job, or correlation identifier

Logging every internal detail is not automatically useful. Logs consume storage, may expose sensitive data, and can become noisy during an incident.

Time and clocks

Time appears simple until a system needs to measure durations, order events, schedule work, or display timestamps across machines.

Linux exposes different clock concepts for different purposes.

Real-time clock

Wall-clock time represents the current calendar time. It can be adjusted by synchronization, an administrator, a time zone change, or a manual update. It is appropriate for displaying timestamps and comparing events to calendar dates.

It is not always safe for measuring elapsed time. If the clock moves backward or forward while an operation runs, subtracting two wall-clock readings can produce an incorrect duration.

Monotonic clock

A monotonic clock moves forward for the purpose of elapsed-time measurement. It is appropriate for timeouts, deadlines, rate calculations, and measuring operation duration.

It does not represent a calendar date and should not be shown to users as a timestamp.

Display timestamp      → wall-clock time
Request timeout        → monotonic time
Operation duration     → monotonic time
Log event timestamp    → wall-clock time

Choosing the wrong clock can create bugs that appear only when the system clock is corrected or synchronized.

Timers

A timer asks the operating system to notify or wake a process after a duration or at a deadline. Timers can be affected by scheduling delays, process suspension, system load, and clock choice.

“The timer expired” does not mean the process runs at that exact instant. It means the timer event is ready and the scheduler will run the process when it can.

Hostnames and identity

A hostname is a name associated with a machine or network namespace. It is useful for logs, prompts, service discovery, configuration, and human identification.

The hostname is not automatically a secure identity. It may change, may not be globally unique, and may resolve differently depending on DNS or local configuration.

Systems often have several identity concepts:

• Kernel hostname
• Fully qualified DNS name
• Machine ID
• Instance ID from a cloud platform
• Container hostname
• Service identity or certificate identity

Confusing these can cause incorrect routing, certificate failures, duplicate registration, or misleading logs.

Environment variables and process configuration

An environment variable is a key-value string passed to a process, usually inherited from its parent. It can configure paths, logging, credentials, feature flags, time zones, and runtime behavior.

flowchart LR
    Parent[Parent process environment] -->|inherit at exec| Child[Child process]
    Child --> Override[Process-specific changes]
    Override --> Behavior[Runtime behavior]

Environment variables are convenient, but they have limitations:

• They are untyped strings.
• A child inherits them unless they are changed.
• They can appear in process inspection or crash diagnostics.
• Changing the parent environment does not update an already-running child.
• A missing value may silently select a dangerous default.

Configuration should validate required values, define defaults clearly, and avoid placing secrets in environments where process inspection or logging can expose them.

The same interface can expose and change state

Linux system interfaces often support both observation and control.

Reading /proc/<pid>/status observes process state. Writing a value to a control interface under /proc/sys may change a kernel setting. Reading /sys may show a device property. Writing to a writable sysfs attribute may change a device or driver setting. Sending a signal changes process state through a separate interface.

The difference between observation and control should be clear in tools and documentation. A diagnostic command that accidentally changes a live setting is dangerous.

A small code example: inspect a process from user space

The following Go function reads a process’s status file. It uses an ordinary file API, but the path refers to a live kernel-generated view rather than persistent application data.

func processStatus(pid int) ([]byte, error) {
 path := fmt.Sprintf("/proc/%d/status", pid)
 data, err := os.ReadFile(path)
 if err != nil {
  return nil, fmt.Errorf("read %s: %w", path, err)
 }
 return data, nil
}

The function can fail because the process does not exist, the caller lacks permission, /proc is not mounted, or the process exits between path construction and the read. The data is also a view of state at the time the kernel generated it; it is not a permanent record.

This small example shows why a file-like interface does not remove operating-system concerns. The program still needs to handle permissions, races, changing state, and platform assumptions.

Inspection tools are views built on these interfaces

Many familiar Linux tools read from these interfaces or use related system calls.

Question	Useful tools or interfaces
Which processes are running?	ps, pstree, /proc
What is a process doing?	/proc/<pid>/status, strace, top
What files and sockets are open?	lsof, /proc/<pid>/fd, ss
What memory is mapped?	/proc/<pid>/maps, pmap, debugger
What devices exist?	/sys, udevadm, /dev
What did the kernel report?	dmesg, journalctl -k
Why did a service fail?	systemctl, journalctl -u
What time behavior is available?	clock_gettime, timedatectl, /proc/uptime

The tool is an interface to evidence, not an explanation by itself. ps may show that a process is sleeping, but a trace or stack inspection may be needed to learn what it is waiting for.

Race conditions while inspecting state

Inspection can race with the system changing. A process may exit after a tool lists its PID but before the tool reads /proc/<pid>/status. A file descriptor may close between listing /proc/<pid>/fd and reading one of its links. A device may disappear during a hardware event.

Tools should handle these races as normal conditions. A missing entry does not always mean the original observation was wrong; it may mean the system changed between operations.

This is one reason a single snapshot is not always enough for diagnosis. Repeated observations, timestamps, tracing, and correlation with service logs can provide a more reliable explanation.

A realistic production example

Imagine a service that is reported as “running,” but requests are failing. The supervisor shows that the process has a live PID. The first assumption is that the application is healthy.

The engineer checks /proc/<pid>/status and sees that the process has many threads but little CPU activity. The file-descriptor directory shows a large number of sockets. ss shows many connections waiting in a state associated with slow clients. Service logs show request deadlines being exceeded, while kernel and network statistics show no hardware failure.

The process is alive but unable to make useful progress because resources are tied up by slow connections. The fix may involve connection timeouts, bounded concurrency, backpressure, or a change in how responses are streamed. Restarting the process may restore service temporarily, but the interfaces reveal the resource behavior that caused the problem.

How experienced engineers use these interfaces

They start with a question rather than opening every file in /proc.

For a process problem, they might ask:

• Is it alive, blocked, or repeatedly restarting?
• What resources does it hold?
• Which files, sockets, and memory mappings are open?
• What identity and limits apply?
• Which threads are waiting?

For a device problem, they might ask:

• Does the kernel see the device?
• Which driver is attached?
• What does the device state say in /sys?
• Did the kernel log an error?
• Are permissions or device policies blocking access?

For a service problem, they might ask:

• Did the service start with the expected environment?
• Which unit owns the process?
• What did the journal record before the failure?
• Is the manager restarting it?
• Did the clock, hostname, or configuration change?

The goal is to turn a vague symptom into a system-level hypothesis that can be checked.

Interview definitions

What is /proc?

/proc is a Linux pseudo-filesystem that exposes live process information and selected kernel runtime state through file-like interfaces.

What is /sys?

/sys is a pseudo-filesystem that exposes the kernel’s device, driver, hardware, and object relationships, along with selected control attributes.

What is /dev?

/dev contains device nodes and special entries that let programs access devices or kernel-managed resources through file-like operations.

What is the difference between /proc and /sys?

/proc focuses mainly on processes and general runtime state, while /sys presents the kernel’s device and object model. Both are virtual kernel interfaces rather than ordinary disk directories.

What is a pseudo-filesystem?

A pseudo-filesystem is a file-like interface whose entries are generated from live system state or kernel objects instead of being stored as ordinary persistent files.

What is the difference between wall-clock and monotonic time?

Wall-clock time represents calendar time and can be adjusted. Monotonic time is intended for measuring elapsed time and deadlines because it moves forward without calendar adjustments changing the result.

Interview follow-up questions

Why does Linux expose kernel state through files?

File operations provide a familiar interface that tools and programs can use to inspect or control state. The file-like representation does not mean the data is stored on disk; it is often generated dynamically by the kernel.

Can /proc data be treated as a consistent snapshot?

Not necessarily. Processes and resources can change while the files are being read, so observations from multiple entries may represent slightly different moments.

What is the difference between /dev/null and a regular file?

/dev/null is a device interface implemented by the kernel. Writes are discarded and reads return end-of-file; it does not contain persistent file data on disk.

Why is monotonic time better for timeouts?

Wall-clock time can move because of synchronization or manual adjustment. A monotonic clock is designed for elapsed-time measurement, so a clock correction does not unexpectedly extend or shorten a timeout.

Why might a process be alive but unhealthy?

It may be blocked on a resource, stuck in a retry loop, unable to accept work, holding exhausted connections, or failing every request. A live PID proves only that the process has not exited.

Why might a /proc entry disappear during inspection?

The process may have exited between the directory listing and the read. /proc exposes live state, so programs and tools must handle changes and races.

Common misconceptions

“Everything under /proc is a normal file.”

The entries use file-like operations, but many are generated dynamically and have behavior that differs from persistent files.

“Writing to /sys edits a configuration file.”

Writing to a sysfs attribute usually sends a control request to the kernel or driver. The change may be immediate, temporary, restricted, or hardware-affecting.

“/dev/sda contains the entire disk as a normal file.”

It is a device node that provides access to a block-device driver. Operations on it have device and kernel semantics, not just ordinary file semantics.

“A process list is a reliable snapshot of the machine.”

Processes can start, exit, and change state while the list is being collected. It is an observation taken over a period, not necessarily one atomic view.

“A hostname is a secure machine identity.”

A hostname is a name used by the local system and network configuration. It may change, may not be unique, and does not by itself authenticate a machine.

“Environment variables are safe configuration storage.”

They are convenient process inputs, but they are untyped, inherited, and sometimes visible through process inspection or diagnostics. Secrets and dangerous configuration need stronger handling.

Summary

Linux exposes live system state through file-like interfaces. /proc shows process and kernel runtime information, /sys presents devices and kernel-object relationships, and /dev provides access to devices and special resources. Kernel logs, service journals, clocks, hostnames, and environment variables expose other parts of the running system.

These interfaces are powerful because they let ordinary tools inspect a complex machine without requiring every tool to know the kernel’s internal data structures. They also have limits: state can change while it is being read, permissions can hide information, and file-like behavior does not mean ordinary persistent-file semantics.

The systems-engineering habit is to begin with a question, inspect the interface that can provide evidence, understand its consistency and permission limits, and connect the observation to a process, resource, device, or service hypothesis.

If you want to build this later

Build a small Linux system-inspection command that reports one target process in a readable format.

Read /proc/<pid>/status, /proc/<pid>/limits, /proc/<pid>/maps, /proc/<pid>/fd, and /proc/<pid>/cmdline. Add options to show memory, open descriptors, threads, and resource limits. Handle processes that exit during inspection and explain in the output that the values are observations rather than one atomic snapshot.

Then add a device mode that lists network interfaces through /sys/class/net and reports their state. The project should teach you to treat /proc, /sys, and /dev as system interfaces with contracts, permissions, races, and changing state.