Linux interview questions

Each file has read/write/execute bits for owner, group, and others (e.g. 640 = rw-r-----). On directories, execute means "can enter"; read means "can list".

$ ls -l app.sh
-rwxr-x--- 1 deploy ops 812 Jul  4 10:20 app.sh
#  user group other  = rwx r-x ---  = 750
chmod 640 app.sh        # rw- r-- ---
chmod u+x,o-r app.sh    # symbolic edit

Numeric encodes rwx as bits per class: r=4, w=2, x=1, summed per owner/group/other (755 = rwxr-xr-x). Symbolic edits relative to the current mode (u+x, go-w), which is safer when you only want to flip one bit.

chmod 644 file      # rw-r--r--
chmod 755 dir       # rwxr-xr-x
chmod g+w,o-rwx file

SUID runs a file with the owner’s privileges (e.g. passwd); SGID runs with the group’s, or makes new files inherit a directory’s group; the sticky bit on a directory (like /tmp) lets only the owner delete their files.

SUID/SGID binaries are a classic privilege-escalation path, so audit them regularly. A capital S in the mode means the bit is set but the file is not executable — usually a mistake.

$ ls -l /usr/bin/passwd
-rwsr-xr-x 1 root root ... /usr/bin/passwd   # the s = SUID
find / -perm -4000 -type f 2>/dev/null       # audit all SUID

A hard link is another name for the same inode (same data, same filesystem, survives deleting the original); a symlink is a pointer to a path that can cross filesystems and break if the target moves.

ln file hard        # same inode
ln -s file soft     # pointer to the path
$ ls -li file hard
1234 file
1234 hard           # identical inode number

A mask subtracted from default permissions when files are created — 022 yields 644 files and 755 directories. It sets the default security posture for new files per user or service.

$ umask
0022
$ touch new; ls -l new
-rw-r--r--   # 666 with 022 masked off = 644

When plain user/group/other is not enough (e.g. two groups need different access), ACLs grant per-user or per-group entries with setfacl/getfacl. A + after the mode in ls -l marks a file that carries ACLs.

setfacl -m u:alice:rw report.txt
getfacl report.txt
$ ls -l report.txt
-rw-rw-r--+ 1 ...        # the + = ACL present

A process has its own address space; threads share one process’s memory. In Linux both are tasks scheduled by the kernel — threads just share more via clone() flags.

ps -eLf | grep nginx           # -L shows threads (LWP column)
cat /proc/<pid>/status | grep Threads

A zombie is a finished child whose exit status the parent has not reaped via wait(); it holds only a PID slot. An orphan is a child whose parent died — init/systemd adopts and reaps it.

You cannot kill a zombie — it is already dead; you fix or restart the parent so it reaps (or the parent dies and init reaps it). A pile of zombies signals a buggy parent that never calls wait().

$ ps -eo pid,ppid,stat,cmd | awk '$3 ~ /Z/'
 4823  4800  Z+  [worker] <defunct>

SIGTERM politely asks a process to stop and can be caught for cleanup; SIGKILL cannot be caught or ignored and is enforced by the kernel; SIGHUP historically means "reload config" for daemons.

kill -TERM 4823     # graceful (default)
kill -HUP  4823     # reload config
kill -9    4823     # SIGKILL, last resort
kill -l             # list all signal names

Find it by CPU/memory in top or ps, confirm it is safe to stop, then send SIGTERM first and only escalate to SIGKILL if it ignores you.

ps -eo pid,%cpu,%mem,cmd --sort=-%cpu | head
kill -TERM <pid>    # give it a chance to clean up
kill -9 <pid>       # only if it will not exit

A unit file declares ExecStart, dependencies (After/Requires), restart policy, and resource limits; systemd supervises it as a cgroup, captures logs to the journal, and handles ordering and socket activation at boot.

Because each service is a cgroup, systemd tracks every child process (no forks escaping supervision) and can enforce memory/CPU limits and restart policy. Logs go to the journal, queryable by unit and time.

# /etc/systemd/system/app.service
[Service]
ExecStart=/usr/bin/app
Restart=on-failure
MemoryMax=512M

systemctl enable --now app
journalctl -u app -f

The nice value (-20 to 19) hints CPU scheduling priority — lower is greedier. Use it to keep a batch job from starving interactive work; for real isolation use cgroup CPU limits instead.

nice -n 10 ./batch.sh      # start it nicer
renice 15 -p <pid>         # renice a running process

Check load and the four resources: CPU (top/mpstat), memory (free, swap), disk I/O (iostat), and network (ss, sar). Correlate with recent changes and logs (journalctl) to find the constrained resource.

A structured pass beats guessing — the USE method: for each resource check Utilization, Saturation, and Errors. Start broad (load, then CPU/mem/disk/net), correlate with recent deploys, and only then drill into one process.

The average number of runnable plus uninterruptible (D-state, usually I/O) tasks over 1/5/15 minutes. Compare it to core count — load 4 on 4 cores is fully busy; sustained values well above cores mean saturation.

$ uptime
 10:20:01 up 5 days,  load average: 3.90, 2.10, 1.40
$ nproc
4          # 3.9 on 4 cores = ~fully busy, not overloaded

Identify the process (top/pidstat), then whether it is burning user or system time, then where it spends cycles (perf/strace). System-heavy usually means syscalls or I/O; user-heavy means app code.

top -o %CPU
pidstat -u 1
perf top -p <pid>

Look at available memory, not free — Linux uses spare RAM for reclaimable page cache. Real pressure shows as low available, heavy swap in/out, or OOM kills in dmesg.

$ free -m
              total   used   free  buff/cache  available
Mem:          15900   9200    400        6300        6100
# available 6.1G = fine, even though free is low

A deleted-but-open file still consumes blocks until the holding process closes it; df counts it, ls does not. Find it with `lsof | grep deleted` and restart or truncate the holder.

df counts allocated blocks; a deleted file still open by a process keeps its blocks until the fd closes, and du (which walks the directory tree) cannot see it. Restarting or truncating the holder frees the space immediately.

df -h /var
lsof +L1 | grep deleted        # link count 0 but still open
: > /proc/<pid>/fd/3           # truncate without a restart

strace to see its syscalls (e.g. which file/permission fails), lsof for open files and sockets, /proc/<pid>/ for fds, maps, and status, and perf for CPU hotspots. For latency, correlate with the USE method.

strace -f -e trace=openat,connect -p <pid>
lsof -p <pid>                  # open files + sockets
cat /proc/<pid>/status         # memory, threads, state

The kernel kills a process when memory (or a cgroup memory limit) is exhausted, picking by oom_score. Check dmesg/journal for "Out of memory", the cgroup limits, and whether the app’s heap settings respect the container limit.

The OOM-killer fires when the kernel cannot reclaim memory; it scores processes (oom_score) and kills the highest. In containers a per-cgroup limit triggers it locally, so one greedy pod is killed instead of taking the whole node.

dmesg -T | grep -i -A1 'killed process'
journalctl -k | grep -i 'out of memory'

ss -tulpn (or lsof -i :PORT) lists listening sockets with the owning PID and process, which you can trace back to a unit or container.

$ ss -tulpn | grep :8080
tcp LISTEN 0 128 *:8080 users:(("app",pid=4823,fd=6))
# then: ps -p 4823 -o cmd

Client sends SYN, server replies SYN-ACK, client sends ACK — then the connection is ESTABLISHED. Teardown is a four-way FIN/ACK exchange, leaving TIME_WAIT on the closer to absorb stray packets.

The handshake syncs sequence numbers before any data flows. A half-open connection (SYN received, never ACKed) is what SYN floods abuse; TIME_WAIT on the active closer holds the tuple ~2xMSL so a late packet cannot corrupt a new connection.

nsswitch.conf decides the order (files, DNS, mDNS); /etc/hosts is checked first, then a resolver or systemd-resolved queries DNS. Containers get a generated resolv.conf pointing at the cluster/embedded DNS.

Apps call getaddrinfo, which follows nsswitch.conf order — typically files (/etc/hosts) then DNS or systemd-resolved. Test with getent/resolvectl to see the real path an app takes, not just dig, which talks straight to a DNS server.

getent hosts api.internal      # uses nsswitch order
resolvectl query api.internal  # systemd-resolved view

Work up the stack: is it a name, a route, or a firewall? Resolve the name, ping the IP, check the route, then check for a filter or a dead service on the port.

getent hosts host          # DNS resolves?
ping -c1 10.0.0.5          # L3 reachable?
ip route get 10.0.0.5      # which route / iface?
nc -vz 10.0.0.5 443        # port open?

Packets traverse hooks (prerouting, input, forward, output, postrouting); rules in chains match and take a verdict (accept/drop/reject) or jump to another chain. nftables is the modern replacement with one unified ruleset.

nft list ruleset                     # nftables
iptables -L -n -v --line-numbers     # legacy view

Firmware/UEFI -> bootloader (GRUB) -> kernel + initramfs mounts root -> PID 1 (systemd) brings up targets and units -> getty/display manager offers login. Failures at each stage have distinct symptoms and logs.

Each stage hands off to the next and logs distinctly: firmware POST, GRUB loads the kernel + initramfs, the kernel mounts root and execs PID 1 (systemd), which activates targets until getty or a display manager offers login. A hang localizes to whichever stage last logged.