The Linux Command Line Every Developer Needs to Know

The Linux command line is a language for combining small programs, and learning it means learning the grammar before the vocabulary. Once you know how a command line is parsed, how files and permissions work, and how processes and streams connect, the individual commands are quick to pick up from their manual pages. The examples here ran on Debian 12 (bookworm) with GNU coreutils 9.1, Bash 5.2 and util-linux 2.38, in a clean container, and each section says where other systems differ. Writing these commands into files that run unattended is the subject of the Bash scripting tutorial.

The guide is arranged in the order a developer needs things: the grammar of the shell, moving through the filesystem, reading and changing files, permissions and ownership, finding things, filtering text, managing processes, checking disks and networks, archives, and the habits that make all of it repeatable. Where a topic has a deeper guide of its own, it links there.

The grammar of a command line

A command line is a program name followed by arguments, separated by spaces. The shell splits the line into words, expands variables and wildcards, then starts the program with the resulting list. The program never sees the wildcard or the quotes; it sees the words. That fact explains most of the surprises in shell use.

ls -l -a /etc
ls -la /etc
ls --all --format=long /etc

Options usually start with a dash. Short options are single letters and can be combined, so -la is -l -a. Long options start with two dashes and are spelled out. Arguments that are not options are usually files. A lone -- ends options, so rm -- -file removes a file named -file. Everything on the line is text; there are no typed arguments, which is why quoting matters so much.

Read the manual page first

man shows the manual page for a command, man -k word searches summaries for a topic, and command --help prints a shorter usage text. The sections matter: section 1 is commands, 5 is file formats such as crontab(5), 7 is overviews such as signal(7), and 8 is system administration. Inside the pager, / searches, n jumps to the next match and q quits.

man 5 crontab
man -k "disk usage"
type cd ls

cd is a shell builtin
ls is hashed (/usr/bin/ls)

type tells you whether a name is a builtin, an alias, a function or a file on disk. That matters because builtins such as cd have their documentation in Bash's own manual and not in man cd. Read the GNU coreutils manual for the core commands and the Linux man-pages project for system calls and file formats. Options differ between GNU, BSD and BusyBox implementations, so check the page that belongs to the system in front of you.

Where you are: the working directory

Every process has a current working directory, and relative paths are resolved against it. pwd prints it, cd changes it, cd - returns to the previous one, and cd alone goes home. A path starting with / is absolute; . is the current directory and .. is its parent; ~ expands to the home directory.

pwd
cd /tmp/demo/docs
pwd
cd ..
cd -

/tmp/demo
/tmp/demo/docs
/tmp/demo
/tmp/demo/docs

A child process cannot change its parent's directory, so a script that runs cd does not move your terminal. This is why cd must be a builtin, and why a script that needs to change your shell's directory has to be sourced rather than executed.

The filesystem layout

Linux presents one tree rooted at /. The Filesystem Hierarchy Standard, which most distributions follow, gives the main directories their jobs. /etc holds system configuration. /home holds users' files. /var holds data that changes, such as logs in /var/log and spool files. /tmp is temporary and is often cleared at boot. /usr holds installed programs and libraries, and on modern systems /bin, /sbin and /lib are symbolic links into it. /proc and /sys are virtual and expose kernel state, and /dev holds device files.

ls -l /bin /sbin | head -3
uname -sr
head -2 /etc/os-release

lrwxrwxrwx 1 root root 7 Mar  1 09:12 /bin -> usr/bin
Linux 7.0.14-linuxkit
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"

Knowing the layout lets you guess where to look: a service's config is under /etc, its logs under /var/log, and its binaries under /usr/bin or /usr/local/bin. The standard is documented in the Filesystem Hierarchy Standard.

List files with ls

ls lists a directory. The flags worth knowing are -l for long format, -a to include dotfiles, -h for human-readable sizes, -t to sort by modification time, -S to sort by size, -r to reverse and -d to show a directory itself and not its contents.

cd /tmp/demo
ls -lah

total 20K
drwxr-xr-x 4 root root 4.0K Nov  6 05:43 .
drwxrwxrwt 1 root root 4.0K Nov  6 05:43 ..
-rw-r--r-- 1 root root    5 Nov  6 05:43 Makefile
drwxr-xr-x 2 root root 4.0K Nov  6 05:43 docs
drwxr-xr-x 2 root root 4.0K Nov  6 05:43 src

The first column is the file type and permissions, then the link count, owner, group, size and modification time. The first character of the mode is - for a regular file, d for a directory and l for a symbolic link. Never parse ls output in a script, because names with spaces and newlines break it. Use globs, or find and stat, which are built for machines.

Inspect a file with stat and file

stat prints the inode's metadata, and with -c a format of your choice. file guesses what a file contains from its first bytes, which is more reliable than the extension.

stat -c '%n %s bytes %A owner=%U mtime=%y' docs/notes.txt
file docs/notes.txt src/main.c

docs/notes.txt 45 bytes -rw-r--r-- owner=root mtime=2026-10-01 10:00:00.000000000 +0000
docs/notes.txt: ASCII text
src/main.c:     C source, ASCII text

Linux files carry three times: modification (mtime, content changed), change (ctime, inode changed, such as permissions) and access (atime, usually updated lazily). There is no creation time field that every filesystem supports. The BSD and macOS stat uses a different option syntax, so scripts that call it are not portable without a branch.

Create, copy, move and remove

mkdir -p creates a directory and any missing parents, and does not complain if it exists. touch creates an empty file or updates a timestamp. cp -a copies preserving mode, ownership, times and links, and -r copies directories. mv renames or moves, and on the same filesystem it is an atomic rename. rm -r removes a tree, and rm -i asks first.

mkdir -p build/logs
cp -a src build/src-copy
mv build/src-copy build/src
rm -r build/src
rmdir build/logs

There is no trash can and no undo for rm. Preview wildcard deletions by running the same pattern with ls or echo first, and be wary of rm -rf with a variable that could be empty. For files you can recreate, rm is fine; for the rest, copy first. Tools like trash-cli add a recoverable delete if you want one.

Links: hard and symbolic

A filename is a directory entry that points at an inode. A hard link is a second entry pointing at the same inode, so the data persists until the last name is removed. A symbolic link is a tiny file that stores a path, followed at access time. Hard links cannot cross filesystems or point at directories; symbolic links can do both and can dangle if the target disappears.

ln -s docs/notes.txt link
ls -l link
ln docs/notes.txt hard
ls -li docs/notes.txt hard

lrwxrwxrwx 1 root root 14 Nov  6 05:43 link -> docs/notes.txt

The -i flag shows inode numbers, and two hard links to one file show the same number. A relative symbolic link is resolved from the directory that contains the link, not from where you ran ln, which is the most common reason a new link does not work. Deploy tools use ln -sfn to repoint a current link at a new release in one step.

Read files: cat, less, head and tail

cat prints files in order, and it is for concatenating, not for paging. less pages a file with search, so use it for anything longer than a screen. head -n 2 and tail -n 1 show the start and end, and tail -f follows a file as it grows, which is how you watch a log. wc -l counts lines.

head -n 2 docs/notes.txt
tail -n 1 docs/notes.txt
wc -l docs/notes.txt
tail -F /var/log/syslog

line one
line two
line four
4 docs/notes.txt

Use tail -F with a capital F for logs, because it keeps following across log rotation, where lowercase -f keeps watching the old, renamed file. Inside less, press F to follow, / to search, g and G to jump to the ends, and -S to toggle line wrapping for wide output.

Streams, pipes and redirection

Every process starts with standard input (0), standard output (1) and standard error (2). A pipe | connects one command's output to the next one's input, and redirections point the streams at files. The model is simple and everything else depends on it.

grep -c line docs/notes.txt
grep line docs/notes.txt | sort -r | head -2
ls /nonexistent 2> errors.txt
ls /tmp /nonexistent > both.txt 2>&1
command >> log.txt

Greater-than truncates and writes, double greater-than appends, and 2>&1 sends standard error to wherever standard output currently points. Order matters: the redirections are processed left to right. The special file /dev/null discards whatever is written to it, so command > /dev/null 2>&1 silences everything. The tee command copies its input to a file and to standard output, which is how you save the output of a pipeline and still see it.

Quoting

Quotes control what the shell expands before the program runs. Double quotes allow variable and command expansion but prevent word splitting and globbing. Single quotes prevent everything. A backslash escapes the next character.

name="two words"
printf '[%s]\n' $name
printf '[%s]\n' "$name"
printf '[%s]\n' '$name'

[two]
[words]
[two words]
[$name]

The unquoted form became two arguments. Quote every variable expansion unless you specifically want splitting, which is rare. Use single quotes for patterns passed to grep, sed and awk, so the shell does not interpret their special characters first. The security consequences of unquoted data are covered in the shell script security guide.

Wildcards and globbing

The shell expands wildcards into matching file names before the command starts. * matches any string, ? one character, [abc] one of a set, and [!abc] anything else. Brace expansion generates strings without checking files: {a,b} and {1..3}. Hidden files, whose names start with a dot, are not matched by *.

echo *.txt
echo docs/*
echo file{1..3}.log
echo {src,docs}/*

A pattern with no matches is passed through literally, so ls *.xyz reports an error about a file named *.xyz. Because the shell, not the command, expands wildcards, find -name '*.txt' needs the quotes so that find receives the pattern. Globs are not regular expressions: they are a separate, simpler language, and mixing them up is a frequent stumbling block that the regular expressions guide clears up.

Permissions and ownership

Each file has an owner, a group and a mode with three triples: owner, group and others, each of read, write and execute. For a directory, read means listing names, write means creating and removing entries, and execute means entering it. The octal form adds 4, 2 and 1 for each triple, so 640 is owner read and write, group read, others nothing.

chmod 640 docs/notes.txt
ls -l docs/notes.txt
chmod u+x,g-r docs/notes.txt
ls -l docs/notes.txt
stat -c %a docs/notes.txt

-rw-r----- 1 root root 45 Oct  1 10:00 docs/notes.txt
-rwx------ 1 root root 45 Oct  1 10:00 docs/notes.txt
700

Symbolic modes read as who, operator, what: u+x adds execute for the owner, g-r removes read from the group, a=r sets read for all. chown user:group file changes ownership and normally needs root. Use chmod -R with care: a recursive 755 makes every data file executable, and the capital X (chmod -R u+rwX,go+rX) sets execute only on directories and files already executable.

umask, setuid and the sticky bit

New files start from a base mode of 666 for files and 777 for directories, and the umask removes bits from it. The common umask of 022 yields 644 files and 755 directories. A stricter 077 yields private files, which suits scripts that write secrets.

umask
touch f1; mkdir d1
ls -ld f1 d1

0022
drwxr-xr-x 2 root root 4096 Nov  6 05:43 d1
-rw-r--r-- 1 root root    0 Nov  6 05:43 f1

Three special bits sit above the nine. Setuid on an executable makes it run as the file's owner, which is how passwd can write the password database. Setgid on a directory makes new files inherit its group, which suits shared project directories. The sticky bit on a directory, as on /tmp, lets only a file's owner delete it. They show up as s and t in the listing. Setuid scripts are ignored by Linux for safety, and every setuid binary on a system is an attack surface worth auditing with find / -perm -4000 -type f.

Identity and sudo

id prints the user ID, group ID and supplementary groups, whoami prints the name, and groups lists memberships. sudo command runs one command as root (or -u user as another user) after checking /etc/sudoers. Prefer it over a long-lived root shell, because every use is logged.

id
sudo -l
sudo -u www-data whoami

uid=0(root) gid=0(root) groups=0(root)

A redirection is performed by your shell before sudo starts, so sudo echo x > /etc/file fails with permission denied. Use echo x | sudo tee /etc/file instead. Edit the sudoers file only with visudo, which checks syntax before saving, because a broken file can lock out every administrator.

Find files with find

find walks a directory tree and tests each entry. Tests include -name (a glob, quoted), -type f or d, -mtime +7 for older than seven days, -size +100M, -user, -perm and -newer file. Combine tests with implicit AND, -o for OR, ! for NOT and parentheses escaped for the shell.

find . -name '*.c' -o -name '*.txt'
find . -type f -mtime +7
find . -type f \( -name '*.c' -o -name '*.h' \) -size -1M

./src/main.c
./docs/notes.txt
./docs/notes.txt

The first command lists both because -o has lower precedence than the implied AND; add parentheses when mixing. The second printed only the file with a timestamp from October. Find evaluates tests left to right and stops at the first false one, so put cheap tests such as -name before expensive ones such as -exec. The GNU findutils manual documents every test.

Act on what find finds

Three actions cover most uses. -print prints names, and is the default when no action is given. -exec command {} + runs a command with many files at once, and -exec command {} \; runs it once per file. -delete removes matches. For names with unusual characters, -print0 separates with a NUL byte, which pairs with xargs -0.

find /tmp/demo -name '*.tmp' -mtime +7 -delete
find . -name '*.c' -exec wc -l {} +
find . -type f -print0 | xargs -0 grep -l 'error'
find . -type d -empty -print

Put -delete last; placed before the tests, it deletes everything. Preview by running the command with -print first. The + form is faster because it starts fewer processes, much like xargs. For simple name searches on a big tree, locate uses a prebuilt index and is instant, but its index can be stale.

Search text with grep

grep prints lines that match a pattern. The options to learn are -i ignore case, -v invert, -n line numbers, -r recurse, -l file names only, -c count, -w whole words, -F fixed strings, -E extended regular expressions and -A, -B, -C for context lines.

grep -n error docs/notes.txt
grep -rn --include='*.c' 'main' src
grep -c . docs/notes.txt
grep -v '^#' /etc/hosts | grep -v '^$'

3:error: disk full

The exit status is 0 for a match, 1 for none and 2 for an error, which makes if grep -q pattern file the standard way to test for text. A leading - in the pattern needs -e or --. For code, tools such as ripgrep respect .gitignore and run faster, but GNU grep is on every machine. The deeper treatment, including sed and awk, is in the grep, sed and awk guide.

Filter and reshape with sort, uniq, cut and tr

These four small tools do most of the everyday work on columns and lists. sort orders lines, and its useful options are -n numeric, -h human sizes, -r reverse, -k a key field, -t a separator and -u unique. uniq collapses adjacent duplicates, so sort first; -c counts them. cut -d: -f1,7 picks fields. tr translates or deletes characters.

printf 'b\na\nb\nc\nb\n' | sort | uniq -c | sort -rn
echo 'root:x:0:0:root:/root:/bin/bash' | cut -d: -f1,7
echo 'Hello World' | tr a-z A-Z

      3 b
      1 c
      1 a
root:/bin/bash
HELLO WORLD

The pattern sort | uniq -c | sort -rn is the classic frequency table, and it turns up in every log analysis. Sorting follows the locale; set LC_ALL=C for plain byte order, which is faster and gives the same result on every machine. sort -k2,2n sorts by the second field numerically, and the comma matters: without it the key runs to the end of the line.

Run commands on many items with xargs

xargs reads items from standard input and builds command lines from them. It exists because the kernel limits the length of an argument list, and because it lets you turn a stream of names into arguments. Use -n to cap items per call, -P to run several in parallel, and -0 with -print0 for safe names.

printf 'a b\n' | xargs -n1 echo item
find . -name '*.log' -print0 | xargs -0 -P4 gzip
printf '%s\n' web01 web02 | xargs -I{} ssh {} uptime

item a
item b

By default xargs splits on whitespace, which breaks on names with spaces; the NUL form is the reliable one. With -I{} each item replaces the placeholder and runs once. If the input is empty, GNU xargs still runs the command once unless you add -r (--no-run-if-empty), which is the cause of many strange "ran with no arguments" errors.

Processes: ps, top and pgrep

A process is a running program with a PID, a parent, an owner and a state. ps shows a snapshot, top and htop show it live, pgrep finds PIDs by name, and pstree draws the parent-child tree. The container image used for the other examples ships without ps; on Debian it comes from the procps package.

ps -eo pid,ppid,user,stat,etime,comm --sort=-etime | head
ps aux --sort=-%mem | head -5
pgrep -af nginx
pstree -p $$

The STAT column letters are worth knowing: R running, S sleeping, D uninterruptible sleep, usually waiting on disk or network storage, T stopped and Z zombie, a finished process whose parent has not collected its status. A pile of D processes explains a high load average with an idle CPU. The BSD-style ps aux and the standard ps -ef both work on Linux; mixing the styles in one command gives warnings. The ps manual page lists every column.

Signals and kill

A signal is a small message to a process. kill PID sends SIGTERM (15), a polite request to exit that a program can catch to clean up. kill -HUP often asks a daemon to reload configuration. kill -9 sends SIGKILL, which cannot be caught or ignored and gives the process no chance to clean up. Use SIGTERM first and escalate only if it does not exit.

sleep 300 &
kill %1
pkill -f 'python worker.py'
kill -l | head -3

Ctrl-C sends SIGINT to the foreground job, Ctrl-Z sends SIGTSTP to pause it, bg resumes it in the background and fg brings it back. jobs lists the shell's jobs. The full list and default actions are in signal(7). A process stuck in state D will not die from SIGKILL until its disk or network request returns, which is a property of the kernel and not a bug in the command.

Keep a job running: tmux, nohup and systemd-run

A job started in a terminal gets SIGHUP when the terminal closes. For long jobs over SSH, run them in tmux (or screen), which keeps a session alive on the server so you can detach and reattach. nohup makes a command ignore SIGHUP and write to nohup.out. On a systemd host, systemd-run --user starts a command in its own transient unit, with logs in the journal.

tmux new -s build
# Ctrl-b d to detach
tmux attach -t build
nohup ./long-job.sh > job.log 2>&1 &
systemd-run --user --unit=import ./import.sh

For something that must start at boot and restart on failure, a transient job is the wrong tool; write a service or timer unit, as described in the cron and systemd timers guide. SSH multiplexing and reconnect behavior, which affect long sessions, are in the SSH guide.

Open files and ports with lsof and ss

When something is using a file or a port, lsof and ss say which process. lsof path lists processes with a file open, lsof -i :8080 finds who is bound to a port, and lsof +L1 lists deleted files that are still held open, which is the usual cause of a disk that is full while du shows free space. ss -ltnp lists listening TCP sockets with their processes.

ss -ltnp
lsof -i :8080
lsof +L1
fuser -v /var/log/app.log

State   Recv-Q  Send-Q  Local Address:Port  Peer Address:Port  Process
LISTEN  0       511     0.0.0.0:80          0.0.0.0:*          users:(("nginx",pid=812,fd=6))
LISTEN  0       128     0.0.0.0:22          0.0.0.0:*          users:(("sshd",pid=611,fd=3))

Seeing other users' processes needs root, so run these with sudo when the Process column is empty. ss replaced netstat, which is deprecated and absent from many current images. The -n flag skips name resolution, which both speeds the command up and shows real addresses.

Disk space: df, du and ncdu

df -h shows free space per filesystem, df -i shows free inodes, and du -sh totals a directory. A filesystem can be full by blocks or by inodes, and a directory with millions of tiny files exhausts inodes while df -h still shows free space.

df -h /tmp
du -sh .
du -ah . | sort -rh | head -3
df -i /

Filesystem      Size  Used Avail Use% Mounted on
overlay         189G  137G   43G  77% /
24K	.
24K	.
8.0K	./src
8.0K	./docs

The sorted du output lists the total first because . contains everything; use du -h --max-depth=1 to see one level at a time. When df and du disagree, a deleted file is still open by a process, which lsof +L1 shows; restarting that process frees the space. ncdu is an interactive version of the same investigation and is worth installing on servers you manage.

Memory, load and system facts

free -m shows memory in megabytes, with available being the number that matters, not free, because Linux uses spare memory as cache. uptime prints load averages for 1, 5 and 15 minutes, which count runnable and uninterruptible tasks and should be read against the number of CPUs from nproc. uname -a, lscpu and /etc/os-release describe the machine.

free -m
uptime
nproc
cat /proc/meminfo | head -3

A load of 4 on a 4-CPU machine is full; on a 32-CPU machine it is idle. A rising load with low CPU use points at storage or network waits. The /proc files are the source of truth for these tools, and reading them directly is how scripts get numbers without parsing another program's formatting.

Networking basics: ip, ping, curl and dig

ip addr shows addresses and ip route the routing table; they replace ifconfig and route. ping -c 3 host tests reachability. dig +short name resolves DNS names. curl speaks HTTP and many other protocols, and curl -fsS is the script-friendly form that fails on HTTP errors and prints only errors.

ip -br addr
ip route get 1.1.1.1
dig +short example.org
curl -fsS -o /dev/null -w '%{http_code} %{time_total}s\n' https://example.org/

200 0.231s

The -w format prints timing and status without the body, which suits health checks. Without -f, curl exits 0 on a 404 or 500, because the transfer itself worked. Add --max-time and --retry in scripts so a stalled server cannot hang the job. For remote login and file transfer, move on to the SSH guide.

Archives and compression

tar bundles files into one archive and gzip, xz or zstd compress it. The flags are c create, x extract, t list, f file, z gzip, J xz and v verbose. Use -C dir to choose where to work.

tar -czf ../demo.tgz .
tar -tzf ../demo.tgz | head -4
mkdir /tmp/restore && tar -xzf ../demo.tgz -C /tmp/restore
gzip -k big.log
zcat big.log.gz | grep -c error

./
./Makefile
./src/
./src/main.c

List before extracting an archive you did not make, because a tarball can contain absolute paths or .. components. GNU tar strips leading slashes by default, but check. Pick the compressor by the trade: gzip is fast and universal, xz compresses best and is slow, zstd gives most of xz's ratio at much higher speed. zcat, zgrep and zless read compressed files without unpacking.

Environment variables and PATH

Variables exported with export are inherited by child processes. env prints the environment, and env -i starts a command with an empty one, which is the honest test of what a script needs. PATH is the colon-separated list of directories searched for commands, in order, and the first match wins.

echo "$PATH"
env -i PATH=/bin bash -c 'echo "$HOME-$PATH"'
export EDITOR=vim
PATH="$HOME/bin:$PATH"

-/bin

That second command printed an empty HOME and the one directory we gave it, which is closer to what cron provides than your login shell. Put per-user setup in ~/.profile for login shells and ~/.bashrc for interactive Bash, and system-wide values in /etc/environment or /etc/profile.d/. Never put . in PATH, because a malicious file in a directory you visit then runs as a command.

Shell history and line editing

Bash keeps a history, searched with Ctrl-R as you type. !! repeats the last command, !$ is its last argument, and sudo !! reruns it as root. Line editing follows Emacs bindings: Ctrl-A and Ctrl-E jump to the ends, Alt-B and Alt-F move by word, Ctrl-W deletes a word and Ctrl-U deletes to the start. Settings such as HISTSIZE, HISTCONTROL=ignoreboth and HISTTIMEFORMAT in ~/.bashrc tune how much is kept.

history | tail -3
sudo !!
cd !$

History is written when the shell exits, so a terminal that crashes loses its commands; shopt -s histappend and PROMPT_COMMAND='history -a' write after every command. History files can contain secrets typed on command lines, so treat ~/.bash_history as sensitive, and prefix a command with a space (with ignorespace set) to keep it out.

Compose commands into a pipeline: a worked example

Real questions are answered by chaining tools. Suppose a web server log has lines like 203.0.113.9 - - [05/Nov/2026:10:01:02 +0000] "GET /a HTTP/1.1" 200 512 and you want the five client addresses that made the most failing requests. Build the pipeline one stage at a time, running it after each addition.

awk '$9 >= 500 { print $1 }' access.log \
  | sort \
  | uniq -c \
  | sort -rn \
  | head -5

     42 203.0.113.9
     17 198.51.100.4
      3 192.0.2.77

awk keeps the client column of lines whose status field is 500 or above, sort groups equal addresses, uniq -c counts each group, the second sort -rn ranks by count and head keeps the top. The shape is general: select, normalize, group, count, rank, limit. Awk and its alternatives are explained properly in the text-processing guide. When a pipeline grows past a few stages, name its parts in a script, as in the Bash tutorial.

Turn a command line into a script

Once a command line works, save it. A script is the same commands in a file with a shebang and the executable bit, plus the quoting, argument handling and error checks that an interactive session lets you skip. Start by pasting the working line, replace the hard-coded values with variables, quote them, and add set -euo pipefail and a usage message. Then run it under shellcheck and fix what it reports.

#!/usr/bin/env bash
set -euo pipefail
log="${1:?usage: top-errors.sh LOGFILE [N]}"
n="${2:-5}"
awk '$9 >= 500 { print $1 }' "$log" | sort | uniq -c | sort -rn | head -n "$n"

This is the point at which the command line stops being exploration and becomes a tool, and the fuller treatment is in the Bash scripting tutorial and the testing guide. If you plan to run the result on a schedule, the cron and systemd timers guide covers the environment differences that make a working interactive command fail overnight.

Compare files with diff and cmp

diff shows the differences between two text files, and diff -u produces the unified format that patches and code review use. diff -r compares directories. cmp compares bytes and reports the first difference, which suits binary files. comm compares two sorted lists and prints lines unique to each and common to both.

diff -u old.conf new.conf
diff -rq dir1 dir2
cmp a.bin b.bin
comm -12 <(sort list1) <(sort list2)

The exit status is 0 for identical, 1 for different and 2 for trouble, which makes diff -q usable as a test. A unified diff can be applied with patch -p1 < change.diff, though in a repository git diff and git apply do the same with better error handling.

Edit files from the terminal

Every developer on servers needs one terminal editor. nano is the gentlest, showing its keys at the bottom. vim is modal and fast once learned: i inserts, Esc returns to normal mode, :w saves, :q quits, :q! quits without saving, /text searches, and dd deletes a line. Set EDITOR and VISUAL so programs such as git commit and crontab -e open the editor you chose. For one-line changes in a script, sed -i edits a file in place, and the deeper tour is in the text-processing guide.

export EDITOR=nano
sed -i.bak 's/old/new/g' config.ini

The .bak suffix keeps a copy of the original, which is the cheap insurance for any in-place edit and also the portable form, since BSD sed requires a suffix argument.

Users, groups and the files that define them

Accounts live in plain text files. /etc/passwd lists users, with name, UID, GID, home and shell. /etc/shadow holds password hashes and is readable only by root. /etc/group lists groups. getent passwd name queries these and any directory service, such as LDAP, in one command, and adduser, usermod and userdel manage accounts. A user must log out and in again for a new group membership to apply.

getent passwd root
getent group sudo
sudo usermod -aG docker deploy

root:x:0:0:root:/root:/bin/bash

The -a in usermod -aG means append; leaving it off replaces all of the user's supplementary groups with the one named, which has locked people out of sudo. Giving an account membership in the docker group is equivalent to giving it root on the host, a trade-off to make deliberately.

Packages and services

Install software with the distribution's package manager: apt on Debian and Ubuntu, dnf on Fedora and RHEL, apk on Alpine and pacman on Arch. Update the package index before installing, query what owns a file with dpkg -S or rpm -qf, and list a package's files with dpkg -L. On systemd systems, services are controlled with systemctl and read with journalctl.

sudo apt update && sudo apt install -y jq
dpkg -S /usr/bin/jq
systemctl status ssh --no-pager
journalctl -u ssh --since '1 hour ago' --no-pager | tail -5

Treat curl ... | sh installers with suspicion, as the script can change between your reading and running it. Download, read and then run, or prefer the package manager. In containers, run apt-get update and the install in the same RUN layer, and clean the lists afterward, to avoid stale indexes and large images.

Common mistakes and the habit behind each

• Running a destructive command with a wildcard unchecked. Run it with ls or echo first.
• Using sudo with a redirection. The shell opens the file before sudo runs; use tee.
• Using kill -9 first. Send SIGTERM and wait, so the program can clean up.
• Trusting ls output in a script. Use globs, find and stat.
• Recursive chmod 777 to make a permission error go away. Find the real owner and mode problem.
• Forgetting that cd in a script does not move your shell. Source the file or print the path.
• Forgetting -F on tail across rotation. Lowercase -f follows the renamed file.
• Comparing environments by memory. Use env -i to see what a clean launch has.

Most of these are the same error: acting on what you meant instead of on what the shell will do. Reading the expansion first, with echo or set -x, closes the gap, and the debugging guide builds on that habit.

Time, dates and scheduling at a glance

date prints and formats the current time, date -u prints UTC, and date +%s prints seconds since the epoch. timedatectl shows the time zone and whether the clock is synchronized. Servers should run in UTC with NTP enabled, so logs from different machines line up and daylight saving changes cannot cause a job to run twice or not at all. For one-off delayed jobs, at runs a command at a given time, sleep waits, and watch -n 5 command repeats a command and redraws the output, which is a quick way to monitor a changing value. Recurring jobs belong to cron or a systemd timer.

date -u +%FT%TZ
timedatectl | head -4
watch -n 5 'df -h /var | tail -1'
echo 'tar -czf /tmp/b.tgz /srv/data' | at 02:30

A clock that drifts breaks TLS validation, token expiry and log correlation in ways that look unrelated to time, so check timedatectl early when a certificate or token error makes no sense.

Compare tools by what they cost

Every command you chain costs a process, and every process costs a fork, an exec and some memory. For a handful of files it does not matter; for a million it dominates. A few rules of thumb come from that. A single awk replaces a grep | cut | sort chain with one process, though on three million lines both ran in about half a second in Debian 12, so measure before rewriting. A find ... -exec cmd {} + is faster than -exec cmd {} \; because it starts far fewer processes. grep -F is faster than grep with a regular expression when you only need a fixed string. LC_ALL=C sort avoids locale collation. Piping a file into a tool that can open it, as in cat file | grep x, adds a process and loses nothing but a little clarity, so write grep x file. Measure with time before and after any change claimed to be an optimization, since the slow step is rarely the one you suspect.

time grep -c error big.log
time LC_ALL=C grep -c error big.log
time awk '/error/ {n++} END {print n}' big.log

The method for finding the slow part of a longer script, with counters and traces, is described in the debugging guide.

Keep a personal toolbox

People who work fast at a terminal have usually built a small, deliberate toolbox. Keep scripts you reuse in ~/bin and add it to PATH, with each script named for what it does and passing ShellCheck. Keep dotfiles in a Git repository so a new machine takes minutes. Define a few aliases for commands you type daily, such as alias ll='ls -lah', and a handful of functions for anything that needs arguments, because aliases cannot take them. Prefer functions in ~/.bashrc over scripts for anything that must change the shell's own state, such as moving to a project directory. Resist adding shortcuts you will forget: a toolbox of ten things you use beats one of a hundred you do not.

alias ll='ls -lah'
mkcd() { mkdir -p -- "$1" && cd -- "$1"; }
up() { cd "$(printf '../%.0s' $(seq 1 "${1:-1}"))"; }

Functions like mkcd must live in the shell's startup file or be sourced, because a script cannot change its parent's directory, as shown at the start of the guide.

Practice on a disposable machine

The fastest way to become fluent is to practice somewhere a mistake costs nothing. A container gives you a clean Linux in seconds and a fresh one every time, so you can break it, delete the wrong directory, and start again. Run docker run --rm -it debian:12 bash and work through the sections of this guide inside it. Install the missing tools with apt-get update and apt-get install -y procps when you need ps or free. A virtual machine or a spare cloud instance serves when you need systemd, which most containers lack. Set yourself small tasks: find the five largest files under /usr, list every listening port with its process, write a one-line command that counts distinct users in /etc/passwd by shell. Each task uses three or four of the commands above and teaches you which options you actually reach for.

Filesystems, mounts and fstab

Every file you touch lives on a filesystem, and the tree under / is assembled by mounting several of them. lsblk lists block devices as a tree, findmnt shows what is mounted where and with which options, and blkid prints the UUID and type of each device. Mounts that should survive a reboot are listed in /etc/fstab, one per line, with the device, mount point, type, options, dump flag and fsck order. Identify devices by UUID, not by names like /dev/sdb1, because the kernel can renumber disks between boots.

lsblk -f
findmnt /var
sudo blkid /dev/sdb1
sudo mount -o ro /dev/sdb1 /mnt
sudo umount /mnt

# /etc/fstab
UUID=3c1f6a52-0d0e-4b52-9b0a-6a8d8b1f2c11  /srv/data  ext4  defaults,noatime,nofail  0  2

The option nofail lets the system boot when the disk is missing, which is the difference between a server that comes up without its data volume and one that drops to an emergency shell. Test a new entry with sudo mount -a before rebooting: it mounts everything in the file and reports errors, and a typo found now is cheaper than one found at boot. Mount options worth knowing are noatime, which skips access-time updates, ro for read-only, and noexec, nosuid and nodev for untrusted volumes such as /tmp or removable media.

Access control lists and extended attributes

The owner, group and others model covers most cases, and sooner or later someone needs to grant one more user access without changing the group. POSIX ACLs add per-user and per-group entries to a file. setfacl sets them and getfacl reads them. A plus sign at the end of the mode in ls -l tells you that a file has an ACL. A default ACL on a directory is inherited by new files, which is the tool for shared project folders.

setfacl -m u:alice:rw report.txt
setfacl -d -m g:team:rwX /srv/project
getfacl report.txt
ls -l report.txt

Check that the filesystem was mounted with ACL support; ext4 and XFS enable it by default on current kernels. ACLs are invisible to tools that copy files without preserving them: cp -a, rsync -A and tar --acls keep them, and plain cp may not. Since ACLs add a second place where permissions can hide, document any that you use, and prefer plain groups when they are enough. Related is the immutable flag set with chattr +i, which stops even root from changing a file until it is cleared, a useful guard for critical configuration.

Logs: where things are written and how to read them

On systemd distributions most service output goes to the journal, read with journalctl. Filter by unit with -u, by time with --since and --until, by priority with -p, by boot with -b, and follow with -f. Traditional text logs still live in /var/log: syslog or messages for general events, auth.log or secure for logins and sudo, kern.log for the kernel, and per-application directories for web servers and databases.

journalctl -u nginx --since '1 hour ago' -p warning --no-pager
journalctl -b -1 -p err
journalctl -k | tail -20
sudo tail -F /var/log/auth.log
ls -lh /var/log | sort -k5 -h | tail -5

Logs grow until something stops them. logrotate runs daily from cron or a timer, reading rules from /etc/logrotate.conf and /etc/logrotate.d/: rotate weekly, keep four copies, compress, and signal the service to reopen its file. A service that writes to a file and ignores the reopen signal keeps writing to the rotated file, which is why a disk can stay full after rotation; the copytruncate option is a workaround at the cost of possibly losing a few lines. The journal has its own limits, set with SystemMaxUse= in /etc/systemd/journald.conf.

A triage routine for a slow or struggling machine

When a server is slow, work through the resources in a fixed order: CPU, memory, disk, network. For each, ask three questions: how busy is it, is anything waiting on it, and is it failing. Brendan Gregg's USE method, utilization, saturation and errors, is the name for that checklist, and the commands below cover it with tools present on any Linux system.

uptime
top -b -n1 | head -15
free -m
vmstat 1 5
iostat -x 1 3
df -h; df -i
ss -s
dmesg -T | tail -20

Read them in order. uptime gives the load averages; compare them with nproc. top sorted by CPU shows the busy process. free shows whether available memory is low, and vmstat columns si and so show swapping, the clearest sign of memory pressure. iostat -x shows disk utilization and the average wait; a device at 100 percent with high waits is saturated. df shows full disks or inodes. ss -s summarizes sockets, and dmesg shows kernel complaints such as out-of-memory kills or disk errors. The result is a short list of facts, and the fix usually follows from the one that stands out. Write down what you saw before you change anything.

Memory: what the numbers mean

Linux uses free memory for the page cache, so a healthy machine shows little free memory. The figure to watch is available in free, an estimate of how much can be given to programs without swapping. Swap use alone is not a problem, since the kernel moves idle pages out to make room for cache; continuous swapping in and out is. When memory truly runs out, the kernel's out-of-memory killer chooses a process to end, and it records the decision in the kernel log.

free -m
ps aux --sort=-%mem | head -5
grep -E 'MemAvailable|SwapFree' /proc/meminfo
dmesg -T | grep -i 'killed process'
cat /proc/$(pgrep -o nginx)/status | grep -E 'VmRSS|VmSwap'

The fields VmRSS and VmSwap in a process's status file give its resident and swapped memory, which is more precise than the sorted ps list. If a process is killed with status 137, check the kernel log for the OOM message before blaming the program. Containers add their own limit, enforced by cgroups, and a process can be killed for exceeding it while the host has plenty free.

Firewalls in one page

The kernel's packet filter is netfilter, and the current front end is nftables, with iptables commands often translated onto it. Most people meet a simpler wrapper: ufw on Ubuntu and firewalld on Fedora and RHEL. The model is the same in each: rules match traffic by direction, protocol, port and address, and a default policy decides what the rules do not mention. For a server the sound default is to deny incoming traffic and allow only the services it provides, plus SSH from known addresses.

sudo ufw default deny incoming
sudo ufw allow from 203.0.113.0/24 to any port 22 proto tcp
sudo ufw allow 443/tcp
sudo ufw enable
sudo ufw status numbered
sudo nft list ruleset | head -20

Allow SSH before enabling the firewall on a remote machine, or you lock yourself out. A firewall hides services; it is not a replacement for not running them, so also check what is listening with ss -ltnp and shut down what you do not need. Container engines add their own rules, and publishing a port with Docker can bypass ufw, which surprises people who assumed the firewall covered it.

Where distributions differ

Most of this guide works on any Linux. A few areas depend on the family, and knowing the map avoids a confusing hour. Debian, Ubuntu and their derivatives use apt and dpkg, ufw for firewalls, AppArmor for confinement, and keep network configuration in netplan or /etc/network/interfaces. Fedora, RHEL, Rocky and Alma use dnf and rpm, firewalld, SELinux and NetworkManager. Alpine uses apk, BusyBox utilities and OpenRC. Arch uses pacman and a rolling release.

The commands in the rest of the guide behave the same on all of them. When you write a script that must run across families, wrap the differences in functions and test on each, following the portable shell guide.

Mandatory access control in brief

Standard permissions let the owner decide who can access a file. Mandatory access control adds a policy the owner cannot override: a confined process may touch only what the policy names, even when running as root. SELinux labels every file and process and checks the labels against rules. AppArmor confines programs by path with profiles. Both stop a compromised service from reading unrelated files, and both are a regular cause of "permission denied" with correct modes.

getenforce
ls -Z /var/www/html | head -3
sudo ausearch -m avc -ts recent | tail -5
sudo aa-status | head -5
sudo journalctl -k | grep -i apparmor | tail -3

When a service fails with permission errors and the modes look right, check these logs before changing modes or disabling the policy. Switching SELinux to permissive mode to test is reasonable; leaving it off is giving up a layer of defence for the sake of a missing label, and restorecon or a one-line policy boolean usually fixes the real problem.

Containers and namespaces, as seen from the shell

A container is an ordinary process with a restricted view, built from kernel namespaces for process IDs, mounts, networks and users, and cgroups for resource limits. Seeing that from the command line removes much of the mystery. Every process has its namespaces listed under /proc/PID/ns, and lsns lists them. A process inside a container appears on the host with a normal PID.

ls -l /proc/self/ns
lsns -t pid
docker top mycontainer
cat /proc/self/cgroup
systemd-cgls | head -15

This explains why host tools see container processes, why a container killed by the OOM killer leaves a message in the host's dmesg, and why files created in a bind-mounted directory have numeric owners that mean different users on each side. When you debug a container, docker exec or nsenter places you inside its namespaces with the host's tools still available if you use nsenter -t PID -n for the network only, which is a way to run ss or tcpdump in a container that has neither.

Recovering from common mistakes

Everyone eventually runs a command they regret. A few recoveries are worth knowing before you need them. A deleted file that a process still holds open can be copied back from /proc/PID/fd/N. A shell left with echo off after a crashed password prompt is fixed with stty sane or reset. A file with a name you cannot type, such as one that starts with a dash or holds control characters, is removed with rm -- ./-name or find . -inum N -delete, using the inode number from ls -i. A wrong chmod -R on a system directory is the hard one: restore modes from a package with dpkg --verify and apt install --reinstall, or from a backup.

sudo lsof +L1 | grep deleted
cp /proc/4242/fd/5 /tmp/recovered.log
ls -i
find . -maxdepth 1 -inum 131075 -delete
sudo dpkg --verify | head

The lesson in each is the same: stop writing to the disk, think, and copy before you change. A habit of taking a quick snapshot or copy before a risky bulk operation, with cp -a, an LVM or filesystem snapshot, or a version control commit of /etc, turns most of these from emergencies into inconveniences.

Where this does not apply

macOS and the BSDs. The commands share names but not always flags. BSD sed -i needs a suffix, BSD stat uses -f for formats, ps columns differ, and /proc does not exist. Install GNU coreutils from Homebrew, or check each page.

Alpine and BusyBox. Many commands are reduced applets with fewer options. find -printf, grep -P and long options may be missing. The portable shell guide lists the differences in more detail.

Containers and distroless images. There may be no shell, no ps, no package manager. Debug with an ephemeral debug container or by mounting tools from another image, and do not assume the host's commands exist inside.

Large-scale operations. One-line pipelines are fine for a server, but for hundreds of hosts, use configuration management or a proper orchestrator. Commands that are reliable on one machine become unmanageable across many.

What to learn after this

The command line rewards depth in a few tools over breadth across many. After the basics here, the highest-return topics are text processing, remote work and automation. The grep, sed and awk guide and the regular expressions guide cover text. The SSH guide covers remote work. The cron and systemd timers guide covers running things on a schedule, and the shell script security guide covers doing all of it safely.

Primary sources used here: the GNU coreutils manual, the Bash Reference Manual, the GNU findutils manual, the GNU grep manual, the GNU tar manual, the Filesystem Hierarchy Standard, the signal manual page, the ss manual page and the systemctl manual page.