Linux & Shell Fundamentals

Command line mastery, bash scripting basics, and essential system administration skills

Why Master the Linux Shell?

The command line is the most powerful interface to Linux systems, the foundation of servers, containers, Kubernetes, cloud instances, and automation. Mastering the shell lets you troubleshoot faster, automate everything, write infrastructure as code, and operate production systems confidently.

Linux File System Hierarchy

Understanding where things live in Linux is fundamental. Everything in Linux is a file, even devices and processes.

LINUX FILE SYSTEM STRUCTURE

/                      (Root - everything starts here)
│
├── bin/               Essential user commands (ls, cat, cp)
├── sbin/              System admin commands (reboot, ifconfig)
├── etc/               Configuration files (nginx.conf, hosts)
├── home/              User home directories (/home/username)
│   └── username/      Your personal files and configs
├── root/              Root user's home directory
├── var/               Variable data (logs, databases, caches)
│   ├── log/           System and application logs
│   └── www/           Web server files
├── tmp/               Temporary files (cleared on reboot)
├── usr/               User programs and data
│   ├── bin/           User commands
│   ├── local/         Locally installed software
│   └── share/         Shared data
├── opt/               Optional/third-party software
├── dev/               Device files (hard drives, USB)
├── proc/              Process and kernel information
└── mnt/               Mount points for external drives

Key Locations for Engineers:
/etc/, All config files live here
/var/log/, Logs for debugging
/home/user/, Your personal workspace
/tmp/, Safe space for temp files (auto-cleaned)

Navigation Basics, Moving Around

Before doing anything, you need to know where you are and how to move around.

Command	What It Does	Example
`pwd`	Print Working Directory, shows where you are	`pwd` → `/home/user`
`cd [dir]`	Change Directory, move to a directory	`cd /var/log`
`cd ..`	Go up one directory level	`cd ..`
`cd ~`	Go to home directory	`cd ~` → `/home/user`
`cd -`	Go to previous directory	`cd -`

Listing Files, ls Progression

# Basic listing
$ ls
Documents  Downloads  Pictures  projects

# Long format with details
$ ls -l
drwxr-xr-x  5 user user 4096 Jan 15 10:30 Documents
drwxr-xr-x  3 user user 4096 Jan 14 09:15 Downloads
drwxr-xr-x  2 user user 4096 Jan 10 14:20 Pictures

# Show all files (including hidden)
$ ls -la
total 48
drwxr-xr-x 12 user user 4096 Jan 19 12:00 .
drwxr-xr-x  8 root root 4096 Dec 10 08:30 ..
-rw-------  1 user user 3842 Jan 19 11:45 .bash_history
-rw-r--r--  1 user user  220 Jan 01 10:00 .bash_logout
drwxr-xr-x  5 user user 4096 Jan 15 10:30 Documents

# Human-readable sizes
$ ls -lah
-rw-r--r-- 1 user user  15M Jan 19 12:00 large_file.zip
-rw-r--r-- 1 user user 2.3K Jan 18 09:30 config.yaml

Pro-Tip: Hidden files start with a dot (.). Config files like .bashrc and .gitconfig are hidden. Use ls -la to reveal them.

Absolute vs Relative Paths

Absolute Path

Starts from root (/). Works from anywhere.

cd /var/log/nginx

Always starts with /

Relative Path

Relative to current location. Shorter but context-dependent.

cd projects/my-app

No leading /

File Viewing, Reading Content

Different tools for different viewing needs, from quick peeks to scrolling through logs.

Command	When to Use	Example
`cat`	View entire file (small files)	`cat config.yaml`
`head`	View first 10 lines	`head -n 20 app.log`
`tail`	View last 10 lines (logs!)	`tail -n 50 error.log`
`tail -f`	Follow file in real-time (live logs)	`tail -f /var/log/nginx/access.log`
`less`	Scroll through large files (use arrows, q to quit)	`less application.log`
`more`	Older version of less (still useful)	`more file.txt`

# View last 100 lines of error log
$ tail -n 100 /var/log/nginx/error.log

# Follow log in real-time (Ctrl+C to stop)
$ tail -f /var/log/application/app.log
2026-01-19 12:34:56 INFO  Server started on port 3000
2026-01-19 12:35:02 DEBUG Database connection established
2026-01-19 12:35:15 ERROR Failed to fetch user data
^C

# View file with scrolling (press 'q' to quit)
$ less huge_log_file.log

Pro-Tip: tail -f is essential for watching logs in production. Use it to debug issues in real-time.

File Operations, Creating, Copying, Moving

Basic file manipulation, the building blocks of shell scripting and automation.

Command	Purpose	Example
`touch`	Create empty file or update timestamp	`touch newfile.txt`
`mkdir`	Create directory	`mkdir my-project`
`mkdir -p`	Create nested directories	`mkdir -p /var/www/app/logs`
`cp`	Copy file	`cp config.yaml config.backup.yaml`
`cp -r`	Copy directory recursively	`cp -r /src /backup`
`mv`	Move or rename file	`mv oldname.txt newname.txt`
`rm`	Remove file	`rm unwanted.log`
`rm -r`	Remove directory recursively	`rm -r old_project/`

# Create backup of config before editing
$ cp nginx.conf nginx.conf.backup

# Copy entire project directory
$ cp -r my-project my-project-backup

# Rename file
$ mv draft.md final-report.md

# Move file to different directory
$ mv report.pdf ~/Documents/

# Create nested directory structure
$ mkdir -p ~/projects/web-app/src/components

Wildcards, Pattern Matching

Wildcard	Matches	Example
`*`	Any characters (zero or more)	`*.log` → all .log files
`?`	Single character	`file?.txt` → file1.txt, fileA.txt
`[abc]`	Any character in brackets	`file[123].txt` → file1.txt, file2.txt
`[a-z]`	Range of characters	`[A-Z]*.pdf` → files starting with capitals

# Delete all .log files
$ rm *.log

# Copy all .txt files to backup directory
$ cp *.txt ~/backup/

# List all files starting with 'test'
$ ls test*

# Remove all numbered backup files (backup1, backup2, etc.)
$ rm backup[0-9]

Danger: No Undo!

Linux has NO trash/recycle bin for command line operations. rm is permanent. Always double-check before hitting enter, especially with wildcards!

Finding Files, Locate Anything

Lost a config file? Need to find all .log files? These commands are your search party.

find, The Powerful Search Tool

# Find by name
$ find /var/log -name "*.log"
/var/log/nginx/access.log
/var/log/nginx/error.log
/var/log/auth.log

# Find files modified in last 7 days
$ find /home/user -type f -mtime -7

# Find large files (over 100MB)
$ find / -type f -size +100M

# Find and delete old log files (older than 30 days)
$ find /var/log -name "*.log" -mtime +30 -delete

# Find files by permission
$ find /var/www -type f -perm 777

grep, Search Inside Files

# Search for text in a file
$ grep "error" application.log

# Case-insensitive search
$ grep -i "Error" application.log

# Recursive search in all files
$ grep -r "TODO" /home/user/projects

# Show line numbers
$ grep -n "function" script.js
15:function processData() {
42:function validateInput() {

# Search for whole word only
$ grep -w "test" file.txt

# Invert match (show lines that DON'T contain pattern)
$ grep -v "DEBUG" app.log

Real-world example: Find all TODO comments in your codebase:
grep -r "TODO" ~/projects/my-app/src

Text Editors, Editing Files in the Terminal

You'll need to edit config files on servers. Here are the essential editors.

nano, Beginner-Friendly Editor

# Open file for editing
$ nano config.yaml

# Common keyboard shortcuts (shown at bottom):
Ctrl+O    Save (WriteOut)
Ctrl+X    Exit
Ctrl+K    Cut line
Ctrl+U    Paste
Ctrl+W    Search
Ctrl+G    Help

When to use nano: Quick config edits, beginner-friendly, server administration. It shows all shortcuts at the bottom, no need to memorize!

vim, The Power Editor (Survival Guide)

vim is powerful but has a learning curve. Here's how to survive your first encounter:

# Open file
$ vim file.txt

# vim has MODES. You start in NORMAL mode.

━━━ SURVIVAL COMMANDS ━━━
:q          Quit (if no changes)
:q!         Quit without saving (force quit)
:wq         Write and quit (save and exit)
:w          Write (save)

━━━ EDITING ━━━
i           Enter INSERT mode (now you can type!)
Esc         Return to NORMAL mode
dd          Delete current line
yy          Copy current line
p           Paste
u           Undo
/text       Search for "text"
n           Next search result

━━━ THE GOLDEN RULE ━━━
Press 'Esc' first, then type commands!

Stuck in vim?

Press Esc multiple times, then type :q! and press Enter. This force-quits without saving. Don't worry, everyone gets stuck in vim at first!

Permissions, Who Can Do What

Linux security starts with file permissions. Understanding this is critical for security and troubleshooting.

Understanding Permission Structure

PERMISSION BREAKDOWN

-rwxr-xr--  1 user  group  4096 Jan 19 12:00 script.sh
│││││││││
││││││││└─ Others permissions (everyone else)
│││││└┴┴── Group permissions (users in the group)
│││└┴┴──── Owner permissions (file owner)
││└─────── Number of hard links
│└──────── File type (- = file, d = directory, l = link)

Permission bits:
r = read    (4)  → Can view file content
w = write   (2)  → Can modify file
x = execute (1)  → Can run as program

Examples:
rwx = 7  (4+2+1)  → Read, Write, Execute
rw- = 6  (4+2+0)  → Read, Write
r-x = 5  (4+0+1)  → Read, Execute
r-- = 4  (4+0+0)  → Read only
--- = 0  (0+0+0)  → No permissions

chmod, Change Permissions (Octal Notation)

chmod Value	Permissions	Common Use Case
`chmod 755`	rwxr-xr-x	Scripts, executables (owner can modify, others can run)
`chmod 644`	rw-r--r--	Regular files, configs (owner can edit, others can read)
`chmod 600`	rw-------	Private files, API keys (only owner can read/write)
`chmod 400`	r--------	SSH private keys (only owner can read, nobody can modify)
`chmod 777`	rwxrwxrwx	⚠️ DANGEROUS! Everyone can do everything (avoid!)

# Make script executable
$ chmod +x deploy.sh
$ ./deploy.sh

# Secure SSH key (required by SSH)
$ chmod 400 ~/.ssh/id_rsa

# Standard file permissions
$ chmod 644 config.yaml

# Make directory accessible
$ chmod 755 /var/www/html

# Recursive permission change (be careful!)
$ chmod -R 755 /var/www/app

chmod, Symbolic Notation (Alternative)

# Add execute permission for owner
$ chmod u+x script.sh

# Remove write permission for group
$ chmod g-w file.txt

# Add read permission for others
$ chmod o+r document.pdf

# Give everyone read and execute
$ chmod a+rx public_script.sh

# u = user (owner), g = group, o = others, a = all
# + = add, - = remove, = = set exactly

chown, Change Ownership

# Change owner
$ sudo chown nginx /var/www/html/index.html

# Change owner and group
$ sudo chown user:group file.txt

# Recursive ownership change
$ sudo chown -R www-data:www-data /var/www/app

# Change only group
$ sudo chgrp developers project/

Real-world example: After deploying code, you often need to ensure web server (nginx/apache) can read files:
sudo chown -R www-data:www-data /var/www/myapp
sudo chmod -R 755 /var/www/myapp

Text Processing & Pipes, Command Chaining Power

The real power of the shell: chain small commands into powerful data processors.

The Power of Pipes

Command	What it does in this pipeline
`cat access.log`	Reads the file and sends the entire stream of text to the next command
`grep "/admin"`	Acts as a filter. Only allows lines containing "/admin" to pass through
`awk '{print $1}'`	Acts as a transformer. Extracts only the first column (IP address)
`sort`	Alphabetically orders the IP addresses (required for uniq to work properly)
`uniq -c`	The aggregator. Collapses duplicate lines and adds count prefix

FULL PIPELINE

$ cat access.log | grep "/admin" | awk '{print $1}' | sort | uniq -c | sort -nr

     42 192.168.1.105
     12 10.0.0.45
      5 172.16.0.1

Result: Found which IPs accessed /admin most frequently, 192.168.1.105 hit it 42 times!

Essential Text Processing Tools

# wc - Word count, line count
$ wc -l access.log
10523 access.log       # 10,523 lines

# cut - Extract columns
$ cut -d',' -f1,3 data.csv    # Get columns 1 and 3 from CSV

# sed - Find and replace
$ sed 's/localhost/127.0.0.1/g' config.txt
$ sed -i 's/old/new/g' file.txt    # Replace in-place

# tr - Translate/delete characters
$ echo "HELLO" | tr '[:upper:]' '[:lower:]'
hello

# sort - Sort lines
$ sort names.txt
$ sort -r names.txt    # Reverse order
$ sort -n numbers.txt  # Numeric sort

# uniq - Remove duplicates (requires sorted input)
$ sort names.txt | uniq

Pro-Tip: Add | sort -nr at the end of pipelines to sort numbers highest-first. Essential for finding top consumers in logs.

Process Management, Controlling Running Programs

Everything running on Linux is a process. Learn to view, control, and kill them.

Viewing Running Processes

# View all running processes
$ ps aux
USER       PID  %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1   0.0  0.1 168820 12804 ?        Ss   Jan15   0:23 /sbin/init
user      1234   2.5  5.2 2847392 425364 ?      Sl   08:30   1:42 node server.js
user      5678   0.0  0.3  45892  28416 pts/0   R+   12:34   0:00 ps aux

# View processes in tree format
$ ps auxf

# View processes for specific user
$ ps -u username

# Real-time process viewer
$ top
# Press 'q' to quit, 'k' to kill a process

# Better top alternative (if installed)
$ htop

Understanding ps output:
PID = Process ID (use this to kill processes)
%CPU = CPU usage
%MEM = Memory usage
COMMAND = What's running

Killing Processes

Command	What It Does
`kill [PID]`	Politely ask process to terminate (SIGTERM)
`kill -9 [PID]`	Force kill immediately (SIGKILL), last resort!
`killall [name]`	Kill all processes with this name
`pkill [pattern]`	Kill processes matching pattern

# Find process ID
$ ps aux | grep node
user  12345  2.5  5.2  node server.js

# Kill by PID (graceful)
$ kill 12345

# Force kill if it won't die
$ kill -9 12345

# Kill all node processes
$ killall node

# Kill process by pattern
$ pkill -f "python app.py"

Background Jobs & Job Control

# Run command in background
$ long-running-script.sh &
[1] 23456

# View background jobs
$ jobs
[1]+  Running    long-running-script.sh &

# Bring job to foreground
$ fg %1

# Send running job to background
# 1. Press Ctrl+Z (suspend)
# 2. Type: bg

# Run command that continues after logout
$ nohup python server.py &
$ exit    # Server keeps running!

# Check if still running
$ ps aux | grep server.py

When to background processes: Long tasks (backups, data processing), development servers you want to keep running while using the terminal.

10. System Monitoring, Health Checks

When systems slow down or fail, these commands tell you what's happening.

Disk Space Monitoring

# Check disk space
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       100G   85G   10G  90% /
/dev/sdb1       500G  120G  350G  26% /data

# Check directory sizes
$ du -sh */
2.5G    Documents/
850M    Downloads/
15G     projects/
120K    scripts/

# Find largest directories
$ du -sh /* | sort -rh | head -10

# Check disk usage with alert
$ df -h | awk '$5+0 > 80 {print "WARNING: " $1 " is at " $5}'

Memory Monitoring

# Check memory usage
$ free -h
              total        used        free      shared  buff/cache   available
Mem:           16Gi       8.2Gi       2.1Gi       523Mi       5.7Gi       7.2Gi
Swap:         8.0Gi       1.2Gi       6.8Gi

# Memory by process (top 10)
$ ps aux --sort=-%mem | head -11

CPU & System Load

# System uptime and load average
$ uptime
12:45:32 up 23 days,  3:15,  2 users,  load average: 1.25, 0.98, 0.85
                                                      └──1min──5min──15min

# Load average meaning (relative to CPU cores):
# < num_cores  = System has capacity
# = num_cores  = System fully utilized (e.g., 4.0 on 4-core)
# > num_cores  = Tasks queuing for CPU (potential issue)
# Check cores: nproc

# System information
$ uname -a
Linux hostname 5.15.0-91-generic #101-Ubuntu SMP x86_64 GNU/Linux

# Who's logged in
$ who
user1    pts/0    2026-01-19 08:30 (192.168.1.100)
user2    pts/1    2026-01-19 10:15 (10.0.0.50)

Health check script example:
Check if disk >90%, memory >90%, or load >4.0 → send alert

11. Redirection & I/O, Controlling Input/Output

Redirect command output to files, chain commands, and handle errors like a pro.

Operator	Purpose	Example
`>`	Redirect output (overwrite file)	`ls > files.txt`
`>>`	Redirect output (append to file)	`echo "log" >> app.log`
`<`	Redirect input (read from file)	`sort < names.txt`
`2>`	Redirect errors only	`command 2> errors.log`
`&>`	Redirect both output and errors	`script.sh &> output.log`
`\|`	Pipe (send output to another command)	`cat file.txt \| grep "error"`

# Save command output to file (overwrite)
$ ls -la > directory_listing.txt

# Append to file
$ date >> build.log
$ echo "Build completed" >> build.log

# Redirect errors to file
$ make build 2> errors.log

# Redirect both output and errors
$ ./deploy.sh &> deploy.log

# Discard output (send to /dev/null black hole)
$ noisy-command > /dev/null 2>&1

# Separate output and errors
$ command > output.log 2> errors.log

# Real-world example: Save successful output, show errors
$ backup.sh > backup.log 2>&1
$ cat backup.log

Pro-Tip: Use > /dev/null 2>&1 to silence commands completely. Useful for cron jobs where you don't want email spam.

12. Archives & Compression, Packaging Files

tar is everywhere, backups, downloads, deployments. Master these patterns.

tar, The Archive Tool

Command	Purpose
`tar -czf archive.tar.gz folder/`	Create compressed archive
`tar -xzf archive.tar.gz`	Extract compressed archive
`tar -tzf archive.tar.gz`	List contents without extracting
`tar -xzf archive.tar.gz -C /path/`	Extract to specific directory

# Create backup of project
$ tar -czf project-backup-$(date +%Y%m%d).tar.gz ~/projects/my-app/
# Creates: project-backup-20260119.tar.gz

# Extract archive
$ tar -xzf project-backup-20260119.tar.gz

# List what's inside
$ tar -tzf backup.tar.gz

# Extract to specific location
$ tar -xzf archive.tar.gz -C /tmp/

# Create archive excluding files
$ tar -czf backup.tar.gz --exclude='*.log' --exclude='node_modules' project/

# Mnemonic for flags:
# c = create, x = extract, t = list
# z = gzip compression
# f = file (must be last before filename)
# v = verbose (show progress)

Common pattern: Backup with date:
tar -czf backup-$(date +%Y%m%d-%H%M).tar.gz important-folder/

13. Package Management, Installing Software

Different Linux distributions use different package managers. Here's how they work.

apt, Ubuntu/Debian Package Manager

# Update package list (do this first!)
$ sudo apt update

# Upgrade all packages
$ sudo apt upgrade

# Install package
$ sudo apt install nginx

# Install multiple packages
$ sudo apt install curl git htop vim

# Remove package
$ sudo apt remove nginx

# Remove package and config files
$ sudo apt purge nginx

# Search for packages
$ apt search postgresql

# Show package info
$ apt show nginx

# Clean up unused packages
$ sudo apt autoremove

Update vs Upgrade:
apt update, Downloads package lists (what's available)
apt upgrade, Actually installs updates
Always run apt update before apt install!

Other Package Managers

yum/dnf (RedHat/CentOS/Fedora)

sudo yum update
sudo yum install nginx
sudo dnf install nginx

brew (macOS)

brew update
brew install nginx
brew upgrade

14. Environment Variables & Shell Configuration

Customize your shell, set variables, and configure your environment.

Environment Variables

# View all environment variables
$ env

# View specific variable
$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

$ echo $HOME
/home/user

# Set variable for current session
$ export MY_VAR="hello world"
$ echo $MY_VAR
hello world

# Add to PATH (temporary)
$ export PATH=$PATH:/new/directory

# Common important variables
$ echo $USER        # Current username
$ echo $SHELL       # Current shell
$ echo $PWD         # Current directory
$ echo $OLDPWD      # Previous directory

Shell Configuration Files

File	Purpose
`~/.bashrc`	Runs for every new interactive shell (aliases, functions)
`~/.bash_profile`	Runs once at login (environment variables, PATH)
`~/.profile`	Similar to bash_profile (more universal)
`~/.bash_aliases`	Store aliases separately (optional, good practice)

# Add to ~/.bashrc

# Custom aliases
alias ll='ls -lah'
alias gs='git status'
alias gp='git pull'
alias dc='docker-compose'

# Custom functions
mkcd() {
    mkdir -p "$1" && cd "$1"
}

# Add directory to PATH
export PATH="$HOME/bin:$PATH"

# Custom prompt (optional)
export PS1="\u@\h:\w\$ "

# After editing ~/.bashrc, reload it:
$ source ~/.bashrc

# Or
$ . ~/.bashrc

Pro-Tip: Create aliases for commands you use frequently. alias k=kubectl saves tons of typing!

Key Takeaways

File System: Everything lives under /, know where configs, logs, and programs live
Navigation: pwd, cd, ls are your compass, master absolute vs relative paths
File Operations: cat, head, tail, cp, mv, find, view and manipulate files efficiently
Editors: nano for quick edits, vim for power (know how to exit!)
Permissions: rwx, chmod, chown, control who can do what
Text Processing: Pipes and grep turn logs into insights
Processes: ps, top, kill, monitor and control running programs
System Health: df, free, uptime, catch problems before they crash systems
I/O Redirection: >, >>, |, control where output goes
Archives: tar for backups and deployments
Packages: apt/yum for installing software
Environment: Customize your shell with .bashrc and aliases

Back to "Git & Version Control"

Infrastructure for EngineersLesson 2

Next: "Networking Essentials"