Linux System Administration.
Cheat sheet.

Comprehensive guide to Linux system administration covering essential commands, file systems, user management, networking, and security. Master the command line, system services, performance tuning, and troubleshooting. Fundamental knowledge for managing production Linux servers and demonstrating sysadmin expertise.

Essential Commands

pwd                 # Print working directory
ls -la              # List all files with details
cd /path/to/dir     # Change directory
cd ..               # Go up one directory
cd ~                # Go to home directory
mkdir -p dir1/dir2  # Create nested directories
rmdir dirname       # Remove empty directory
rm -rf dirname      # Remove directory and contents
cp -r src dest      # Copy recursively
mv old new          # Move/rename files
touch file.txt      # Create empty file
cat file.txt        # Display file content
less file.txt       # View file page by page
head -n 10 file     # Show first 10 lines
tail -f logfile     # Follow log file in real-time

File Operations

find /path -name "*.log"              # Find files by name
find /path -mtime -7                  # Files modified in last 7 days
find /path -size +100M                # Files larger than 100MB
grep -r "pattern" /path               # Search text recursively
grep -i "pattern" file                # Case-insensitive search
sed -i 's/old/new/g' file            # Replace text in file
awk '{print $1,$3}' file             # Print columns 1 and 3
sort file.txt | uniq                  # Sort and remove duplicates
wc -l file.txt                       # Count lines

File System Hierarchy

/         # Root directory
/bin      # Essential user binaries
/etc      # System configuration files
/home     # User home directories
/var      # Variable data (logs, caches)
/usr      # User programs and data
/tmp      # Temporary files
/dev      # Device files
/proc     # Process and kernel information
/opt      # Optional software packages

Permissions

# Permission format: rwxrwxrwx (owner|group|others)
# r=4, w=2, x=1

chmod 755 file      # rwxr-xr-x
chmod u+x file      # Add execute for owner
chmod g-w file      # Remove write for group
chmod -R 644 dir    # Recursive permission change
chown user:group file   # Change ownership
chown -R user dir       # Recursive ownership change
umask 022              # Set default permissions (755 for dirs, 644 for files)

Special Permissions

chmod u+s file      # SUID - Run as file owner
chmod g+s dir       # SGID - New files inherit group
chmod +t dir        # Sticky bit - Only owner can delete

User Management

useradd -m -s /bin/bash username     # Create user with home dir
usermod -aG sudo username            # Add user to sudo group
userdel -r username                  # Delete user and home dir
passwd username                      # Set/change password
su - username                        # Switch user
sudo command                         # Run as root
whoami                              # Current username
id username                         # User and group info

Group Management

groupadd groupname          # Create group
groupdel groupname          # Delete group
groups username             # Show user's groups
gpasswd -a user group      # Add user to group
gpasswd -d user group      # Remove user from group

/etc Files

/etc/passwd     # User accounts
/etc/shadow     # Encrypted passwords
/etc/group      # Group information
/etc/sudoers    # Sudo permissions
/etc/skel       # Default files for new users

Process Commands

ps aux                      # Show all processes
ps -ef                      # Full format listing
top                         # Interactive process viewer
htop                        # Enhanced process viewer
pgrep process_name          # Find process ID by name
kill PID                    # Terminate process
kill -9 PID                 # Force kill process
killall process_name        # Kill all processes by name
pkill -f pattern           # Kill processes matching pattern

Process Control

command &                   # Run in background
jobs                       # List background jobs
fg %1                      # Bring job 1 to foreground
bg %1                      # Resume job 1 in background
nohup command &            # Run immune to hangups
nice -n 10 command         # Run with lower priority
renice -n 5 -p PID        # Change process priority

System Information

uptime                     # System uptime and load
free -h                    # Memory usage
df -h                      # Disk usage
du -sh /path              # Directory size
lscpu                     # CPU information
lsblk                     # Block devices
lsusb                     # USB devices
dmidecode                 # Hardware information

Debian/Ubuntu (APT)

apt update                     # Update package list
apt upgrade                    # Upgrade installed packages
apt install package            # Install package
apt remove package             # Remove package
apt purge package             # Remove package and config
apt search keyword            # Search packages
apt show package              # Show package info
apt autoremove               # Remove unused dependencies
dpkg -i package.deb          # Install local .deb file

RHEL/CentOS (YUM/DNF)

yum update                    # Update all packages
yum install package           # Install package
yum remove package            # Remove package
yum search keyword           # Search packages
yum info package             # Package information
yum clean all                # Clean cache
rpm -ivh package.rpm         # Install local RPM
rpm -qa | grep package       # List installed packages

Network Commands

ip addr show                  # Show IP addresses
ip route show                 # Show routing table
ifconfig                      # Legacy network interface config
ping -c 4 google.com         # Test connectivity
traceroute google.com        # Trace network path
netstat -tulpn               # Show listening ports
ss -tulpn                    # Modern socket statistics
nslookup domain.com          # DNS lookup
dig domain.com               # Detailed DNS lookup
curl -I https://site.com     # Get HTTP headers
wget https://site.com/file   # Download file

Network Configuration

# /etc/network/interfaces (Debian/Ubuntu)
auto eth0
iface eth0 inet static
    address 192.168.1.100
    netmask 255.255.255.0
    gateway 192.168.1.1

# /etc/sysconfig/network-scripts/ifcfg-eth0 (RHEL/CentOS)
DEVICE=eth0
BOOTPROTO=static
IPADDR=192.168.1.100
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
ONBOOT=yes

Firewall (iptables/firewalld)

# iptables
iptables -L                           # List rules
iptables -A INPUT -p tcp --dport 22 -j ACCEPT    # Allow SSH
iptables -A INPUT -j DROP            # Drop all other input
iptables-save > /etc/iptables/rules.v4           # Save rules

# firewalld
firewall-cmd --list-all              # Show configuration
firewall-cmd --add-port=80/tcp --permanent       # Open port
firewall-cmd --reload                # Apply changes

Service Commands

systemctl start service         # Start service
systemctl stop service          # Stop service
systemctl restart service       # Restart service
systemctl status service        # Check service status
systemctl enable service        # Enable at boot
systemctl disable service       # Disable at boot
systemctl daemon-reload         # Reload systemd configs
systemctl list-units           # List all units
journalctl -u service          # View service logs
journalctl -f                  # Follow system logs

Creating a Service

# /etc/systemd/system/myapp.service
[Unit]
Description=My Application
After=network.target

[Service]
Type=simple
User=myuser
WorkingDirectory=/opt/myapp
ExecStart=/usr/bin/python3 /opt/myapp/app.py
Restart=always

[Install]
WantedBy=multi-user.target

Disk Management

fdisk -l                    # List disk partitions
fdisk /dev/sdb             # Partition disk
mkfs.ext4 /dev/sdb1        # Format partition
mount /dev/sdb1 /mnt       # Mount partition
umount /mnt                # Unmount partition
blkid                      # Show block device UUIDs

/etc/fstab Example

# <device>                               <mount>  <type>  <options>       <dump> <pass>
UUID=abc123-def456                       /        ext4    defaults        0      1
UUID=789012-345678                       /home    ext4    defaults        0      2
/dev/sdb1                               /data    ext4    defaults,noatime 0      2

LVM (Logical Volume Management)

pvcreate /dev/sdb              # Create physical volume
vgcreate vg_data /dev/sdb      # Create volume group
lvcreate -L 10G -n lv_app vg_data    # Create logical volume
lvextend -L +5G /dev/vg_data/lv_app  # Extend volume
resize2fs /dev/vg_data/lv_app        # Resize filesystem

Script Structure

#!/bin/bash
# Script description

# Variables
NAME="Linux"
COUNT=5

# Conditionals
if [ "$NAME" = "Linux" ]; then
    echo "It's Linux!"
elif [ $COUNT -gt 3 ]; then
    echo "Count is greater than 3"
else
    echo "Other condition"
fi

# Loops
for i in {1..5}; do
    echo "Number: $i"
done

while [ $COUNT -gt 0 ]; do
    echo "Count: $COUNT"
    COUNT=$((COUNT-1))
done

# Functions
function greet() {
    echo "Hello, $1!"
}
greet "World"

# Command substitution
DATE=$(date +%Y-%m-%d)
FILES=`ls *.txt`

Common Script Patterns

# Check if running as root
if [ "$EUID" -ne 0 ]; then 
    echo "Please run as root"
    exit 1
fi

# Check if file exists
if [ -f "/path/to/file" ]; then
    echo "File exists"
fi

# Check if directory exists
if [ -d "/path/to/dir" ]; then
    echo "Directory exists"
fi

# Read user input
read -p "Enter your name: " username
echo "Hello, $username"

# Exit on error
set -e

# Debug mode
set -x

Monitoring Commands

top                         # Real-time process viewer
htop                        # Enhanced process viewer
vmstat 1                    # Virtual memory statistics
iostat -x 1                 # I/O statistics
sar -u 1 10                # CPU usage (10 samples)
mpstat -P ALL              # Per-CPU statistics
free -h                    # Memory usage
df -h                      # Disk usage
du -sh *                   # Directory sizes
netstat -i                 # Network interface statistics
iftop                      # Network bandwidth usage

Performance Analysis

# CPU
uptime                     # Load average
top -b -n 1               # Batch mode snapshot
ps aux --sort=-%cpu       # Sort by CPU usage

# Memory
free -m                   # Memory in MB
cat /proc/meminfo        # Detailed memory info
vmstat -s                # Memory statistics

# Disk I/O
iotop                    # I/O by process
iostat -d 1              # Disk statistics
hdparm -t /dev/sda       # Disk read speed test

# Network
nethogs                  # Bandwidth by process
iftop -i eth0           # Interface bandwidth
tcpdump -i eth0         # Packet capture

Important Log Files

/var/log/syslog          # System logs (Debian/Ubuntu)
/var/log/messages        # System logs (RHEL/CentOS)
/var/log/auth.log        # Authentication logs
/var/log/kern.log        # Kernel logs
/var/log/boot.log        # Boot logs
/var/log/dmesg           # Kernel ring buffer
/var/log/apache2/        # Apache logs
/var/log/nginx/          # Nginx logs
/var/log/mysql/          # MySQL logs

Log Analysis

tail -f /var/log/syslog              # Follow log in real-time
grep ERROR /var/log/syslog           # Search for errors
journalctl -xe                       # systemd journal
journalctl --since "1 hour ago"      # Recent logs
journalctl -u ssh                    # Service-specific logs
zgrep pattern /var/log/syslog.*.gz   # Search compressed logs

Log Rotation (/etc/logrotate.conf)

/var/log/myapp/*.log {
    daily
    rotate 7
    compress
    missingok
    notifempty
    create 0644 myuser mygroup
    postrotate
        systemctl reload myapp
    endscript
}

SSH Configuration

# /etc/ssh/sshd_config
Port 22
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
AllowUsers user1 user2

# SSH key management
ssh-keygen -t rsa -b 4096           # Generate SSH key
ssh-copy-id user@server             # Copy key to server
ssh-add ~/.ssh/id_rsa               # Add key to agent

SELinux/AppArmor

# SELinux
getenforce                          # Get SELinux status
setenforce 0                        # Set permissive mode
sestatus                           # Detailed status
restorecon -R /path                # Restore contexts

# AppArmor
aa-status                          # Check status
aa-enforce /path/to/profile        # Enforce profile
aa-complain /path/to/profile       # Complain mode

Security Tools

fail2ban-client status              # Check fail2ban
last                               # Last logins
lastlog                            # All user last login
w                                  # Who is logged in
chage -l username                  # Password aging info
auditctl -l                        # List audit rules

Backup Commands

# tar
tar -czf backup.tar.gz /path/to/dir    # Create compressed backup
tar -xzf backup.tar.gz                  # Extract backup
tar -tzf backup.tar.gz                  # List contents

# rsync
rsync -avz /source/ /destination/       # Local sync
rsync -avz -e ssh /source/ user@remote:/dest/   # Remote sync
rsync -avz --delete /source/ /dest/     # Mirror with deletion

# dd
dd if=/dev/sda of=/backup/disk.img     # Disk image backup
dd if=/dev/zero of=/swapfile bs=1G count=4      # Create swap file

Cron Jobs

crontab -e                 # Edit user crontab
crontab -l                 # List cron jobs
/etc/crontab              # System crontab

# Cron format: minute hour day month weekday command
0 2 * * * /backup/script.sh         # Daily at 2 AM
*/15 * * * * /monitor/check.sh      # Every 15 minutes
0 0 * * 0 /weekly/maintenance.sh    # Weekly on Sunday

System Issues

dmesg | tail -50           # Recent kernel messages
systemctl --failed         # Failed services
journalctl -p err          # Error messages
lsof +D /path             # Files open in directory
lsof -i :80               # Process using port 80
strace -p PID             # Trace system calls
tcpdump -i any port 80    # Capture HTTP traffic

Disk Issues

fsck /dev/sdb1            # File system check
smartctl -a /dev/sda      # SMART disk info
badblocks -v /dev/sdb     # Check for bad blocks
mount -o remount,rw /     # Remount root as read-write

Performance Issues

# High CPU
top -b -n 1 | head -20
ps aux --sort=-%cpu | head
mpstat 1 5

# High Memory
ps aux --sort=-%mem | head
vmstat 1 5
slabtop

# High I/O
iotop -b -n 1
iostat -x 1 5

Kernel Management

uname -r                   # Current kernel version
dpkg --list | grep linux-image    # Installed kernels (Debian)
rpm -qa | grep kernel             # Installed kernels (RHEL)
grub-mkconfig -o /boot/grub/grub.cfg    # Update GRUB

System Tuning

# /etc/sysctl.conf
vm.swappiness = 10                    # Reduce swap usage
net.ipv4.tcp_keepalive_time = 300    # TCP keepalive
fs.file-max = 100000                  # Max file descriptors

sysctl -p                             # Apply changes
ulimit -n 65535                       # Set file descriptor limit

Container Basics (Docker)

docker ps                             # List running containers
docker ps -a                          # List all containers
docker images                         # List images
docker run -d -p 80:80 nginx         # Run nginx container
docker exec -it container_id bash     # Enter container
docker logs container_id              # View logs
docker-compose up -d                  # Start services

Automation (Ansible Example)

# playbook.yml
---
- hosts: webservers
  tasks:
    - name: Install nginx
      apt:
        name: nginx
        state: present
    - name: Start nginx
      service:
        name: nginx
        state: started
        enabled: yes

Core Competencies

1. System Fundamentals

   • Boot process: BIOS/UEFI → GRUB → Kernel → systemd
   • File system hierarchy and FHS compliance
   • Process management and inter-process communication
   • Memory management and virtual memory

2. Command Line Mastery

   • Text processing: grep, sed, awk, cut, sort
   • File operations: find, xargs, rsync
   • System monitoring: top, htop, vmstat, iostat
   • Network tools: netstat, ss, tcpdump, nc

3. System Administration

   • User/group management and sudo configuration
   • Service management with systemd
   • Package management (apt/yum/dnf)
   • Storage management (LVM, RAID, file systems)

4. Security Hardening

   • SSH configuration and key management
   • Firewall rules (iptables/firewalld)
   • SELinux/AppArmor contexts
   • File permissions and special bits (SUID/SGID/sticky)

5. Performance Optimization

   • Kernel parameter tuning (sysctl)
   • I/O scheduling and disk optimization
   • Memory management and swap tuning
   • Network stack optimization

System Analysis

Q: How do you troubleshoot high load average?

• Check load average trend: uptime, w
• Identify CPU vs I/O wait: top, vmstat 1
• Find resource-heavy processes: ps aux --sort=-%cpu
• Check disk I/O: iotop, iostat -x 1
• Review system logs: journalctl -xe

File System Deep Dive

Q: Explain hard vs soft links with use cases

• Hard Link: Same inode, can't cross file systems, survives original deletion
  • Use case: Backup without duplication
• Soft Link: Different inode, can cross file systems, breaks if target deleted
  • Use case: Versioning, shortcuts

Process Management

Q: Difference between kill signals

• SIGTERM (15): Graceful shutdown, allows cleanup
• SIGKILL (9): Immediate termination, no cleanup
• SIGHUP (1): Reload configuration
• SIGSTOP (19): Pause process
• SIGCONT (18): Resume process

Performance Tuning

Q: Explain load average values

• Three values: 1-min, 5-min, 15-min averages
• Represents processes in running + uninterruptible sleep state
• Rule of thumb: Load < number of CPU cores = healthy
• High load with low CPU = I/O bottleneck

Advanced Topics

Q: How do you optimize a database server?

• Tune kernel parameters: vm.swappiness=10
• Adjust file descriptors: ulimit -n 65535
• Configure huge pages for memory
• Set I/O scheduler to deadline/noop for SSDs
• Implement NUMA affinity for multi-socket systems

Golden Rule: Linux administration is about understanding the system holistically. Master the fundamentals, automate repetitive tasks, monitor proactively, and always have a rollback plan. The command line is your most powerful tool - learn to wield it effectively.