Linux Administration
Comprehensive Linux system administration for managing servers, deploying applications, and troubleshooting production issues in modern cloud-native environments.
Purpose
This skill teaches fundamental and intermediate Linux administration for DevOps engineers, SREs, backend developers, and platform engineers. Focus on systemd-based distributions (Ubuntu, RHEL, Debian, Fedora) covering service management, process monitoring, filesystem operations, user administration, performance tuning, log analysis, and network configuration.
Modern infrastructure requires solid Linux fundamentals even with containerization. Container hosts run Linux, Kubernetes nodes need optimization, and troubleshooting production issues requires understanding systemd, processes, and logs.
Not Covered:
- Advanced networking (BGP, OSPF) - see
network-architectureskill - Deep security hardening (compliance, pentesting) - see
security-hardeningskill - Configuration management at scale (Ansible, Puppet) - see
configuration-managementskill - Container orchestration - see
kubernetes-operationsskill
When to Use This Skill
Use when deploying custom applications, troubleshooting slow systems, investigating service failures, optimizing workloads, managing users, configuring SSH, monitoring disk space, scheduling tasks, diagnosing network issues, or applying performance tuning.
Quick Start
Essential Commands
Service Management:
systemctl start nginx # Start service
systemctl stop nginx # Stop service
systemctl restart nginx # Restart service
systemctl status nginx # Check status
systemctl enable nginx # Enable at boot
journalctl -u nginx -f # Follow service logs
Process Monitoring:
top # Interactive process monitor
htop # Enhanced process monitor
ps aux | grep process_name # Find specific process
kill -15 PID # Graceful shutdown (SIGTERM)
kill -9 PID # Force kill (SIGKILL)
Disk Usage:
df -h # Filesystem usage
du -sh /path/to/dir # Directory size
ncdu /path # Interactive disk analyzer
Log Analysis:
journalctl -f # Follow all logs
journalctl -u service -f # Follow service logs
journalctl --since "1 hour ago" # Filter by time
journalctl -p err # Show errors only
User Management:
useradd -m -s /bin/bash username # Create user with home dir
passwd username # Set password
usermod -aG sudo username # Add to sudo group
userdel -r username # Delete user and home dir
Core Concepts
Systemd Architecture
Systemd is the standard init system and service manager. Systemd units define services, timers, targets, and other system resources.
Unit File Locations (priority order):
/etc/systemd/system/- Custom units (highest priority)/run/systemd/system/- Runtime units (transient)/lib/systemd/system/- System-provided units (don't modify)
Key Unit Types: .service (services), .timer (scheduled tasks), .target (unit groups), .socket (socket-activated)
Essential systemctl Commands:
systemctl daemon-reload # Reload unit files after changes
systemctl list-units --type=service
systemctl list-timers # Show all timers
systemctl cat nginx.service # Show unit file content
systemctl edit nginx.service # Create override file
For detailed systemd reference, see references/systemd-guide.md.
Process Management
Processes are running programs with unique PIDs. Understanding process states, signals, and resource usage is essential for troubleshooting.
Process States: R (running), S (sleeping), D (uninterruptible sleep/I/O), Z (zombie), T (stopped)
Common Signals: SIGTERM (15) graceful, SIGKILL (9) force, SIGHUP (1) reload config
Process Priority:
nice -n 10 command # Start with lower priority
renice -n 5 -p PID # Change priority of running process
Filesystem Hierarchy
Essential directories: / (root), /etc/ (config), /var/ (variable data), /opt/ (optional software), /usr/ (user programs), /home/ (user directories), /tmp/ (temporary), /boot/ (boot loader)
Filesystem Types Quick Reference:
- ext4 - General purpose (default)
- XFS - Large files, databases (RHEL default)
- Btrfs - Snapshots, copy-on-write
- ZFS - Enterprise, data integrity, NAS
For filesystem management details including LVM and RAID, see references/filesystem-management.md.
Package Management
Ubuntu/Debian (apt):
apt update && apt upgrade # Update system
apt install package # Install package
apt remove package # Remove package
apt search keyword # Search packages
RHEL/CentOS/Fedora (dnf):
dnf update # Update all packages
dnf install package # Install package
dnf remove package # Remove package
dnf search keyword # Search packages
Use native package managers for system services; snap/flatpak for desktop apps and cross-distro compatibility.
Decision Frameworks
Troubleshooting Performance Issues
Investigation Workflow:
-
Identify bottleneck:
top # Quick overview uptime # Load averages -
CPU Issues (usage >80%):
top # Press Shift+P to sort by CPU ps aux --sort=-%cpu | head -
Memory Issues (swap used):
free -h # Memory usage top # Press Shift+M to sort by memory -
Disk I/O Issues (high wa%):
iostat -x 1 # Disk statistics iotop # I/O by process -
Network Issues:
ss -tunap # Active connections iftop # Bandwidth monitor
For comprehensive troubleshooting, see references/troubleshooting-guide.md.
Filesystem Selection
Quick Decision:
- Default/General → ext4
- Database servers → XFS
- Large file storage → XFS or ZFS
- NAS/File server → ZFS
- Need snapshots → Btrfs or ZFS
Common Workflows
Creating a Systemd Service
Step 1: Create unit file
sudo nano /etc/systemd/system/myapp.service
Step 2: Unit file content
[Unit]
Description=My Web Application
After=network.target postgresql.service
Requires=postgresql.service
[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
Environment="PORT=8080"
ExecStart=/opt/myapp/bin/server
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s
StandardOutput=journal
# Security hardening
PrivateTmp=true
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/var/lib/myapp
[Install]
WantedBy=multi-user.target
Step 3: Deploy and start
sudo useradd -r -s /bin/false myapp
sudo mkdir -p /var/lib/myapp
sudo chown myapp:myapp /var/lib/myapp
sudo systemctl daemon-reload
sudo systemctl enable myapp.service
sudo systemctl start myapp.service
sudo systemctl status myapp.service
For complete examples, see examples/systemd-units/.
Systemd Timer (Cron Replacement)
Create service and timer units for scheduled tasks. Timer unit specifies OnCalendar= schedule and Persistent=true for missed jobs. Service unit has Type=oneshot. See examples/systemd-units/backup.timer and backup.service for complete examples.
SSH Hardening
Generate SSH key:
ssh-keygen -t ed2551