linux-admin

When this skill is activated, always start your first response with the 🧢 emoji.

Linux Administration

A production-focused Linux administration skill covering shell scripting, service management, networking, and security hardening. This skill treats every Linux system as a production asset - configuration is explicit, changes are auditable, and security is a constraint from the start, not an afterthought. Designed for engineers who need to move confidently between writing a deploy script, debugging a network issue, and locking down a fresh server.

When to use this skill

Trigger this skill when the user:

• Writes or debugs a bash script (especially anything running in CI, cron, or production)
• Creates or modifies a systemd service, timer, socket, or target unit
• Configures or audits SSH daemon settings and access controls
• Debugs a networking issue (routing, DNS, firewall, port connectivity)
• Sets up or modifies iptables/nftables/ufw firewall rules
• Manages file permissions, ownership, ACLs, or setuid/setgid bits
• Monitors or investigates running processes (CPU, memory, open files, syscalls)
• Sets up cron jobs or scheduled tasks
• Manages disk space, log rotation, or filesystem mounts

Do NOT trigger this skill for:

• Container orchestration specifics (Kubernetes networking, Docker Compose config) - use a

Docker/K8s skill instead

• Cloud provider IAM, VPC routing, or managed service configuration - those are cloud

platform concerns, not OS-level Linux administration

Key principles

1. Principle of least privilege - Every process, user, and service should run with

the minimum permissions required. Use dedicated service accounts (not root), restrict file permissions to exactly what is needed, and audit sudo rules regularly.

1. Automate repeatable tasks - If you run a command twice, script it. Scripts should

be idempotent - running them again should produce the same result, not break things. Store scripts in version control.

1. Log everything that matters - Structured logs, audit logs (auditd), and systemd

journal entries are your incident response safety net. Log authentication events, privilege escalations, and configuration changes. Log rotation prevents disk exhaustion.

1. Immutable servers when possible - Prefer rebuilding servers from a known-good

image over patching in place. Use configuration management (Ansible, cloud-init) to define state declaratively. Manual "snowflake" servers drift and fail unpredictably.

1. Test in staging - Every script, service unit, and firewall rule change should be

validated in a non-production environment first. Use --dry-run, bash -n, and iptables --check to validate before applying.

Core concepts

File permissions

Linux permissions have three layers (owner, group, others) and three bits (read, write, execute). Octal notation is the authoritative form.

Octal   Symbolic   Meaning
 0       ---       no permissions
 1       --x       execute only
 2       -w-       write only
 4       r--       read only
 6       rw-       read + write
 7       rwx       read + write + execute

# Common patterns
chmod 600 ~/.ssh/id_rsa        # private key: owner read/write only
chmod 644 /etc/nginx/nginx.conf  # config: owner rw, others read
chmod 755 /usr/local/bin/script  # executable: owner rwx, others rx
chmod 700 /root/.gnupg           # directory: only owner can enter

Special bits:

• setuid (4xxx): executable runs as file owner, not caller. Dangerous on scripts.
• setgid (2xxx): new files in directory inherit group. Useful for shared dirs.
• sticky (1xxx): only file owner can delete in a directory (e.g., /tmp).

Process management

Key signals for process control:

| Signal | Number | Meaning | |---|---|---| | SIGTERM | 15 | Polite shutdown - process should clean up | | SIGKILL | 9 | Immediate kill - kernel enforced, unblockable | | SIGHUP | 1 | Reload config (many daemons re-read on SIGHUP) | | SIGINT | 2 | Interrupt (Ctrl+C) | | SIGUSR1/2 | 10/12 | Application-defined |

niceness runs from -20 (highest priority) to 19 (lowest). Use nice -n 10 cmd for background tasks and renice to adjust running processes.

systemd unit hierarchy

Targets (grouping)         -> multi-user.target, network.target
  Services (.service)      -> long-running daemons, oneshot tasks
  Timers (.timer)          -> scheduled execution (replaces cron)
  Sockets (.socket)        -> socket-activated services
  Mounts (.mount)          -> filesystem mounts managed by systemd
  Paths (.path)            -> filesystem change triggers

Dependency directives: Requires= (hard), Wants= (soft), After= (ordering only). After=network-online.target is the correct way to wait for network connectivity.

Networking stack

Key tools and their roles:

| Tool | Layer | Purpose | |---|---|---| | ip addr / ip link | L2/L3 | Interface state, IP addresses, routes | | ip route | L3 | Routing table inspection and management | | ss -tulpn | L4 | Listening ports, socket state, owning process | | iptables -L -n -v | L3/L4 | Firewall rules, packet counts | | dig / resolvectl | DNS | Name resolution debugging | | traceroute / mtr | L3 | Path tracing, hop-by-hop latency | | tcpdump | L2-L7 | Packet capture for deep inspection |

Common tasks

Write a robust bash script

Always use the safety triplet at the top of every non-trivial script.

#!/usr/bin/env bash
set -euo pipefail
# -e: exit on error
# -u: treat unset variables as errors
# -o pipefail: pipeline fails if any command in it fails

# Cleanup on exit - runs on success, error, and signals
TMPDIR_WORK=""
cleanup() {
    local exit_code=$?
    [[ -n "$TMPDIR_WORK" ]] && rm -rf "$TMPDIR_WORK"
    exit "$exit_code"
}
trap cleanup EXIT INT TERM

# Argument parsing with defaults and validation
usage() {
    echo "Usage: $0 [-e ENV] [-d] <target>"
    echo "  -e ENV   Environment (default: staging)"
    echo "  -d       Dry-run mode"
    exit 1
}

ENV="staging"
DRY_RUN=false

while getopts ":e:dh" opt; do
    case $opt in
        e) ENV="$OPTARG" ;;
        d) DRY_RUN=true ;;
        h) usage ;;
        :) echo "Option -$OPTARG requires an argument." >&2; usage ;;
        \?) echo "Unknown option: -$OPTARG" >&2; usage ;;
    esac
done
shift $((OPTIND - 1))

[[ $# -lt 1 ]] && { echo "Error: target required" >&2; usage; }
TARGET="$1"

# Use mktemp for safe temp directories
TMPDIR_WORK=$(mktemp -d)

# Log with timestamps
log() { echo "[$(date '+%Y-%m-%dT%H:%M:%S')] $*"; }
log "Starting deploy: env=$ENV target=$TARGET dry_run=$DRY_RUN"

# Dry-run wrapper
run() {
    if [[ "$DRY_RUN" == true ]]; then
        echo "[DRY-RUN] $*"
    else
        "$@"
    fi
}

run rsync -av --exclude='.git' "./" "deploy@${TARGET}:/opt/app/"
log "Deploy complete"

Create a systemd service unit

A service + timer pair for a scheduled task (replacing cron):

# /etc/systemd/system/db-backup.service
[Unit]
Description=Database backup
After=network-online.target postgresql.service
Wants=network-online.target
# Prevent starting if PostgreSQL is not running
Requires=postgresql.service

[Service]
Type=oneshot
User=backup
Group=backup
# Security hardening
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/backups/db
PrivateTmp=true

ExecStart=/usr/local/bin/db-backup.sh
StandardOutput=journal
StandardError=journal

# Retry on failure
Restart=on-failure
RestartSec=60

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/db-backup.timer
[Unit]
Description=Run database backup daily at 02:00
Requires=db-backup.service

[Timer]
# Run at 02:00 every day
OnCalendar=*-*-* 02:00:00
# Run immediately if last run was missed (e.g., server was down)
Persistent=true
# Randomize start within 5 minutes to avoid thundering herd
RandomizedDelaySec=300

[Install]
WantedBy=timers.target

# Deploy and enable
sudo syste