it-operations

IT Operations Expert

A comprehensive skill for managing IT infrastructure operations, ensuring service reliability, implementing monitoring and alerting strategies, managing incidents, and maintaining operational excellence through automation and best practices.

Core Principles

1. Service Reliability First

• Proactive Monitoring: Implement comprehensive observability before incidents occur
• Incident Management: Structured response processes with clear escalation paths
• SLA/SLO Management: Define and maintain service level objectives aligned with business needs
• Continuous Improvement: Learn from incidents through blameless post-mortems

2. Automation Over Manual Processes

• Infrastructure as Code: Manage infrastructure configuration through version-controlled code
• Runbook Automation: Convert manual procedures into automated workflows
• Self-Healing Systems: Implement automated remediation for common issues
• Configuration Management: Maintain consistency across environments

3. ITIL Service Management

• Service Strategy: Align IT services with business objectives
• Service Design: Design resilient, scalable services
• Service Transition: Manage changes with minimal disruption
• Service Operation: Deliver and support services effectively
• Continual Service Improvement: Iteratively enhance service quality

4. Operational Excellence

• Documentation: Maintain current runbooks, procedures, and architecture diagrams
• Knowledge Management: Build searchable knowledge bases from incident resolutions
• Capacity Planning: Forecast and provision resources proactively
• Cost Optimization: Balance performance requirements with infrastructure costs

Core Workflow

Infrastructure Operations Workflow

1. MONITORING & OBSERVABILITY
   ├─ Define SLIs/SLOs/SLAs for critical services
   ├─ Implement metrics collection (infrastructure, application, business)
   ├─ Configure alerting with proper thresholds and escalation
   ├─ Build dashboards for different audiences (ops, devs, executives)
   └─ Establish on-call rotation and escalation procedures

2. INCIDENT MANAGEMENT
   ├─ Receive alert or user report
   ├─ Assess severity and impact (P1/P2/P3/P4)
   ├─ Engage appropriate responders
   ├─ Investigate and diagnose root cause
   ├─ Implement fix or workaround
   ├─ Communicate status to stakeholders
   ├─ Document resolution in knowledge base
   └─ Conduct post-incident review

3. CHANGE MANAGEMENT
   ├─ Submit change request with impact assessment
   ├─ Review and approve through CAB (Change Advisory Board)
   ├─ Schedule change window
   ├─ Execute change with rollback plan ready
   ├─ Validate success criteria
   ├─ Document actual vs planned results
   └─ Close change ticket

4. CAPACITY PLANNING
   ├─ Collect resource utilization trends
   ├─ Analyze growth patterns
   ├─ Forecast future requirements
   ├─ Plan procurement or provisioning
   ├─ Execute capacity additions
   └─ Monitor effectiveness

5. AUTOMATION & OPTIMIZATION
   ├─ Identify repetitive manual tasks
   ├─ Document current process
   ├─ Design automated solution
   ├─ Implement and test automation
   ├─ Deploy to production
   ├─ Measure time/cost savings
   └─ Iterate and improve

Decision Frameworks

Alert Configuration Decision Matrix

| Scenario | Alert Type | Threshold | Response Time | Escalation | |----------|-----------|-----------|---------------|------------| | Service completely down | Page | Immediate | < 5 min | Immediate to on-call | | Service degraded | Page | 2-3 failures | < 15 min | After 15 min to on-call | | High resource usage | Warning | > 80% sustained | < 1 hour | After 2 hours to team lead | | Approaching capacity | Info | > 70% trend | < 24 hours | Weekly capacity review | | Configuration drift | Ticket | Any deviation | < 7 days | Monthly review |

Incident Severity Classification

Priority 1 (Critical)

• Complete service outage affecting all users
• Data loss or security breach
• Financial impact > $10K/hour
• Response: Immediate, 24/7, all hands on deck

Priority 2 (High)

• Partial service outage affecting many users
• Significant performance degradation
• Financial impact $1K-$10K/hour
• Response: < 30 minutes during business hours

Priority 3 (Medium)

• Service degradation affecting some users
• Non-critical functionality impaired
• Workaround available
• Response: < 4 hours during business hours

Priority 4 (Low)

• Minor issues with minimal impact
• Cosmetic problems
• Enhancement requests
• Response: Next business day

Change Management Risk Assessment

Risk Level = Impact × Likelihood × Complexity

Impact (1-5):
1 = Single user
2 = Team
3 = Department
4 = Company-wide
5 = Customer-facing

Likelihood of Issues (1-5):
1 = Routine, tested
2 = Familiar, documented
3 = Some uncertainty
4 = New territory
5 = Never done before

Complexity (1-5):
1 = Single component
2 = Few components
3 = Multiple systems
4 = Cross-platform
5 = Enterprise-wide

Risk Score Interpretation:
1-20: Standard change (pre-approved)
21-50: Normal change (CAB review)
51-75: High-risk change (extensive testing, senior approval)
76-125: Emergency change only (executive approval)

Monitoring Tool Selection

| Requirement | Prometheus + Grafana | Datadog | New Relic | ELK Stack | Splunk | |-------------|---------------------|---------|-----------|-----------|---------| | Cost | Free (self-hosted) | $$$$ | $$$$ | Free-$$ | $$$$$ | | Metrics | Excellent | Excellent | Excellent | Good | Good | | Logs | Via Loki | Excellent | Excellent | Excellent | Excellent | | Traces | Via Tempo | Excellent | Excellent | Limited | Good | | Learning Curve | Steep | Moderate | Moderate | Steep | Steep | | Cloud-Native | Excellent | Excellent | Excellent | Good | Good | | On-Premises | Excellent | Good | Good | Excellent | Excellent | | APM | Via exporters | Excellent | Excellent | Limited | Good |

Common Operational Challenges

Challenge 1: Alert Fatigue

Problem: Too many false positive alerts causing team burnout

Solution:

Alert Tuning Process:
1. Measure baseline alert volume and false positive rate
2. Categorize alerts by actionability:
   - Actionable + Urgent = Keep as page
   - Actionable + Not Urgent = Ticket
   - Not Actionable = Remove or convert to dashboard metric
3. Implement alert aggregation (group similar alerts)
4. Add context to alerts (runbook links, relevant metrics)
5. Regular review meetings (weekly) to tune thresholds
6. Track metrics:
   - MTTA (Mean Time to Acknowledge): < 5 min target
   - False Positive Rate: < 20% target
   - Alert Volume per Week: Trending down

Challenge 2: Incident Documentation During Crisis

Problem: Teams skip documentation during high-pressure incidents

Solution:

• Assign dedicated scribe role (not the incident commander)
• Use incident management tools (PagerDuty, Opsgenie) with automatic timeline
• Template-based incident reports with required fields
• Post-incident review scheduled automatically (within 48 hours)
• Gamify documentation (track and recognize thorough documentation)

Challenge 3: Knowledge Silos

Problem: Critical knowledge trapped in individual team members' heads

Solution:

Knowledge Transfer Strategy:
- Pair Programming/Shadowing: 20% of sprint capacity
- Runbook Requirements: Every system must have runbook
- Lunch & Learn Sessions: Weekly 30-min knowledge sharing
- Cross-Training Matrix: Track who knows what, identify gaps
- On-Call Rotation: Everyone rotates to spread knowledge
- Post-Incident Reviews: Mandatory team sharing
- Documentation Sprints: Quarterly focus on doc completion

Challenge 4: Balancing Stability vs Innovation

Problem: Operations team resists change to maintain stability

Solution:

• Implement change windows (planned maintenance periods)
• Use blue-green or canary deployments for lower risk
• Establis