When this skill is activated, always start your first response with the 🧢 emoji.

Computer Vision

Computer vision enables machines to interpret and reason about visual data - images, video, and multi-modal inputs. Modern CV pipelines are built on deep neural networks pretrained on large datasets (ImageNet, COCO, ADE20K) and fine-tuned for specific domains. PyTorch and its ecosystem (torchvision, timm, ultralytics, albumentations) cover the full stack from data loading through deployment. Foundation models like SAM, DINOv2, and OpenCLIP have shifted best practice toward prompt-based and zero-shot approaches before committing to full training runs.

When to use this skill

Trigger this skill when the user:

• Trains or fine-tunes an image classifier on a custom dataset
• Runs inference with YOLO, DETR, or other detection models
• Builds a semantic or instance segmentation pipeline
• Implements data augmentation for CV training
• Preprocesses images for model ingestion (resize, normalize, batch)
• Exports a vision model to ONNX or optimizes with TensorRT
• Evaluates a vision model (mAP, confusion matrix, per-class metrics)
• Implements a U-Net, DeepLabV3, or similar segmentation architecture

Do NOT trigger this skill for:

• Pure NLP tasks with no visual component (use a language-model skill instead)
• 3D point-cloud processing or LiDAR-only pipelines (overlap is limited; check domain)

Key principles

1. Start with pretrained models - Fine-tune ImageNet/COCO weights before training from scratch. Even a frozen backbone with a new head beats random init on small datasets.
2. Augment data aggressively - Real-world distribution shifts are unavoidable. Use albumentations with geometric, color, and noise transforms. Target-aware augments (mosaic, copy-paste) matter especially for detection.
3. Validate on representative data - Always hold out data from the exact deployment distribution. Benchmark on in-distribution AND out-of-distribution splits separately.
4. Optimize inference separately from training - Training precision (FP32/AMP) and inference precision (INT8/FP16) have different tradeoffs. Profile, export to ONNX, then apply TensorRT or OpenVINO post-training quantization.
5. Monitor for distribution shift - Production images drift from training data (lighting changes, new object classes, compression artifacts). Log prediction confidence distributions and trigger retraining pipelines when they degrade.

Core concepts

Task taxonomy

Backbone architectures

Anchor-free vs anchor-based detection

• Anchor-based (YOLOv5, Faster R-CNN) - predefined box aspect ratios per grid cell. Fast training convergence, tuning required for unusual object scales.
• Anchor-free (YOLO11/v8, FCOS, DETR) - predict box center + offsets directly. Cleaner training, no anchor hyperparameter search, now the default for new projects.

Loss functions

Common tasks

Fine-tune an image classifier

import torch
import torch.nn as nn
from torch.utils.data import DataLoader
from torchvision import datasets, transforms, models

# 1. Data transforms
train_tf = transforms.Compose([
    transforms.RandomResizedCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.ColorJitter(brightness=0.3, contrast=0.3, saturation=0.2),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
val_tf = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

train_ds = datasets.ImageFolder("data/train", transform=train_tf)
val_ds   = datasets.ImageFolder("data/val",   transform=val_tf)
train_loader = DataLoader(train_ds, batch_size=32, shuffle=True,  num_workers=4)
val_loader   = DataLoader(val_ds,   batch_size=64, shuffle=False, num_workers=4)

# 2. Load pretrained backbone, replace head
NUM_CLASSES = len(train_ds.classes)
model = models.efficientnet_b0(weights=models.EfficientNet_B0_Weights.DEFAULT)
model.classifier[1] = nn.Linear(model.classifier[1].in_features, NUM_CLASSES)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)

# 3. Two-phase training: head first, then unfreeze backbone
optimizer = torch.optim.AdamW(model.classifier.parameters(), lr=1e-3)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=5)
criterion = nn.CrossEntropyLoss(label_smoothing=0.1)

def train_one_epoch(loader):
    model.train()
    for imgs, labels in loader:
        imgs, labels = imgs.to(device), labels.to(device)
        optimizer.zero_grad()
        loss = criterion(model(imgs), labels)
        loss.backward()
        optimizer.step()
    scheduler.step()

# Phase 1 - head only (5 epochs)
for epoch in range(5):
    train_one_epoch(train_loader)

# Phase 2 - unfreeze everything with lower LR
for p in model.parameters():
    p.requires_grad = True
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4, weight_decay=0.01)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=10)
for epoch in range(10):
    train_one_epoch(train_loader)

torch.save(model.state_dict(), "classifier.pth")

Run object detection with YOLO

from ultralytics import YOLO

# --- Inference ---
model = YOLO("yolo11n.pt")  # nano; swap for yolo11s/m/l/x for accuracy
results = model.predict("image.jpg", conf=0.25, iou=0.45, device=0)

for r in results:
    for box in r.boxes:
        cls   = int(box.cls[0])
        label = model.names[cls]
        conf  = float(box.conf[0])
        xyxy  = box.xyxy[0].tolist()   # [x1, y1, x2, y2]
        print(f"{label}: {conf:.2f}  {xyxy}")

# --- Fine-tune on custom dataset ---
# Expects data.yaml with train/val paths and class names
model = YOLO("yolo11s.pt")
results = model.train(
    data="data.yaml",
    epochs=100,
    imgsz=640,
    batch=16,
    device=0,
    optimizer="AdamW",
    lr0=1e-3,
    weight_decay=0.0005,
    augment=True,         # built-in mosaic, mixup, copy-paste
    cos_lr=True,
    patience=20,          # early stopping
    project="runs/detect",
    name="custom_v1",
)
print(results.results_dict)  # mAP50, mAP50-95, precision, recall

Implement a data augmentation pipeline

import albumentations as A
from albumentations.pytorch import ToTensorV2
import numpy as np

# Classification pipeline
clf_transform = A.Compose([
    A.RandomResizedCrop(height=224, width=224, scale=(0.6, 1.0)),
    A.HorizontalFlip(p=0.5),
    A.ShiftScaleRotate(shift_limit=0.05, scale_limit=0.1, rotate_limit=15, p=0.5),
    A.OneOf([
        A.GaussNoise(var