product-analytics

Product Analytics

Core Principles

• Metrics over vanity — Focus on actionable metrics tied to business outcomes
• Data-driven decisions — Hypothesize, measure, learn, iterate
• User-centric measurement — Track behavior, not just pageviews
• Statistical rigor — Understand significance, avoid false positives
• Privacy-first — Respect user data, comply with GDPR/CCPA
• North Star focus — Align all teams around one key metric

Hard Rules (Must Follow)

These rules are mandatory. Violating them means the skill is not working correctly.

No PII in Events

Events must NEVER contain personally identifiable information.

// ❌ FORBIDDEN: PII in event properties
track('user_signed_up', {
  email: 'user@example.com',     // PII!
  name: 'John Doe',              // PII!
  phone: '+1234567890',          // PII!
  ip_address: '192.168.1.1',     // PII!
  credit_card: '4111...',        // NEVER!
});

// ✅ REQUIRED: Anonymized/hashed identifiers only
track('user_signed_up', {
  user_id: hash('user@example.com'),  // Hashed
  plan: 'pro',
  source: 'organic',
  country: 'US',                       // Broad location OK
});

// Masking utilities
const maskEmail = (email) => {
  const [name, domain] = email.split('@');
  return `${name[0]}***@${domain}`;
};

Object_Action Event Naming

All event names must follow the object_action snake_case format.

// ❌ FORBIDDEN: Inconsistent naming
track('signup');                    // No object
track('newProject');                // camelCase
track('Upload File');               // Spaces and PascalCase
track('user-created');              // kebab-case
track('BUTTON_CLICKED');            // SCREAMING_CASE

// ✅ REQUIRED: object_action snake_case
track('user_signed_up');
track('project_created');
track('file_uploaded');
track('payment_completed');
track('checkout_started');

Actionable Metrics Only

Track metrics that drive decisions, not vanity metrics.

// ❌ FORBIDDEN: Vanity metrics without context
track('page_viewed');               // No insight
track('button_clicked');            // Too generic
track('app_opened');                // Doesn't indicate value

// ✅ REQUIRED: Actionable metrics tied to outcomes
track('feature_activated', {
  feature: 'dark_mode',
  time_to_activation_hours: 2.5,
  user_segment: 'power_user',
});

track('checkout_completed', {
  order_value: 99.99,
  items_count: 3,
  payment_method: 'credit_card',
  coupon_applied: true,
});

Statistical Rigor for Experiments

A/B tests must have proper sample size and significance thresholds.

// ❌ FORBIDDEN: Drawing conclusions too early
// "After 100 users, variant B has 5% higher conversion!"
// This is not statistically significant.

// ✅ REQUIRED: Proper experiment setup
const experimentConfig = {
  name: 'new_checkout_flow',
  hypothesis: 'New flow increases conversion by 10%',

  // Statistical requirements
  significance_level: 0.05,      // 95% confidence
  power: 0.80,                   // 80% power
  minimum_detectable_effect: 0.10, // 10% lift

  // Calculated sample size
  sample_size_per_variant: 3842,

  // Guardrails
  max_duration_days: 14,
  stop_if_degradation: -0.05,    // Stop if 5% worse
};

Quick Reference

When to Use What

| Scenario | Framework/Tool | Key Metric | |----------|---------------|------------| | Overall product health | North Star Metric | Time spent listening (Spotify), Nights booked (Airbnb) | | Growth optimization | AARRR (Pirate Metrics) | Conversion rates per stage | | Feature validation | A/B Testing | Statistical significance (p < 0.05) | | User engagement | Cohort Analysis | Day 1/7/30 retention rates | | Conversion optimization | Funnel Analysis | Drop-off rates per step | | Feature impact | Attribution Modeling | Multi-touch attribution | | Experiment success | Statistical Testing | Power, significance, effect size |

North Star Metric

Definition

A North Star Metric is the one metric that best captures the core value your product delivers to customers. When this metric grows sustainably, your business succeeds.

Characteristics of Good NSMs

✓ Captures product value delivery
✓ Correlates with revenue/growth
✓ Measurable and trackable
✓ Movable by product/engineering
✓ Understandable by entire org
✓ Leading (not lagging) indicator

Examples by Company

| Company | North Star Metric | Why It Works | |---------|------------------|--------------| | Spotify | Time Spent Listening | Core value = music enjoyment | | Airbnb | Nights Booked | Revenue driver + value delivered | | Slack | Daily Active Teams | Engagement = product stickiness | | Facebook | Monthly Active Users | Network effect foundation | | Amplitude | Weekly Learning Users | Value = analytics insights | | Dropbox | Active Users Sharing Files | Core product behavior |

NSM Framework

North Star Metric
       ↓
┌──────┴──────┬──────────┬──────────┐
│             │          │          │
Input 1    Input 2   Input 3   Input 4
(Supporting metrics that drive NSM)

Example: Spotify
NSM: Time Spent Listening
├── Daily Active Users
├── Playlists Created
├── Songs Added to Library
└── Share/Social Actions

How to Define Your NSM

1. Identify core value proposition
2. What job does your product do for users?
3. When do users get "aha!" moment?

1. Find the metric that represents this value
2. Transaction completed? (e.g., Nights Booked)
3. Time engaged? (e.g., Time Listening)
4. Content created? (e.g., Messages Sent)

1. Validate it correlates with business success
2. Does NSM increase → revenue increases?
3. Can product changes move this metric?

1. Define supporting input metrics
2. What user behaviors drive NSM?
3. Break into 3-5 key inputs

AARRR Framework (Pirate Metrics)

Overview

The AARRR framework tracks the customer lifecycle across five stages:

ACQUISITION → ACTIVATION → RETENTION → REFERRAL → REVENUE

Stage Definitions

1. Acquisition

When users discover your product

Key Questions:

• Where do users come from?
• Which channels have best quality users?
• What's the cost per acquisition (CPA)?

Metrics:

• Website visitors
• App installs
• Sign-ups per channel
• Cost per acquisition (CPA)
• Channel conversion rates

Example Events:

// Landing page view
track('page_viewed', {
  page: 'landing',
  utm_source: 'google',
  utm_medium: 'cpc',
  utm_campaign: 'brand_search'
});

// Sign-up started
track('signup_started', {
  source: 'homepage_cta'
});

2. Activation

When users experience core product value

Key Questions:

• What's the "aha!" moment?
• How long to first value?
• What % reach activation?

Metrics:

• Time to first action
• Activation rate (% completing key action)
• Setup completion rate
• Feature adoption rate

Example "Aha!" Moments:

Slack:     Send 2,000 messages in team
Twitter:   Follow 30 users
Dropbox:   Upload first file
LinkedIn:  Connect with 5 people

Example Events:

// Activation milestone
track('activated', {
  user_id: 'usr_123',
  activation_action: 'first_project_created',
  time_to_activation_hours: 2.5
});

3. Retention

When users keep coming back

Key Questions:

• What's Day 1/7/30 retention?
• Which cohorts retain best?
• What drives churn?

Metrics:

• Day 1/7/30 retention rate
• Weekly/Monthly active users (WAU/MAU)
• Churn rate
• Usage frequency
• Feature stickiness (DAU/MAU)

Retention Calculation:

Day X Retention = Users returning on Day X / Total users in cohort

Example:
Cohort: 1000 users signed up Jan 1
Day 7: 300 returned
Day 7 Retention = 300/1000 = 30%

Example Events:

// Daily engagement
track('session_started', {
  user_id: 'usr_123',
  session_count: 42,
  days_since_signup: 1