cloud-browser-automation

Cloud Browser Automation

Use cloud browser services (Browserbase, Browser Use Cloud) for web scraping when local tools fail due to Cloudflare protection, JavaScript rendering requirements, or anti-bot measures.

Usage Strategy

CRITICAL: On-demand only, close immediately

Always follow this pattern:

1. Create session ONLY when needed
2. Do the task (scrape/navigate/extract)
3. Close session IMMEDIATELY after done
4. Never leave sessions running idle

Why:

• Save credits (pay per minute)
• Avoid hitting concurrent session limits (e.g., Browserbase max 3)
• Clean resource usage
• No waste

When to Use

Escalation ladder (always try cheapest/fastest first):

1. Try Hermes web_fetch first
   ↓ (if fails or empty content)
2. Try direct curl with proper headers (Pattern 0 — works for most static sites)
   ↓ (if blocked or needs JS)
3. Try Hermes browser
   ↓ (if blocked by Cloudflare or sandbox issues)
4. Use cloud browser (Browserbase)

Use cloud browser when:

• ✅ Cloudflare Turnstile blocking requests
• ✅ JavaScript-heavy sites (SPA, dynamic content)
• ✅ Anti-bot detection (fingerprinting, TLS checks)
• ✅ Complex multi-step automation (login, navigate, extract)

DON'T use cloud browser when:

• ❌ Simple HTTP requests (use web_fetch)
• ❌ Static content (use web_fetch)
• ❌ Cost-sensitive tasks (cloud = pay per minute)

Browserbase Setup

API Configuration

# Credentials (saved in memory)
API_KEY="bb_live_O6tVgdzl8B6WquBSj1XpuaC5hMc"
PROJECT_ID="a5008864-bbaa-4966-96e2-2272497b003d"
BASE_URL="https://www.browserbase.com/v1"

# Limits
MAX_CONCURRENT_SESSIONS=3
SESSION_TIMEOUT=5  # minutes (default)

Session Management

CRITICAL USER PREFERENCE:

"gini jadi lo pakai browserbase saat dibutuhkan aja, nah sesi langsung close saat beres"

Translation: Use on-demand only, close immediately after done.

Why:

• ✅ Save credits (pay per minute)
• ✅ Avoid hitting concurrent limit (max 3)
• ✅ Clean resource usage
• ✅ No waste

Pattern:

# 1. Create session ONLY when needed
session = create_session()

# 2. Do the task
data = scrape_page(session)

# 3. Close IMMEDIATELY after done
close_session(session)

DON'T:

• ❌ Create session at start of script "just in case"
• ❌ Leave sessions running between tasks
• ❌ Reuse sessions across multiple unrelated tasks
• ❌ Forget to close sessions

Workflow

Step 1: Create Session

curl -X POST "https://www.browserbase.com/v1/sessions" \
  -H "X-BB-API-Key: $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "projectId": "'$PROJECT_ID'",
    "browserSettings": {
      "viewport": {"width": 1920, "height": 1080}
    }
  }'

Response:

{
  "id": "session-id-here",
  "status": "RUNNING",
  "connectUrl": "wss://connect-v2.usw2.browserbase.com/?signingKey=...",
  "seleniumRemoteUrl": "http://connect-v2.usw2.browserbase.com/webdriver",
  "expiresAt": "2026-05-03T13:00:00.000Z"
}

Step 2: Automate (Requires SDK)

Option A: Playwright (Python)

from playwright.async_api import async_playwright

async with async_playwright() as p:
    browser = await p.chromium.connect_over_cdp(connect_url)
    page = await browser.new_page()
    await page.goto("https://example.com")
    data = await page.evaluate("() => document.body.innerText")
    await browser.close()

Option B: Selenium (Python)

from selenium import webdriver

options = webdriver.ChromeOptions()
driver = webdriver.Remote(
    command_executor=selenium_remote_url,
    options=options
)
driver.get("https://example.com")
data = driver.find_element(By.TAG_NAME, "body").text
driver.quit()

Option C: Manual (Fallback)

# Get debug URL
curl "https://www.browserbase.com/v1/sessions/$SESSION_ID/debug" \
  -H "X-BB-API-Key: $API_KEY"

# Open debug URL in browser
# Navigate manually
# Screenshot/extract data manually

Step 3: Close Session

curl -X POST "https://www.browserbase.com/v1/sessions/$SESSION_ID/stop" \
  -H "X-BB-API-Key: $API_KEY"

ALWAYS close, even if task fails:

try:
    data = scrape_with_browserbase()
finally:
    close_session()  # Always close!

VPS Constraints

Problem: Ubuntu 24.04 strict package management blocks pip install playwright/selenium.

Solutions:

Option 1: Use Docker (Recommended)

# Run Playwright in container
docker run -it mcr.microsoft.com/playwright/python:v1.40.0-jammy \
  python3 /tmp/scraper.py

Pros: Isolated, no pip conflicts Cons: Need Docker (~500MB)

Option 2: Manual Fallback

# Create session
SESSION_ID=$(curl -X POST ... | jq -r '.id')

# Get debug URL
DEBUG_URL=$(curl ... | jq -r '.debuggerFullscreenUrl')

# User opens URL manually
echo "Open: $DEBUG_URL"
echo "Navigate to target site"
echo "Screenshot and send to me"

# Close session
curl -X POST ".../sessions/$SESSION_ID/stop" ...

Option 3: Delegate to Subagent

# If automation blocked, delegate research
delegate_task(
    goal="Research [topic] via web search",
    toolsets=["web", "terminal"]
)

Common Patterns

Pattern 0: Direct Curl for Static Sites (No Browserbase)

Challenge: Site returns HTML but might have Cloudflare or anti-bot checks

Solution:

# Direct curl with proper headers (often bypasses basic protection)
curl -s -L -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" \
  -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" \
  -H "Accept-Language: en-US,en;q=0.5" \
  -H "Connection: keep-alive" \
  -H "Upgrade-Insecure-Requests: 1" \
  "https://example.com" > /tmp/page.html

# If response is gzip-encoded, check file type first
file /tmp/page.html  # Should show "HTML document" not "gzip compressed"

# Extract data with grep/sed/python
grep -oP '<title>\K[^<]+' /tmp/page.html

When to use:

• ✅ Static HTML sites (marketing pages, landing pages)
• ✅ Sites with basic Cloudflare (not Turnstile)
• ✅ Public content (no login required)
• ✅ VPS without Docker/Playwright

When to escalate:

• ❌ Cloudflare Turnstile challenge appears
• ❌ JavaScript-heavy SPA (empty body in curl response)
• ❌ Login-required content

Pattern 0b: TinyFish Browser for Cloudflare Bypass (Recommended)

Challenge: API returns Cloudflare challenge page, need full browser automation

Solution:

import requests
import asyncio
import websockets
import json
import time

TINYFISH_API_KEY = "your-api-key"

async def fetch_via_tinyfish(url):
    # 1. Create browser session
    headers = {"X-API-Key": TINYFISH_API_KEY, "Content-Type": "application/json"}
    payload = {
        "url": url,
        "timeout_seconds": 300,
        "browser_profile": "stealth",
        "proxy_config": {"enabled": True, "type": "tetra", "country_code": "US"}
    }
    resp = requests.post("https://api.browser.tinyfish.ai", headers=headers, json=payload, timeout=60)
    
    info = resp.json()
    session_id = info["session_id"]
    base_url = info["base_url"].rstrip("/")
    
    # 2. Wait for Cloudflare bypass (CRITICAL: 30 attempts × 6 seconds)
    page_ws_url = None
    for attempt in range(30):  # Not 20!
        time.sleep(6)  # Not 5!
        try:
            pages_resp = requests.get(f"{base_url}/pages", timeout=20)
            pages = pages_resp.json()
            
            if pages:
                page = pages[0]
                title = page.get("title", "")
                url_check = page.get("url", "")
                
                # Check BOTH title AND url (not just title!)
                if "Just a moment" not in title and url in url_check:
                    page_ws_url = page.get("webSocketDebuggerUrl")
                    break
        except:
            pass
    
    if not page_ws_url:
        requests.delete(f"https://api.browser.tinyfish.ai/{session_id}", headers={"X-API-Key"