Skip to main content

Firecrawl & Playwright - Complete Implementation Guide

A comprehensive guide to the Smart Firecrawl API and Smart Playwright Service implementations. This document centralizes the documentation for both services, providing a complete understanding of their architecture, features, and integration patterns.

Table of Contents​

System Architecture​

The Smart Firecrawl API and Smart Playwright Service form a comprehensive web scraping ecosystem with the following architecture:

┌─────────────────────────────────────────────────────────────┐
│                    Client Applications                     │
└─────────────────────┬─────────────────────────────────────┘
                      │
┌─────────────────────▼─────────────────────────────────────┐
│                Smart Firecrawl API                       │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────┐│
│  │   Controllers   │  │   Services       │  │   Queue     ││
│  │   (v0/v1)      │  │   (Billing)      │  │   (BullMQ)  ││
│  └─────────────────┘  └─────────────────┘  └─────────────┘│
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────┐│
│  │   Scrapers      │  │   Extractors    │  │   AI/LLM    ││
│  │   (Playwright)  │  │   (Content)     │  │   (Claude)  ││
│  └─────────────────┘  └─────────────────┘  └─────────────┘│
└─────────────────────┬─────────────────────────────────────┘
                      │
┌─────────────────────▼─────────────────────────────────────┐
│              Smart Playwright Service                     │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────┐│
│  │   Browser       │  │   Stealth       │  │   Resource  ││
│  │   Automation   │  │   Features      │  │   Blocking  ││
│  └─────────────────┘  └─────────────────┘  └─────────────┘│
└─────────────────────┬─────────────────────────────────────┘
                      │
┌─────────────────────▼─────────────────────────────────────┐
│              External Services                            │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────┐│
│  │   Redis         │  │   Database      │  │   Storage   ││
│  │   (Queue)       │  │   (Supabase)    │  │   (GCS)     ││
│  └─────────────────┘  └─────────────────┘  └─────────────┘│
└─────────────────────────────────────────────────────────────┘

Key Components​

  • Smart Firecrawl API: Main orchestration service with AI-powered extraction
  • Smart Playwright Service: Microservice for complex browser automation
  • Queue System: BullMQ with Redis for job management
  • AI Integration: Multiple LLM providers (OpenAI, Anthropic, Google, Groq)
  • Content Processing: Native libraries for HTML-to-Markdown, PDF parsing

Smart Firecrawl API​

The Smart Firecrawl API is a comprehensive web scraping and crawling service built with Node.js, TypeScript, and Express. It provides powerful web scraping capabilities with AI-powered content extraction, queue-based processing, and enterprise features.

Project Structure​

smart-firecrawl-api/
├── src/                          # Main source code
│   ├── controllers/              # API controllers
│   │   ├── v0/                   # Legacy API endpoints
│   │   └── v1/                   # Current API endpoints
│   ├── lib/                      # Core libraries and utilities
│   │   ├── extract/              # Content extraction modules
│   │   ├── deep-research/        # AI-powered research features
│   │   └── generate-llmstxt/      # LLM text generation
│   ├── routes/                   # Express route definitions
│   ├── scraper/                  # Web scraping engines
│   │   ├── scrapeURL/            # URL scraping logic
│   │   └── WebScraper/           # Web scraper implementations
│   ├── services/                 # Background services
│   │   ├── billing/              # Credit and billing management
│   │   ├── queue-service.ts      # Queue management
│   │   └── redis.ts              # Redis configuration
│   ├── search/                   # Search functionality
│   └── types/                    # TypeScript type definitions
├── sharedLibs/                   # Native libraries
│   ├── crawler/                  # Rust-based crawler
│   ├── go-html-to-md/           # Go-based HTML to Markdown
│   ├── html-transformer/        # Rust HTML transformer
│   └── pdf-parser/              # Rust PDF parser
├── chrome-extension/            # Browser extension
├── firecrawl-extension/         # Firecrawl extension
├── page-discovery-extension/   # Page discovery extension
├── dist/                       # Compiled JavaScript output
├── node_modules/               # Node.js dependencies
└── tests/                      # Test files

Getting Started​

Prerequisites​

  • Node.js 22+
  • pnpm (recommended) or npm
  • Redis server
  • Docker (for containerized deployment)

Installation​

  1. Clone and install dependencies
git clone <repository-url>
cd smart-firecrawl-api
pnpm install
  1. Set up environment variables
# Database
DATABASE_URL=your_database_url
REDIS_URL=redis://localhost:6379

# Authentication
JWT_SECRET=your_jwt_secret

# External Services
SENTRY_DSN=your_sentry_dsn
SLACK_WEBHOOK_URL=your_slack_webhook

# AI Services
OPENAI_API_KEY=your_openai_key
ANTHROPIC_API_KEY=your_anthropic_key

# Storage
GCS_FIRE_ENGINE_BUCKET_NAME=your_bucket_name
  1. Start services
# Start Redis
docker run -d -p 6379:6379 redis:latest

# Start development server
npm run start:dev

# Start queue workers (separate terminals)
npm run workers
npm run index-worker

API Endpoints​

Scraping Endpoints​

  • POST /v1/scrape - Scrape a single URL
  • GET /v1/scrape/:jobId - Get scrape job status

Crawling Endpoints​

  • POST /v1/crawl - Start a website crawl
  • GET /v1/crawl/:jobId - Get crawl job status
  • DELETE /v1/crawl/:jobId - Cancel a crawl job

Batch Operations​

  • POST /v1/batch/scrape - Batch scrape multiple URLs
  • GET /v1/batch/scrape/:jobId - Get batch job status

AI Features​

  • POST /v1/extract - Extract structured data using AI
  • POST /v1/deep-research - Perform deep research on topics
  • POST /v1/llmstxt - Generate LLM-optimized text

Search & Mapping​

  • POST /v1/search - Search the web
  • POST /v1/map - Map website structure

Core Features​

  1. Multi-Engine Scraping: Supports various scraping engines including Playwright, Puppeteer, and custom implementations

  2. AI-Powered Extraction:

    • Structured data extraction using LLMs
    • Content summarization and analysis
    • Deep research capabilities

  3. Queue-Based Processing:

    • BullMQ for job management
    • Redis for caching and coordination
    • Priority-based job processing

  4. Content Processing:

    • HTML to Markdown conversion
    • PDF parsing and extraction
    • Image and media handling

  5. Rate Limiting & Authentication:

    • Team-based authentication
    • Credit-based billing system
    • Rate limiting per endpoint

Basic Usage Examples​

Scrape Single URL​

// Using the API directly
const response = await fetch("http://localhost:3002/v1/scrape", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    Authorization: "Bearer your-api-key",
  },
  body: JSON.stringify({
    url: "https://example.com",
    formats: ["markdown", "html"],
    onlyMainContent: true,
  }),
});

const result = await response.json();
console.log(result.data.markdown);

Crawl Website​

// Start a crawl job
const crawlResponse = await fetch("http://localhost:3002/v1/crawl", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    Authorization: "Bearer your-api-key",
  },
  body: JSON.stringify({
    url: "https://example.com",
    crawlerOptions: {
      includes: ["/blog/*", "/articles/*"],
      excludes: ["/admin/*", "/login"],
      limit: 10,
    },
    pageOptions: {
      formats: ["markdown"],
      onlyMainContent: true,
    },
  }),
});

const crawlJob = await crawlResponse.json();
console.log("Crawl job ID:", crawlJob.jobId);

Smart Playwright Service​

The Smart Playwright Service is a lightweight, high-performance web scraping microservice built with Node.js, TypeScript, and Playwright. It provides efficient web scraping capabilities with advanced browser automation, stealth features, and optimized performance for production environments.

Project Structure​

smart-playwright-service/
├── api.ts                    # Main API server implementation
├── helpers/                  # Utility functions
│   └── get_error.ts         # HTTP error handling
├── dist/                    # Compiled JavaScript output
├── node_modules/            # Node.js dependencies
├── package.json             # Project configuration
├── tsconfig.json            # TypeScript configuration
├── Dockerfile              # Docker container setup
├── .dockerignore           # Docker ignore patterns
├── .gitignore              # Git ignore patterns
└── README.md               # Project documentation

Getting Started​

Prerequisites​

  • Node.js 18+
  • npm or pnpm
  • Docker (for containerized deployment)

Installation​

  1. Clone and install dependencies
git clone <repository-url>
cd smart-playwright-service
npm install
npx playwright install
  1. Set up environment variables
# Server Configuration
PORT=3003

# Proxy Configuration (Optional)
PROXY_SERVER=your_proxy_server
PROXY_USERNAME=your_proxy_username
PROXY_PASSWORD=your_proxy_password

# Performance Settings
BLOCK_MEDIA=true
  1. Start the service
# Development
npm run dev

# Production
npm run build
npm start

API Endpoints​

Health Check​

  • GET /health - Service health status

Web Scraping​

  • POST /scrape - Scrape a single URL with advanced options

Scraping Request Format​

{
  "url": "https://example.com",
  "wait_after_load": 1000,
  "timeout": 15000,
  "headers": {
    "Custom-Header": "value",
    "User-Agent": "Mozilla/5.0...",
    "Cookie": "session=abc123; user=john"
  },
  "check_selector": "#content"
}

Response Format​

{
  "content": "<html>...</html>",
  "pageStatusCode": 200,
  "contentType": "text/html",
  "pageError": null
}

Key Features​

  1. Advanced Browser Automation:

    • Playwright-based browser automation
    • Stealth mode to avoid detection
    • Random user-agent rotation
    • Custom viewport and device simulation

  2. Smart Request Blocking:

    • Blocks ad-serving domains automatically
    • Optional media file blocking for performance
    • Configurable request filtering

  3. Cookie Management:

    • Automatic cookie parsing from headers
    • Support for secure cookies (**Secure-, **Host-)
    • Domain-specific cookie handling
    • Google services cookie optimization

  4. Proxy Support:

    • HTTP/HTTPS proxy configuration
    • Authentication support
    • Environment-based proxy settings

  5. Performance Optimizations:

    • Fresh browser instance per request
    • Resource blocking for faster loading
    • Configurable timeouts and waits
    • Memory-efficient cleanup

Usage Examples​

Basic Scraping​

curl -X POST http://localhost:3003/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "timeout": 15000
  }'

Advanced Scraping with Headers​

curl -X POST http://localhost:3003/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "wait_after_load": 2000,
    "timeout": 30000,
    "headers": {
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
      "Cookie": "session=abc123; user=john"
    },
    "check_selector": "#main-content"
  }'

Integration Patterns​

Service Communication​

The Smart Firecrawl API and Smart Playwright Service work together through a microservice architecture:

// Firecrawl API configuration for Playwright integration
const config = {
  playwrightService: {
    url:
      process.env.PLAYWRIGHT_MICROSERVICE_URL || "http://localhost:3003/scrape",
    timeout: 30000,
    retries: 3,
  },
};

Hybrid Scraping Strategy​

async function intelligentScrape(url, options = {}) {
  // 1. Try Firecrawl first for simple content
  try {
    const firecrawlResult = await firecrawl.scrapeUrl(url, {
      formats: ["markdown"],
      onlyMainContent: true,
      ...options,
    });

    if (firecrawlResult.success) {
      return {
        method: "firecrawl",
        data: firecrawlResult.data,
      };
    }
  } catch (error) {
    console.log("Firecrawl failed, trying Playwright...");
  }

  // 2. Fallback to Playwright for complex content
  try {
    const playwrightResult = await fetch(config.playwrightService.url, {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        url,
        wait_after_load: 2000,
        timeout: 30000,
        ...options,
      }),
    });

    const data = await playwrightResult.json();

    return {
      method: "playwright",
      data: {
        content: data.content,
        statusCode: data.pageStatusCode,
      },
    };
  } catch (error) {
    throw new Error(`Both scraping methods failed: ${error.message}`);
  }
}

Queue-Based Processing​

// Firecrawl API queue integration
const Queue = require("bull");
const redis = require("redis");

const scrapeQueue = new Queue("scrape", {
  redis: { host: "localhost", port: 6379 },
});

// Add job to queue
const job = await scrapeQueue.add("scrape-url", {
  url: "https://example.com",
  options: {
    usePlaywright: true,
    formats: ["markdown"],
  },
});

// Process job
scrapeQueue.process("scrape-url", async (job) => {
  const { url, options } = job.data;

  if (options.usePlaywright) {
    // Use Playwright service
    return await callPlaywrightService(url, options);
  } else {
    // Use Firecrawl directly
    return await firecrawl.scrapeUrl(url, options);
  }
});

Deployment & Operations​

Docker Deployment​

Smart Firecrawl API​

# Dockerfile for Smart Firecrawl API
FROM node:22-alpine

WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production

COPY . .
RUN npm run build

EXPOSE 3002
CMD ["npm", "start"]
# Build and run
docker build -t smart-firecrawl-api .
docker run -p 3002:3002 \
  -e REDIS_URL=redis://host.docker.internal:6379 \
  -e DATABASE_URL=your_database_url \
  smart-firecrawl-api

Smart Playwright Service​

# Dockerfile for Smart Playwright Service
FROM mcr.microsoft.com/playwright:v1.40.0-focal

WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production

COPY . .
RUN npm run build

EXPOSE 3003
CMD ["npm", "start"]

Docker Compose Setup​

version: "3.8"
services:
  firecrawl-api:
    build: ./smart-firecrawl-api
    ports:
      - "3002:3002"
    environment:
      - REDIS_URL=redis://redis:6379
      - DATABASE_URL=your_database_url
      - PLAYWRIGHT_MICROSERVICE_URL=http://playwright-service:3003/scrape
    depends_on:
      - redis
      - playwright-service

  playwright-service:
    build: ./smart-playwright-service
    ports:
      - "3003:3003"
    environment:
      - BLOCK_MEDIA=true
    volumes:
      - /tmp/.cache:/tmp/.cache

  redis:
    image: redis:latest
    ports:
      - "6379:6379"

Environment Configuration​

Firecrawl API Environment​

# Production Environment
IS_PRODUCTION=true
ENV=production
PORT=3002
HOST=0.0.0.0

# Database
DATABASE_URL=postgresql://user:pass@localhost:5432/firecrawl
REDIS_URL=redis://localhost:6379

# AI Services
OPENAI_API_KEY=your_openai_key
ANTHROPIC_API_KEY=your_anthropic_key
GOOGLE_API_KEY=your_google_key
GROQ_API_KEY=your_groq_key

# Storage
GCS_FIRE_ENGINE_BUCKET_NAME=your_bucket_name

# Monitoring
SENTRY_DSN=your_sentry_dsn
SLACK_WEBHOOK_URL=your_slack_webhook

Playwright Service Environment​

# Production Environment
NODE_ENV=production
PORT=3003

# Proxy Configuration
PROXY_SERVER=your_proxy_server
PROXY_USERNAME=your_proxy_username
PROXY_PASSWORD=your_proxy_password

# Performance
BLOCK_MEDIA=true

Monitoring & Health Checks​

Firecrawl API Health Endpoints​

# Server health
curl http://localhost:3002/serverHealthCheck

# Production status
curl http://localhost:3002/is-production

# Queue monitoring (with auth)
curl http://localhost:3002/admin/{BULL_AUTH_KEY}/queues

Playwright Service Health​

# Service health
curl http://localhost:3003/health

Scaling Considerations​

Horizontal Scaling​

# Docker Compose with multiple instances
services:
  firecrawl-api-1:
    build: ./smart-firecrawl-api
    ports: ["3002:3002"]

  firecrawl-api-2:
    build: ./smart-firecrawl-api
    ports: ["3003:3002"]

  playwright-service-1:
    build: ./smart-playwright-service
    ports: ["3004:3003"]

  playwright-service-2:
    build: ./smart-playwright-service
    ports: ["3005:3003"]

Load Balancing​

# Nginx configuration
upstream firecrawl_api {
    server localhost:3002;
    server localhost:3003;
}

upstream playwright_service {
    server localhost:3004;
    server localhost:3005;
}

server {
    listen 80;

    location /api/ {
        proxy_pass http://firecrawl_api;
    }

    location /scrape/ {
        proxy_pass http://playwright_service;
    }
}

Advanced Features​

AI-Powered Content Extraction​

Custom LLM Extraction​

// Firecrawl API - Custom extraction with multiple LLM providers
const extractResult = await fetch("http://localhost:3002/v1/extract", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    Authorization: "Bearer your-api-key",
  },
  body: JSON.stringify({
    url: "https://example.com",
    extractionPrompt:
      "Extract all product information including name, price, and description",
    extractionSchema: {
      type: "object",
      properties: {
        products: {
          type: "array",
          items: {
            type: "object",
            properties: {
              name: { type: "string" },
              price: { type: "string" },
              description: { type: "string" },
            },
          },
        },
      },
    },
    llmProvider: "anthropic", // or 'openai', 'google', 'groq'
  }),
});

Deep Research Capabilities​

// Firecrawl API - Deep research on topics
const researchResult = await fetch("http://localhost:3002/v1/deep-research", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    Authorization: "Bearer your-api-key",
  },
  body: JSON.stringify({
    topic: "AI tools for developers",
    depth: "comprehensive",
    sources: 10,
    includeAnalysis: true,
  }),
});

Native Library Integration​

Rust-based Crawler​

// Using native Rust crawler for high-performance crawling
const crawlResult = await fetch("http://localhost:3002/v1/crawl", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    Authorization: "Bearer your-api-key",
  },
  body: JSON.stringify({
    url: "https://example.com",
    crawlerOptions: {
      engine: "rust", // Use Rust-based crawler
      maxConcurrency: 10,
      respectRobotsTxt: true,
      delayBetweenRequests: 1000,
    },
  }),
});

Go HTML-to-Markdown​

// High-performance HTML to Markdown conversion
const markdownResult = await fetch("http://localhost:3002/v1/scrape", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    Authorization: "Bearer your-api-key",
  },
  body: JSON.stringify({
    url: "https://example.com",
    formats: ["markdown"],
    converter: "go", // Use Go-based converter
  }),
});

Stealth and Anti-Detection​

Playwright Service Stealth Features​

// Advanced stealth configuration
const stealthResult = await fetch("http://localhost:3003/scrape", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    url: "https://example.com",
    stealth: {
      userAgent: "random", // Random user agent rotation
      viewport: { width: 1920, height: 1080 },
      timezone: "America/New_York",
      locale: "en-US",
      geolocation: { latitude: 40.7128, longitude: -74.006 },
    },
    antiDetection: {
      removeWebdriver: true,
      spoofChrome: true,
      randomizeFingerprint: true,
    },
  }),
});

Performance Optimizations​

Resource Blocking​

// Block unnecessary resources for faster scraping
const optimizedResult = await fetch("http://localhost:3003/scrape", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    url: "https://example.com",
    blockResources: {
      images: true,
      stylesheets: false,
      fonts: true,
      media: true,
      ads: true,
      analytics: true,
    },
    customBlockList: [
      "*.google-analytics.com",
      "*.facebook.com",
      "*.doubleclick.net",
    ],
  }),
});

Best Practices​

Error Handling and Retries​

// Robust error handling with exponential backoff
async function robustScrape(url, maxRetries = 3) {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      // Try Firecrawl first
      const firecrawlResult = await fetch("http://localhost:3002/v1/scrape", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({ url, formats: ["markdown"] }),
      });

      if (firecrawlResult.ok) {
        return await firecrawlResult.json();
      }
    } catch (error) {
      console.log(`Firecrawl attempt ${attempt} failed:`, error.message);
    }

    // Fallback to Playwright
    try {
      const playwrightResult = await fetch("http://localhost:3003/scrape", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({ url, timeout: 30000 }),
      });

      if (playwrightResult.ok) {
        return await playwrightResult.json();
      }
    } catch (error) {
      console.log(`Playwright attempt ${attempt} failed:`, error.message);
    }

    if (attempt === maxRetries) {
      throw new Error(`Failed after ${maxRetries} attempts`);
    }

    // Exponential backoff
    await new Promise((resolve) =>
      setTimeout(resolve, Math.pow(2, attempt) * 1000)
    );
  }
}

Rate Limiting and Respectful Scraping​

class RespectfulScraper {
  constructor(delayMs = 1000) {
    this.delayMs = delayMs;
    this.lastRequest = 0;
  }

  async scrape(url) {
    // Respect rate limits
    const timeSinceLastRequest = Date.now() - this.lastRequest;
    if (timeSinceLastRequest < this.delayMs) {
      await new Promise((resolve) =>
        setTimeout(resolve, this.delayMs - timeSinceLastRequest)
      );
    }

    this.lastRequest = Date.now();
    return await this.performScrape(url);
  }
}

Resource Management​

// Proper resource cleanup for Playwright Service
async function scrapeWithCleanup(urls) {
  const results = [];

  for (const url of urls) {
    try {
      const response = await fetch("http://localhost:3003/scrape", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({ url }),
      });

      const data = await response.json();
      results.push({ url, data });
    } catch (error) {
      console.error(`Failed to scrape ${url}:`, error);
    }
  }

  return results;
}

Configuration Management​

// Environment-based configuration
const config = {
  development: {
    firecrawlUrl: "http://localhost:3002",
    playwrightUrl: "http://localhost:3003",
    timeout: 30000,
    retries: 3,
  },
  production: {
    firecrawlUrl: "https://api.firecrawl.dev",
    playwrightUrl: "https://playwright.firecrawl.dev",
    timeout: 60000,
    retries: 5,
  },
};

const env = process.env.NODE_ENV || "development";
const currentConfig = config[env];

Troubleshooting​

Common Issues​

1. Service Communication Failures​

// Health check both services
async function checkServices() {
  try {
    const firecrawlHealth = await fetch(
      "http://localhost:3002/serverHealthCheck"
    );
    const playwrightHealth = await fetch("http://localhost:3003/health");

    console.log("Firecrawl API:", firecrawlHealth.ok ? "OK" : "FAILED");
    console.log("Playwright Service:", playwrightHealth.ok ? "OK" : "FAILED");
  } catch (error) {
    console.error("Service check failed:", error);
  }
}

2. Timeout Issues​

// Solution: Increase timeouts and add retries
const scrapeWithTimeout = async (url) => {
  try {
    const response = await fetch("http://localhost:3003/scrape", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        url,
        timeout: 60000, // 60 seconds
        wait_after_load: 5000, // Wait 5 seconds after load
      }),
    });

    return await response.json();
  } catch (error) {
    console.error("Timeout error:", error);
    throw error;
  }
};

3. Memory Leaks​

// Solution: Proper resource cleanup
async function scrapeWithCleanup(urls) {
  const results = [];

  for (const url of urls) {
    try {
      // Each request gets a fresh browser instance
      const response = await fetch("http://localhost:3003/scrape", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({ url }),
      });

      const data = await response.json();
      results.push({ url, data });
    } catch (error) {
      console.error(`Failed to scrape ${url}:`, error);
    }
  }

  return results;
}

4. Anti-Bot Detection​

// Solution: Use stealth mode and realistic behavior
const stealthScrape = async (url) => {
  const response = await fetch("http://localhost:3003/scrape", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      url,
      stealth: {
        userAgent: "random",
        viewport: { width: 1920, height: 1080 },
        timezone: "America/New_York",
      },
      antiDetection: {
        removeWebdriver: true,
        spoofChrome: true,
        randomizeFingerprint: true,
      },
    }),
  });

  return await response.json();
};

Debugging Tips​

// Enable debugging for both services
const debugScrape = async (url) => {
  console.log("Starting scrape for:", url);

  try {
    // Try Firecrawl first
    const firecrawlResult = await fetch("http://localhost:3002/v1/scrape", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ url, formats: ["markdown"] }),
    });

    if (firecrawlResult.ok) {
      console.log("Firecrawl success");
      return await firecrawlResult.json();
    }
  } catch (error) {
    console.log("Firecrawl failed:", error.message);
  }

  // Fallback to Playwright
  try {
    const playwrightResult = await fetch("http://localhost:3003/scrape", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ url }),
    });

    console.log("Playwright success");
    return await playwrightResult.json();
  } catch (error) {
    console.error("Both services failed:", error);
    throw error;
  }
};

Use Cases​

1. E-commerce Product Monitoring​

// Monitor product prices and availability
async function monitorProducts(productUrls) {
  const results = [];

  for (const url of productUrls) {
    try {
      // Use Playwright for dynamic e-commerce sites
      const response = await fetch("http://localhost:3003/scrape", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({
          url,
          wait_after_load: 3000,
          check_selector: ".product-info",
        }),
      });

      const data = await response.json();

      // Extract product data using Firecrawl AI
      const extractResponse = await fetch("http://localhost:3002/v1/extract", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({
          url,
          extractionPrompt:
            "Extract product name, price, availability, and images",
          extractionSchema: {
            type: "object",
            properties: {
              name: { type: "string" },
              price: { type: "string" },
              availability: { type: "string" },
              images: { type: "array", items: { type: "string" } },
            },
          },
        }),
      });

      const extractedData = await extractResponse.json();
      results.push({ url, ...extractedData, timestamp: new Date() });
    } catch (error) {
      console.error(`Failed to monitor ${url}:`, error);
    }
  }

  return results;
}

2. Social Media Content Extraction​

// Extract social media posts and interactions
async function extractSocialContent(url) {
  try {
    // Use Playwright for dynamic social media content
    const response = await fetch("http://localhost:3003/scrape", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        url,
        wait_after_load: 5000,
        check_selector: '[data-testid="post"]',
        stealth: {
          userAgent: "random",
          viewport: { width: 1920, height: 1080 },
        },
      }),
    });

    const data = await response.json();

    // Use Firecrawl AI to extract structured data
    const extractResponse = await fetch("http://localhost:3002/v1/extract", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        url,
        extractionPrompt:
          "Extract all posts with author, text, timestamp, and engagement metrics",
        extractionSchema: {
          type: "object",
          properties: {
            posts: {
              type: "array",
              items: {
                type: "object",
                properties: {
                  author: { type: "string" },
                  text: { type: "string" },
                  timestamp: { type: "string" },
                  likes: { type: "number" },
                  shares: { type: "number" },
                },
              },
            },
          },
        },
      }),
    });

    return await extractResponse.json();
  } catch (error) {
    console.error("Social content extraction failed:", error);
    throw error;
  }
}

3. Research and Content Aggregation​

// Deep research on topics using both services
async function deepResearch(topic) {
  try {
    // Use Firecrawl's deep research capabilities
    const researchResponse = await fetch(
      "http://localhost:3002/v1/deep-research",
      {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({
          topic,
          depth: "comprehensive",
          sources: 20,
          includeAnalysis: true,
        }),
      }
    );

    const researchData = await researchResponse.json();

    // Use Playwright for additional dynamic content
    const additionalSources = await Promise.all(
      researchData.sources.slice(0, 5).map(async (source) => {
        try {
          const response = await fetch("http://localhost:3003/scrape", {
            method: "POST",
            headers: { "Content-Type": "application/json" },
            body: JSON.stringify({
              url: source.url,
              wait_after_load: 3000,
            }),
          });

          return await response.json();
        } catch (error) {
          console.error(`Failed to scrape ${source.url}:`, error);
          return null;
        }
      })
    );

    return {
      research: researchData,
      additionalContent: additionalSources.filter(Boolean),
    };
  } catch (error) {
    console.error("Deep research failed:", error);
    throw error;
  }
}

4. Form Automation and Data Collection​

// Automate form submissions and data collection
async function automateFormSubmission(formUrl, formData) {
  try {
    // Use Playwright for form automation
    const response = await fetch("http://localhost:3003/scrape", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        url: formUrl,
        formData,
        wait_after_load: 2000,
        check_selector: ".success-message, .error-message",
      }),
    });

    const result = await response.json();

    // Use Firecrawl to extract the result page content
    const extractResponse = await fetch("http://localhost:3002/v1/scrape", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        url: formUrl,
        formats: ["markdown"],
        onlyMainContent: true,
      }),
    });

    const extractedContent = await extractResponse.json();

    return {
      formResult: result,
      pageContent: extractedContent,
    };
  } catch (error) {
    console.error("Form automation failed:", error);
    throw error;
  }
}

5. Competitive Intelligence​

// Monitor competitor websites and extract insights
async function competitiveIntelligence(competitorUrls) {
  const results = [];

  for (const url of competitorUrls) {
    try {
      // Use both services for comprehensive analysis
      const [firecrawlResult, playwrightResult] = await Promise.all([
        fetch("http://localhost:3002/v1/scrape", {
          method: "POST",
          headers: { "Content-Type": "application/json" },
          body: JSON.stringify({
            url,
            formats: ["markdown"],
            onlyMainContent: true,
          }),
        }),
        fetch("http://localhost:3003/scrape", {
          method: "POST",
          headers: { "Content-Type": "application/json" },
          body: JSON.stringify({
            url,
            wait_after_load: 3000,
          }),
        }),
      ]);

      const firecrawlData = await firecrawlResult.json();
      const playwrightData = await playwrightResult.json();

      // Use AI to extract competitive insights
      const insightsResponse = await fetch("http://localhost:3002/v1/extract", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({
          url,
          extractionPrompt:
            "Extract key business insights, pricing, features, and competitive advantages",
          extractionSchema: {
            type: "object",
            properties: {
              pricing: { type: "string" },
              features: { type: "array", items: { type: "string" } },
              advantages: { type: "array", items: { type: "string" } },
              weaknesses: { type: "array", items: { type: "string" } },
            },
          },
        }),
      });

      const insights = await insightsResponse.json();

      results.push({
        url,
        firecrawl: firecrawlData,
        playwright: playwrightData,
        insights,
      });
    } catch (error) {
      console.error(`Failed to analyze ${url}:`, error);
    }
  }

  return results;
}

Resources​

Smart Firecrawl API​

  • Documentation: Comprehensive API documentation with examples
  • GitHub: Smart Firecrawl API Repository
  • Docker Hub: Pre-built Docker images for easy deployment
  • API Reference: Complete endpoint documentation with request/response examples

Smart Playwright Service​

  • Documentation: Service-specific documentation and configuration
  • GitHub: Smart Playwright Service Repository
  • Docker Hub: Optimized Docker images with Playwright browsers
  • Configuration Guide: Environment variables and deployment options

Integration Resources​

  • Architecture Diagrams: Visual representations of system components
  • Deployment Guides: Step-by-step deployment instructions
  • Monitoring Setup: Health checks, logging, and alerting configuration
  • Performance Tuning: Optimization guides for production environments

Conclusion​

The Smart Firecrawl API and Smart Playwright Service form a comprehensive web scraping ecosystem that combines the best of both worlds:

Key Advantages​

  1. Intelligent Content Extraction: AI-powered extraction with multiple LLM providers
  2. Advanced Browser Automation: Stealth features and anti-detection capabilities
  3. Scalable Architecture: Queue-based processing with Redis coordination
  4. Native Performance: Rust and Go libraries for high-performance operations
  5. Enterprise Features: Authentication, billing, monitoring, and rate limiting

Use Cases Covered​

  • E-commerce Monitoring: Product tracking and price monitoring
  • Social Media Analysis: Content extraction and sentiment analysis
  • Research Automation: Deep research with AI-powered insights
  • Form Automation: Complex form submissions and data collection
  • Competitive Intelligence: Market analysis and competitor monitoring

Best Practices​

  • Respectful Scraping: Rate limiting and ethical practices
  • Error Handling: Robust retry mechanisms and fallback strategies
  • Resource Management: Proper cleanup and memory management
  • Security: Authentication, authorization, and data protection
  • Monitoring: Health checks, logging, and performance tracking

Getting Started​

  1. Deploy Services: Use Docker Compose for easy setup
  2. Configure Environment: Set up API keys and database connections
  3. Test Integration: Verify both services are working correctly
  4. Implement Use Cases: Start with simple scraping and scale up
  5. Monitor Performance: Set up logging and alerting

This comprehensive solution provides everything needed for modern web scraping applications, from simple content extraction to complex AI-powered data processing. The combination of Firecrawl's intelligent extraction capabilities and Playwright's advanced browser automation creates a powerful platform for any web scraping needs.

Remember to always respect website terms of service, implement proper error handling, and use reasonable delays to ensure ethical and sustainable scraping practices.