playwright-scraper-skill

Overview

Playwright Scraper Skill is a powerful web scraping OpenClaw Skill that leverages Playwright for anti-bot protection. It supports various use cases, from simple sites to complex Cloudflare-protected ones.

Key Features

• Supports various use cases, including regular sites, dynamic sites, and Cloudflare-protected sites
• Includes anti-bot protection techniques, such as hiding automation markers and using realistic User-Agent
• Offers customization options, including environment variables for screenshot path, wait time, and User-Agent
• Supports headful mode for debugging

How It Works

The skill uses Playwright to scrape websites, with different scripts for different use cases. The web_fetch tool is used for simple sites, while playwright-simple.js and playwright-stealth.js are used for dynamic and Cloudflare-protected sites, respectively.

Use Cases

• Regular sites: Use OpenClaw’s built-in web_fetch tool
• Dynamic sites: Use playwright-simple.js
• Cloudflare-protected sites: Use playwright-stealth.js
• YouTube video transcripts: Use deep-scraper (install separately)
• Reddit: Use reddit-scraper (install separately)

Installation

1. Clone the repository
2. Run npm install and npx playwright install chromium

Quick Start

1. Use OpenClaw’s built-in web_fetch tool for simple sites
2. Use playwright-simple.js for dynamic sites
3. Use playwright-stealth.js for Cloudflare-protected sites
4. Use deep-scraper for YouTube video transcripts
5. Use reddit-scraper for Reddit