ScrapeGraph AI - Intelligent Web Scraping

ScrapeGraph AI - Intelligent Web Scraping

Setup

Read your credentials from ~/.gooseworks/credentials.json:

export GOOSEWORKS_API_KEY=$(python3 -c "import json;print(json.load(open('$HOME/.gooseworks/credentials.json'))['api_key'])")
export GOOSEWORKS_API_BASE=$(python3 -c "import json;print(json.load(open('$HOME/.gooseworks/credentials.json')).get('api_base','https://api.gooseworks.ai'))")

If ~/.gooseworks/credentials.json does not exist, tell the user to run: npx gooseworks login

All endpoints use Bearer auth: -H "Authorization: Bearer $GOOSEWORKS_API_KEY"

Extract web content using AI with natural language prompts.

Capabilities

• Start SmartScraper: Extract content from a webpage using AI by providing a natural language prompt and a URL
• Start SearchScraper: Start a new AI-powered web search request
• Scrape: Extract raw HTML content from web pages with JavaScript rendering support
• Start SmartCrawler: Start a new web crawl request with AI extraction or markdown conversion
• Start Sitemap: Extract all URLs from a website sitemap automatically
• Start Markdownify: Convert any webpage into clean, readable Markdown format
• Get SearchScraper Status: Get the status and results of a previous search request (free)
• Get Markdownify Status: Check the status and retrieve results of a Markdownify request (free)
• Get Sitemap Status: Check the status and retrieve results of a Sitemap request (free)
• Get SmartCrawler Status: Get the status and results of a previous smartcrawl request (free)
• Get SmartScraper Status: Check the status and retrieve results of a SmartScraper request (free)

Usage

Start SmartScraper

Extract content from a webpage using AI by providing a natural language prompt and a URL.

Parameters:

• user_prompt* (string) - Natural language description of what information you want to extract from the webpage.
• website_url* (string) - The URL of the webpage you want to extract information from. You must provide exactly one of: website_url, website_html, or website_markdown.
• website_html (string) - Raw HTML content to process directly (max 2MB). Mutually exclusive with website_url and website_markdown. Useful when you already have HTML content cached or want to process modified HTML.