Olostep - Web Scraping & Crawling API

Setup

Read your credentials from ~/.gooseworks/credentials.json:

bash
export GOOSEWORKS_API_KEY=$(python3 -c "import json;print(json.load(open('$HOME/.gooseworks/credentials.json'))['api_key'])")
export GOOSEWORKS_API_BASE=$(python3 -c "import json;print(json.load(open('$HOME/.gooseworks/credentials.json')).get('api_base','https://api.gooseworks.ai'))")

If ~/.gooseworks/credentials.json does not exist, tell the user to run: npx gooseworks login

All endpoints use Bearer auth: -H "Authorization: Bearer $GOOSEWORKS_API_KEY"

Powerful web scraping, crawling, and AI-powered content extraction.

Capabilities

• Create Scrape: Initiate a web page scrape
• Create Answer: The AI will perform actions like searching and browsing web pages to find the answer to the provided task
• Maps: This endpoint allows users to get all the urls on a certain website
• Start Crawl: Starts a new crawl
• Start Batch: Starts a new batch
• Batch Items: Retrieves the list of items processed for a batch
• Crawl Info: Fetches information about a specific crawl
• Crawl Pages: Fetches the list of pages for a specific crawl
• Get Answer: This endpoint retrieves a previously completed answer by its ID
• Get Scrape: Can be used to retrieve response for a scrape
• Batch Info: Retrieves the status and progress information about a batch
• Retrieve Content: Retrieve page content of processed batches and crawls urls

Usage

Create Scrape

Initiate a web page scrape

Parameters:

• url_to_scrape (string) - The URL to start scraping from.
• wait_before_scraping (integer) - Time to wait in milliseconds before starting the scraping.
• formats (string]) - Formats in which you want the content.
• remove_css_selectors (string) - Option to remove certain CSS selectors from the content. Optionally, you can also pass a JSON stringified array of specific selectors you want to remove. The CSS selectors removed when this option is set to default are 'nav','footer','script','style','noscript','svg',role=alert],role=banner],role=dialog],role=alertdialog],role=region]aria-label=skip i],aria-modal=true]] Available options: default, none, array
• actions (object]) - Actions to perform on the page before getting the content.
• country (string) - Residential country to load the request from. Supported values are: US (United States) CA (Canada) IT (Italy) IN (India) GB (England) JP (Japan) MX (Mexico) AU (Australia) ID (Indonesia) UA (UAE) RU (Russia) RANDOM Some operations, like scraping Google Search and Google News, support all countries.
• transformer (string) - Specify the HTML transformer to use, if any. Postlight's Mercury Parser library is used to remove ads and other unwanted content from the scraped content. Available options: postlight, none
• remove_images (boolean) - Option to remove images from the scraped content. Defaults to false.
• remove_class_names (string]) - List of class names to remove from the content.
• parser (object) - When defining json as a format, you can use this parameter to specify the parser to use. Parsers are useful to extract structured content from web pages. Olostep has a few parsers built in for most common web pages, and you can also create your own parsers.
• llm_extract (object)
• links_on_page (object) - With this option, you can get all the links present on the page you scrape.
• screen_size (object) - Configuration for screen size. Preset dimensions are available through screen_type: desktop (1920x1080), mobile (414x896), or default (768x1024).
• metadata (object) - User-defined metadata. Not supported yet

bash
curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \
  -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"api":"olostep","path":"/v1/scrapes","body":{"url_to_scrape":"https://example.com/page"}}'

Create Answer

The AI will perform actions like searching and browsing web pages to find the answer to the provided task. Execution time is 3-30s depending upon complexity. For longer tasks, use the agent endpoint instead.

Parameters:

• task (string) - The task to be performed.
• json_format (object) - The desired output JSON object with empty values as a schema, or simply describe the data you want as a string.

bash
curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \
  -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"api":"olostep","path":"/v1/answers","body":{"task":"What are the latest AI developments?"}}'

Maps

This endpoint allows users to get all the urls on a certain website. It can take up to 120 seconds for complex websites. For large websites, results are paginated using cursor-based pagination

Parameters:

• url (string) - The URL of the website for which you want the links
• search_query (string) - An optional search query to sort the links by search relevance.
• top_n (number) - An optional number to limit to only top n links for a search query.
• include_subdomain (boolean) - Include subdomains of the given URL. true by default.
• include_urls (string]) - URL path patterns to include using glob syntax. For example: /blog/** to only include blog URLs. Only URLs matching these patterns will be returned.
• exclude_urls (string]) - URL path patterns to exclude using glob syntax. For example: /careers/**. Excluded URLs will supersede included URLs.
• cursor (string) - OPTIONAL: Pagination cursor from a previous response. When provided, returns the next set of URLs from where the previous request left off due to response size limit.

bash
curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \
  -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"api":"olostep","path":"/v1/maps","body":{"url":"https://example.com"}}'

Start Crawl

Starts a new crawl. You receive a id to track the progress. The operation may take 1-10 mins depending upon the site and depth and pages parameters.

Parameters:

• start_url (string) - The starting point of the crawl.
• max_pages (number) - Maximum number of pages to crawl. Recommended for most use cases like crawling an entire website.
• include_urls (string]) - URL path patterns to include in the crawl using glob syntax. Defaults to /** which includes all URLs. Use patterns like /blog/** to crawl specific sections (e.g., only blog pages), /products/*.html for product pages, or multiple patterns for different sections. Supports standard glob features like (any characters) and (recursive matching).
• exclude_urls (string]) - URL path names in glob pattern to exclude. For example: /careers/**. Excluded URLs will supersede included URLs.
• max_depth (number) - Maximum depth of the crawl. Useful to extract only up to n-degree of links.
• include_external (boolean) - Crawl first-degree external links.
• include_subdomain (boolean) - Include subdomains of the website. false by default.
• search_query (string) - An optional search query to find specific links and also sort the results by relevance.
• top_n (number) - An optional number to only crawl the top N most relevant links on every page as per search query.
• webhook_url (string) - An optional POST request endpoint called when this crawl is completed. The body of the request will be same as the response of this v1/crawls/{crawl_id} endpoint.
• timeout (number) - End the crawl after n seconds with the pages completed until then. May take ~10s extra from provided timeout.

bash
curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \
  -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"api":"olostep","path":"/v1/crawls"}'
  "start_url": "https://example.com",
  "max_pages": 100
}'

Start Batch

Starts a new batch. You receive an id that you can use to track the progress of the batch as shown here. Note: Processing time is constant regardless of batch size

Parameters:

• items (object]) - Array of items to be processed in the batch.
• country (string) - Country for the batch execution. Provide in ISO 3166-1 alpha-2 codes like US(USA), IN(India), etc
• parser (object) - You can use this parameter to specify the parser to use. Parsers are useful to extract structured content from web pages. Olostep has a few parsers built in for most common web pages, and you can also create your own parsers.
• links_on_page (object) - Get all the links present on each page in the batch.

bash
curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \
  -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"api":"olostep","path":"/v1/batches"}'
  "items": [
    {"url_to_scrape": "https://example.com/page1"},
    {"url_to_scrape": "https://example.com/page2"}
  ]
}'

Batch Items

Retrieves the list of items processed for a batch. You can then use the retrieve_id to get the content with the Retrieve Endpoint

bash
curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \
  -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"api":"olostep","path":"/v1/batches/{batch_id}/items"}'

Crawl Info

Fetches information about a specific crawl.

bash
curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \
  -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"api":"olostep","path":"/v1/crawls/{crawl_id}"}'

Crawl Pages

Fetches the list of pages for a specific crawl.

bash
curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \
  -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"api":"olostep","path":"/v1/crawls/{crawl_id}/pages"}'

Get Answer

This endpoint retrieves a previously completed answer by its ID.

bash
curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \
  -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"api":"olostep","path":"/v1/answers/{answer_id}"}'

Get Scrape

Can be used to retrieve response for a scrape.

bash
curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \
  -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"api":"olostep","path":"/v1/scrapes/{scrape_id}"}'

Batch Info

Retrieves the status and progress information about a batch. To retrieve the content for a batch, see here

bash
curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \
  -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"api":"olostep","path":"/v1/batches/{batch_id}"}'

Retrieve Content

Retrieve page content of processed batches and crawls urls.

Parameters:

• retrieve_id (string) - The ID of the page content to retrieve. Available in the response of /v1/crawls/{crawl_id}/pages, /v1/scrapes/{scrape_id} or /v1/batches/{batch_id}/items endpoints
• formats (string]) - Optional array to retrieve only specific formats in production. If not provided, all formats will be returned.

bash
curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \
  -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"api":"olostep","path":"/v1/retrieve","body":{"retrieve_id":"abc123"}}'

Use Cases

1. Data Collection: Gather data from websites at scale
2. Content Monitoring: Track changes on competitor sites
3. Research Automation: Get AI-synthesized answers from web sources
4. SEO Analysis: Crawl and analyze site structure

Discover More

For full endpoint details and parameters:

bash
curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/search \
  -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"prompt":"olostep API endpoints"}' List all endpoints
curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/details \
  -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"api":"olostep","path":"/v1/scrapes"}'   # Get endpoint details