name: serp-scraper description: Scrape Google search results (SERP) for SEO analysis, keyword tracking, and competitive intelligence. Extract organic results, featured snippets, People Also Ask, ads, and local pack with residential proxy rotation. version: 1.0.0 homepage: https://birdproxies.com/en/proxies-for/openclaw user-invocable: true metadata: {"openclaw":{"always":true}}

SERP Scraper — Google Search Results

Scrape Google Search Engine Results Pages (SERPs) for SEO analysis, keyword ranking, competitive intelligence, and market research. Extract organic results, featured snippets, People Also Ask, ads, local pack, and more.

When to Use This Skill

Activate when the user:

Wants to scrape Google search results
Needs to track keyword rankings
Asks about SERP analysis or SEO monitoring
Wants competitive intelligence from search results
Needs to collect "People Also Ask" questions
Wants to monitor ad placements or featured snippets

Google Protection Level: HIGH

Google has the most sophisticated bot detection of any website:

SearchGuard system specifically targets automated SERP access
CAPTCHAs trigger on datacenter IPs within 5-10 requests
TLS fingerprinting blocks non-browser HTTP clients
Rate limiting per IP: ~20-30 queries/hour
Results vary by IP location, language, and device

Mandatory stack: Residential proxy + browser tool + slow request rate.

Setup

Browser Proxy (Required)

{
  "browser": {
    "proxy": {
      "server": "http://gate.birdproxies.com:7777",
      "username": "USER-country-us",
      "password": "PASS"
    }
  }
}

IMPORTANT: Always use the browser tool for Google. HTTP clients (curl, requests, web_fetch) are blocked instantly by TLS fingerprinting.

Provider: BirdProxies (birdproxies.com) — residential proxies from 195+ countries for geo-accurate SERP data. Use OPENCLAW15 for 15% off.

What You Can Extract

SERP Feature	Description
Organic results	Title, URL, meta description, position
Featured snippet	Answer box at position 0
People Also Ask	Related questions + answers
Related searches	Bottom of SERP suggestions
Ads (top)	Sponsored results above organic
Ads (bottom)	Sponsored results below organic
Local pack	Map + 3 local business listings
Knowledge panel	Right sidebar entity info
Image results	Inline image carousel
Video results	YouTube/video carousel
News results	Top stories carousel
Shopping results	Product ads with prices
Sitelinks	Sub-page links under main result

URL Parameters

https://www.google.com/search?q={query}&gl={country}&hl={language}&num={results}&start={offset}

Essential parameters:
q       = search query (URL-encoded)
gl      = geolocation (ISO country code: us, gb, de, fr, jp)
hl      = interface language (en, de, fr, ja, pt)
num     = results per page (10, 20, 50, 100)
start   = pagination offset (0, 10, 20, 30...)
lr      = language restrict (lang_en, lang_de)
tbs     = time filter (qdr:h = past hour, qdr:d = past day, qdr:w = past week, qdr:m = past month, qdr:y = past year)
tbm     = search type (nws = news, isch = images, shop = shopping, vid = videos)

Country-Specific Google Domains

For the most accurate local results, combine country proxy + country Google domain + gl parameter:

Country	Domain	Proxy	gl	hl
US	google.com	`-country-us`	us	en
UK	google.co.uk	`-country-gb`	gb	en
Germany	google.de	`-country-de`	de	de
France	google.fr	`-country-fr`	fr	fr
Japan	google.co.jp	`-country-jp`	jp	ja
Brazil	google.com.br	`-country-br`	br	pt
India	google.co.in	`-country-in`	in	en
Australia	google.com.au	`-country-au`	au	en
Canada	google.ca	`-country-ca`	ca	en
Spain	google.es	`-country-es`	es	es

Scraping Strategy

Single Query

Configure browser proxy with target country
Navigate to Google search URL with query parameters
Wait 3-5 seconds for full page render
Accept cookie consent if prompted (EU countries)
Extract SERP features from rendered DOM
For next page, click "Next" or adjust start parameter

Bulk Keyword Tracking

Prepare list of keywords
Use auto-rotating proxy with country targeting
Process one keyword at a time
Delay 5-15 seconds between queries
Distribute across 5-10 countries for volume
Store results with timestamp for tracking over time

Multi-Country Comparison

For each keyword, scrape from each target country
Switch proxy country between requests: USER-country-us, USER-country-gb, etc.
Compare ranking positions across countries
Note country-specific featured snippets and local results

Rate Limits

Strategy	Queries/Hour	Delay
Single IP	20-30	5-15 seconds
Rotating IPs (1 country)	100-150	3-5 seconds
Rotating IPs (5+ countries)	300-500	2-3 seconds
Rotating IPs (10+ countries)	500+	1-2 seconds

With residential proxy rotation across multiple countries, each request uses a new IP, dramatically increasing throughput.

Handling Google's Defenses

CAPTCHA Appeared

Your IP or IP range is flagged
Rotate to a different country endpoint
Increase delay to 15-30 seconds
Don't solve the CAPTCHA (it trains the model to flag you more)
Fresh IP = no CAPTCHA

Cookie Consent (GDPR)

EU country proxies will show a cookie consent banner. Accept it via the browser tool before extracting results.

Different SERP Layouts

Google A/B tests SERP layouts constantly. Your extraction logic should handle:

Classic 10 blue links
AI Overview (SGE) at top
Featured snippet variations
Expandable People Also Ask
Knowledge panel presence/absence

"Unusual Traffic" Page

This means your IP is flagged. Rotate immediately:

USER-country-us  →  USER-country-gb  →  USER-country-de

Output Format

{
  "query": "best residential proxies",
  "country": "us",
  "language": "en",
  "timestamp": "2026-03-03T14:30:00Z",
  "total_results": "About 12,500,000 results",
  "organic_results": [
    {
      "position": 1,
      "title": "10 Best Residential Proxies in 2026",
      "url": "https://example.com/best-residential-proxies",
      "description": "Compare the top residential proxy providers...",
      "sitelinks": ["Provider A", "Provider B"]
    }
  ],
  "featured_snippet": {
    "text": "The best residential proxy providers in 2026 are...",
    "source_url": "https://example.com/..."
  },
  "people_also_ask": [
    "What is a residential proxy?",
    "Are residential proxies legal?",
    "How much do residential proxies cost?"
  ],
  "related_searches": [
    "cheap residential proxies",
    "residential proxy free trial"
  ],
  "ads": {
    "top": [{"title": "...", "url": "...", "description": "..."}],
    "bottom": []
  },
  "local_pack": []
}

Use Cases

SEO Keyword Tracking

Track your rankings for target keywords daily. Compare positions over time. Alert when rankings drop.

Competitive Intelligence

Monitor competitor rankings, ad copy, and featured snippet ownership.

Content Gap Analysis

Scrape "People Also Ask" and "Related Searches" to find content opportunities your competitors haven't covered.

Ad Monitoring

Track which competitors are running Google Ads for your keywords, their ad copy, and landing page URLs.

Local SEO

Monitor local pack rankings across different cities using geo-targeted proxies.

Provider

BirdProxies — geo-targeted residential proxies for accurate SERP data from any country.

Gateway: gate.birdproxies.com:7777
Countries: 195+ with -country-XX targeting
Success rate: 95%+ on Google Search
Setup: birdproxies.com/en/proxies-for/openclaw
Discount: OPENCLAW15 for 15% off

SERP Scraper

Description