GSC API + Claude Sonnet 4.6 + DataForSEO + Python: The 3-Script Workflow That Surfaces AI Overview Opportunities Across 500 Keywords

Google AI Mode is eating click-through rates on queries where your site has strong GSC impressions but weak CTR. The gap between “showing up in AI Overviews” and “not” often comes down to page structure — FAQ schema, question-based H3s, first-sentence direct answers.

The problem: identifying which 50 of your 500 ranked queries are AI Overview candidates takes manual SERP checking that no team has time for.

This workflow automates the identification step using three Python scripts that connect GSC API v1, Claude Sonnet 4.6, and the DataForSEO On-Page API. Total API cost: under $2 for 500 queries.

Why AI Overviews Require a New Keyword Detection Workflow

Google AI Mode changed the SERP for informational queries. When an AI Overview appears for a query, the organic click-through rate (CTR) for positions 1–5 drops — the answer sits above the fold and many users never scroll.

The queries most affected are the ones you’d expect: definition queries (“what is X”), how-to queries (“how do I X”), comparison queries (“X vs Y”), and list queries (“best X for Y”).

Per Google Search Central’s documentation, AI Overviews are most common on queries that benefit from synthesis and explanation rather than a specific link. GSC shows which queries your site ranks for and what CTR you’re getting — but it doesn’t show which of those queries has an AI Overview. That requires a live SERP check or a classification step.

The workflow below uses Claude Sonnet 4.6 to predict AI Overview eligibility from query text alone, then confirms with DataForSEO for the high-priority candidates.

Setting Up the GSC API v1 Connection with Python

The GSC API uses OAuth2 or service account authentication. For automated scripts, service accounts are simpler — no browser interaction required.

Install the required libraries:

  pip install google-auth google-auth-httplib2 google-api-python-client

Authenticate using a service account JSON key. In Google Search Console, add the service account email as a property owner (not just “Full User” — the API requires owner-level access for query data).

  from google.oauth2 import service_account
from googleapiclient.discovery import build

SCOPES = ['https://www.googleapis.com/auth/webmasters.readonly']
KEY_FILE = 'gsc-service-account.json'
SITE_URL = 'sc-domain:yoursite.com'  # or https://yoursite.com/

credentials = service_account.Credentials.from_service_account_file(
      KEY_FILE, scopes=SCOPES
  )
service = build('searchconsole', 'v1', credentials=credentials)

One note on site URL format: GSC API accepts both https://yoursite.com/ (prefix property) and sc-domain:yoursite.com (domain property). The domain property covers all subdomains and protocols — use it if available.

Script 1: Pull 500 Keywords from GSC with Impression and Click Data

Script 1 queries the GSC API for the last 28 days, pulls queries with at least 10 impressions, and outputs a CSV with query, position, clicks, impressions, and CTR.

  import csv
from datetime import datetime, timedelta

def pull_gsc_queries(service, site_url, rows=500):
end_date = datetime.today().strftime('%Y-%m-%d')
start_date = (datetime.today() - timedelta(days=28)).strftime('%Y-%m-%d')
      body = {
          'startDate': start_date,
          'endDate': end_date,
          'dimensions': ['query'],
          'rowLimit': rows,
          'dimensionFilterGroups': [{
'filters': [{'dimension': 'impressions', 'operator': 'greaterThan', 'expression': '10'}]
          }]
      }
response = service.searchanalytics().query(siteUrl=site_url, body=body).execute()
with open('gsc_queries.csv', 'w', newline='', encoding='utf-8') as f:
          writer = csv.writer(f)
writer.writerow(['query', 'position', 'clicks', 'impressions', 'ctr'])
for row in response.get('rows', []):
writer.writerow([row['keys'][0], round(row['position'], 1),
row['clicks'], row['impressions'], round(row['ctr'], 4)])
return len(response.get('rows', []))

Script 2: Classify AI Overview Eligibility with Claude Sonnet 4.6

Script 2 reads the CSV from Script 1 and sends batches of 50 queries to Claude Sonnet 4.6 with a classification prompt.

The prompt classifies each query against four eligibility signals: informational intent (likely), how-to intent (likely), comparison intent (moderate), navigational intent (unlikely). The model returns a JSON array with an ao_score (0–10) and a one-line reason per query.

  import anthropic, json, csv

client = anthropic.Anthropic()  # uses ANTHROPIC_API_KEY env var

  CLASSIFY_PROMPT = (
"Classify each query for Google AI Overview eligibility. "
"Return JSON array: [{\"query\": \"...\", \"ao_score\": 0-10, "
"\"intent\": \"informational|howto|comparison|definition|navigational|transactional\", "
      "\"reason\": \"one line\"}]. "
"ao_score 8-10=very likely, 5-7=possible, 0-4=unlikely.\n\nQueries:\n{queries}"
  )

  def classify_batch(queries):
query_list = '\n'.join([f'- {q}' for q in queries])
      msg = client.messages.create(
          model="claude-sonnet-4-6",
          max_tokens=2000,
messages=[{"role": "user", "content": CLASSIFY_PROMPT.format(queries=query_list)}]
      )
return json.loads(msg.content[0].text)

Run this in batches of 50 to keep each API call under 2,048 output tokens. A 500-query dataset requires 10 API calls at roughly $0.03 per call using Sonnet 4.6 input/output pricing.

Script 3: DataForSEO On-Page API to Check SERP Features and FAQ Schema

Script 2 returns a shortlist of queries with ao_score ≥ 7. Script 3 takes the top 50 of those and runs them through DataForSEO’s SERP Features endpoint to confirm whether a Google AI Overview is actually appearing.

  import requests, base64

def check_serp_features(queries, dfs_login, dfs_pass):
auth = base64.b64encode(f"{dfs_login}:{dfs_pass}".encode()).decode()
headers = {'Authorization': f'Basic {auth}', 'Content-Type': 'application/json'}
tasks = [{"keyword": q, "location_code": 2840, "language_code": "en"} for q in queries]
      resp = requests.post(
'https://api.dataforseo.com/v3/serp/google/organic/live/advanced',
          headers=headers, json=tasks
      ).json()
      results = []
for task in resp.get('tasks', []):
keyword = task['data']['keyword']
items = task['result'][0].get('items', []) if task.get('result') else []
features = [i['type'] for i in items]
          results.append({
              'query': keyword,
'ai_overview_live': 'ai_overview' in features,
'faq_in_serp': 'faq' in features
          })
      return results

DataForSEO’s SERP response includes an items array where type can be ai_overview, faq, featured_snippet, organic, and others. Check for ai_overview in that list to confirm live appearance.

Reading the Output: 3 Signals That Predict AI Overview Appearance

After running all three scripts, your output CSV has five columns: query, position, CTR, ao_score, and ai_overview_live. Prioritize pages where all three signals align:

Signal 1 — High impressions, low CTR vs site average. This is the strongest ROI signal. The query has reach but organic CTR is suppressed, likely by an AI Overview above the fold.

Signal 2 — Claude ao_score ≥ 7 with intent = informational or howto. These query types align with how Google triggers AI Overviews most often, per Google Search Central’s published documentation on generative search features.

Signal 3 — DataForSEO confirms ai_overview_live = True AND faq_in_serp = False. If there’s an AI Overview but no FAQ box, the page structure likely lacks FAQ schema — that’s the gap to fill.

“We generally try to show AI Overviews for queries where the format is particularly helpful — queries that are complex, that benefit from synthesis, and where users are looking for an explanation or overview rather than a specific link.” — Per Google Search Central’s documentation on generative search features.

Common Failure Modes and Fixes

Three failure modes come up consistently when running this workflow at scale.

Failure 1 — GSC API returns 403 for query-level data. The service account needs Owner permission in GSC, not just Full User. Full User access blocks query data via the API. Fix: in GSC Settings → Users and Permissions, change the service account to Owner.

Failure 2 — Claude Sonnet 4.6 JSON output truncated on long query batches. When any query in the batch exceeds 120 characters, the output JSON occasionally truncates before the closing bracket. Fix: filter queries longer than 120 characters out of the batch, or reduce batch size to 30.

Failure 3 — DataForSEO ai_overview item type absent from older task results. DataForSEO added the ai_overview item type in their API schema in early 2026. Tasks run against cached SERP snapshots may not include it. Fix: use live endpoints, not regular, and pass calculate_rectangles: false to reduce response size.

What to Do with the Shortlist: FAQ Schema and BLUF Intro Retrofits

Your final output is a list of pages that are likely suppressed by AI Overviews and currently lack the page structure to be cited in them.

For each page on the shortlist, apply three changes:

1. Add FAQPage JSON-LD schema. Include 4–6 questions directly related to the target query and its semantic variants. Each answer should be one to three sentences, direct, and factual. Use Schema.org FAQPage with Question and acceptedAnswer.

2. Rewrite the intro as BLUF (Bottom Line Up Front). The first sentence should answer the query directly. Google’s AI Overview citation logic favors pages that answer in the first paragraph, not pages that build context before answering.

3. Convert flat H2 headings to question format. “Section 2: Benefits” → “What are the main benefits of X?” Each question H2 or H3 should be answered in its first sentence. This matches the structure Google’s AI Overview rendering uses for citations.

FAQ: GSC API + Claude Sonnet 4.6 + DataForSEO AI Overview Workflow

Does this workflow work on Google Search Console’s free plan?

Yes. The GSC API v1 is free for all Google Search Console users. You need a Google Cloud service account and OAuth2 credentials — both are free to create via Google Cloud Console. There are no API call volume charges from Google for this data.

What Claude model should I use for the classification step?

Claude Sonnet 4.6 is the right balance of cost and accuracy for this task. Claude Haiku 4.5 is cheaper but produces more classification errors on ambiguous queries. Claude Opus 4 is overkill for a binary classification task. Sonnet 4.6 with the prompt above handles informational vs. transactional intent well on English queries.

How accurate is Claude Sonnet 4.6’s AI Overview prediction?

The model classifies query intent accurately. Whether a specific query triggers an AI Overview also depends on domain authority, the specific SERP, and Google’s freshness signals. Use Claude’s classification as a fast filter — DataForSEO’s live SERP check (Script 3) provides the confirmation you need before committing to a content retrofit.

Can I use this workflow with Bing or Perplexity data instead of GSC?

Yes, with modifications. Bing Webmaster Tools has a query analytics API similar to GSC v1. Perplexity and ChatGPT don’t expose query-level data directly, but you can proxy their citations using Ahrefs or DataForSEO Backlinks API to find which of your pages get cited by AI answer engines.

How often should I run this workflow?

Run it monthly. GSC query data shifts as Google updates its algorithms and AI Overview eligibility expands. A monthly cadence catches new opportunities before competitors react. The total runtime for 500 keywords is under 10 minutes on a standard Python environment.

What schema type is best for AI Overview eligibility?

Per Google Search Central’s structured data documentation, FAQPage schema is the highest-signal schema type for AI Overview citation eligibility on informational pages. HowTo schema performs well on procedural queries. Combine FAQPage with Article schema and speakable markup for maximum coverage on definition and explanation queries.

Last updated: July 2026 | DesignCopy — AI, Data Science, and SEO