Use Screaming Frog's AI Integration for Faster Audit Analysis

What This Does

Screaming Frog now connects directly to OpenAI or Google Gemini, letting you run AI analysis on crawl data in bulk — generate meta descriptions, categorize issues, and analyze pages at scale without copying anything out of the tool.

Before You Start

• Screaming Frog SEO Spider installed (free version works for sites under 500 URLs; paid licence for larger sites — £259/year)
• An OpenAI API key OR a Google Gemini API key (you'll need a paid OpenAI account with API credits, or a Gemini API key from Google AI Studio — starts at ~$0.002 per request)
• A completed crawl of the site you're auditing

Steps

1. Connect your AI API key

With Screaming Frog open, navigate to Configuration → AI → Settings. Enter your OpenAI API key (from platform.openai.com → API keys) or Google Gemini key. Click Save and Test — you'll see a confirmation if the connection works.

What you should see: A green checkmark or "Connection successful" message.

Troubleshooting: If it fails, check that your API key has billing set up (OpenAI requires a payment method even for small usage).

2. Run a crawl

Click Start to crawl your target URL. For the AI integration to work most efficiently, wait for the full crawl to complete. A 500-page site typically takes 5-10 minutes.

3. Use AI bulk analysis on a column

After crawling, select the Crawl Overview tab and navigate to a specific report — for example, Response Codes → 200 OK pages. In the column view, right-click any column header and look for Custom Extraction or select pages, then from the Bulk Export menu choose AI Custom Analysis.

A dialog box appears. Type your prompt — for example: "Based on this page's title and meta description, write an improved meta description under 155 characters that naturally includes the target keyword."

What you should see: Screaming Frog sends each selected page's data to the AI API and returns results in a new column. This runs in the background and can process hundreds of pages.

4. Export the AI-generated output

Once the AI column populates, export to CSV via Bulk Export → Custom → Select your AI column. This gives you a spreadsheet with URLs + AI-generated content ready for review and CMS upload.

Real Example

Scenario: You're auditing a 300-page SaaS site. 180 pages have thin or duplicate meta descriptions. You want unique, optimized meta descriptions for all of them without writing each one manually.

What you do: Crawl the site → filter to pages with duplicate/missing metas → select all 180 → run AI bulk analysis with prompt: "Write a unique, SEO-optimized meta description under 155 characters for this page. Title: {page title}. Include a clear benefit and call to action." → export to CSV → QA the output → bulk upload to CMS.

Time saved: 180 × 10 min manually = 30 hours → 1 hour with AI bulk analysis.

Tips

• Start with a small test batch (10-20 pages) before running the AI analysis on your full crawl — verify the output quality before spending API credits on 500 pages.
• The AI integration costs real money at scale. A 500-page audit using GPT-4o will cost roughly $0.50-$2.00 depending on prompt length — very cheap for the time saved.
• Use this for FAQ content extraction too: prompt Screaming Frog to "identify 3 questions users likely have about this page topic" — use the output to build FAQ sections.

Tool interfaces change — if the AI menu has moved, check under Configuration → AI or the Bulk Export menu for current AI options.