n8n templatesn8n templates

Vision-Based AI Agent Scraper - Integrating Google Sheets, ScrapingBee, and Gemini

This workflow combines visual intelligence AI and HTML scraping to automatically extract structured data from webpage screenshots. It supports e-commerce information monitoring, competitor data collection, and market analysis. It can automatically supplement data when the screenshot information is insufficient, ensuring high accuracy and completeness. Ultimately, the extracted information is converted into JSON format for easier subsequent processing and analysis. This solution significantly enhances the automation of data collection and is suitable for users who need to quickly obtain multidimensional information from webpages.

Workflow Diagram
Vision-Based AI Agent Scraper - Integrating Google Sheets, ScrapingBee, and Gemini Workflow diagram

Workflow Name

Vision-Based AI Agent Scraper - Integrating Google Sheets, ScrapingBee, and Gemini

Key Features and Highlights

This workflow leverages a vision-intelligent AI agent combined with Google Sheets, ScrapingBee, and the Google Gemini-1.5-Pro model to automatically extract structured data from webpage screenshots. It primarily supports full-page screenshot-based visual data scraping and, when screenshot information is insufficient, automatically supplements it with HTML scraping to ensure data completeness and accuracy. Using a structured output parser, the extracted data is converted into JSON format for easy downstream processing and analysis. Additionally, HTML content is transformed into Markdown format to enhance processing efficiency and reduce costs.

Core Problems Addressed

  • Traditional web scraping often relies on HTML code, which can struggle to accurately extract data from complex page structures or dynamically rendered content.
  • Visual recognition methods can directly extract information from page screenshots but may risk missing or incomplete data recognition.
  • This workflow combines AI vision with HTML scraping, automatically detecting and supplementing missing data, significantly improving scraping accuracy and robustness.
  • Enables automated, structured data collection, reducing manual annotation and processing workload.

Application Scenarios

  • Monitoring e-commerce product information such as prices, brands, and promotional statuses through automated collection.
  • Competitor website data gathering.
  • Market research and analysis, especially scenarios requiring visual-level understanding of page content.
  • Business processes that require consolidating multi-page data into spreadsheets for subsequent processing.

Main Workflow Steps

  1. Manually trigger the workflow start.
  2. Retrieve the list of URLs to be scraped from Google Sheets.
  3. Use the ScrapingBee API to capture full-page screenshots of target webpages.
  4. The vision AI agent (based on Google Gemini-1.5-Pro model) analyzes screenshots to identify and extract product titles, prices, brands, and promotional information.
  5. If screenshot extraction is incomplete, automatically invoke the HTML scraping tool to obtain page source code, converting it to Markdown for AI-assisted recognition.
  6. Use the structured output parser to format extracted data into a JSON array.
  7. Split data items via the “Split Out” node and write them individually into the “Results” sheet in Google Sheets.
  8. Save results in real-time for easy review and further use.

Systems and Services Involved

  • Google Sheets: Stores URLs to be scraped and final structured results.
  • ScrapingBee: Responsible for webpage screenshots and HTML content scraping.
  • Google Gemini-1.5-Pro Model: Core vision AI model for screenshot information recognition and text understanding.
  • Built-in n8n Nodes: Assist with task triggering, data splitting, JSON parsing, Markdown conversion, and other processing.

Target Users and Value

  • E-commerce operators and market analysts seeking rapid access to competitor product information and promotional updates.
  • Data scientists and developers building customized visual web data scraping solutions.
  • Business automation specialists aiming to enhance cross-platform data integration efficiency.
  • Any users needing to combine visual and textual information for automated multi-dimensional web data collection.

This workflow template is designed for e-commerce pages with structured fields including product title, price, brand, and promotion details. Users can customize output formats and scraping content according to their specific needs, enabling flexible and diverse visual web data acquisition. Please ensure compliance with relevant legal regulations regarding web scraping before use.