n8n templatesn8n templates

Agent with Custom HTTP Request

This workflow combines intelligent AI agents with the OpenAI GPT-4 model to achieve automatic web content scraping and processing. After the user inputs a chat message, the system automatically generates HTTP request parameters, retrieves web content from a specified URL, performs deep cleaning of the HTML, and finally outputs it in Markdown format. It supports both complete and simplified scraping modes, intelligently handles request errors, and provides feedback and suggestions. This workflow is suitable for content monitoring, information collection, and AI question-answering systems, enhancing information retrieval efficiency and reducing manual intervention.

Workflow Diagram
Agent with Custom HTTP Request Workflow diagram

Workflow Name

Agent with Custom HTTP Request

Key Features and Highlights

This workflow leverages an intelligent AI Agent (ReAct AI Agent) combined with the OpenAI GPT-4 model to process user-initiated chat messages. It intelligently generates query parameters conforming to HTTP request formats, fetches web content from specified URLs, and performs deep cleaning and simplification of the webpage HTML. The cleaned content is then converted into Markdown format for output. It supports two content extraction modes (full and simplified) and automatically handles request errors by providing appropriate feedback and adjustment suggestions.

Core Problems Addressed

  • Automates web content scraping and intelligent extraction of relevant information, eliminating the complexity and inefficiency of manual webpage parsing.
  • Utilizes an AI Agent to guide the construction of request parameters, reducing API call complexity and error rates.
  • Cleans redundant scripts, styles, multimedia tags, and other irrelevant elements from webpages to minimize noise.
  • Implements content length restrictions to prevent overly long content from affecting subsequent processing and storage.
  • Offers a simplified mode to further compress content size, catering to diverse application scenarios.

Application Scenarios

  • Automated workflows requiring web content scraping and intelligent organization, such as content monitoring, information gathering, and pre-processing for data analysis.
  • AI Q&A systems that enhance response accuracy by integrating real-time web data.
  • Developers or business users needing quick access to concise webpage text for further processing or display.
  • Automated customer service or knowledge management systems that automatically update webpage content summaries in the backend.

Main Workflow Steps

  1. Listen for user-triggered manual chat messages (On new manual Chat Message).
  2. Use the ReAct AI Agent to process input and generate HTTP request parameters (e.g., URL and method).
  3. Parse query parameters and set content length limits (CONFIG node).
  4. Send HTTP requests to retrieve webpage HTML.
  5. Check for request errors; if any, generate appropriate error messages.
  6. Extract content within the webpage’s tag.
  7. Remove
  8. Based on the method parameter, decide whether to simplify content by removing links and image URLs.
  9. Convert the cleaned HTML into Markdown format.
  10. Verify content length and return an error message if it exceeds limits.
  11. Output the final page content for downstream use.

Involved Systems or Services

  • OpenAI GPT-4 (invoked via OpenAI Chat Model node)
  • Custom HTTP requests (n8n built-in HTTP Request node)
  • n8n Langchain plugin (ReAct AI Agent and related tool nodes)
  • Markdown conversion node (converts HTML to Markdown)

Target Users and Value

  • Automation developers and technical professionals building intelligent content collection and processing tools.
  • Content operators and data analysts seeking rapid access to structured webpage text.
  • AI application developers enhancing intelligent Q&A and knowledge bases with real-time web data.
  • Enterprises and teams aiming to improve information acquisition efficiency and reduce manual intervention.

This workflow integrates AI intelligence with web data scraping, significantly enhancing the automation and intelligence of content acquisition. It serves as a foundational component for building modern information service platforms.