n8n templatesn8n templates

Convert URL HTML to Markdown and Extract Page Links

This workflow is designed to convert webpage HTML content into structured Markdown format and extract all links from the webpage. By utilizing the Firecrawl.dev API, it supports batch processing of URLs, automatically managing request rates to ensure stable and efficient content crawling and conversion. It is suitable for scenarios such as data analysis, content aggregation, and market research, helping users quickly acquire and process large amounts of webpage information, reducing manual operations and improving work efficiency.

Workflow Diagram
Convert URL HTML to Markdown and Extract Page Links Workflow diagram

Workflow Name

Convert URL HTML to Markdown and Extract Page Links

Key Features and Highlights

This workflow leverages the Firecrawl.dev API to convert webpage HTML content into a structured Markdown format while simultaneously extracting all hyperlinks from the page. It supports batch processing of URLs and automatically manages request rates to avoid exceeding API limits, ensuring stable and efficient web content crawling and conversion.

Core Problems Addressed

  • Converting complex webpage HTML into AI-friendly Markdown format by removing redundant HTML tags.
  • Extracting all hyperlinks from webpages to facilitate subsequent data analysis or content mining.
  • Automatically managing API request frequency to prevent service denial due to excessive requests.
  • Supporting bulk URL imports from databases and automatic batch processing to enhance large-scale data crawling efficiency.

Use Cases

  • Structuring large volumes of web content for training or analyzing large language models (LLMs).
  • Content aggregation and information extraction projects requiring both webpage text and internal links.
  • Bulk web content and link crawling for SEO, market research, or competitive analysis.
  • Automating data collection workflows to reduce manual copy-paste operations.

Main Workflow Steps

  1. Manually trigger the workflow to start execution.
  2. Retrieve a list of webpage URLs from a user-defined data source (URL field named “Page”).
  3. Split the URL list into batches (up to 40 URLs, with actual requests sent in batches of 10).
  4. Request webpage content one by one via the Firecrawl.dev API, converting it to Markdown and extracting all page links.
  5. Handle API rate limiting to ensure no more than 10 requests per minute.
  6. Output the extracted data—including title, description, Markdown content, and links—to a user-specified data source (e.g., Airtable).
  7. Complete batch processing and await the next trigger.

Involved Systems or Services

  • Firecrawl.dev API (provides webpage content conversion and link extraction)
  • User-defined data sources (for URL input and result output, supporting databases like Airtable)
  • n8n automation platform (handles workflow orchestration, rate limiting, and batch processing)

Target Users and Value Proposition

  • Data analysts, content operators, and AI developers who need fast, bulk processing and structured output of web content.
  • Technical teams requiring webpage content conversion to Markdown for AI or other downstream systems.
  • Market research, SEO optimization, and content aggregation teams.
  • Businesses and individuals aiming to automate web content crawling workflows to reduce manual effort and improve efficiency.

This workflow is designed by Simon (automake.io). Users only need to configure their Firecrawl API key and data sources to effortlessly automate web content crawling and conversion, enabling efficient data processing and content analysis.