GitHub Stars Pagination Retrieval and Web Data Extraction Example Workflow
This workflow demonstrates how to automate the retrieval and processing of API data, specifically by making paginated requests to fetch the favorite projects of GitHub users. It supports automatic incrementing of page numbers, determining the end condition for data, and achieving complete data retrieval. Additionally, this process illustrates how to extract article titles from random Wikipedia pages, combining HTTP requests with HTML content extraction. It is suitable for scenarios that require batch scraping and processing of data from multiple sources, helping users efficiently build automated workflows.
Workflow Name
GitHub Stars Pagination Retrieval and Web Data Extraction Example Workflow
Key Features and Highlights
This workflow demonstrates how to leverage n8n's HTTP Request node to handle common data acquisition scenarios, with a focus on automatic pagination looping, web content scraping, and HTML element extraction. By configuring pagination parameters, it automatically increments page numbers and evaluates termination conditions to comprehensively retrieve GitHub users' starred repositories. Additionally, it showcases fetching random Wikipedia pages to extract article titles, illustrating the combined use of HTTP requests and HTML extraction.
Core Problems Addressed
- Automates API pagination handling to avoid manual repetitive calls, ensuring complete data retrieval.
- Parses webpage binary content to extract specified HTML element information, supporting web data scraping.
- Breaks down complex HTTP response data into manageable individual items for easier downstream processing.
Application Scenarios
- Automated batch data retrieval from APIs supporting pagination, such as fetching user favorites, history logs, or order lists.
- Scraping specific content from web pages (e.g., article titles, news summaries, product details) for automated processing.
- Integrating heterogeneous data from multiple sources and building automated workflows.
Main Workflow Steps
- Manually trigger the workflow start.
- Initialize pagination parameters (current page number, items per page) and target GitHub username using the Set node.
- Send paginated requests to the GitHub API to retrieve the user’s starred repositories.
- Use the If node to check if the current request returns empty data, determining whether to continue pagination.
- If not finished, increment the page number via the Set node and loop to request the next page.
- Concurrently execute requests to fetch Mock Albums data, splitting the response into individual items.
- Request a random Wikipedia page and extract the page title to demonstrate HTML content extraction capabilities.
Involved Systems or Services
- GitHub API: for paginated retrieval of user starred repositories data.
- JSONPlaceholder API: mock data interface used for demonstration purposes.
- Wikipedia: random page fetching and HTML content extraction.
- n8n native nodes: HTTP Request, Set, If, Item Lists, HTML Extract, Manual Trigger.
Target Audience and Use Value
- Automation developers and operations personnel needing to quickly build API data retrieval and processing workflows.
- Data analysts and product managers interested in automated multi-source data acquisition and integration.
- Tech enthusiasts learning API pagination handling, web scraping, and n8n node orchestration.
- Enterprises aiming to reduce manual data collection costs and implement data-driven business process automation.