Question

How can you scrape data from multiple pages when the URL doesn't change (e.g., "Load more" button)?

I’m looking for an automated way to scrape data from multiple pages where the URL doesn’t change, such as with a “Load more” button, so I can capture all the data for analysis in Power BI.

Answer 1

You would need to mimic the user interaction that initiates the "Load more" action in order to automate the process of scraping data from multiple pages with a static URL (such as a "Load more" button) and collect the extra data. You would need to use Power Query in conjunction with web scraping techniques or external tools like Python or APIs because Power BI does not have native support for scraping data like this.

Here are a few strategies you can use to deal with this:

Method 1: If JavaScript isn't loading the "Load More" button dynamically, use Power Query with web scraping.
Examine the webpage:

Using browser developer tools, open the webpage in your browser and examine the network traffic. Find an endpoint for an API or a network request that fetches the data when the "Load More" button is clicked.


This will often be an AJAX request or an API call returning data in a JSON format.


Use Power Query to Pull Data from the API:


In Power BI, go to Home > Get Data > Web and enter the API URL.


If the data comes in multiple pages, you can pass the page number or offset as a query parameter in the API URL.


Example URL:

https://example.com/api/data?page=1

Put Pagination Logic into Practice:

Once the first page's data has been obtained, loop through the following pages in Power Query to load all of the data that is available.

To call the API for every page and append the results collectively, you can create a custom function.

Utilize the "Combine" feature in Power Query:

To add the information from every page to a single query result, use the "Combine" feature.

Example Power Query M Code:

let
    PageNumber = 1,  // Starting page
    GetPageData = (page as number) => 
        let
            Source = Json.Document(Web.Contents("https://example.com/api/data?page=" & Text.From(page))),
            Data = Source[data]  // Assuming the data is inside the "data" field
        in
            Data,
    AllPagesData = List.Generate(
        () => [Page = PageNumber, Data = GetPageData(PageNumber)],
        each List.Count([Data]) > 0,
        each [Page = [Page] + 1, Data = GetPageData([Page])],
        each [Data]
    ),
    CombinedData = List.Combine(AllPagesData)
in
    CombinedData