Simple Web Scraper in Wappler

Server Side Components General
#1

Hi

for my project (nodeJS) I need to scrape some Data from an other website. I have done similar things in the past with php and python, but I don’t know how to approach this with wappler.

I have not seen any action in the workflow-area that would allow me to get the html from a url and than phrase it so that I can extract certain elements from it.

There are several packages for this in nodeJS (https://www.twilio.com/blog/4-tools-for-web-scraping-in-node-js) but how can I make use of them within my workflows?

#2

You will need to build a custom extension. Check the extensions category in this forum.

2 Likes
#3

oh, I see. thanks for pointing me in the right direction

#4

You can utilize external Node.js modules for web scraping or crm data enrichment. For example, the axios library for making HTTP requests and cheerio for parsing HTML. You can install these modules using npm.
npm install axios cheerio

Write a Node.js script that includes the necessary logic for scraping the website. For example:
const axios = require(‘axios’);
const cheerio = require(‘cheerio’);

async function scrapeWebsite(url) {
try {
const response = await axios.get(url);
const $ = cheerio.load(response.data);

    // Extract data using cheerio selectors
    const title = $('title').text();
    const paragraphs = $('p').map((index, element) => $(element).text()).get();

    return { title, paragraphs };
} catch (error) {
    console.error('Error:', error.message);
    throw error;
}

}

// Example usage
const targetUrl = ‘https://example.com’;
scrapeWebsite(targetUrl)
.then(data => console.log(data))
.catch(error => console.error(‘Failed to scrape website:’, error));

1 Like