← Learn Javascript

Javascript – Data Scraping

Methods & strategies

About

• Web scraping to get data on web pages can be thought of like this: if you can perform actions to copy / paste data by hand, there is a way to do it automatically with software.
• Many scraping-specific tools exist for Python and PHP
• The following Javascript-based strategies are listed in increasing complexity.

Browser Extensions

• Browser extensions have access to everything all the data on a page.
• A user would have to visit each page scraped, which can actually be a bonus if the information requires a login.
• See topics/extensions for more

Browser Testing Tools

• Using browser testing tools, one can instruct them to act a user to visit locally or remotely hosted websites, submit forms, and other actions in the browser.
• For example, the Nightwatch Javascript testing library can automate browser activity through Selenium WebDriver
• Solomon Eseme Guide To Web Scraping With JavaScript And Selenium (2022)

Node

• Kevin Sahin Web Scraping With JavaScript And NodeJS (2022)
• Daniel Ni The Ultimate Guide to Web Scraping with Node.js (2018)