The problem
A ticket scraper needed to collect information from three additional websites. The existing Puppeteer implementation ran sequentially, took about five minutes and often failed before finishing. It ran every two hours.
The scraper ran on AWS Lambda. The target pages could be fetched and parsed without a browser.
What I did
I assessed the existing Node.js implementation and the information needed from the target websites. The key decisions were whether a browser was actually required and which scraping operations could run concurrently.
I replaced Puppeteer with Axios for HTTP requests and Cheerio for HTML parsing, and ran scraping work concurrently.
Technical details
The new implementation fetched pages directly with Axios and extracted information using Cheerio.
Independent scraping operations ran concurrently, reducing overall execution time and avoiding unnecessary browser startup overhead.
Before
- Scheduled Lambda
- Sequential browser tasks
- Extract ticket information
- ~5 minutes
After
- Scheduled Lambda
- Concurrent Axios requests
- Cheerio extraction
- ~15 seconds
Direct HTTP fetching avoids browser overhead when target websites do not require JavaScript execution.
Browser automation remains appropriate for dynamically rendered websites, while concurrent requests must respect upstream service limits.
The result
Execution time improved from approximately five minutes to 15 seconds for this scraper. The client described the revised Node.js code as nearly 20× faster.