Dealing with timeouts in Puppeteer 🐢️

You are most likely using Puppeteer to interact with the web in some form. Either you are taking screenshots of a website using Puppeteer, you scrape content, you render content or you crawl the web. Sooner or later you will run into the question of how to deal with timeouts and how to tell Puppeteer what you want.

Let's dive right into understanding and solving navigational timeouts with Puppeteer.

The Problem: "TimeoutError: Navigation Timeout Exceeded: 30000ms exceeded" #

Whenever your target site (either your own or some other site) fails to serve you in reasonable manner you might get this error:

error { TimeoutError: Navigation Timeout Exceeded: 30000ms exceeded
    at Promise.then (.../node_modules/puppeteer/lib/NavigatorWatcher.js:74:21)
    at <anonymous> name: 'TimeoutError' }

Here you see the default Puppeteer timeout of 30 seconds in action. Of course only, if your Function as a Service hasn't timed you out at this point.

Solution 1: Set or disable the timeout for a navigation/request #

Some requests take longer than others. If you need to wait for a request that regularly exceeds the Puppeteer default of 30 seconds you can increase it:

// 60k milliseconds = 60s
await page.goto(url, {waitUntil: 'load', timeout: 60000});

If need to wait for a request to finish no matter what, you want to disable the timeout:

// 0 = disabled
await page.goto(url, {waitUntil: 'load', timeout: 0});

This will work for the following navigational events

It's best practice to set a very high value instead of completely disabling the timeout. The default is 30000 milliseconds. Set the value to 90k milliseconds or even higher if needed, but avoid

Here is the source, in case you are curious: Ticket, Commit

Solution 2: Set or disable timeouts for a Puppeteer page/tab #

If you want to increase the timeout for all requests and navigational events in a Puppeteer page/tab you can use setDefaultNavigationTimeout:

// 120s
page.setDefaultNavigationTimeout(120000);

If you aren't concerned about timeouts but about results, you can simply deactivate the timeouts for a page/tab in Puppeteer:

page.setDefaultNavigationTimeout(0);

This might be more a case for internal applications than for web crawlers.

More content in the Docu.

Over to you. #

Now it's over to you. You should think about what times you can and have to set. Consider that network connectivity always is a risk you can't control.

Even internal applications aren't always safe. If you use networkidle0 and your external fonts take ages, you've got a problem. Gracefully handling any delays is key.

🙏🙏🙏

Since you've made it this far, sharing this article on your favorite social media network would be highly appreciated 💖! For feedback, please ping me on Twitter.

Published