How to whitelist Netpeak Spider's user agent in Cloudflare

Modified on Tue, 16 Jul at 2:18 PM

Cloudflare typically blocks site crawlers.

However, you can whitelist the Netpeak Spider user agent in the Cloudflare settings.

Here are the steps to do so:

1. Set custom user agent at Netpeak Spider

2. Log in to the Cloudflare account, select your website.

3. Go the Security tab from the left menu and select WAF. As a result it shows the Web Application Firewall security settings.

4. Click on Create Rule.

5. Give a name to the Firewall rule.

6. Set Field to User Agent, Equals as the Operator, and Value to the name from the Netpeak Spider settings.

7. Set action to Skip All remaining custom rules.

8. Finally, click Deploy in the bottom-right corner after completing the form.