Settings – authentication

Netpeak Spider allows performing SEO-audit of a website at the development stage and crawling it before search engine robots do.

The ‘Authentication‘ allows getting access to websites secured by basic authentication. It is when a username and a password are included as a part of HTTP GET-request. 

Follow these steps to use this feature:

1. Go to the ‘Authentication‘ tab in the program settings.
2. Tick the ‘Use basic authentication‘ checkbox to activate ‘Login‘ and ‘Password‘ fields. Click on the ‘OK’ button.
3. Enter a website address into the corresponding field and start crawling using the ‘Start‘ button.       

Netpeak Spider also supports URL links with the format https://username:password@example.com/, which may be useful when it is necessary to crawl a list of URLs from different websites.