Netpeak Spider 3.6: Search Console Queries, Custom HTTP Headers

You probably haven't recovered from joy in the wake of Netpeak Spider 3.5 release, but we're ready to strike you with a new version of the crawler! Yes, we don't allow your ideas to get bottlenecked ? Normally, it happens like this:

We decided not to stiffen our development zeal. Here you go, mates, meet not only export of queries from Google Search Console but also custom HTTP headers.

So each in turn, below ?

• 1. Export of Search Queries from Google Search Console (Pro plan)
• 2. Customization of HTTP Request Headers
• Perks for You 
• In a Nutshell

1. Export of Search Queries from Google Search Console

1.1. Why Do You Need This Feature?

Export of search queries from Google Search Console allows you to:

1. Analyze data from GSC in the format ‘URL – search query – major metrics (impressions, clicks, CTR, average position)’in Netpeak Spider. It significantly empowers your opportunities, since in GSC itself, you're confined only to two options: analyze main indicators by URL, or by search queries.
   

   Search queries analysis in GSC

   

   URL analysis in GSC
   
   This is how it looks in a single report in Netpeak Spider

2. Quickly export absolutely all queries URL ranks for. While in GSC, you're restrained to 1000 results only ?
3. Search for the missed optimization opportunities and find the growth points in your project → for instance, export only those queries which are close to the top but not in the top-10 (from 10 to 30 position) and optimize pages to drive more traffic to them.
4. Optimize meta tags: steer by relevant queries from GSC – it helps catch inspiration when all ideas for a unique title, description, or h1 left you.
5. Assess your website performance, enriching GSC data by additional parameters: device (mobile, desktop) and country. Thus you can see which types of devices and countries respectively get a small number of impressions / clicks, low CTR / average position. Or where the things are too bright ?
6. Compare dynamics → you can find new, missed, increased, or decreased queries. The algorithm is simple:
   • Export data on one period to the file
   • Do the same for the previous period
   • Compare two sets of data from files in the pivot table in Google Sheets or Excel. View the example of table →

1.2. This Is What We Added

• A new section with queries settings on the 'Google Analytics & Search Console' tab
• New data enrichment parameters (device / country)
• 'GSC: Queries' parameter
• An inner table that allows approaching received data
• XL report ‘Queries from Search Console' in the 'Export' menu
• 'GSC: queries' item in the main table context menu

2. Customization of HTTP Request Headers

We've implemented the opportunity to configure HTTP request headers thanks to our users' feedback mentioning the range of some specific tasks:

1. Crawling of test websites with HTTP headers protection. Sometimes, developers set up protection on the website that allows you to get to it only after entering a particular HTTP header (usually they deploy IP protection or login / password protection, but we've already covered these cases).
2. Visiting the site logged in as a specific user. To do so, go to the website in a browser, log in, copy cookie, enter it in the program, and Netpeak Spider will crawl website on behalf of the user. Be cautious and attentive when performing this task to ensure that Spider crawls the website correctly and doesn't turn everything upside down. Potentially, it can follow the links an administrator has access to and delete pages, create new topics, etc.
3. Testing the website in various situations. For example, if you need to check how a website determines with 'Accept-language' header what language of the page to show to the user → with 'Source code and HTTP headers analysis' tool, these checks are quick-fire. Likewise, you can check any other headers (take, for example, 'If-Modified-Since'). Our QA team regularly uses Netpeak Spider for testing – they save their time and feel upbeat about it.

You can choose any of the 19 draft headers, or write your custom one. Also, this header is passed to the inner browser for crawling in JavaScript rendering mode (to put it mildly, it wasn't that easy at all ?).

Perks for You

Need assistance in harnessing new features? Our specialists will be delighted to help you get through all these hard knocks and bumps! Sign up for webinar to learn all the cases and ways of using new features. On our webinars even the most advanced SEOs learn new cases about website promotion.

 Sign Up for Webinar

In a Nutshell

In the updated Netpeak Spider 3.6, we've implemented:

• Export of search queries from Google Search Console – professional features for comprehensive analysis of website performance. Export all queries URLs rank for from GSC, find missed optimization opportunities, and new growth points. Top is the limit ?
• Customization of HTTP headers – an important feature for users with specific tasks, be it a versatile website testing or crawling of the websites with HTTP headers protection, or visiting the website under the guise of a specific user.

That's the way the cookie crumbles! Please leave comments, tell your friends and relatives, subscribe to channel, smash that thumbs up button... well, you got it ?