Netpeak Spider 3.6: Search Console Queries, Custom HTTP Headers

Modified on Tue, 05 Dec 2023 at 06:08 PM

Netpeak Spider 3.6: Search Console Queries, Custom HTTP Headers

You probably haven't recovered from joy in the wake of Netpeak Spider 3.5 release, but we're ready to strike you with a new version of the crawler! Yes, we don't allow your ideas to get bottlenecked ? Normally, it happens like this:

This is how our Product Manager suggested cool feature, and Lead Software Engineer managed to implement it in just two weeks

We decided not to stiffen our development zeal. Here you go, mates, meet not only export of queries from Google Search Console but also custom HTTP headers.

So each in turn, below ?

  • 1. Export of Search Queries from Google Search Console (Pro plan)
  • 2. Customization of HTTP Request Headers
  • Perks for You 
  • In a Nutshell

1. Export of Search Queries from Google Search Console

Export of search queries from Google Search Console is a Pro feature of Netpeak Spider. Eager to have access to this and other PROfessional features? They include:

  • white label reports that allow branding
  • multi-domain crawling
  • export of reports to Google Drive / Sheets, etc.

Hit the button to purchase the Pro plan, and get your inspiring insights!

 Buy Netpeak Spider Pro and Feel Its Power

Here's a post which describes four optimization insights to using this feature: 'How to Use Export of Queries from Google Search Console (GSC) in Netpeak Spider'.

1.1. Why Do You Need This Feature?

Export of search queries from Google Search Console allows you to:

  1. Analyze data from GSC in the format ‘URL – search query – major metrics (impressions, clicks, CTR, average position)’in Netpeak Spider. It significantly empowers your opportunities, since in GSC itself, you're confined only to two options: analyze main indicators by URL, or by search queries.

    Queries in Google Search Console are displayed in a separate ‘Queries’ tab. To see the URL, you need to open another tab
    Search queries analysis in GSC
    This is how the URLs that match the queries are displayed in Google Search Console
    URL analysis in GSC
    In the database table of Netpeak Spider you can compare this and other data from Google Search Console in one table at once
    This is how it looks in a single report in Netpeak Spider

  2. Quickly export absolutely all queries URL ranks for. While in GSC, you're restrained to 1000 results only ?
  3. Search for the missed optimization opportunities and find the growth points in your project → for instance, export only those queries which are close to the top but not in the top-10 (from 10 to 30 position) and optimize pages to drive more traffic to them.
  4. Optimize meta tags: steer by relevant queries from GSC – it helps catch inspiration when all ideas for a unique title, description, or h1 left you.
  5. Assess your website performance, enriching GSC data by additional parameters: device (mobile, desktop) and country. Thus you can see which types of devices and countries respectively get a small number of impressions / clicks, low CTR / average position. Or where the things are too bright ?
  6. Compare dynamics → you can find new, missed, increased, or decreased queries. The algorithm is simple:
    • Export data on one period to the file
    • Do the same for the previous period
    • Compare two sets of data from files in the pivot table in Google Sheets or Excel. View the example of table →

1.2. This Is What We Added

  • A new section with queries settings on the 'Google Analytics & Search Console' tab
  • New data enrichment parameters (device / country)
  • 'GSC: Queries' parameter
  • An inner table that allows approaching received data
  • XL report ‘Queries from Search Console' in the 'Export' menu
  • 'GSC: queries' item in the main table context menu

New featutes that we added in Netpeak Spider 3.6

Read all details in the User Guide: 'Export of Search Queries in Google Search Console.'

2. Customization of HTTP Request Headers

We've implemented the opportunity to configure HTTP request headers thanks to our users' feedback mentioning the range of some specific tasks:

  1. Crawling of test websites with HTTP headers protection. Sometimes, developers set up protection on the website that allows you to get to it only after entering a particular HTTP header (usually they deploy IP protection or login / password protection, but we've already covered these cases).
  2. Visiting the site logged in as a specific user. To do so, go to the website in a browser, log in, copy cookie, enter it in the program, and Netpeak Spider will crawl website on behalf of the user. Be cautious and attentive when performing this task to ensure that Spider crawls the website correctly and doesn't turn everything upside down. Potentially, it can follow the links an administrator has access to and delete pages, create new topics, etc.
  3. Testing the website in various situations. For example, if you need to check how a website determines with 'Accept-language' header what language of the page to show to the user → with 'Source code and HTTP headers analysis' tool, these checks are quick-fire. Likewise, you can check any other headers (take, for example, 'If-Modified-Since'). Our QA team regularly uses Netpeak Spider for testing – they save their time and feel upbeat about it.

Note that you can perform these tasks with a free version of Netpeak Spider that is not limited by the term of use and the number of analyzed URLs. Other basic features are also available in the Freemium version of the program.

To get access to free Netpeak Spider, you just need to sign up, download, and launch the program ?

 Sign Up and Download Freemium Version of Netpeak Spider

P.S. Right after signup, you'll also have the opportunity to try all paid functionality and then compare all our plans and pick the most suitable for you.

You can choose any of the 19 draft headers, or write your custom one. Also, this header is passed to the inner browser for crawling in JavaScript rendering mode (to put it mildly, it wasn't that easy at all ?).

On the 'HTTP headers' settings tab, set your own headers, open the 'Source Code and HTTP headers analysis' tool, enter the URL you need and see how the crawler will request the server with the given headers

We'll show you the ropes in the User Guide: 'Custom HTTP Headers.'

Perks for You

Need assistance in harnessing new features? Our specialists will be delighted to help you get through all these hard knocks and bumps! Sign up for webinar to learn all the cases and ways of using new features. On our webinars even the most advanced SEOs learn new cases about website promotion.

 Sign Up for Webinar

In a Nutshell

In the updated Netpeak Spider 3.6, we've implemented:

  • Export of search queries from Google Search Console – professional features for comprehensive analysis of website performance. Export all queries URLs rank for from GSC, find missed optimization opportunities, and new growth points. Top is the limit ?
  • Customization of HTTP headers – an important feature for users with specific tasks, be it a versatile website testing or crawling of the websites with HTTP headers protection, or visiting the website under the guise of a specific user.

That's the way the cookie crumbles! Please leave comments, tell your friends and relatives, subscribe to channel, smash that thumbs up button... well, you got it ?

Was this article helpful?

That’s Great!

Thank you for your feedback

Sorry! We couldn't be helpful

Thank you for your feedback

Let us know how can we improve this article!

Select atleast one of the reasons
CAPTCHA verification is required.

Feedback sent

We appreciate your effort and will try to fix the article