How to Analyze Competitors’ Content with Netpeak Spider & Checker

Creating and improving your website content are some of the most complex parts of comprehensive website promotion. How to find proper post topic and how to make it more interesting and viral? If you focus only on your own website, it will be difficult to find answers to these questions. Our advice is to look closely at your competitors. That's right, we propose you to conduct competitor marketing analysis to use your competitors' experience and automatically figure out which of their posts are most popular and why.

In this article, you will learn how to:

• find the most popular posts on your competitors’ websites
• scrape automatically their user engagement rates
• figure out the length of the most popular posts
• get new ideas from competitors for your publications

As an example, we used businessoffashion.com website.

To perform these tasks we’ll use Netpeak Spider. We’ll analyze all necessary parameters using general crawling settings (or crawling only the directory with analyzed content) and scraping with a few simultaneous searches.

1. Engagement Rates: Scraping Settings

To crawl and scrape data from competitors’ websites, we’ll set two simultaneous scraping searches: by shares and by comments. Keep in mind that metrics to measure engagement on selected websites can differ from comments and shares to views, likes (upvotes), etc.

To set and launch scraping, perform the following actions:

1. Open one of the content pages.

2. Find shares (comments, views) counter and highlight it.

3. Right-button click on it and select ‘Inspect’.

4. In the opened window with page source code you’ll find an element responsible for showing number of shares (it will be highlighted when you hover the mouse over).

5. Right-click on it and select ‘Copy’ → ‘Copy XPath’. Keep in mind that the type of data which is used for scraping (XPath, RegExp, CSS-selector) may differ according to the website structure. In most cases, this task is carried out through XPath.

6. Launch Netpeak Spider and open ‘Settings’ → ‘Scraping’ and check ‘Use HTML scraping’.

7. To make the process clear, set the name of the conditions according to the type of scraping data (‘Comments’, ‘Views’, ‘Shares’, etc.).

8. Select type of data (XPath) and Data Extraction mode (Inner text).

9. Insert the piece of resulting code into the search box.

10. Repeat points 2-9 for each metric you interested in and create additional scraping searches.

11. Open ‘General’ tab in the Settings window.

2. Crawling Settings: Basic parameters

Before crawling and scraping you need to set several key parameters:

1. Find ‘Basic crawling settings’ in the ‘General’ tab . Turn off ‘check images’, ‘check PDF’, ‘check CSS’, ‘check JavaScript’ and ‘check other MIME types’.

2. Choose scan mode. If you’re interested in content from the whole website, you need to choose ‘Crawl all subdomains’. If you’re interested in selected category or directory choose ‘Crawl only in directory’. In case of Business of Fashion we’ve chosen /articles/ directory and set a limit for crawled pages – 1000 URL.

3. Set custom crawling ‘Rules’, if you need some specific pages not to be limited by one directory. Details on how to perform this procedure are available here.

4. In the Netpeak Spider main window open ‘All results’ tab and right-click on the line with rows names (painted in blue). Turn off all parameters excluding ‘Title’, ‘Description’, ‘Words in <p>’, ‘Characters in <p>’ and other potentially useful indicators.

5. Start crawling.

3. Results Export and Analysis

Now Netpeak Spider final results table includes only the parameters you need for building your content strategy (paragraph 2, point 4). Based on this data, you can draw a conclusion about optimal post length and method of building winning Titles and Descriptions.

When the crawling is completed, you can export the results in a convenient table format. Push the ‘Export’ button placed above crawling results window.

Received report will contain information about all parameters you have chosen in basic crawling settings and scraped data. If you need to download scraped data as a separate file, you can proceed as follows:

• Open 'Reports' tab in sidebar and go to 'Scraping'.
• To see scraping results table push the ‘All Results’ button.

In the opened window you’ll see only the scraping data (in our case it’s a number of shares and comments). By sorting out results by one or the other parameter you can see posts with the biggest (or smallest) number of comments, shares, likes or views.

On the basis of businessoffashion.com crawling we’ve sorted gained data and figured out which posts were the most popular. Among those, we’ve highlighted the following leaders with the biggest number of shares or comments:

• Miroslava Duma and Ulyana Sergeenko Accused of Racism, Homophobia and Transphobia: 77 comments.
• Gucci Bans Fur: ‘It’s Not Modern’: 44 888 shares.

However, do not draw conclusions based only on the two (or more if you have more than two key metrics) most popular posts: we highly recommend to analyze more representative samples with a high number of shares or/and comments.

Based on the received data, we can determine some parameters of highly demanded content: its topic, length (number of words or symbols) and way of writing title and description.

It is worth mentioning that each analyzed URL gets its own number. It means that if you work with two separate reports, one URL in both of them will have the same number and it makes your work with data much easier.

1. Crawl website in Netpeak Spider to get URLs.

2. Transfer crawled URLs in any convenient way.

3. Go to the ‘Services’ settings tab → ‘Facebook’, and enter the Facebook token.

4. In a sidebar, choose the parameter ‘Facebook’ → ‘Shares’. If necessary, you can tick other parameters, for instance, comments and reactions, data from Open Graph markup.

1. Start crawling.

2. When the crawling is completed, all received data will be displayed in the main table.

Summary

To analyze your competitors content and find their most popular posts (most viewed, commented, viral), you need to follow these easy steps:

1. Decide which user engagement metrics you need to analyze first.
2. Copy their XPath.
3. Choose crawling mode and set corresponding Netpeak Spider settings.
4. Launch crawling and scraping.
5. Filter resulting data.
6. Export crawling and scraping results and draw the appropriate conclusions.