Site icon MiltonMarketing.com – Bernard Aybout's Blog

How to Spy on Competitors with Python & Data Studio

How to Spy on Competitors with Python & Data Studio

How to Spy on Competitors with Python & Data Studio

Spying on Competitors using Python & Data Studio

Spying on competitors using Python and Data Studio involves several steps, including data collection, analysis, and visualization. Here’s a high-level overview:

1. Identify Key Competitors and Metrics

  • Competitors: List down who your main competitors are.
  • Metrics: Decide on what metrics you want to track (e.g., website traffic, SEO rankings, social media engagement, etc.).

2. Data Collection with Python

To collect data, you can use Python libraries and APIs related to the metrics you want to track. Here are some tools you might use:

  • Web Scraping: Libraries like Beautiful Soup or Scrapy can scrape public data from competitor websites.
  • APIs: Use social media APIs (Twitter API, Facebook Graph API, etc.) to gather data on social media performance.
  • SEO Tools: Utilize APIs from SEO tools like Ahrefs, SEMrush, or Moz to get data on search rankings and backlinks.

Example: Scraping Social Media Followers

import requests
from bs4 import BeautifulSoup

# Example for Instagram - Adjust for the specific platform and legality
url = 'https://www.instagram.com/competitorusername/'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
followers = soup.find('meta', property='og:description')['content']
print(followers)

3. Data Cleaning and Analysis

Once you have collected the data, you might need to clean and analyze it to get meaningful insights. Python’s Pandas library is excellent for data manipulation and analysis.

4. Visualization with Google Data Studio

Google Data Studio is a tool that can transform your data into informative, easy-to-read, and easy-to-share reports and dashboards.

  • Connect Your Data: Import your dataset into Google Data Studio.
  • Create Reports: Use Data Studio to create visual reports.
  • Share Insights: Share your reports with stakeholders or use them to inform your strategic decisions.

Best Practices

  • Automation: Consider automating the data collection and analysis process.
  • Compliance: Ensure all data collection methods comply with legal regulations and terms of service.
  • Continuous Monitoring: Set up a system for ongoing monitoring rather than one-time analysis.

How to Spy on Competitors with Python & Data Studio. Whether you’re new to SEO or a seasoned pro, investigating random drops in keyword rankings is just part of the trade.

There are many methods to investigating why your targeted keywords may have dropped, but using Python and Data Studio together is a game changer!

A quick disclaimer: I did not create this script myself. A fellow SEO and developer, Evan from Architek, collaborated with me to solve a particular need I had. Evan was the brains behind creating the script.

I always wanted to be able to view a large set of Google results at scale, without having to conduct these searches manually one-by-one.

Evan mentioned that Python may be the ideal solution for solving my SEO investigation needs.

Why Python Is Relevant to SEO

Python is an incredibly powerful programing language that can do just about anything. One of the more common uses for Python is automating daily monotonous tasks.

One of the coolest things about Python is that there are several different ways to accomplish the same task. However, this also adds a new level of difficulty.

Most of the sample Python scripts out there can be a bit outdated, so you’ll find yourself doing a lot of trial and error.

Python has many applications for analyzing SEO data. The key is to have the right idea. If you have an idea to automate a task, chances are that there’s a script that can be created for it.

What This Specific Script Does & Doesn’t Do

Most keyword ranking tools report an average ranking for a keyword over a specified timeframe. This Python script runs a single crawl at the time you run it from your IP address. This script is not meant to track keyword rankings.

The purpose of this script was to solve an issue I was having with investigating sudden drops of rankings across my client and their competition.

Most keyword ranking tools will tell you which pages your domain is ranking for, but not your competitor’s highest ranking pages per keyword.

So Why’s That Important?

In this scenario, we aren’t tacking page performance for the long-term. We’re simply trying to get quick data.

This script allows us to quickly identify trends across the organic landscape and see which pages are performing best.

What You’ll Need to Get Started

If you’re new to Python, I recommend checking out the official Python tutorial or Automate the Boring Stuff.

For this tutorial I’m using PyCharm CE, but you can use Sublime Text or whatever your preferred development environment is.

This script is written in Python 3 and may be a bit advanced for folks new to this programming language.

If you have not found an interpreter or set up your first virtual environment yet, this guide can help you get started.

Once you’re set up with a new virtual environment, you’ll need the following libraries:

  • urllib
  • lxml
  • requests

Now that you’re all set up, let’s dive into some research together.

1. Make a List of Keywords to Investigate – How to Spy on Competitors with Python & Data Studio

We’re going to be using some sample data to investigate some keywords we’re pretending to track.

Let’s pretend that you looked at your keyword tracking software and noticed that the following keywords dropped more than five positions:

  • SEO Tips
  • Local SEO Advice
  • Learn SEO
  • Search Engine Optimization Articles
  • SEO Blog
  • SEO Basics

Disclaimer: *Searching for too many keywords may result in your IP getting temporarily banned. Pinging Google for this many results at once may appear spammy and will drain their resources. Use with caution and moderation.

The first thing we will do is place these keywords in a simple text file. The keywords should be separated with a line break, as shown in the screenshot below.

2. Run the Python Ranking Investigation Script – How to Spy on Competitors with Python & Data Studio

The TL;DR of this script is that it does three basic functions:

  • Locates and opens your searches.txt file.
  • Uses those keywords and searches the first page of Google for each result.
  • Creates a new CSV file and prints the results (Keyword, URLs, and page titles).

For this script to work properly, you will need to run it in sections. First, we will need to request our libraries. Copy and paste the command below.

[python]

from urllib.parse import urlencode, urlparse, parse_qs

from lxml.html import fromstring
from requests import get
import csv

[/python]

Next, you’ll be able to input the main function of this script in a single copy/paste action. This part of the script sets the actual steps taken, but will not execute the command until the third step.

[python] def scrape_run():
with open(‘searches.txt’) as searches:
for search in searches:
userQuery = search
raw = get(“https://www.google.com/search?q=” + userQuery).text
page = fromstring(raw)
links = page.cssselect(‘.r a’)
csvfile = ‘data.csv’
for row in links:
raw_url = row.get(‘href’)
title = row.text_content()
if raw_url.startswith(“/url?”):
url = parse_qs(urlparse(raw_url).query)[‘q’] csvRow = [userQuery, url[0], title] with open(csvfile, ‘a’) as data:
writer = csv.writer(data)
writer.writerow(csvRow)[/python]

Now you’re ready to run the command. The final step is to copy/paste the command below and click the return key.

[python]

scrape_run()

[/python]

That’s it!

3. Use Data Studio to Analyze the Results

By running this command, you may notice that a new CSV file has been created called data.csv. These are your raw results, which we’ll need for the final step.

SEJ agency has created a handy Data Studio template for analyzing your results. In order to use this free report, you’ll need to paste your results into Google Sheets.

The page in the link above has in-depth instructions on how to set up your Data Studio report.

How to Analyze Our Results

Now that you have your new Data Studio report implemented, it’s time to make sense of all this data.

What we’re looking for are patterns. Yes, you can find patterns in the raw data, but this Data Studio template has a handy feature that allows us to quickly identify which pages are ranking the most frequently for our targeted keywords.

This is helpful, because it allows us to see which competitors are performing well, and which specific pages are performing well.

As you can see in the Data Studio screenshot above, Moz and Ahrefs are the top two competitors ranking for our searched keywords.

However, that doesn’t really help us figure out exactly what they’re doing to rank for those keywords.

That’s where the second chart comes in handy. This displays each ranking page and how many times they occur for all of our search queries. We’re quickly able to identify the top three performing pages for our keywords.

Need to filter down to a page or keyword specific level?

We’ve included filters at the top of the Data Studio template to simplify this.

Once you’ve made a list of the best-performing pages, you can conduct further on-page and off-page analysis to figure out why those pages are performing so well.

Getting Stuck?

If you’re getting stuck, reach out to the inventor of this script for tips or custom programming solutions. Or contact us at MiltonMarketing.com for FREE help and support with your SEO.

What’s Your Idea?

Hopefully, this has sparked some creative ideas on how you can use Python to help automate your SEO processes.


Related Videos:

Related Posts:

What is the use of New Keyword in VB.NET?

Cybersecurity Analyst (CSA+) Exam Study Guide

Unlocking Digital Curiosities: The Top 1000 Google Queries Unveiled(Opens in a new browser tab)

Unveiling the Top 1000 Google Queries: Insights into Global Curiosities and Search Trends(Opens in a new browser tab)

SEO(Opens in a new browser tab)

Monster Jam – Hamilton(Opens in a new browser tab)

Spy Camera stealth gear equipment and gadgets(Opens in a new browser tab)

Google is about to have a lot more ads on phones

Automatically Link to Twitter Usernames in Content

How To Build a Website With WordPress…Fast!

Increase User Engagement & Why It Matters for SEO

What does your car know about you? We hacked a Chevy to find out

Google brings one-click install for Android Studio on Chrome OS

10 Visual Studio Code extensions for every developer

Android Studios

Security experts say health care industry is prized target for cyber criminals

What is SEO?

Hacker Lauri Love denied bid to get computers back

Deep learning neural network could reveal why the world works the way it does

Cloud database removed after exposing details on 80 million US households

What is Kodi or XBMC?

Exit mobile version