Welcome to the Fastest Backlink Discovery Tool Ever

It’s 2021, you need your digital marketing to move fast. 

To compete, you’ll need the fastest backlink analytics tools and the freshest database on the market to match with full reporting capabilities. 

To help you find, report and manage backlinks, Semrush is launching the fastest backlink analytics checker in the market.

It's not just rebuilt, it's completely new! 

Welcome to the future of dynamic backlink management.

Semrush is Now the Fastest Backlink Checker Ever

We know that the speed of link discovery, reporting, and audits are important to digital marketers tracking SEO campaigns. 

Links are an essential metric, and if you can’t report them fast, you could fall behind.

Backlink data is a major piece of our SEO platform, and one of the top requests we’ve heard from you, our users over the years. So what did we do?

We listened, invested, and got to work.

This meant dedicating over 30,000 expert engineering and data scientist hours, adding over 500 new servers, and setting up 14,000 TBs of storage building a brand new backlinks analytics tool. 

To give those numbers a little perspective... 14,000 terabytes could hold up to 238 million hours of music, or roughly 2,700 years of listening.

And 30,000 hours of work is equivalent to a team of 10 working full time for a year and a half.

After eighteen months of tireless work from our engineers, we’ve moved our backlink system to a completely new architecture. 

This architecture works with a new data collection methodology that dramatically increases how fast we detect and report new links.

To see just how much faster we are now, check out the results from the study we ran a little bit lower in this post.

Semrush 2021 Backlinks Update: What's New?

In this post we’ll talk about what’s new. 

If you want to see the technical details of how we did it, read our explainer blog post. 

Our goal of this update was to create the best backlink tool possible, which meant two things: 

1. Being the fastest at reporting new links, and 
2. Holding the largest database of known links.

After a year and a half of dedicated focus and hard work, we’ve achieved both.

What changed?

First, we audited our current system of storing links and updating the user-facing database. 

Then, we decided to upgrade both our hardware and also the way that we store our data. 

By completely rebuilding this tool from scratch, we’ve now dramatically decreased the time between the moment your site receives a backlink and when you can see that link reported in Semrush. 

How Fast Is Semrush Backlink Database Now?

The speed of any backlink tool’s link discovery can be simplified as two steps:

1. Crawling the web and finding backlinks.
2. Refreshing the public-facing database/UI to show these new links.

If you add up the median times it can take for Semrush to go through steps 1 and 2, you could theoretically publish a brand new blog post, receive a backlink 1 minute later, and then see this new backlink reported in our tool 40 minutes after your published time. 

Backlinks Study: Who Has the Fastest Backlink Checker on the Market?

To back up our claim of being the fastest, we ran a study in two parts. 

The first part was a race to see which backlink tool, among known industry leading competitors, would find a new link the fastest and most frequently. For this we used a random sample of 33,000 URLs.

In the second part, we looked into exactly how much time Semrush and the other tools would take to find a new link and then show it in the tool’s UI. 

For this second step we used RSS feeds to fetch highly trafficked parts of the web where we could also see an article’s published time.

Part 1. Who sees new links first: Semrush, Ahrefs, Moz or Majestic?

First, we looked into a random sample of URLs provided by an independent provider — Common Crawl.

The question we wanted to answer was:

Using data from Semrush, Ahrefs, Moz and Majestic, which tool will identify new links to these URLs first?

1. We received a random sample of over 33,000 URLs from Common Crawl.
2. For each URL, we measured the time when it was first discovered as a backlink by Semrush, Ahrefs, Moz, and Majestic.
3. We calculated the number and percentage of cases that each data provider found a backlink at least one day ahead of the other data providers.
4. If two data providers found the same backlink within the same day, we called this a tie and give one point to each provider in the tie.
5. If any provider did not find a backlink by the time of the measurement, this provider was counted as the last to discover this link.

As we can see on the chart below, Semrush outperformed the competition with 36.8% of cases. 

Part 2(a). How fast, exactly, can Semrush find new links?

Next, we wanted to find out exactly how fast this whole process is — from the time you publish your new blog post to the time that you will see your new backlinks in Semrush.

That’s why we decided to take a look at RSS feeds of large websites. 

We used RSS feeds for two reasons:

1. News publishers' RSS feeds of news publications represent highly trafficked sites that would generate new content and almost immediately garner backlinks.
2. RSS feeds provide a publication date & time, which allowed us to track exactly how long it took between the time of a new page’s publication and the moment that a new backlink to that page is seen in each tool.

First, we took roughly 20,000 URLs that we sourced via RSS feeds of various news outlets around the world. 

Next, we measured how fast Semrush discovered links to their new content compared to the other data providers.

Then we measured the median amount of time it took for each tool to check and discover a new backlink. 

To do that we conducted the following steps:

1. We identified the most popular news outlet RSS feeds on the web.
2. Over approximately one month we collected every URL from those feeds until we collected 20,000 URLs.
3. For each URL we calculated the difference between the time the URL was published according to the RSS feed and the time Semrush first saw this URL.

DiscoveryTime = FirstSeenTime - PublishedTime

4. Then we calculated the median of those differences.
5. We repeated the above process for Ahrefs, Moz, and Majestic.
6. Any URLs that were not seen by one of the SEO tools within 30 days from the publication date were discarded for each provider mentioned above.

Amazingly, the median amount of time Semrush needed was only 19 minutes to show the new link after the target page was created! 

Please note: We had to exclude Moz and Majestic from this chart because they don’t offer timestamps on the time of day in which a link was seen. 

It’s important to have not only the day, but the exact time the link appeared to have proper results. Moz and Majestic don’t have it in the API, that’s why we decided to exclude them from this chart.

Part 2(b). Who won the most backlink races in the RSS feed sample?

Finally, we measured the percentage of cases in which each data provider discovered a backlink at least one day before any other provider, the same timeframe we used with the Common Crawl data sample.

The results are very similar to what we’ve got there with Semrush winning at 32.9% cases.

Semrush Dynamic Backlink Management

Combining our new architecture, the fastest backlink analytics tool, and freshest database with our five tools for backlink management, Semrush is the market leader for businesses that need dynamic backlink management marketing solutions.

To succeed in SEO and link building/management you’ll need a strategy. You’ll want to identify your top competitors and run competitive analysis.

You’ll need a way to build new links and manage any pre-existing ones. 

And since SEO is competitive, you’ll want to always be in this process, getting results faster and more consistently than your competitors. Here’s what you can do.

The best way to understand how good the data is to try it yourself!

Especially if you are tasked with building links and reporting that growth, we’d love to have you try out our backlinks analytics tool and tell us what you think.

Want to dive deeper and understand how we did this? 

Check out this follow-up blog post with the details of our process so you can understand exactly what we’ve done.