Everything we know about IndexNow so far

Hopefully, you’ve heard of IndexNow by now.

We covered your announcement in October and in December I had Fabrice Canel, Senior Program Manager at Microsoft Bing, at the SEJ Show to talk about this revolution in the way search engines discover new and updated content.

But what exactly is it and what do you need to know about it for SEO?

This is what we know so far.

Why do we need IndexNow

Search engines have been discovering content in the same way for a long time.

Way back in 1993, JumpStation became the first internet discovery tool to send bots to follow (or crawl) links from one website to another and create a map or index of content.

In the years that followed, other search engines adopted this process as a way to find new content.

This process was fed into an algorithm that also considered content, relevance, and other factors, and search engine optimization as we know it was born.

It wasn’t long before SEO-savvy web publishers recognized the importance of indexing to ensure high search engine rankings. In fact, without indexed content, a website will not generate organic traffic.

But therein lies the problem: unless you’re a large, heavily trafficked website, it can take days or even weeks before a search engine crawls your site and indexes new pages. Heck, even if you’re a big e-commerce or marketplace site, it can take weeks for engines to spot new arrivals or out-of-stock products, especially when you rely on discoverability with basic crawls.

In the meantime, you may have lost significant traffic to your competitors.

To streamline this process, Microsoft Bing launched a new initiative in October 2021 called IndexNow.

So what is IndexNow?

IndexNow is an open source protocol that allows website publishers to instantly index on participating search engines, updating results based on the latest content changes.

In a nutshell, it’s a simple ping that notifies search engines that a URL and its content have been added, updated, or removed.

By eliminating the need for exploratory crawls, IndexNow seeks to improve the overall efficiency of the Internet.

How does IndexNow work?

There are two different ways search engines get index data: pulling and pushing.

Pull indexing occurs when a search engine visits your site to request web pages and collects data from the server.

This is how search engines traditionally work.

Automatic indexing is when the web editor or content management system notifies the search engine that pages have been added or removed or other changes have been made.

That’s what’s really innovative about IndexNow: it allows all URLs submitted to any IndexNow-enabled search engine to be simultaneously submitted to all other search engines that use the protocol.

You can read more about the technical aspects of how IndexNow works here.

What search engines use IndexNow?

At the time of publication, Bing and Yandex use IndexNow.

A handful of others are reportedly testing the protocol, according to an anonymous expert.

So far, DuckDuckGo has not announced any plans to adopt IndexNow.

However, it is worth noting that the privacy-focused search engine relies on up to 400 sources in addition to using its own DuckDuckBot web crawler in compiling its search results.

Since Bing and Yandex are among those sources, you could also see updated content appearing on DuckDuckGo faster due to IndexNow.

Even Bing’s biggest competitor, Google, the biggest of all search engines, has confirmed that it will test the IndexNow protocol.

And while they haven’t gone so far as to announce whether or not they’ll be adopting it, the fact that they’re experimenting with it shows an appreciation of its capabilities.

Google’s main concerns about IndexNow seem to be related to sustainability and efficiency.

Currently, Google uses HTTP/2, a fundamental and efficient data transfer protocol for more than half of all crawls, and can ultimately decide which is best for its needs.

Or they may decide to create their own alternative if they determine an API-based approach is needed, and it’s not a foregone conclusion that they will.

But not everyone is on board.

There are many people in the SEO community who don’t see the need for this protocol, and instead claim that XML sitemaps already accomplish what IndexNow is trying to do.

You can read more about the debate here.

How is IndexNow used?

Notifying search engines of updates to your web content using IndexNow is a simple process:

  1. Generate an API key – This is sent along with the URLs to ensure ownership of the domain. You can do this by using an online key generator tool like the one found here.
  2. Host API Key – Your API key is located in the root directory in txt format.
  3. Send URL with parameters – You can send URLs individually or in bulk. Send your key location as a parameter.
  4. Check your shipments – Using the Bing Webmaster Tools portal, you can check which URLs have been submitted and discovered.

For more detailed examples and instructions, you can visit the IndexNow site.

More advanced users can use Python for bulk indexing and URL submission automation.

This requires a basic understanding of Python syntax and familiarity with Python libraries and packages.

We have a handy step-by-step guide to walk you through this process here.

What tools sync with IndexNow?

Any website developer can use the process outlined here to take advantage of IndexNow, but what’s really exciting is that you don’t have to know HDMI HTML to reap the benefits.

Several large companies, including LinkedIn, MSN, and GitHub, are planning migrations to IndexNow.

Additionally, other search solution providers such as Botify, OnCrawl, Onely, and Yext are in the process of adopting this protocol.

Content management systems (CMS) are also coming on board.

WordPress already offers an IndexNow plugin. Duda supports this proactive approach to web crawling, and Wix plans to integrate soon.

Bing also recently announced the integration of Rank Math and All In One SEO for IndexNow.

Content Delivery Networks (CDNs) that support IndexNow include Cloudflare and Akamai.

Through your proxy servers, this will speed up data delivery and bring it closer to users.

What are the benefits of using IndexNow?

In addition to emphasizing the evolution of indexing from pull to push, the main benefit of IndexNow for publishers is the elimination of time between updates being made and search engines discovering them.

By allowing webmasters to report all participating URLs with a single API call, it speeds up search engine content discovery, which in turn makes the web more efficient.

This is advantageous for site owners as increasing requirements on the server are reduced.

Search engine spiders no longer have to perform exploratory crawls to determine if a page has been updated.

This allows servers to run more efficiently without the need for redundant site load, which in turn reduces energy usage.

And unlike an XML sitemap, which only redirects SEO crawlers to pages within your website, IndexNow allows you to report status code pages other than 200 to search engines.

IndexNow FAQ

What happens after I submit a URL?

IndexNow transmits your changes to search engines, which in turn apply them to page rankings.

Use of IndexNow does not guarantee that web pages will be crawled or indexed and changes may take time to be reflected.

How many URLs can I submit using IndexNow?

You can provide search engines with a list of up to 10,000 URLs with a single API call.

Will I rank better in search results if I index my content faster?

Yes, the sooner your pages are indexed, the faster they can start competing for high ranking places in the search engines.

What does this all mean?

The march of technology is inexorable.

And as such a vital part of the modern world, it’s only natural that changes to the Internet, and search engines in particular, will be part of it.

By changing the site indexing process from a crawling process to something faster and more efficient, IndexNow appears to be on the cusp of a revolution.

And the fact that companies like MSN, GitHub, and WordPress are joining in seems to hint that this protocol will soon become the norm.

More resources:


Featured Image: ra2 studio/Shutterstock