Googlebot starts crawling using the HTTP / 2 protocol

Google updated its support page for Googlebot developers to indicate that Google can now attempt to download pages using the latest HTTP / 2 protocol. This will be effective as of November 2020. The Googlebot developer page was updated on November 12, 2020 to reflect this change.

This change was announced back in September. The change is now officially in effect.

According to Google:

“In general, Googlebot crawls over HTTP / 1.1. However, as of November 2020, Googlebot will be able to crawl websites that may benefit over HTTP / 2 if the website supports it. “

Why HTTP / 2 Network Protocol?

HTTP / 2 is currently the newest network protocol. It enables faster and more efficient data transfer between a server and a browser (or Googlebot).

HTTP / 2 reduces the time it takes for a web page to be transmitted from a browser to a server. HTTP / 2 also reduces overhead by compressing HTTP header fields.

advertising

Read on below

With the previous network protocol (HTTP / 1), multiple streams would have to be downloaded in parallel, as only one request was activated under the old HTTP / 1 version.

With HTTP / 2, Googlebot and browser can take advantage of the new “multiplexed“Quality. This means that multiple resources can be downloaded from one connection within a stream instead of having to request multiple streams from multiple connections to download the same webpage.”

According to an official IETF FAQ page on Github:

“HTTP / 1.x has a problem called” head-of-line blocking “which is effectively only one request for a connection at a time.

… Multiplexing solves these problems by sending multiple request and response messages at the same time. It is even possible to mix parts of one message with another on the wire.

This way, a client can only use one connection per origin to load a page. “

advertising

Read on below

The functions of HTTP / 2 mean less server overload and save server resources.

Minimizing the load on server resources is good for websites. Sometimes not only Googlebot but also many other bots hit a website at the same time.

The result is that the site is slow to respond because of so much server resources being used. This is bad for users trying to view web pages, and bad for the publisher if Googlebot cannot crawl a website because the server is being pushed to the limit by rogue bots like scrapers and hackers.

According to Google:

“… As of November 2020, Googlebot will be able to crawl websites that can benefit from it via HTTP / 2 if the website supports it.

This may save computing resources (e.g. CPU, RAM) for the site and Googlebot, but otherwise will not affect the indexing or ranking of your site. “

Publishers can disable HTTP / 2 crawling

It is possible to disable HTTP / 2 crawling. The server must be configured to send a 421 server response code.

Status code 421 is described by the Internet Engineering Task Force (IETF.org) as a misdirected request. This means that a request for HTTP / 2 will be misrouted if it is not available.

According to the IETF::

“The status code 421 (Misdirected Request) indicates that the request was directed to a server that cannot generate a response.
This can be sent from a server that is not configured to generate responses for the combination of schema and authorization contained in the request URI. “

Google’s developer site recommends:

“To disable crawling over HTTP / 2, instruct the server hosting your website to respond with an HTTP status code of 421 when Googlebot tries to crawl your website over HTTP / 2. If that is not possible, you can send a message to the Googlebot team (but this solution is only temporary). “

advertising

Read on below

Is your site eligible for HTTP / 2 crawling?

Many websites should be ready for HTTP / 2 crawling. Finding out is easy. You can verify that your server is qualified using an HTTP / 2 checking tool.

HTTP / 2 crawling is a welcome change

Googlebot’s ability to crawl using the HTTP / 2 protocol is good news for publishers. This reduces the server load and makes it easier for Googlebot to crawl websites.

Quote

Googlebot developer site
https://www.google.com/webmasters/tools/googlebot-report

}// end of scroll user