Gary Illyes from Google said that Google’s crawlers do not only send HTTP requests such as GET and POST but also for HEAD, OPTIONS, PUT, PATCH, and DELETE. He said the latter make up for less than 1.5% of the total number of requests all Google’s crawlers send. Google sends them for JavaScript, he said.
He wrote this on LinkedIn after saying, “For some reason I got a few questions the past few weeks about whether Google’s crawlers send HTTP requests other than GET and POST.”
Gary said, “The short answer is yes, they do, but HEAD, OPTIONS, PUT, PATCH, and DELETE account for less than 1.5% of the total number of requests all Google’s crawlers send.”
He then explained why Google sends these types of HTTP requests. It is for JavaScript. “Basically it seems that some JavaScript nonsense is initiating these requests and rendering does its rendering things,” he wrote.
Forum discussion at LinkedIn.
