Google has introduced a brand new Googlebot, a brand new Google crawler, named Google-Prolonged that you should use to manage in case your content material will help enhance Bard and Vertex AI generative APIs or future Google AI merchandise. So if you wish to disallow Bard from utilizing your content material, you specify so in your robots.txt with the consumer agent Google-Prolonged.
Google will not crawl from Google-Prolonged, Google will nonetheless crawl from its regular Googlebot or different bots. However utilizing Google-Prolonged will talk to Google to not use that content material for Bard or different AI Google tasks. A Google spokesperson instructed me, “Google-Prolonged will inform Google to not use the positioning’s content material for Bard and Vertex AI generative APIs.” “For Search, web site directors ought to proceed to make use of the Googlebot consumer agent via robots.txt and the NOINDEX meta tag to handle their content material in search outcomes, together with experiments like Search Generative Expertise,” Google added.
Primarily this lets you permit Google Search to crawl, index and rank your web site however disallow Bard or different Google AI tasks from utilizing your content material.
This comes after Bing supplied controls to dam Bing Chat AI from utilizing your web site per week in the past.
“At the moment we’re asserting Google-Prolonged, a brand new management that internet publishers can use to handle whether or not their websites assist enhance Bard and Vertex AI generative APIs, together with future generations of fashions that energy these merchandise. By utilizing Google-Prolonged to manage entry to content material on a web site, an internet site administrator can select whether or not to assist these AI fashions turn into extra correct and succesful over time,” Google wrote.
Google-Prolonged is a “standalone product token that internet publishers can use to handle whether or not their websites assist enhance Bard and Vertex AI generative APIs, together with future generations of fashions that energy these merchandise,” Google defined.
The Person agent token is Google-Prolonged
“Google-Prolonged does not have a separate HTTP request consumer agent string. Crawling is completed with present Google consumer agent strings; the robots.txt user-agent token is utilized in a management capability,” Google added.
I’m not positive if that is the different method for robots.txt for AI…
Huge information on the AI entrance. You possibly can implement through robots.txt -> Saying Google-Prolonged, a brand new management that internet publishers can use to handle whether or not their websites assist enhance Bard & Vertex AI generative APIs, together with future generations of fashions https://t.co/L73rm6mwzM pic.twitter.com/BtcQ5kaATP
— Glenn Gabe (@glenngabe) September 28, 2023
Word, Google Information bot additionally works an analogous manner, the place it doesn’t crawl however makes use of the directive for utilizing that content material in Google Information:
Sure, like Googlebot-Information.
— John, aka “a complete bell cheese” (@JohnMu) September 29, 2023
Discussion board dialogue at X.