{"draft":"draft-koster-rep-12","doc_id":"RFC9309","title":"Robots Exclusion Protocol","authors":["M. Koster","G. Illyes","H. Zeller","L. Sassman"],"format":["HTML","TEXT","PDF","XML"],"page_count":"12","pub_status":"PROPOSED STANDARD","status":"PROPOSED STANDARD","source":"IETF - NON WORKING GROUP","abstract":"This document specifies and extends the \"Robots Exclusion Protocol\"\r\nmethod originally defined by Martijn Koster in 1994 for service\r\nowners to control how content served by their services may be\r\naccessed, if at all, by automatic clients known as crawlers.\r\nSpecifically, it adds definition language for the protocol,\r\ninstructions for handling errors, and instructions for caching.","pub_date":"September 2022","keywords":["robot","crawler","robots.txt"],"obsoletes":[],"obsoleted_by":[],"updates":[],"updated_by":[],"see_also":[],"doi":"10.17487\/RFC9309","errata_url":"https:\/\/www.rfc-editor.org\/errata\/rfc9309"}