CocoBot

The crawler behind Coco Search, an open search engine for the web's contents.

CocoBot reads a small, chosen part of the web — the feeds and pages Coco follows, and the papers, code, videos and files they link to — so that people and small projects can search it through one free API instead of each writing a scraper.

How it behaves

• It says who it is: CocoBot/0.1 (+https://search.linearcode.io/bot). It never pretends to be a browser.
• It reads your robots.txt first and obeys the group for CocoBot, else the one for *, including Crawl-delay.
• It asks one page of a site at a time, at least a second apart, and backs off when a site answers 429 or 503.
• It honours noindex and nofollow in <meta name="robots"> and X-Robots-Tag.
• It never solves a captcha, never signs in, never reads what is behind a paywall, and never drives a browser to get past a bot check. A page that refuses it is simply not in the index.
• Search results carry a title, a short snippet and the link to your page; the text itself is kept only to match questions.

To leave

Add this to your robots.txt; CocoBot reads it again within a day:

User-agent: CocoBot
Disallow: /

To be taken out of the index at once, write to lc@linearcode.io with your site's address.