DuskwingBot
DuskwingBot is the crawler of Duskwing, an independent search engine that builds its own index from scratch. It identifies itself as:
DuskwingBot/0.1 (+https://duskwing.app/bot)
What it respects
robots.txt(RFC 9309), fetched before any page of a site and cached for a day. Groups forDuskwingBottake precedence over*.Crawl-delay— it only ever makes the crawler slower. Without it, DuskwingBot waits at least 1000 ms between requests to the same site and keeps one request in flight per site.noindex/nofollowin<meta name="robots">,<meta name="duskwingbot">and theX-Robots-Tagheader, plusrel="canonical".429and503responses back off the whole site, honouringRetry-After.- It never sends or stores cookies, and only fetches HTML.
Blocking it
User-agent: DuskwingBot Disallow: /