AI scrapers that ignore robots.txt
| 1 min read
LLM trawlers are becoming more aggressive and can consume substantial bandwidth and CPU power. For that and other reasons, I would guess most Yunohost users don’t want their sites to be trawled. Old approaches like robots.txt are not guaranteed to work, so other ways of stopping and/or poisoning LLMs are being developed.
Here is an interesting read on the subject... Thoughts...