[CollEc] helos overload

Thomas Krichel krichel at openlib.org
Sun Jul 30 12:09:22 UTC 2023


  Thomas Krichel writes

> Now I will reboot helos.

  Reboot is a succes but bytespider is still there.

krichel at trabbi~$ GET https://collec.repec.org/robots.txt
# makes requests faster than we can handle
User-agent: Bytespider
Disallow: /
# Crawl-delay: 10

 
-- 
  Written by Thomas Krichel http://openlib.org/home/krichel on his 21240th day.



More information about the CollEc-run mailing list