[CollEc] helos overload

[Düben, Christian]

I added a cron job removing crashed app containers

*/30 * * * * docker ps --filter "ancestor=collec_app" --filter "status=exited" | xargs docker rm

I also stopped and removed the running app containers. Obviously, I do not add that as a cron job as that would also abort legitimate user sessions.

Robots continue to be a problem. The current system is not well suited to distinguish robots from human users. This should be managed at the middleware level which I do not develop myself.

Robots will be much less of an issue with CollEc 3.0. Though, as you know, I will only work on that towards the end of the year.

Christian Düben
Doctoral Candidate
Chair of Macroeconomics
Hamburg University
Germany
christian.dueben at uni-hamburg.de
https://www.christian-dueben.com

-----Original Message-----
From: CollEc-run <collec-run-bounces at lists.openlib.org> On Behalf Of Thomas Krichel
Sent: Sonntag, 30. Juli 2023 14:09
To: CollEc Run <collec-run at lists.openlib.org>
Cc: spider-feedback at bytedance.com
Subject: Re: [CollEc] helos overload

  Thomas Krichel writes

> Now I will reboot helos.

  Reboot is a succes but bytespider is still there.

krichel at trabbi~$ GET https://collec.repec.org/robots.txt
# makes requests faster than we can handle
User-agent: Bytespider
Disallow: /
# Crawl-delay: 10

 
-- 
  Written by Thomas Krichel http://openlib.org/home/krichel on his 21240th day.

_______________________________________________
CollEc-run mailing list
CollEc-run at lists.openlib.org
http://lists.openlib.org/cgi-bin/mailman/listinfo/collec-run