Re^3: Perlmonks site has become far too slow

Since the crawlers have not signed up yet, all such changes would only affect Anonymous Monk.

See Anubis for something that could be implemented with relative ease. Instead of building a babel-tower of front-end proxies all needing a container to keep up, Anubis itself could be ported to a Perl function instead by somebody enterprising.

Comment on Re^3: Perlmonks site has become far too slow

Replies are listed 'Best First'.
Re^4: Perlmonks site has become far too slow by soonix (Chancellor) on Oct 21, 2025 at 13:03 UTC
in the CB you mentioned "ticket system" and "SQLite" very close to each other recently, which reminded me of Fossil's ticket system, using (and used by) SQLite. Their Defense Against Robots works thus: (of course, complemented by other measures) most dynamic links are shown to logged-in users only there is an "anonymous" login (with dynamic password) for anonymous humans Of course the design principles of Everything₂ and Fossil are different enough, but I like the concept of an "anonymous login" 😉 and most (perhaps all?) of the reasons why PM still allows anonymous posts would still be respected by this…	[reply]
Re^5: Perlmonks site has become far too slow by jdporter (Paladin) on Oct 21, 2025 at 20:38 UTC
That sounds like it could be viable. But, help me understand: what is meant by "dynamic links"? One thing to remember: we don't want to completely prevent robots from spidering the site; we want people to be able to use g0ggle (etc.) to find information in perlmonks. We're just looking at ways to throttle robots so they don't (inadvertently or otherwise) DDOS the site.	[reply]
Re^6: Perlmonks site has become far too slow by soonix (Chancellor) on Oct 22, 2025 at 06:42 UTC
From the text there, it sounds like more or less any link. However, when looking at the site without logging in, there are a great number of valid links. For Perlmonks, it would probably be the Super Search, RAT (at least over a defined nesting depth), and nodelets with dynamic content (e.g. Other Users, CB) that would be "unlinked". But I think these are already blocked for robots. So, probably not much to gain here :-/ especially given the fact that this concerns only "A HREF"s, not incoming links from elsewhere (or self-generated ones). we don't want to completely prevent robots from spidering the site Yes, that's understandable. Don't know how Fossil handles that. OTOH, they seem to be less well known, maybe as a result from exactly that 🙈	[reply]