If you are browing Perlmonks using Google Chrome and are not logged in, your browser needs to send the sec-ch-ua header, otherwise your anonymous access to the site is blocked.

We are constantly being hit by crawlers claiming to be Google Chrome, so this is a first step in the rat race.

Replies are listed 'Best First'.
Re: Anonymous Google Chrome browsers now under additional scrutiny
by cavac (Prior) on May 05, 2025 at 14:01 UTC

    I have the same problem on my own sites. I've added a blackhole that seems to, at least partially, mitigate the problem.

    1. Added an entry in my robots.txt:
      User-agent: * Disallow: /secret/bla.php
    2. Added a hidden link to the main page to an inbetween page that is never visibly linked anywhere. This mainly prevents that annoying pre-loading in chrome triggers anything.
    3. The inbetween page links to /secret/bla.php
    4. All IPs of clients navigating to /secret/bla.php get firewalled.

    It's not perfect, but i was able to reduce bot traffic (no matter what UserAgent was set) by roughly 50%-80%. It's only a temporary win in the war against china and the silicon valley crowd, of course. But every little bit helps.

    PerlMonks XP is useless? Not anymore: XPD - Do more with your PerlMonks XP
    Also check out my sisters artwork and my weekly webcomics
      Well yes, a node behind the honeypot could automatically block the IP.

      But probably it's better to redirect blacklisted IPs to some static and link-wise shallow fake content to feed them a little. ¹

      Otherwise HTTP-Errors are easily detected, so someone might be triggered to improve the attack.

      From my understanding they are using IP-farms to attack us and share the harvested links among them for the next requests. Like that we might trap far more IPs in our "honeyfarm"

      Cheers Rolf
      (addicted to the Perl Programming Language :)
      see Wikisyntax for the Monastery

      ¹) like a wget-mirror of the last week in a dedicated node-id range, there is a huge unused gap there, which we could reuse.

        Hmm, for some (other) attacks on my server, my backend actually grabs the WHOIS information and tries to extract the network range, which then gets iptabled.

        Of course, for my private server i can be a lot more aggresive in blocking large swathes of the internet.

        PerlMonks XP is useless? Not anymore: XPD - Do more with your PerlMonks XP
        Also check out my sisters artwork and my weekly webcomics
      Wow, that's really clever! :)

        Wasn't my idea. Found the basics on some forum (can't remember which) and then just iterated over the idea. There are some other "blackhole" URIs as well, mostly stuff check by bots searching for vulnerabilities. You know, Wordpress config pages and such...

        PerlMonks XP is useless? Not anymore: XPD - Do more with your PerlMonks XP
        Also check out my sisters artwork and my weekly webcomics
Re: Anonymous Google Chrome browsers now under additional scrutiny
by LanX (Saint) on May 11, 2025 at 14:53 UTC
    What's happening to me when using Chrome - be it mobile or desktop - is that I get a 403 when using another domain, i.e. not being logged in AND I'm having JS disabled by default. ¹

    prohibited You don't have permission to access this resource.

    Now the problem I see:

    What about new users coming here with Chrome and no JS for the first time. Are they able to create a new account if all pages are prohibited?

    Or am I missing something?

    Cheers Rolf
    (addicted to the Perl Programming Language :)
    see Wikisyntax for the Monastery

    ¹) not sure how this is related to the sec-ch-ua header.

    update

    so when checking headers in the console it looks like sec-ch-ua only appears when JS is on:

    sec-ch-ua: "Not A(Brand";v="8", "Chromium";v="132"

    so AnoMonks just need to make sure JS is activated or create a login when they want JS to be off.

      Yes. Such is life.

Re: Anonymous Google Chrome browsers now under additional scrutiny
by Co-Rion (Pilgrim) on Sep 12, 2025 at 20:19 UTC

    If you're sending a Chrome User-Agent, you now also need to make sure that your Referer header and the Sec-Fetch-Site header agree.