Forum Problems

[timer]

With the old forum, we were plagued by three bots - Google, Yahoo & "Majestic". These were indexing our BB and thus publicising us on-line in the respective search engines.

We now have two new ones - Amazon and Semrush.

Do they do anything useful? Or should we unleash our Spam Assassins on them?

Semrush is where companies go to find info on other companies and people. They provide us with nothing of use, just parasitic wasters. I have them banned on my companies web site. Semrush kindly provide a page for white listing themselves, which can of course be used to black list them. They say their IP addresses are these two ranges 85.208.98.32/28 and 85.208.98.48/28.

Amazon, well that could be some hacker using AWS but again there's nothing to sell or buy here so the sales side of Amazon should not care about us either.

That's may take on them.

agree )
if we know the ranges of IP addresses,
we can block them at the nginx or even CloudFlare level - before they reach the forum.


This morning I changed the caching setting - from file to memcached. I wonder if we can sense a change in the forum's responsiveness...

[phkb]

I wonder if we can sense a change in the forum's responsiveness...

I can. It definitely feels snappier.

[Cody]

Or should we unleash our Spam Assassins on them?

The grey bots are beyond our reach.

[timer]

Bytespider, SemrushBot, Amazonbot - these bastards flooded the log with their stupid requests - closed them on CloudFlare.
It became much calmer

[hiran]

Bytespider, SemrushBot, Amazonbot - these bastards flooded the log with their stupid requests - closed them on CloudFlare.
It became much calmer

[timer]

Bytespider, SemrushBot, Amazonbot - these bastards flooded the log with their stupid requests - closed them on CloudFlare.
It became much calmer

@hiran, if anything, add other villains here:
SSL/TLS -> WAF -> [first tab] Custom rules
There is only one rule so far - it’s easy to edit )

[timer]

Code: Select all

MJ12bot
DotBot
AhrefsBot

sent to exile

[Cholmondely]

sent to exile

Merely sent to exile?

Why not defenestrated? Or publicly impaled? Or hung, drawn and quartered?

[hiran]

sent to exile

Merely sent to exile?

Why not defenestrated? Or publicly impaled? Or hung, drawn and quartered?

Watch Matrix and you may understand it is hard to kill programs, regardless of the chosen spectacularity. But your list is lacking Bamboo.

[timer]

sent to exile

Merely sent to exile?

Why not defenestrated? Or publicly impaled? Or hung, drawn and quartered?

Oooh, it seems there is a lot we don't know about our Archivist...

[Cody]

I think there's mention of "bamboo torture/execution" in Conan Doyle's novel The Lost World.

[Redspear]

Why not defenestrated? Or publicly impaled?

You know, those two could be combined...

But your list is lacking Bamboo.

...along with this one.

[Cody]

Was trying to test registering, but I only have two email addresses (which are already in use), so abandoned that.

Found this message though:

You have exceeded the maximum number of login attempts allowed. In addition to your username and password the invisible reCAPTCHA v3 will be used to authenticate your session

reCAPTCHA v3 is invisible? <wanders off, singing something about radio gnome>

[hiran]

Was trying to test registering, but I only have two email addresses (which are already in use), so abandoned that.

Found this message though:

You have exceeded the maximum number of login attempts allowed. In addition to your username and password the invisible reCAPTCHA v3 will be used to authenticate your session

reCAPTCHA v3 is invisible? <wanders off, singing something about radio gnome>

CAPTCHA is technology to tell users apart from bots.
So websites present riddles that are easy to solve for humans and difficult for bots.
reCaptcha tries to present meaningful riddles - stuff that needs human attention anyway. As a result a lot of data is passed between the user's browser and Google.

reCaptcha v3 is even able to detect users without having them answer questions. It means you can pass it while wandering and singing. Google gets to analyze the user's behaviour - even more data is passed to Google.

We will have to add a site banner informing users their data is getting disclosed.

[timer]

As far as I understand, the recaptcha only appears on the registration page. Form data is not sent anywhere. If I correctly understand what I see in the system logs, as well as the fact that the forum mailbox has been empty for the last few days, the activity of bots undergoing registration has either disappeared or has decreased significantly.

Found another spider in the logs
http://www.trendiction.de/bot
what do you think? I am inclined to think that it is useless to us and is only wasting our small server with requests.