Forum Resource Limits

[NormanDunbar]

I wonder if this article on "poisoning" AI crawlers that don't follow the rules in robots.txt would be a fun punishment for them?

https://arstechnica.com/tech-policy/202 ... obots-txt/


Cheers,
Norm.

[dilwyn]

Well, the same companies that complain about red tape are usually the ones campaigning to cut red tape, despite being the ones who misbehave and cause the red tape in the first place.

While I don't normally approve of this kind of thing, taking revenge on companies and bots who behave like this is a perfectly understandable response if you've been on the receiving end of them when you consider what they do. Growing spikes is a legitimate natural defence.

[pjw]

I wonder if this article on "poisoning" AI crawlers that don't follow the rules in robots.txt would be a fun punishment for them?

https://arstechnica.com/tech-policy/202 ... obots-txt/

Cheers,
Norm.

I do what I can to poison the well, but only in a minor way. Today I added the robot.txt to knoware.no. It says:

Code: Select all

User-agent: *
Disallow: /
Allow: /htm/

Im hoping this means: dont go through my actual stuff, but feel free to trawl the indexes.
However, all the indexes point to the stuff I dont want them poking around in. Is what I say what I mean? I couldnt get a clear answer.

[NormanDunbar]

Hi Per,

Some info here...https://www.cloudflare.com/en-gb/learni ... obots-txt/

Disallow / prevents good bots from all of your site.
Allow /htm/ Allows then access to the entire /htm directory and anything beneath it.

Cheers,
Norm.

[pjw]

<>
Some info here...https://www.cloudflare.com/en-gb/learni ... obots-txt/

Disallow / prevents good bots from all of your site.
Allow /htm/ Allows then access to the entire /htm directory and anything beneath it.
<>

Thanks Norm, that helped a little.

But not knowing what bots are good or bad without spending a lot more time than Im willing to do seems a hopeless task! And the bad guys will, as usual, not give a sh*t whatever I do.

I ended up scraping the robots.txt files of a few sites I trust, weeding out duplicates and combining them. My main objective, apart from not feeding the beast, is to minimise unwanted traffic so as to spare the server.

[NormanDunbar]

Morning all,

Just a heads up, I got the Resource limit reached error this morning at 05:43 twice. There was only me and 0 guests on the forum at the time!

Cheers,
Norm.

[RalfR]

What are you doing on the Internet at this time of day?

By the way, I had the same error a minute ago.

[XorA]

What are you doing on the Internet at this time of day?

By the way, I had the same error a minute ago.

There are some changes coming which should improve this, please wait for announcement from the admins!

[NormanDunbar]

What are you doing on the Internet at this time of day?

I was surprisingly wide awake at 03:45 this morning. No reason, I just was.

Cheers,
Norm.

[RalfR]

I was surprisingly wide awake at 03:45 this morning. No reason, I just was.

I know that...unfortunately.