One day I set up the world's lamest content farm. You can see it here:
https://www.web.sp.am/
While humans tend not to find its six billion pages very interesting,
some web spiders are entranced. In the past week or so, Amazon's
amazonbot has visited it 6 million times, and OpenAI's gptbot 2.6
million. (If you were wondering what they use to train ChatGPT, now
you know.) I don't care that googlebot comes by every 5 or 10 minutes,
but gptbot is every few seconds and amazon as fast as the server will
respond.
They both come from predictable IPs so I can set packet filters but
they're still hammering pretty hard. Each has a URL in the user agent
string, Amazon's page has an address to write to but OpenAI's doesn't.
I wrote to the Amazon address, no response.
If anyone has contacts at either I would appreciate it. A few years
ago the bingbot got trapped but fortunately I knew someone at
Microsoft who could pass the word. He reported back that while he
could not go into detail, there was a great deal of animated
conversation at the other end of the hall, and shortly after that it
stopped.
R's,
John
https://www.web.sp.am/
While humans tend not to find its six billion pages very interesting,
some web spiders are entranced. In the past week or so, Amazon's
amazonbot has visited it 6 million times, and OpenAI's gptbot 2.6
million. (If you were wondering what they use to train ChatGPT, now
you know.) I don't care that googlebot comes by every 5 or 10 minutes,
but gptbot is every few seconds and amazon as fast as the server will
respond.
They both come from predictable IPs so I can set packet filters but
they're still hammering pretty hard. Each has a URL in the user agent
string, Amazon's page has an address to write to but OpenAI's doesn't.
I wrote to the Amazon address, no response.
If anyone has contacts at either I would appreciate it. A few years
ago the bingbot got trapped but fortunately I knew someone at
Microsoft who could pass the word. He reported back that while he
could not go into detail, there was a great deal of animated
conversation at the other end of the hall, and shortly after that it
stopped.
R's,
John