4gptsbot

What it is

4gptsbot is the web crawler run by 4GPTs. It reads public product pages: the pages any visitor can open.

How to recognise it

Requests from 4gptsbot carry the User-Agent 4gptsbot/1.0 (+https://4gpts.com/bot) and a Web Bot Auth signature (HTTP Message Signatures) that you can verify against its key directory at https://4gpts.com/.well-known/http-message-signatures-directory.

robots.txt

4gptsbot obeys the robots.txt group for the user-agent token 4gptsbot, or the * group when there is none for 4gptsbot, including Crawl-delay.

Rate limits and refusals

When a site answers 403 Forbidden, 429 Too Many Requests or 503 Service Unavailable, 4gptsbot pauses the whole site, not just the page: for at least as long as Retry-After asks, and never less than 15 minutes, doubling on each refusal up to a day.

How to block it

To block it, add these lines to your robots.txt:

User-agent: 4gptsbot
Disallow: /

How to slow it

To slow it, add these lines to your robots.txt, where 10 is the number of seconds between requests:

User-agent: 4gptsbot
Crawl-delay: 10

Contact

For anything about 4gptsbot, write to contact@4gpts.com.