Inside The Atlantic’s AI bot blocking strategy

The Atlantic has built a scorecard for AI crawlers, identifying which bots actually send readers back and which just strip content. Only those with value get through.

This approach led it to block a single AI crawler that tried to recrawl its site 564,000 times in the past seven days.

Some publishers have taken a hard line on AI crawlers, blocking all that they don’t have a licensing deal with. The Atlantic has a licensing deal with OpenAI, which it doesn’t block, but has determined that other AI crawlers must drive back traffic or bring in new subscribers for it to unblock them. The hope is that if the AI engines want access to its content to improve their LLMs’ outputs, then they’ll pay a licensing fee.

Continue reading this article on digiday.com. Sign up for Digiday newsletters to get the latest on media, marketing and the future of TV.