Cloudflare, along with a majority of the world’s leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for content.
Cloudflare, along with a majority of the world’s leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for content.
This has been tried before many times. The problem is that this exchange can never satisfy all of the parties.
Would site host take 0.01$ for a page? If its Walmart.com then they’d happily lose even 0.10$ or more if competitors can’t analyze their products and other perceived IP damages.
For example, let’s assume they do the business math and come out that if it is anything below 5$ is a no deal - what scraper would pay 5$ for a single product page scrape? Maybe openAI can pay that but is this what we want where public scraping is only accessible to billionaires? What if you’re just a user that wants to track Walmart price to build your own budgeting script? Are you paying 5$ on every request?