Search the 15 million websites in Google’s C4 dataset

Was your website or content used to help train AI systems as part of Google’s C4 dataset? A new search tool from the Washington Post lets you find out.

Why we care. The dataset includes the types of websites and content creators that generative AI could potentially negatively impact or even wipe out, such as news and media publishers, blogs and marketing.

Search. The new search tool can be found in the Post’s article Inside the secret list of websites that make AI like ChatGPT sound smart. It created the list “based on how many ‘tokens’ appeared from each in the data set. Tokens are small bits of text used to process disorganized information — typically a word or phrase,” the story explained.

For example, Search Engine Land was used.

As were Marketing Land (a brand that no longer exists, but did in 2019) and Marketing Land Events, which hosted our SMX and MarTech conference sites.

And Search Engine Land’s parent company site, Third Door Media.

Also, Barry Schwartz’s Search Engine Roundtable was used.

Only part of the data. As a reminder, the C4 (which stands for Colossal Clean Crawled Corpus) is only part of the data used by Google Bard and other large language models. It also uses Wikipedia, Reddit and other sources.

Speaking of Reddit. Reddit wants to get paid when any companies want to use its data to train AI models, the New York Times reported. Reddit has updated its API terms and will now charge some companies (e.g., Google, OpenAI) for access. Said Reddit CEO and co-founder Steve Huffman:

• “The Reddit corpus of data is really valuable. But we don’t need to give all of that value to some of the largest companies in the world for free. Crawling Reddit, generating value and not returning any of that value to our users is something we have a problem with. It’s a good time for us to tighten things up.”

Ironically, Reddit, itself, didn’t even create any of that value. Its users did.