# AI's Data Crisis: Web Sources Drying Up Fast 🚫

**URL:** <https://onehack.st/t/ais-data-crisis-web-sources-drying-up-fast/287363>\
**Category:** News & Articles\
**Tags:** ai\
**Created:** [July 20, 2024, 12:54pm UTC](https://onehack.st/t/ais-data-crisis-web-sources-drying-up-fast/287363 "2024-07-20T12:54:41Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![TheJoker](https://onehack.st/user_avatar/onehack.st/thejoker/32/22660_2.png) [@TheJoker](https://onehack.st/u/TheJoker)\
**Post date:** [July 20, 2024, 12:54pm UTC](https://onehack.st/t/ais-data-crisis-web-sources-drying-up-fast/287363/1 "2024-07-20T12:54:41Z")

</div>

**Summary:**

1. **Emerging Crisis** : A study by the Data Provenance Initiative, an M.I.T.-led research group, has found that many web sources used for AI training data are now restricting access, leading to an “emerging crisis in consent.”

2. **Significant Restrictions** : The study reviewed 14,000 web domains in three AI training datasets (C4, RefinedWeb, and Dolma) and found that 5% of all data and 25% of high-quality data sources have been restricted.

3. **Impact on AI Development** : Restrictions through the Robots Exclusion Protocol and website terms of service could significantly affect AI companies, researchers, and academics, limiting access to essential training data.