In an attempt to safeguard its data and privacy Elon Musk’s X has updated its terms of service, effectively banning crawling and scraping activities on its platform (via TechCrunch).
The decision, which comes into effect on September 29, prohibits any form of data scraping or crawling without obtaining “prior written consent.”
The previous version of the terms permitted crawling if it followed the instructions provided in the robots.txt file. However, the revised terms are now crystal clear in their stance against scraping and crawling without explicit consent.
Notably, X has recently made alterations to its robots.txt file, a file that guides web crawlers on which parts of the site they can access.
The new instructions only permit Google’s bots to crawl the platform, raising questions about the nature of Twitter’s relationship with Google in terms of data access.
Furthermore, X has modified the robots.txt file to restrict crawlers from accessing specific data, including likes, retweets associated with posts, account likes, media, and photos.
This move follows previous actions taken by Twitter, where they temporarily restricted access for logged-out users due to rampant data scraping, negatively impacting the experience for regular users.
Musk has been a vocal critic of companies using Twitter’s data for training AI models. He has previously threatened legal action against Microsoft for alleged misuse of Twitter data and even filed scraping lawsuits against unidentified entities.
Recent changes to X’s privacy policy indicate their intent to utilize public data, including tweets, for AI model training.
Additionally, the updated policy introduces provisions for collecting biometric data, education, and job history of users.
