• Home
  • Latest
  • Fortune 500
  • Finance
  • Tech
  • Leadership
  • Lifestyle
  • Rankings
  • Multimedia

Trendingnow

1

After forcing workers back to the office, Goldman Sachs and JPMorgan Chase are now letting their staff work remotely—but only for the World Cup

2

The Pentagon said Iran War costs $29 billion, but the real cost is closer to $200 billion—and counting

3

Current price of oil as of June 23, 2026

1

After forcing workers back to the office, Goldman Sachs and JPMorgan Chase are now letting their staff work remotely—but only for the World Cup

2

The Pentagon said Iran War costs $29 billion, but the real cost is closer to $200 billion—and counting

3

Current price of oil as of June 23, 2026
TechTikTok

TikTok’s parent launched a web scraper that’s gobbling up the world’s online data 25 times faster than OpenAI

By
Kali Hays
Kali Hays
Down Arrow Button Icon
By
Kali Hays
Kali Hays
Down Arrow Button Icon
October 3, 2024, 9:00 PM ET
A person holding a phone in front of a large TikTok logo
TikTok’s parent company is using a bot that scrapes online content at far greater quantities than rivals. Artur Widak/NurPhoto via Getty Images
Add Fortune on Google for similar content.

ByteDance looks like it’s eager to make up for lost time when it comes to scraping the web for data needed to train its generative AI models.

Recommended Video

The China-based parent company of video app TikTok released its own web crawler or scraper bot, dubbed Bytespider, sometime in April, according to research from Kasada, a company that specializes in bot management for companies with online data. The existence of the bot was also confirmed by Dark Visitors, which monitors scraper bots.

ByteDance’s bot has quickly become one of the most, if not the single most, aggressive scrapers on the internet, the research shows. It’s scraping data at a rate that’s many multiples of other major companies, such as Google, Meta, Amazon, OpenAI, and Anthropic, which use their own scraper bots to help create and improve their large language or multimodal models, known as LLMs or LMMs.

Sam Crowther, the CEO of Kasada, said since Bytespider showed up, it’s been scraping data at about 25 times the rate of GPTbot, which scrapes data for OpenAI’s ChatGPT platform and underlying models, for instance. Bytespider has been scraping at 3,000 times the rate of ClaudeBot, from Anthropic, which operates the Claude platform.

As the months have gone by, Bytespider has become even more aggressive, according to Kasada. Data shows huge spikes in scraping activity from Bytespider over each of the last six weeks.

Representatives of TikTok and ByteDance did not respond to emails seeking comment.

ByteDance’s aggressive scraping comes despite the possibility of TikTok being banned in the U.S. in the coming months. President Joe Biden has signed legislation that requires ByteDance to sell TikTok, due to national security concerns, or shut it down.

The Bytespider bot, much like those of OpenAI and Anthropic, does not respect robots.txt, the research shows. Robots.txt is a line of code that publishers can put into a website that while not legally binding in any way, is supposed to signal to scraper bots that they cannot take that website’s data. 

Web scraping goes back decades, mainly by search engines to gather links to web pages. But the rise of generative AI tools has added a new dimension and made the practice a prime source of lawsuits and controversy. People and organizations whose work has been scraped argue their copyright is being infringed in the process. All of the models that underlay generative AI tools were trained on massive amounts of online data, effectively everything available on the web, particularly written information. Tech companies use scraper bots to essentially copy it all for free and put it into their datasets.

“It’s like they’re trying desperately to catch up,” Crowther said of the aggressive scraping being done by Bytespider. Just last year, ByteDance was reportedly so far behind in the generative AI race that it was using OpenAI to help build ByteDance’s own LLM, which is against OpenAI’s terms of service. Earlier this year, ByteDance released a chat-based LLM called Doubao, but work on that model would have been completed prior to the accumulation of more recent training data scraped by Bytespider.

It’s “clear” that ByteDance is at work on a new LLM, according to one person familiar with the company. As for what ByteDance plans to do with a new LLM, a person familiar with the company’s ambitions said one goal has to do with the search function for TikTok.

Last week, TikTok released an update to its current search function focused on keywords for ads, basically allowing advertisers to search in real time for words that are trending on TikTok. It allows marketers to build an ad with relevant keywords that would ostensibly help the ad show up on the screens of more users.

A new AI model with data on more recent internet trends and topics could expand and improve TikTok’s search environment further, according to the person familiar with the company’s ambitions. 

“Given the audience and the amount of use, TikTok with a search environment that is a completely biddable space with keywords and topics, that would be very interesting to a lot of people spending a ton of money with Google right now,” the person said.

Are you a TikTok or ByteDance employee or someone with insight or a tip to share? Contact Kali Hays securely through Signal at +1-949-280-0267 or at kali.hays@fortune.com.

About the Author
By Kali Hays
See full bioRight Arrow Button Icon
Add Fortune on Google for similar content.

Latest in Tech

Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025

Most Popular

Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Fortune Secondary Logo
Rankings
  • 100 Best Companies
  • Fortune 500
  • Global 500
  • Fortune 500 Europe
  • Most Powerful Women
  • World's Most Admired Companies
  • See All Rankings
  • Lists Calendar
Sections
  • Finance
  • Fortune Crypto
  • Features
  • Leadership
  • Health
  • Commentary
  • Success
  • Retail
  • Mpw
  • Tech
  • Lifestyle
  • CEO Initiative
  • Asia
  • Politics
  • Conferences
  • Europe
  • Newsletters
  • Personal Finance
  • Environment
  • Magazine
  • Education
Customer Support
  • Frequently Asked Questions
  • Customer Service Portal
  • Privacy Policy
  • Terms Of Use
  • Single Issues For Purchase
  • International Print
Commercial Services
  • Advertising
  • Fortune Brand Studio
  • Fortune Analytics
  • Fortune Conferences
  • Business Development
  • Group Subscriptions
About Us
  • About Us
  • Press Center
  • Work At Fortune
  • Terms And Conditions
  • Site Map
  • About Us
  • Press Center
  • Work At Fortune
  • Terms And Conditions
  • Site Map
  • Facebook icon
  • Twitter icon
  • LinkedIn icon
  • Instagram icon
  • Pinterest icon

Latest in Tech

How Home Depot is rebuilding retailing with AI
NewslettersCIO Intelligence
How Home Depot is rebuilding retailing with AI
By John KellJune 24, 2026
1 hour ago
bob
AIbooks
Robert Wright sees an ‘earthquake’ coming from AI that goes far beyond jobs: ‘cultural, political, personal, family, psychological’
By Nick LichtenbergJune 24, 2026
2 hours ago
A man wearing a red and black jacket and a red hat walks down a hallway lined with servers.
InnovationChina
For the first time since 2017, it’s China, not the U.S., that has the world’s most powerful supercomputer
By The Associated PressJune 24, 2026
3 hours ago
Jack Schlossberg, Kennedy scion and sardonic social media star, loses in bid for New York state assembly
PoliticsPolitics
Jack Schlossberg, Kennedy scion and sardonic social media star, loses in bid for New York state assembly
By The Associated Press, Danny Peltz and Anthony IzaguirreJune 24, 2026
3 hours ago
Matt Garman
Successthe future of work
Amazon exec says AI won’t wipe out white-collar jobs—and is hiring 11,000 grads and interns, and has more developers than 2 years ago to prove it
By Preston ForeJune 24, 2026
3 hours ago
NYC Mayor Zohran Mamdani 3 for 3 on his ‘better Democrats’ endorsements: ‘Put working people back at the heart of politics’
PoliticsNew York City
NYC Mayor Zohran Mamdani 3 for 3 on his ‘better Democrats’ endorsements: ‘Put working people back at the heart of politics’
By The Associated Press, Jesse Bedayn, Thomas Beaumont and HUMERA LODHIJune 24, 2026
3 hours ago

Most Popular

After forcing workers back to the office, Goldman Sachs and JPMorgan Chase are now letting their staff work remotely—but only for the World Cup
Success
After forcing workers back to the office, Goldman Sachs and JPMorgan Chase are now letting their staff work remotely—but only for the World Cup
By Orianna Rosa RoyleJune 23, 2026
1 day ago
The Pentagon said Iran War costs $29 billion, but the real cost is closer to $200 billion—and counting
Economy
The Pentagon said Iran War costs $29 billion, but the real cost is closer to $200 billion—and counting
By Jacqueline MunisJune 24, 2026
11 hours ago
Current price of oil as of June 23, 2026
Personal Finance
Current price of oil as of June 23, 2026
By Joseph HostetlerJune 23, 2026
1 day ago
Markets tumble worldwide as Fed resets expectations: $400 billion wiped off SpaceX stock
Banking
Markets tumble worldwide as Fed resets expectations: $400 billion wiped off SpaceX stock
By Jim EdwardsJune 23, 2026
1 day ago
Current price of gold as of June 23, 2026
Personal Finance
Current price of gold as of June 23, 2026
By Danny BakstJune 23, 2026
1 day ago
Texas and Charlotte used to build huge McMansions—now they're copying the California design tricks they once mocked
Real Estate
Texas and Charlotte used to build huge McMansions—now they're copying the California design tricks they once mocked
By Sydney LakeJune 22, 2026
2 days ago

© 2026 Fortune Media IP Limited. All Rights Reserved. Use of this site constitutes acceptance of our Terms of Use and Privacy Policy | CA Notice at Collection and Privacy Notice | Do Not Sell/Share My Personal Information
FORTUNE is a trademark of Fortune Media IP Limited, registered in the U.S. and other countries. FORTUNE may receive compensation for some links to products and services on this website. Offers may be subject to change without notice.