Clicky

  • Login
  • Register
  • Submit Your Content
  • Contact Us
Tuesday, August 5, 2025
World Tribune
No Result
View All Result
  • Home
  • News
  • Business
  • Technology
  • Sports
  • Health
  • Food
Submit
  • Home
  • News
  • Business
  • Technology
  • Sports
  • Health
  • Food
No Result
View All Result
World Tribune
No Result
View All Result

Perplexity is allegedly scraping websites it’s not supposed to, again

August 5, 2025
in Technology
Reading Time: 2 mins read
A A
Perplexity is allegedly scraping websites it’s not supposed to, again
0
SHARES
ShareShareShareShareShare

READ ALSO

How to download the update on your iPhone, new features like Liquid Glass and everything else you need to know

Spotify is raising prices for international customers

Web crawlers deployed by Perplexity to scrape websites are allegedly skirting restrictions, according to a new report from Cloudflare. Specifically, the report claims that the company’s bots appear to be “stealth crawling” sites by disguising their identity to get around robots.txt files and firewalls.

Robots.txt is a simple file websites host that lets web crawlers know if they can scrape a websites’ content or not. Perplexity’s official web crawling bots are “PerplexityBot” and “Perplexity-User.” In Cloudflare’s tests, Perplexity was still able to display the content of a new, unindexed website, even when those specific bots were blocked by robots.txt. The behavior extended to websites with specific Web Application Firewall (WAF) rules that restricted web crawlers, as well.

Perplexity is allegedly scraping websites it’s not supposed to, againPerplexity is allegedly scraping websites it’s not supposed to, again

Cloudflare

Cloudflare believes that Perplexity is getting around those obstacles by using “a generic browser intended to impersonate Google Chrome on macOS” when robots.txt prohibits its normal bots. In Cloudlfare’s tests, the company’s undeclared crawler could also rotate through IP addresses not listed in Perplexity’s official IP range to get through firewalls. Cloudflare says that Perplexity appears to be doing the same thing with autonomous system numbers (ASNs) — an identifier for IP addresses operated by the same business — writing that it spotted the crawler switching ASNs “across tens of thousands of domains and millions of requests per day.”

Engadget has reached out to Perplexity for comment on Cloudflare’s report. We’ll update this article if we hear back.

Up-to-date information from websites is vital to companies training AI models, especially as service’s like Perplexity are used as replacements for search engines. Perplexity has also been caught in the past circumventing the rules to stay up-to-date. Multiple websites reported in 2024 that Perplexity was still accessing their content despite them forbidding it in robots.txt — something the company blamed on the third-party web crawlers it was using at the time. Perplexity later partnered with multiple publishers to share revenue earned from ads displayed alongside their content, seemingly as a make-good for its past behavior.

Stopping companies from scraping content from the web will likely remain a game of whack-a-mole. In the meantime, Cloudflare has removed Perplexity’s bots from its list of verified bots and implemented a way to identify and block Perplexity’s stealth crawler from accessing its customers’ content.

Credit: Source link

ShareTweetSendSharePin
Previous Post

How to download the update on your iPhone, new features like Liquid Glass and everything else you need to know

Next Post

Dan Patrick calls out Bronny James for missing Luka Doncic press conference

Related Posts

How to download the update on your iPhone, new features like Liquid Glass and everything else you need to know
Technology

How to download the update on your iPhone, new features like Liquid Glass and everything else you need to know

August 5, 2025
Spotify is raising prices for international customers
Technology

Spotify is raising prices for international customers

August 4, 2025
The Google Pixel 9a is 0 off right now
Technology

The Google Pixel 9a is $100 off right now

August 4, 2025
Apple’s MacBook Air M4 drops to a record-low price
Technology

Apple’s MacBook Air M4 drops to a record-low price

August 3, 2025
The HORI Piranha Plant camera for Switch 2 is on sale for only
Technology

The HORI Piranha Plant camera for Switch 2 is on sale for only $40

August 3, 2025
Samsung Z Flip 7, Oakley Meta glasses, DJI Osmo 360 and more
Technology

Samsung Z Flip 7, Oakley Meta glasses, DJI Osmo 360 and more

August 2, 2025
Next Post
Dan Patrick calls out Bronny James for missing Luka Doncic press conference

Dan Patrick calls out Bronny James for missing Luka Doncic press conference

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

What's New Here!

People in the happiest relationships do 5 things during weekdays

People in the happiest relationships do 5 things during weekdays

July 13, 2025
In job losses, AI’s role may be bigger than companies say

In job losses, AI’s role may be bigger than companies say

July 20, 2025
Starbucks is pioneering the ‘coffeehouse of the future’ with comfy chairs and couches as its CEO pushes to reimagines its stores

Starbucks is pioneering the ‘coffeehouse of the future’ with comfy chairs and couches as its CEO pushes to reimagines its stores

July 6, 2025
Sheryl Sandberg, Bill Gates and the world’s top CEOs swear by the same daily habit—this career coach says Gen Z can easily steal it for success

Sheryl Sandberg, Bill Gates and the world’s top CEOs swear by the same daily habit—this career coach says Gen Z can easily steal it for success

August 2, 2025
These 7 phrases make you sound like ‘an obnoxious American’ traveling abroad

These 7 phrases make you sound like ‘an obnoxious American’ traveling abroad

July 27, 2025
Mendaera gets FDA 510(k) clearance for handheld robotic system

Mendaera gets FDA 510(k) clearance for handheld robotic system

July 9, 2025
0,000+ to install elevators — these six-figure trade jobs don’t require college degrees

$100,000+ to install elevators — these six-figure trade jobs don’t require college degrees

July 9, 2025

About

World Tribune is an online news portal that shares the latest news on world, business, health, tech, sports, and related topics.

Follow us

Recent Posts

  • Napheesa Collier out at least two weeks in Lynx injury blow
  • After Figma’s red hot IPO, investors say these companies may be next to IPO
  • Hims & Hers (HIMS) Q2 earnings 2025
  • Dan Patrick calls out Bronny James for missing Luka Doncic press conference

Newslatter

Loading
  • Submit Your Content
  • Contact Us
  • Privacy Policy
  • Terms of Use
  • DMCA

© 2024 World Tribune - All Rights Reserved!

No Result
View All Result
  • Home
  • News
  • Business
  • Technology
  • Sports
  • Health
  • Food

© 2024 World Tribune - All Rights Reserved!

Welcome Back!

Login to your account below

Forgotten Password? Sign Up

Create New Account!

Fill the forms below to register

All fields are required. Log In

Retrieve your password

Please enter your username or email address to reset your password.

Log In