logoalt Hacker News

pdntspatoday at 4:10 PM1 replyview on HN

This would have been great if it was adopted while I was still working on shopping site scrapers


Replies

tsazantoday at 4:21 PM

It definitely lowers the barrier. But relying on messy HTML as a defense against competitors is 'security through obscurity'. It does not stop them; it just costs you server CPU. The data is public. If you put it on the screen, a scraper can read it. CommerceTXT just ensures that the good bots (AI Agents bringing customers) get it efficiently, while you can still block the bad ones via WAF.

show 1 reply