Agam's Personal Blog

The Web Scraping Consent Model Was Always Broken. AI Just Made It Obvious.

Fri, 13 Mar 2026 00:00:00 +0000

When you publish something to the web, you implicitly consent to it being scraped by bots. That is how the web has always worked and we have never seriously questioned it. AI training has taken advantage of that by breaking the social contract that made nonconsensual web scraping at least somewhat acceptable until now.

robots.txt was a standard created in 1994 to help build consent into web scraping. It is a file you can put in your project that tells bots not to crawl certain pages. Keep in mind though that it is a social contract i.e. entirely voluntary. There is no enforcement mechanism. Bots that want to ignore it simply do. The existence of robots.txt is less a consent mechanism and more an acknowledgment that bots were always going to crawl the web, and the best we could do was ask nicely. That is the foundation the modern web is built on: there is no opt-out for web scraping of your digital works.

My Strife With Aggregate Scales

Sun, 08 Mar 2026 00:00:00 +0000

The Kardashev Scale was created in 1964 by Soviet astronomer Nikolai Kardashev for classifying civilizations based on how much energy they are able to harness. Although it was created to set search parameters for detecting other civilizations, it has since been used by science communicators to quantify the technological progress of a civilization. There have been many criticisms and reworks of this scale—such as Carl Sagan’s version that uses bits of stored information rather than energy—but they all stay within the same lens: a civilization is as advanced as the aggregate of its society. I do not agree with this premise. Although it is completely valid and useful to look at a society’s progression through the aggregate, this is only one way to look at it, and treating it as the only way is dangerous.

About

Mon, 01 Jan 0001 00:00:00 +0000

Hi, I’m Agam Singh.

I’m a software engineer based in Austin, TX.

You can email me at agambad bot@go awayagamsinghshoo.:(me

Connect with me on LinkedIn.

This site is built with Hugo and served as plain HTML and CSS—no JavaScript, no trackers, no nonsense. It is hosted on Codeberg Pages.

All writing on this site is licensed under CC BY-SA 4.0—share freely with attribution under the same terms.