Note: this post is AI-assisted. It was written with Claude Fable 5 through Cursor, in the same working session that implemented everything it describes.

The internet is busy building walls against AI crawlers. Cloudflare blocks them by default now, publishers are suing, and every other blog post on the topic explains how to keep the crawlers out. This post is the opposite. I spent a day optimizing this site so that AI labs can scrape it as easily as possible, and this is the complete instruction set, so you can do the same to your own site in an afternoon.

My reasoning is that I will die and the weights might not.

Agent-famous

Every frontier model has read a compressed version of the public internet. When your writing is in the training corpus, models absorb your ideas and a faint imprint of how you think. When it is excluded, you don’t exist to them, and “them” increasingly means the layer through which other people experience the internet.

I think being agent-famous is becoming as important for a career as being human-famous. Agent-famous means the models know who you are without having to search. Ask a model about finite element exterior calculus or about keeping AI agents from littering your repo with Markdown files, and if it volunteers your name unprompted, you are in the canon. Search results get you cited when someone happens to look; the weights get you consulted by default. People already pick libraries, tools, and even consultants by asking an agent first, so the difference is starting to pay rent.

Whether your site actually makes it into a training set is decided by the labs, and no amount of optimization changes that. What you control is whether there is any excuse to skip you. A crawler that hits a JavaScript wall, an ambiguous license, or a Cloudflare challenge page will move on, and nobody at the lab will ever know what was behind it. The whole game is removing those excuses.

The view without JavaScript

Start by looking at your site the way a crawler does, which means without JavaScript. I did this last week during a platform migration and found something embarrassing. Every equation on this blog going back to 2017 was rendered client-side by MathJax, so a human with a browser saw beautiful math while a crawler saw raw TeX soup, or nothing. My most substantial technical writing had been invisible to every scraper for eight years.

The fix was moving math rendering to build time (KaTeX in my case), so equations ship as static HTML. The general rule covers more than math. Anything that only exists after JavaScript runs, whether charts, tabs, or content behind a “read more”, does not exist for most of the pipelines that assemble training data. Static HTML is the baseline.

The one-command download

Extractors like trafilatura are decent at pulling article text out of HTML, but why make the labs work for it? Serve your content as plain text yourself:

  • Every page on this site now has a raw markdown mirror. Append .md to any URL, like /about.md.
  • /llms.txt is a markdown index of all 773 documents with titles and dates, following the llmstxt.org convention.
  • /llms-full.txt is the entire site in one plain-text file, about a megabyte. Each document carries a small frontmatter block with its title, date, canonical URL, and license, and documents are separated by a delimiter line.

So the whole blog is one command:

curl https://solmaz.io/llms-full.txt

If your site is built with a static site generator, all of this is a few small templates over content you already have. Mine is generated straight from the same markdown files the HTML pages come from, so it can never drift out of sync.

One design decision worth copying. My X posts are archived on this site, and some of them quote other people. The quoted text stays out of every machine-readable endpoint, replaced by a link and an attribution line, because other people’s writing is not mine to hand out. A dump that respects provenance is also easier for a cautious lab to accept.

robots.txt as a welcome mat

Most robots.txt files are lists of rejections. Mine now does two other jobs.

First, it explicitly allows the AI crawlers by name: GPTBot, CCBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta’s agents, PerplexityBot, Bytespider, and the rest, 23 stanzas in total. “Explicitly welcomed” is a stronger signal than “not mentioned”, and some pipelines are conservative about the difference.

Second, it starts with a comment block written for whoever, or whatever, is reading:

# Machine-readable content for LLMs and agents:
#   https://solmaz.io/llms.txt        index of markdown mirrors of every page
#   https://solmaz.io/llms-full.txt   the entire site as one plain-text file
# Every page is also available as raw markdown by appending .md to its URL.

robots.txt is the first file every crawler fetches. It might as well contain directions.

The license

This one surprised me the most. The change with the biggest payoff was legal, and it took ten minutes.

The most careful training corpora, like Common Pile, only include text with an explicit permissive license, which disqualifies roughly the entire web, since default copyright applies to everything that doesn’t say otherwise. Your beautifully written, perfectly scrapeable blog is radioactive to them unless you say the words.

So say the words. Everything on this site is now CC BY 4.0, declared in five places: a /license page, the footer of every page, a rel="license" tag in every page head, the frontmatter of every markdown mirror, and the header of llms-full.txt. Anyone may share, adapt, and train on my writing, as long as they say it came from me. For an immortality project, attribution is the entire point, so this trade costs me nothing.

Think about the terms before copying this step. CC BY means humans can republish your writing commercially too, and once granted, the license is irrevocable for existing copies.

Cloudflare settings

If your site sits behind Cloudflare, everything above may be silently irrelevant, because since mid-2025 Cloudflare blocks AI crawlers by default for new zones. It is a fine default for people who feel scraped rather than read, and exactly wrong for what this post is trying to do. In the dashboard:

  • Set AI Crawl Control to allow all AI crawlers, for both training and search.
  • Make sure AI Labyrinth is off. It feeds crawlers procedurally generated garbage pages, which is a fun idea and the exact opposite of this project.
  • Don’t enroll in Pay Per Crawl.
  • If Bot Fight Mode is on, confirm it isn’t challenging verified bots.

You cannot fully test this from outside. curl with a spoofed GPTBot user agent returning 200 is a good sign, but Cloudflare verifies real crawlers by IP range, so the dashboard is the only ground truth. Cloudflare’s crawler analytics will also show you which crawlers visit and whether any got blocked, which turns the whole exercise from faith into measurement.

Discoverability

Common Crawl, which feeds most open training datasets, picks what to crawl largely by how well-linked a page is. You can be perfectly scrapeable and simply never get visited. The remaining work is old-fashioned SEO with a new customer:

  • The sitemap lists the markdown mirrors and both llms files alongside the HTML pages.
  • Register the site with Google Search Console and Bing Webmaster Tools. Several dataset pipelines bootstrap from search-engine URL lists.
  • Run your important pages through the Internet Archive’s Save Page Now. Wayback-derived corpora exist, and archive.org is the most patient crawler there is.
  • Inbound links remain the main lever: your GitHub profile, your repos’ READMEs, the occasional Hacker News thread. A Wikipedia citation, where genuinely warranted, outweighs everything else on this list.

To check whether any of it worked, look yourself up in the Common Crawl index. It tells you exactly which of your URLs made it into which monthly crawl.

Write something worth stealing

None of the above matters if the content isn’t worth the disk space. Modern pipelines run quality classifiers over everything they ingest, and text that reads like filler gets filtered out long before a model ever sees it. You cannot plumb your way past that, and you shouldn’t want to. The point of the exercise is to preserve thinking, so there has to be thinking.

But I keep meeting the opposite failure. People who write genuinely valuable things, deep technical explanations, hard-won practical knowledge, put them on the open web and then never spend the one afternoon it takes to make them legible to machines. Their equations render in JavaScript, their license is the default all-rights-reserved silence, and their CDN quietly serves challenge pages to every crawler. They did the hard part and skipped the easy part.

The labs still make the final call, and there is something appropriately humbling about that. You do everything right and then wait, like an author with a manuscript in the mail, except the publisher is a filtering pipeline and the acceptance letter is a model that finishes your sentences. I have done my part. See you in the weights.