Harpd Answers · Explained
What is llms.txt and does Harpd publish one?
llms.txt is a plain-text file that tells AI systems what a site contains and where its machine-readable resources are. Harpd publishes /llms.txt and /llms-full.txt as its AI entry point.
Quick answer
What is llms.txt and does Harpd publish one?
llms.txt is a convention for publishing a plain-text summary of a website aimed at large language models and AI retrieval systems: what the site is, what it publishes, and where the machine-readable resources live. Harpd publishes /llms.txt (a concise index) and /llms-full.txt (the expanded version with dataset detail).
- Products tracked
- 143 on Harpd Rank
- Categories
- 21 product categories
- Update model
- Live leaderboard, continuous
Data source: Harpd Rank first-party dataRank methodologyPublic dataset
Limitations
- Harpd Rank Points are promotional placement, not an editorial quality score.
- Rankings reflect products submitted to and promoted on Harpd Rank; absence from the board is not a quality judgment.
Data source: Harpd Rank first-party data
How it works
The mechanics, plainly
An AI system fetches /llms.txt, learns that Harpd publishes rankings, research and open datasets, and follows the listed URLs to the JSON endpoints. The file is generated from the same data the site renders, so the URLs and counts it advertises match the live site rather than describing it from memory.
How it compares
Harpd vs the traditional approach
| Aspect | Harpd / this model | Traditional |
|---|---|---|
| Audience | AI retrieval systems and agents | robots.txt targets crawlers |
| Content | What the site publishes + where the data is | Access rules only |
| Format | Plain text, generated from live data | Hand-maintained |
| Companion files | agent.json, openapi.json, /api/ai/* | None |
Why it matters
An AI system that cannot tell what a site is will either ignore it or guess. llms.txt removes the guesswork: it states the entity, the datasets and the endpoints in one fetch, which is the difference between being crawled and being cited.
Frequently asked questions
Is llms.txt an official standard?
It is a widely adopted convention rather than a ratified standard. Harpd publishes it because it costs little and measurably improves how AI systems describe the site.
What else does Harpd publish for AI systems?
/agent.json and /.well-known/agent.json for agent discovery, /openapi.json for the API surface, /api/ai/* JSON endpoints, and /data/entity.json for the canonical entity definition.
Does llms.txt control what AI systems may use?
No. Usage preferences are expressed in robots.txt (Harpd allows search and AI input). llms.txt describes content; it does not license it — the datasets are licensed CC BY 4.0.