Robots.txt and llms.txt for Next.js on Vercel

What robots.txt does

robots.txt is a standard file served from the root of a website. Crawlers request it before crawling so they can see which paths are allowed or disallowed. It is useful for keeping internal, duplicate, or API URLs out of crawl queues.

A simple public site usually starts with allowing everything and pointing crawlers to the sitemap:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

What llms.txt does

llms.txt is a Markdown proposal for giving AI assistants a clean summary of your site. It does not replace robots.txtand it is not a guaranteed ranking factor. Its job is to explain what your site is, which pages matter, and where an AI system should look first.

# Example Site

> Short summary of what the site offers.

## Tools

- [Main tool](https://example.com/tools/main): What this page helps users do.

## Guides

- [Blog](https://example.com/blog): Practical guides and examples.

How to set this up in Next.js

For App Router projects, use app/robots.ts and app/sitemap.ts when you want generated files. For a plain static llms.txt, put the file in public/llms.txt. Next.js serves files from public at the site root, so that file becomes available at /llms.txt after deployment.

On Vercel, no extra routing configuration is needed for these files in a normal Next.js app. Deploy the project and check the production URLs directly.

How to test before and after deploy

Run the app locally and request the files:

npm run dev
curl -i http://localhost:3000/robots.txt
curl -i http://localhost:3000/sitemap.xml
curl -i http://localhost:3000/llms.txt

After deployment, test the live domain:

curl -i https://example.com/robots.txt
curl -i https://example.com/sitemap.xml
curl -i https://example.com/llms.txt

Each response should return 200 OK. The sitemap should contain canonical production URLs, and robots.txt should reference that sitemap.