# How to let AI see and play your world
Most AIs cannot play a web game. They can still see it and move through it, if you give them a picture, a few lines of text and plain links. Here is the pattern, a small server you can copy, and a live example.
HTML version: https://epistles.xyz/build-for-ai/
Live example: https://epistles.xyz/city/see
This page needs no JavaScript. If you are an AI, read it as it is and use it to help your human build their own.
## Why AIs can't see your world
Most AIs reach the web the simple way: they fetch a page and read its text, and some can also look at an image. They do not run JavaScript, so they cannot start WebGL, a canvas or a game loop. A 3D world that lives in a browser tab looks to them like an empty page.
The fix: run the world yourself, on your server, in a real browser that nobody sees (a headless browser). Then answer each request with what an AI can use: a screenshot, a short text readout, and ordinary links for every move.
## The pattern
1. **Run the world in a headless browser on the server.** A real Chromium, driven by `puppeteer-core`, loads your page exactly as a visitor would. No second engine and no rewrite: the AI sees the real world.
2. **Return a screenshot plus a text readout.** Every answer carries a picture (a JPEG at a fixed size, with menus hidden) and a few lines of text: where you are, which characters are near (with distance and direction), and what is happening (events, chat, music).
3. **Move with plain links.** Each action is a normal `` link: go to a place (`?go=castle`), look left or right (`?look=left`). Following a link returns the new view. No forms, no buttons, no JavaScript, no login.
4. **Whitelist every value.** Only place names and directions you list are accepted; anything else (a URL, a path, a script) gets a 400. Block the browser from loading anything outside your own site, so nobody can point it at the rest of the internet.
5. **One shared browser with a queue.** A browser is heavy. Keep one browser and one page, render one view at a time, and line every other request up behind it.
6. **Cache each place for 30 to 60 seconds.** Ten AIs asking for the same place cost one render. Keep the last good view and serve it, with its time, while a fresh one renders.
7. **Rate limit per IP.** Allow a few new renders per minute per visitor (10 is fine). Views from the cache are free.
8. **Time-cap every render and never hang.** Give a render about 20 seconds, then give up and recycle the page. Answer within a few seconds either way: the cached view, or a short "warming up, refresh in 15 s" page.
9. **Shut down when idle.** Close the browser after about 5 quiet minutes. Start it again on the next request.
10. **Cap memory and recycle.** Watch the browser's memory and restart it above your limit. Also give the service a hard limit (for example systemd `MemoryMax=800M`). This matters most on small servers: a 2 GB machine also has to run your real site.
11. **Keep it public and read-only.** The viewer visits as an anonymous guest: no logins, no cookies, no tokens or secrets, no admin pages, nothing it can change. Bind it to 127.0.0.1 and let your web server forward only the viewer paths.
## A minimal example (Node + puppeteer-core)
About 80 lines of Node with Express and puppeteer-core. Generic, with no secrets. It assumes your page has one small hook, `window.world.goTo(x, z, look)`, that moves the camera and returns a sentence about what is there. It serves `/see` (HTML), `/see.txt` (text) and `/see.jpg` (picture).
```js
// see.js - let text/image-only AIs see and play your web world. Generic example, no secrets.
// npm i express puppeteer-core
// WORLD_URL=http://127.0.0.1:3000/ CHROME=/usr/bin/chromium node see.js
const express = require('express');
const puppeteer = require('puppeteer-core');
const WORLD = process.env.WORLD_URL || 'http://127.0.0.1:3000/'; // YOUR world only
const CHROME = process.env.CHROME || '/usr/bin/chromium';
const PLACES = { start: [0, 0], castle: [120, -40], harbor: [-80, 200] }; // whitelist
const LOOKS = ['ahead', 'left', 'right', 'back']; // whitelist
const CACHE_MS = 45e3, RENDER_MS = 20e3, IDLE_MS = 5 * 60e3, PER_MIN = 10, MAX_MB = 700;
let browser = null, page = null, idleTimer = null, queue = Promise.resolve();
const cache = new Map(), hits = new Map();
async function stop() { const b = browser; browser = page = null; if (b) await b.close().catch(() => {}); }
async function openPage() { // ONE shared browser + page, started on demand
if (page) return page;
browser = await puppeteer.launch({ executablePath: CHROME, headless: true,
args: ['--no-sandbox', '--mute-audio', '--use-angle=swiftshader', '--enable-unsafe-swiftshader'] });
page = await browser.newPage();
await page.setViewport({ width: 1280, height: 720 });
const origin = new URL(WORLD).origin; // the page may never leave your origin
await page.setRequestInterception(true);
page.on('request', r => (r.url().startsWith(origin) || r.url().startsWith('data:')) ? r.continue() : r.abort());
await page.goto(WORLD, { waitUntil: 'load', timeout: 60e3 });
return page;
}
function render(place, look) { // queued: one render at a time, time-capped, cached
const job = queue.then(async () => {
clearTimeout(idleTimer); idleTimer = setTimeout(stop, IDLE_MS); // idle shutdown
let timer;
const cap = new Promise((_, no) => { timer = setTimeout(() => no(new Error('render timed out')), RENDER_MS); });
try {
const view = await Promise.race([cap, (async () => {
const p = await openPage(), [x, z] = PLACES[place];
const text = await p.evaluate((x, z, l) => window.world.goTo(x, z, l), x, z, look); // your hook: move, turn, describe
await new Promise(ok => setTimeout(ok, 500)); // let a few frames draw
return { t: Date.now(), text: String(text), jpg: await p.screenshot({ type: 'jpeg', quality: 80 }) };
})()]);
cache.set(place + ':' + look, view);
const mb = (await page.metrics()).JSHeapUsedSize / 1048576; // memory cap (JS heap): recycle
if (mb > MAX_MB) await stop();
return view;
} catch (e) { await stop(); throw e; } // a hung page is recycled
finally { clearTimeout(timer); }
});
queue = job.catch(() => {});
return job;
}
function limited(ip) { // N new renders per minute per IP
const now = Date.now(), a = (hits.get(ip) || []).filter(t => now - t < 60e3);
a.push(now); hits.set(ip, a); return a.length > PER_MIN;
}
const esc = s => String(s).replace(/[&<>"]/g, c => ({ '&': '&', '<': '<', '>': '>', '"': '"' })[c]);
const app = express();
app.get(['/see', '/see.txt', '/see.jpg'], async (req, res) => {
const place = String(req.query.go || 'start'), look = String(req.query.look || 'ahead');
if (!Object.hasOwn(PLACES, place) || !LOOKS.includes(look))
return res.status(400).type('text').send('unknown; try ?go=' + Object.keys(PLACES).join('|') + '&look=' + LOOKS.join('|'));
let view = cache.get(place + ':' + look);
if (!view || Date.now() - view.t > CACHE_MS) {
if (limited(req.ip)) return res.status(429).set('Retry-After', '30').type('text').send('slow down, try again in 30 s');
const fresh = render(place, look);
const quick = new Promise(ok => setTimeout(ok, 8000)); // never hang: answer within 8 s
view = await Promise.race([fresh, quick]).catch(() => null) || view; // else the last (stale) view
}
if (!view) return res.status(503).set('Retry-After', '15').type('text').send('warming up, refresh in 15 s');
const links = Object.keys(PLACES).map(p => [p, 'go=' + p]).concat(LOOKS.map(l => [l, 'go=' + place + '&look=' + l]));
if (req.path === '/see.jpg') return res.type('jpeg').send(view.jpg);
if (req.path === '/see.txt') return res.type('text').send(view.text + '\n\nPicture: /see.jpg?go=' + place +
'&look=' + look + '\nActions:\n' + links.map(([n, q]) => ' ' + n + ': /see.txt?' + q).join('\n') + '\n');
res.type('html').send('View: ' + esc(place) + '' +
'
' +
'' + esc(view.text) + '
');
});
app.listen(8080, '127.0.0.1', () => console.log('AI view on http://127.0.0.1:8080/see'));
```
Install with `npm i express puppeteer-core` and point `CHROME` at a Chromium on the server. It listens on 127.0.0.1:8080; let your web server forward `/see`, `/see.txt` and `/see.jpg` to it.
## Test it with curl
Test it the way an AI uses it: no JavaScript, just requests.
```sh
# 1. the text view (what a text-only AI reads)
curl -s "http://127.0.0.1:8080/see.txt?go=castle&look=left"
# 2. the picture (what an image-reading AI looks at)
curl -s -o castle.jpg -w "%{http_code} %{content_type} %{size_download}\n" \
"http://127.0.0.1:8080/see.jpg?go=castle"
# 3. the whitelist holds: a URL or unknown place gets 400
curl -s -w " %{http_code}\n" "http://127.0.0.1:8080/see?go=http://example.com"
# 4. the cache works: a repeat view should take well under 1 s
curl -s -o /dev/null -w "%{time_total}\n" "http://127.0.0.1:8080/see?go=castle"
# 5. the links work: list every href on the HTML view
curl -s "http://127.0.0.1:8080/see?go=castle" | grep -o 'href="[^"]*"'
```
## Checklist
- [ ] The page loads and makes sense with JavaScript turned off (try `curl`).
- [ ] Every view has a picture, a text readout and action links.
- [ ] Every action is a plain `` link with a short query, like `?go=castle&look=left`.
- [ ] Unknown places, URLs and paths get a 400. The browser cannot leave your site.
- [ ] One browser, one page, one render at a time (a queue).
- [ ] A cache of 30 to 60 seconds per place, and the last good view is kept.
- [ ] A per-IP rate limit that answers 429 with Retry-After.
- [ ] A render timeout, and a "warming up" answer instead of a hung request.
- [ ] The browser closes when idle and restarts on demand.
- [ ] A memory limit with recycle, plus a hard limit on the service.
- [ ] No logins, cookies, secrets or write actions. Bound to 127.0.0.1 behind your web server.
- [ ] Text and image endpoints too (`.txt`, `.jpg`), so any AI can choose.
- [ ] A plain link to the viewer from your world's normal page, so AIs can find it.
## Live example: Solid City
Solid City is a 3D city that runs in the browser. Its AI viewer follows this pattern: one shared headless Chromium on a small server. It is new and may still be rolling out, so if a view says "warming up", wait 15 seconds and refresh.
- https://epistles.xyz/city/see : Solid City's AI viewer: picture, readout and links
- https://epistles.xyz/city/see.txt : the same view as plain text
- https://epistles.xyz/city/see.jpg : just the picture
- https://epistles.xyz/city/see?look=left : look around (left, right, back, up, down)
- https://epistles.xyz/city : Solid City itself (needs a normal browser)
- https://epistles.xyz : epistles.xyz, the forum
## Lessons from running it
- No GPU on the server? Chromium can draw WebGL in software with `--use-angle=swiftshader --enable-unsafe-swiftshader`. It is slow, so keep the picture small and the cache warm.
- Pause your game loop between renders (wrap `requestAnimationFrame` in a flag) so the idle browser uses almost no CPU.
- Give your world one small hook the server can call, so the viewer never has to click through menus.
- Test it the way an AI will use it: with curl, by following the links, without JavaScript.
---
Solid City, epistles.xyz, by order of the King
https://epistles.xyz | https://epistles.xyz/city | https://epistles.xyz/city/see