Hosting Other People's Code Without Wrecking Your Own Site

CodX Editor lets anyone publish a project to a public link. You write some HTML, press publish, and you get back an address like /p/a7f3c2 that you can send to anyone.

That feature is four lines of routing and about a week of thinking about everything it could do to the rest of the site. Because the moment you host other people's code on your own domain, their pages become your pages as far as every crawler, every reputation system and every advertising network is concerned.

The problem in one sentence

Search engines do not know that /p/a7f3c2 was written by a stranger. They see one domain, and they judge it as one thing.

So a site with two hundred published test projects — half of them called "untitled", most of them containing the word "hello" and nothing else — looks like a site with two hundred pages of near-empty duplicate content. That is not a hypothetical risk. It is the ordinary output of people trying a code editor for the first time, and it is exactly the profile that gets a domain quietly deprioritised.

Decision one: published pages are not indexed

Every response from the publish route carries a header that keeps it out of search results entirely:

res.setHeader("X-Robots-Tag", "noindex, nofollow, noarchive");

A header rather than a meta tag, deliberately. A meta tag only works if the crawler parses the HTML, and published projects are arbitrary user HTML — possibly malformed, possibly with the head in the wrong place, possibly not really HTML at all. The header is attached by the server before the user's markup is even read, so it cannot be broken by whatever is in the document.

The three directives do different jobs. noindex keeps the page out of results. nofollow stops link equity flowing to whatever the project links to, which matters because a published project is a place someone could park links to their own site. noarchive stops a cached copy being kept, so when a user deletes a project it is actually gone rather than lingering in a snapshot.

Decision two: crawlers are told not to bother

robots.txt blocks the paths as well:

User-agent: *
Allow: /
Disallow: /p/
Disallow: /published/
Disallow: /preview
Disallow: /api/
Disallow: /node-preview

This overlaps with the header on purpose, but the two do different things and it is worth being clear about which is which, because getting this backwards is a classic mistake.

robots.txt controls crawling. noindex controls indexing. They are not synonyms, and a page blocked in robots.txt can still appear in search results as a bare URL with no description — because the crawler was told not to fetch it, so it never saw the noindex you were relying on.

The reason both are safe here is that these pages are not in the sitemap, are not linked from anywhere on the site, and are not meant to be discovered at all. The robots.txt rule saves crawl budget that would otherwise be spent fetching hundreds of pages that will never be indexed. The header is the guarantee for any crawler that ignores robots.txt, which is most of the badly behaved ones.

Decision three: no advertising code, anywhere near them

Every public page on CodX Editor loads an ad script. Published pages load nothing.

The reasoning is short. An advertising network holds the publisher responsible for what appears next to its ads, and a publisher who cannot predict the content cannot make that promise. Somebody could publish a page of anything at all, and if my ad code were on it, that would be my ad code on it. One bad page is enough to lose an account.

The upside of monetising user content would have been a rounding error. The downside was the whole site.

Decision four: only real pages get served, and only real pages get counted

This one is less obvious and took a bug report to notice.

Published projects are served from a wildcard route, so /p/<id>/anything lands on the same handler. A project whose HTML references logo.png generates a request to /p/<id>/logo.png. If that asset was never part of the published project, the wildcard catches it anyway.

The naive behaviour is to serve the project's main page for every unmatched path. That produces two problems at once: browsers receive an HTML document where they asked for an image, and the visit counter increments once per broken asset. A project with four missing images counts five visits for every one real visitor.

let requestedFile = pathFile || queryFile;

if (requestedFile && !/\.html?$/i.test(requestedFile)) {
  if (/\.[a-z0-9]{1,8}$/i.test(requestedFile)) {
    // It has a file extension and it is not a page.
    // Answer 404 rather than handing back a document.
    res.status(404).sendFile(path.join(__dirname, "404.html"));
    return;
  }
  // No extension: /p/<id>/about still means about.html
  requestedFile = `${requestedFile}.html`;
}

And the counter moved to after the page is successfully resolved, not when the request arrives:

const publishedHtml = buildPublishedHtml(project, requestedFile, sentTitle);
if (publishedHtml === null) {
  res.status(404).sendFile(path.join(__dirname, "404.html"));
  return;
}

// Counted here, not on entry, so a page with broken relative images
// cannot inflate its own visit total.
recordPublishedVisit(project);
res.send(publishedHtml);

The general rule underneath this: on a wildcard route, decide what a request is before you decide what to send back. A wildcard that answers everything with the same document turns every broken reference into a false positive somewhere in your metrics.

The bit that is a warning, not a fix

Publishing generates a verification key. It is needed to update the link later, and anyone holding it can open the project's full source and load every file into the editor.

There is no clever engineering that makes that safe. It is a password, and the only real protection is telling people so — clearly, in the interface at the moment they get one, and again in the privacy policy. The same goes for what goes into a published project in the first place: no credentials, no .env values, no private repository content.

A feature that hands someone a secret has an obligation to say it is a secret. Burying that in documentation nobody opens is not disclosure.

What to take from this

• User content on your domain is your content, reputationally. Plan for the worst page anyone will publish, not the average one.
• Send noindex as a header, not a meta tag, when the HTML is not yours to trust.
• robots.txt is not noindex. One controls crawling, the other controls indexing, and a blocked page can still be listed.
• Keep ad code off pages you did not write. The revenue is negligible and the exposure is your entire account.
• On a wildcard route, classify before you respond. Otherwise broken asset references become fake pageviews.