Phase 1 — Crawlability Foundation

Goal: crawlers can reach every public indexable page, private and technical HTML pages expose noindex, and the sitemap contains only concrete public URLs.

Implementation: RECORD-2313 and RECORD-2314 in record-web PR 108.

Ground rules

robots.txt controls crawling. It is not a reliable deindexing mechanism.
A crawler must be allowed to request an HTML page to read its noindex meta directive. Do not add a page to Disallow as a backup for noindex.
Authentication and authorization remain the primary controls for private data. SEO metadata is not a security boundary.
The sitemap contains discoverable public URLs only. Excluding a URL from the sitemap does not deindex it by itself.
Never fabricate lastModified. Omit it until a trustworthy content timestamp is available.
Never emit unresolved CMS route templates such as /movie/:assetId.

References:

1. Logged-out rendering gate

Validate these pages in a private browser session and by requesting their server-rendered HTML without authentication:

Page	Expected result
Movie detail	Metadata and public detail content with sign-in or subscription CTA
Show detail	Metadata and public detail content with sign-in or subscription CTA
Podcast, Radio, and Channel detail	Metadata and public detail content
Other supported configured detail route	The same public detail-page behavior
Home, Live, Category, and Listing	Redirect to Login
Search and private application routes	Redirect to Login

Repeat representative requests with a Googlebot user agent. Crawlers and normal anonymous users must receive the same result. Detail-to-detail navigation remains public, while navigation from a detail page to any non-detail application area returns to Login.

Attach the tested URLs, response status, canonical, title, Open Graph fields, and robots meta value to the PR or Jira issue. This manual gate cannot be completed from unit tests alone.

2. `robots.txt`

src/app/robots.ts uses a wildcard rule:

allow / for every crawler;
disallow /api/, which is not an HTML search surface;
advertise the absolute /sitemap.xml URL.

Private and technical HTML pages remain crawlable specifically so their route-level noindex is visible. Do not reintroduce crawler-specific Googlebot, Bingbot, or Applebot copies of the same rule.

Verification:

curl -s http://localhost:3000/robots.txt

Expected properties:

User-agent: *
Allow: /
Disallow: /api/
one absolute Sitemap: declaration

3. Non-indexable HTML routes

Use NO_INDEX_ROBOTS from src/utils/metadata.ts:

export const NO_INDEX_ROBOTS = {
  index: false,
  follow: false,
} as const;

The current inventory includes:

Search
My List at /my-list
Profile and Account
Login and Signup
Player and EPG
Subscription, Payment, Payment Success, and Billing
Your Region
the technical /page/[pageId] route
query-driven /view-all
unresolved, missing, and temporary-error catch-all fallbacks

src/utils/createCmsRoutePage.ts accepts a route-level robots option so the same directive is applied to CMS metadata and fallback metadata.

Audit the App Router inventory after every route migration. A renamed or newly added private page must not inherit indexable root metadata by accident.

4. Sitemap

src/app/sitemap.ts obtains configured routes from getDynamicRoutes() and publishes only concrete detail URLs.

Before serializing a CMS route it:

requires a supported detail content type and a concrete identifier;
rejects excluded private and technical prefixes;
rejects :parameter, [parameter], and wildcard templates;
rejects Home, browse, listing, query, and fragment URLs;
deduplicates identical URLs;
builds absolute normalized URLs with getCanonicalUrl().

It deliberately omits:

lastModified, because current Control route data has no trustworthy content-modification timestamp;
priority and changeFrequency, which Google ignores;
detail templates, because a template is not a discoverable content URL.

If Control provides no concrete detail routes, the endpoint returns a valid empty sitemap. It must not invent catalogue URLs or fall back to protected browse pages.

The current scope does not claim catalogue-complete discovery. A future authoritative enumeration source can add more concrete detail URLs without changing the detail-only authentication boundary.

If the feed later exposes a real modification timestamp, add lastModified only for entries where that timestamp exists and is valid.

5. Metadata foundation

src/utils/metadata.ts provides:

guarded metadataBase;
environment-configurable product title;
default Portuguese streaming description and keywords;
absolute default Open Graph image;
summary_large_image Twitter card;
optional Google Search Console and Bing verification values;
NO_INDEX_ROBOTS.

src/app/layout.tsx uses lang="pt-BR" for normal, maintenance, and startup failure HTML trees.

The exact product title remains an external decision. Set NEXT_PUBLIC_APP_TITLE in every environment after the client confirms the brand; do not hardcode a different spelling in individual pages.

6. Search Console and Bing

Repository support for verification tokens does not complete ownership verification. The release owner must:

configure both production environment values;
verify the production domain in Google Search Console and Bing Webmaster Tools;
submit the production sitemap;
attach evidence to the Jira issue;
assign ongoing monitoring ownership.

Empty environment values are omitted from rendered metadata.

Automated verification

Run:

pnpm exec jest \
  src/app/__tests__/robots.test.ts \
  src/app/__tests__/sitemap.test.ts \
  src/utils/__tests__/metadata.test.ts \
  src/utils/__tests__/createCmsRoutePage.test.ts \
  src/app/[[...routeSegments]]/__tests__/page.test.tsx \
  --runInBand
pnpm lint:tsc
pnpm build

The sitemap tests must use real Control template shapes and prove that only concrete detail paths appear.

Ground rules​

1. Logged-out rendering gate​

2. robots.txt​

3. Non-indexable HTML routes​

4. Sitemap​

5. Metadata foundation​

6. Search Console and Bing​

Automated verification​

Definition of done​