# Whitelist policy: only the homepage, contact page, policy pages and # spotlight pages (plus the Rendering Photos they embed) may be crawled. # Everything else (designer app, app.js, pricing data, API routes, other # assets) is off-limits to crawlers. User-agent: * Allow: /$ # Kept crawlable on purpose: /index.html now 301s to / (vercel.json). Crawlers # must be able to fetch it to see the redirect and drop the duplicate URL. Allow: /index.html Allow: /contact.html Allow: /kitchen-cabinets-aurora.html Allow: /spotlight.html Allow: /spotlight/ Allow: /learn.html # Free public .kit viewer (kit.html + the two scripts it runs on; nothing else under /kit*). Allow: /kit$ Allow: /kit/zh$ Allow: /kit/fr$ Allow: /kit.html Allow: /kit-zh.html Allow: /kit-fr.html Allow: /kit-viewer.js Allow: /kit2020-import.js # Free public PDF editor (the three language pages, its script and stylesheet, # and the two vendored engines it lazy-loads). Google renders a page before # judging it, so the CSS/JS have to be fetchable or /pdf grades as an unstyled # wall of text — the same trap that bit tw.css and spotlights.json. Allow: /pdf$ Allow: /pdf/zh$ Allow: /pdf/fr$ Allow: /pdf.html Allow: /pdf-zh.html Allow: /pdf-fr.html Allow: /pdf-editor.js Allow: /pdf-editor.css Allow: /assets/pdfjs/ Allow: /assets/pdf-lib/ Allow: /learn/guide/ Allow: /community.html Allow: /privacy.html Allow: /terms.html Allow: /Rendering%20Photos/ # MUST stay crawlable: the public pages' entire stylesheet. Google and Bing # render a page before judging it — block the CSS and they see an unstyled # wall of text (and flag it mobile-unfriendly). This replaced the old # cdn.tailwindcss.com script, which was external and so never robots-gated. Allow: /assets/tw.css # llms.txt — the plain-Markdown site guide written for AI assistants # (ChatGPT, Gemini, Claude, Perplexity). It returned 200 but the catch-all # Disallow below made every compliant AI crawler skip it, so the one file we # wrote specifically for them was the one file they could not read # (found 2026-08-25 — fifth recurrence of the whitelist trap). Allow: /llms.txt # IndexNow / site-verification key file (Bing Webmaster Tools). Allow: /71abb468be9141e38aebbf3d1f7283ae.txt # RSS feed(2026-08-29 新增)。⭐ 白名单制的老规矩:新增任何公开文件都必须在 # 这里点名放行,否则文件末尾那条 catch-all `Disallow: /` 会把它挡死 —— llms.txt、 # sitemap.xml、spotlights.json、tw.css、IndexNow key 已经各栽过一次。 Allow: /feed.xml # The sitemap itself. Google/Bing fetch it under robots rules too — without # this line the catch-all Disallow below makes GSC report "Couldn't fetch". Allow: /sitemap.xml # spotlight.html and the homepage trends section render their article grid by # fetching this JSON. Google's renderer honors robots for subresources too — # with it blocked, spotlight.html rendered as an empty shell and GSC live test # rejected it as a Soft 404 (found 2026-08-23). Allow: /spotlights.json # community.html builds its gallery/Q&A/challenges/pros grids and learn.html # builds its video grid from these endpoints (all public, no auth). Blocked, # the crawler's renderer sees empty grids + "No public designs yet" — the same # Soft-404 class as spotlights.json above (found 2026-08-28, recurrence #6). Allow: /api/community- Allow: /community/inspiration.json Allow: /community/answers-index.json # The redesigned community.html additionally renders door and countertop swatch # strips/tiles whose URLs are built at runtime from the colour catalogs. # Same trap as the JSON above: blocked, Google's renderer sees broken images and # grades the page a Soft 404 (recurrence #7). The real folder is "countertop # colors" with a literal space, percent-encoded here like /Rendering%20Photos/. Allow: /door-colors/ Allow: /countertop%20colors/ # ⭐ 白名单第 8 次栽跟头 (2026-09-05 全站体检 ACQ-001)。上面只放行了图片文件夹, # 却没放行「生成这些图片 URL 的脚本」——community.html 里 /i18n.js 被挡 => # 页面脚本抛 "T is not a function" 后整块中止, Googlebot 拿到的正文从 6,573 字 # 掉到 822 字、 从 16 张掉到 1 张 (learn.html 少掉视频宫格, index.html 少掉 # 整套 i18n 文案)。以下五个都是公开页首屏渲染必需的**数据/文案**脚本, 逐个查过 # 没有任何报价数据 (grep '$'/CAD/USD: door-colors*.js、countertop-colors.js、 # page-i18n.js 全 0 命中; i18n.js 只有 PayPal 最低充值和预算区间这两句 UI 文案, # 本来就印在页面上)。⛔ COMPLETE_PRICING.js 依旧不放行, 报价数据库永远不进白名单。 Allow: /i18n.js Allow: /page-i18n.js Allow: /door-colors.js Allow: /door-colors-extra.js Allow: /countertop-colors.js # Emma 的头像图(index.html 的 用 webp + png 两份,learn.html 用 png)。 # 同样是渲染时可见的图片:挡住 Google 就在首页渲染里看到一张裂图 (2026-09-05)。 Allow: /avatar.webp Allow: /avatar.png Allow: /avatar-full.webp Allow: /avatar-full.png # 干净 URL (2026-09-05 ACQ-010, vercel.json 新增 rewrite): /learn、/community、 # /spotlight 现在返回和对应 .html 完全相同的页面, 而那三个页面都自带指向 .html # 的 canonical, 所以放行它们不会产生重复收录, 只是让分享出去的短链接不至于撞上 # 下面那条 catch-all。⛔ /designer 和 /start 这两条新 rewrite **故意不放行**: # 设计器引擎不进白名单 (Kim 的老规矩), catch-all 已经把它们挡住。 Allow: /learn$ Allow: /community$ Allow: /spotlight$ # 新增公开页 (2026-09-05): /for-pros、/about、/warranty —— 两种写法都点名, # 干净 URL 走 vercel.json 的 rewrite, .html 是自 canonical 的那一份。 Allow: /for-pros$ Allow: /for-pros.html Allow: /about$ Allow: /about.html Allow: /warranty$ Allow: /warranty.html # Brand icons: Google only shows a site's favicon in search results if it can # crawl it. Blocked, every SERP entry shows the generic globe (found 2026-08-28). Allow: /brand/ # In-article images for the guide pages (install steps, designer screenshots). # Blocked, they are invisible to Google Images and missing from the rendered # page Google evaluates (found 2026-08-28). AI-training use is already handled # by the X-Robots-Tag: noimageai response header, which needs the fetch allowed. Allow: /learn/assets/ # The designer tool and its 2D/3D editor interfaces (and the JS that powers # them) must never be crawled or scraped. These are named explicitly below # for clarity; the catch-all Disallow at the end already blocks everything # outside the whitelist above. Disallow: /designer.html Disallow: /start.html Disallow: /settings.html Disallow: /app.js Disallow: /edit-mode-3d.js Disallow: /design-helper-3d.js Disallow: /countertop-edit.js # ⭐⭐ 内部页面**故意一个都不在这里点名**。robots.txt 是公开可读的:在这里 # 写 `Disallow: /<路径>` 等于对所有人发布一份「这个站有哪些内部页面」的目录 # (Kim 2026-08-14「不希望这个链接可以被找到」)。下面那条 catch-all 已经把 # 白名单以外的一切都挡住了,效果完全一样,只是不再点名。内部页面另有 # 和 vercel.json 里的 # X-Robots-Tag 两道,都不依赖这里点名。 # # ⚠️ 所以:**以后新增内部页面时,不要往这个文件里加 Disallow 行。** 加了 # 就是把它公布出去。 # # Catch-all: block everything not explicitly allowed above. Disallow: / Sitemap: https://www.flipkitchen.ca/sitemap.xml