# As a condition of accessing this website, you agree to abide by the following # content signals: # (a) If a Content-Signal = yes, you may collect content for the corresponding # use. # (b) If a Content-Signal = no, you may not collect content for the # corresponding use. # (c) If the website operator does not include a Content-Signal for a # corresponding use, the website operator neither grants nor restricts # permission via Content-Signal with respect to the corresponding use. # The content signals and their meanings are: # search: building a search index and providing search results (e.g., returning # hyperlinks and short excerpts from your website's contents). Search does not # include providing AI-generated search summaries. # ai-input: inputting content into one or more AI models (e.g., retrieval # augmented generation, grounding, or other real-time taking of content for # generative AI search answers). # ai-train: training or fine-tuning AI models. # use: how AI systems may consume the content (immediate, reference, or full). # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET. # BEGIN Cloudflare Managed content User-agent: * Content-Signal: search=yes,ai-train=no,use=reference Allow: / User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CloudflareBrowserRenderingCrawler Disallow: / User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: meta-externalagent Disallow: / # END Cloudflare Managed Content # ours.market — crawl policy. # # ⚠ WHAT IS SERVED IS NOT THIS FILE ALONE. Cloudflare prepends a MANAGED block # at the zone — Content Signals plus its own AI-crawler deny list, fenced by # "BEGIN/END Cloudflare Managed content" — ahead of everything below, and it is # not editable from this repo (Cloudflare dashboard → AI Crawl Control). As of # 2026-07-30 it carries `User-agent: Google-Extended / Disallow: /`, which the # exception below does not override so much as merge with: same-length `Allow: /` # beats `Disallow: /` under the least-restrictive tie-break, so the intent below # survives on a tie-break rather than on the rule. Read the live file, not this # one, before concluding anything: curl -sS https://ours.market/robots.txt # # Real search engines may index the storefront (that is how customers find us). # AI-training / bulk-dataset crawlers are denied wholesale, and the private # capability pages (/t/, /c/, /account/, /checkout/) are off-limits to every # crawler — IN ALL EIGHT LOCALES. # # The `/*/…` twin of each fence is the whole point of this note. The four rules # were root-anchored, and every one of those pages is also served under seven # locale prefixes: Header.astro and CartDrawer.astro link the account+checkout # pair from every page of every prefixed tree, and _redirects answers /ru/t/ # with a 200. So a crawler obeying this file to the letter still walked onto a # customer's receipt — the same locale-prefix blind spot that let a doubled slash # past the middleware's PRIVATE guard and armed the Ads tag on /t/. # The wildcard form is documented by Google and Bing; the bare rules are kept # beside it so a crawler that does not implement `*` is still fenced in French. # # The JSON data surfaces under /api/ are deliberately OPEN as of 2026-07-30. # They are what /.well-known/api-catalog and /openapi.json advertise, and a # catalogue listing endpoints this file forbids is a contradiction a compliant # agent resolves by not fetching them. Bulk harvesting is still refused — by the # user-agent deny list below, which is the lever that actually works; a path # Disallow only ever stopped the crawlers that were already obeying. # # ONE deliberate exception: Google-Extended (below). It is the gate on Google's # generative features, and with it denied the Performance Max campaign reports # "Google AI isn't able to generate assets for your final URL" — the ad account # cannot read the shop it is advertising. The trade is real (Google may use this # content for Gemini training/grounding); it is accepted so the paid channel can # describe the storefront accurately. Re-adding it to the deny list below is the # only change needed to reverse this. # ── AI training + dataset scrapers: full deny ──────────────────────────────── User-agent: GPTBot User-agent: ChatGPT-User User-agent: OAI-SearchBot User-agent: ClaudeBot User-agent: Claude-Web User-agent: anthropic-ai User-agent: Applebot-Extended User-agent: CCBot User-agent: Bytespider User-agent: Amazonbot User-agent: meta-externalagent User-agent: FacebookBot User-agent: PerplexityBot User-agent: Perplexity-User User-agent: cohere-ai User-agent: Diffbot User-agent: ImagesiftBot User-agent: Omgilibot User-agent: Omgili User-agent: DataForSeoBot User-agent: Scrapy Disallow: / # ── Google's generative crawler: storefront + data, not the capability URLs ── # Explicit rather than inherited: the deny list above is the file's default # posture, so an allow that is only implied would read like an oversight. User-agent: Google-Extended Disallow: /t/ Disallow: /c/ Disallow: /account/ Disallow: /checkout/ Disallow: /*/t/ Disallow: /*/c/ Disallow: /*/account/ Disallow: /*/checkout/ Allow: / # ── Google Ads landing-page crawler ────────────────────────────────────────── # AdsBot ignores the `*` group by design, so without its own rules it would be # the one crawler free to walk the customer receipt and portal URLs. User-agent: AdsBot-Google User-agent: AdsBot-Google-Mobile Disallow: /t/ Disallow: /c/ Disallow: /account/ Disallow: /checkout/ Disallow: /*/t/ Disallow: /*/c/ Disallow: /*/account/ Disallow: /*/checkout/ Allow: / # ── everyone else (Googlebot, Bingbot, agents): pages + data, no capability ── # URLs. /.well-known/ is unlisted on purpose: nothing here forbids it, and a # discovery document a crawler must be granted permission to read is useless. User-agent: * Disallow: /t/ Disallow: /c/ Disallow: /account/ Disallow: /checkout/ Disallow: /*/t/ Disallow: /*/c/ Disallow: /*/account/ Disallow: /*/checkout/ Allow: / Sitemap: https://ours.market/sitemap-index.xml