# robots.txt for unimelabs.com
# EXXY® AI Studio — public crawler directives
# Last updated: 2026-07-26
#
# HOW THIS FILE WORKS — read before editing.
# A crawler obeys exactly ONE group: the group whose User-agent line is the most
# specific match for its token. Every other group is ignored entirely. An earlier
# revision of this file added `User-agent: Googlebot` + `Allow: /` blocks "for
# clarity" — because those groups carried no Disallow lines, Googlebot, Bingbot,
# GPTBot, PerplexityBot, ClaudeBot and the rest were silently exempted from every
# Disallow below and were free to crawl /login, /home, /payment/* and /admin.
# So: keep all crawl rules in the single `User-agent: *` group. Only add a named
# group for an agent that genuinely needs DIFFERENT rules, and repeat the full
# Disallow list inside it when you do.

User-agent: *
Allow: /

# --- Public marketing & content (redundant under `Allow: /`, kept as intent) ---
Allow: /about-us
Allow: /privacy-policy
Allow: /terms-of-use
Allow: /help
Allow: /plans
Allow: /blog
Allow: /share/
Allow: /presentations/
# Vietnamese locale is path-prefixed and fully indexable.
Allow: /vi
Allow: /vi/

# --- Authenticated app surfaces — no public content to index ---
Disallow: /home
Disallow: /profile
Disallow: /membership
Disallow: /research/
Disallow: /slides/
Disallow: /collaboration/
Disallow: /3d-model/
Disallow: /social-content
Disallow: /maxmode-request
Disallow: /enterprise-request

# --- Auth & payment flows — never index ---
Disallow: /login
Disallow: /register
Disallow: /auth/
Disallow: /payment/

# --- Admin portal — separate domain, redirect-only ---
Disallow: /admin
Disallow: /admin/

# --- AI / LLM crawler policy -------------------------------------------------
# All of the agents below are deliberately welcome on public content so AI answers
# cite EXXY® AI Studio accurately. They are already covered by `User-agent: *`
# above — giving them their own groups would drop the Disallow rules (see the note
# at the top of this file), so they are documented here instead:
#   Search:   Googlebot, Googlebot-Image, Bingbot, DuckDuckBot, Applebot,
#             YandexBot, Baiduspider, PetalBot, Qwantbot, SeznamBot
#   AI / LLM: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User,
#             Claude-SearchBot, anthropic-ai, PerplexityBot, Perplexity-User,
#             Google-Extended, Google-CloudVertexBot, Applebot-Extended,
#             DuckAssistBot, Amazonbot, Meta-ExternalAgent, MistralAI-User,
#             cohere-ai, YouBot, AI2Bot, CCBot, Bytespider, Diffbot, Timpibot,
#             omgili, ImagesiftBot, FirecrawlAgent
#   Social:   Twitterbot, facebookexternalhit, LinkedInBot, Slackbot,
#             WhatsApp, TelegramBot, Discordbot
# Structured summaries written for LLMs: /llms.txt, /llms-full.txt, /ai.txt

# --- Sitemaps ---------------------------------------------------------------
# sitemap.xml is the primary index. sitemap.rss (RSS 2.0) and blog/feed.xml
# (Atom 1.0) are both accepted as sitemap formats by Google and Bing and give
# faster freshness signals for newly published content.
Sitemap: https://unimelabs.com/sitemap.xml
Sitemap: https://unimelabs.com/sitemap.rss
Sitemap: https://unimelabs.com/blog/feed.xml