# As a condition of accessing this website, you agree to abide by the following # content signals: # (a) If a Content-Signal = yes, you may collect content for the corresponding # use. # (b) If a Content-Signal = no, you may not collect content for the # corresponding use. # (c) If the website operator does not include a Content-Signal for a # corresponding use, the website operator neither grants nor restricts # permission via Content-Signal with respect to the corresponding use. # The content signals and their meanings are: # search: building a search index and providing search results (e.g., returning # hyperlinks and short excerpts from your website's contents). Search does not # include providing AI-generated search summaries. # ai-input: inputting content into one or more AI models (e.g., retrieval # augmented generation, grounding, or other real-time taking of content for # generative AI search answers). # ai-train: training or fine-tuning AI models. # use: how AI systems may consume the content (immediate, reference, or full). # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET. # BEGIN Cloudflare Managed content User-agent: * Content-Signal: search=yes,ai-train=no,use=reference Allow: / User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CloudflareBrowserRenderingCrawler Disallow: / User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: meta-externalagent Disallow: / # END Cloudflare Managed Content # ================================================================= # robots.txt — https://www.schooloffreelancing.com/ # Stack : CWP Control Web Panel · Apache · Varnish Cache # Site : Static HTML / CSS # Purpose : Maximum discoverability for SEO · AEO · GEO · LLM · LLMO # while keeping panel & server paths out of search results. # Standard: RFC 9309 — uses only universally-supported directives # (User-agent, Allow, Disallow, Sitemap), so every bot # parser understands this file. # Updated : 2026-07-20 # # DOMAIN SCOPE — one file covers BOTH properties # The apex schooloffreelancing.com 301-redirects to # https://www.schooloffreelancing.com/ (see .htaccess). Per RFC 9309 # crawlers follow that redirect and read THIS same file for both # hosts, so no second robots.txt is needed at the apex. Add BOTH # properties in Google Search Console + Bing Webmaster Tools; the # canonical host is www and is enforced by 301 + rel=canonical. # # HOW FAST CHANGES GO LIVE # * Page edits — sitemap.xml plus Cache-Control:no-cache # (set in .htaccess) make crawlers refetch on their # next visit, so content updates surface within hours. # * robots.txt — search engines cache this file for up to ~24h on # THEIR side (spec-allowed); that ceiling is not # controllable from the server. No Crawl-delay is set # anywhere below, so crawlers may fetch at full speed. # ================================================================= # ----------------------------------------------------------------- # 1) UNIVERSAL POLICY — applies to EVERY crawler # A single catch-all group is intentional. A bot obeys only its # most-specific matching group, so one universal group guarantees # these rules reach Googlebot, Bingbot, every AI/LLM crawler and # everything else alike. The public site is fully open; only # non-content panel/server paths are withheld from the index. # ----------------------------------------------------------------- User-agent: * Allow: / # CWP control panel (only port-80 path that is web-reachable) Disallow: /webmail Disallow: /webmail/ # Apache status endpoints / internal aliases (no indexable content) Disallow: /server-status Disallow: /server-info Disallow: /cgi-bin/ Disallow: /error/ Disallow: /icons/ # Non-public working directories (kept out of the search index) Disallow: /config/ Disallow: /backup/ Disallow: /backups/ Disallow: /private/ Disallow: /logs/ Disallow: /tmp/ Disallow: /cache/ Disallow: /node_modules/ Disallow: /vendor/ # NOTE — secrets & version-control metadata (.git, .env, .htaccess, # .sql, .log, backups) are blocked at the SERVER with a hard 403 in # .htaccess, which is real access control. They are deliberately NOT # named here: robots.txt is public and only a request, so listing a # secret path would advertise it without protecting it. # ----------------------------------------------------------------- # 2) EXPLICITLY WELCOME (documentation only — no rules here) # Every crawler below is ALREADY granted full access by the # universal Allow in section 1. They are recorded here purely to # document intent. They are intentionally NOT given their own # rule-groups, because a named group would override — and thus # weaken — the universal protections above. # # Search / index ...... Googlebot, Googlebot-Image, Googlebot-Video, # Googlebot-News, GoogleOther, Bingbot, msnbot, # Slurp (Yahoo), DuckDuckBot, Baiduspider, # YandexBot, SeznamBot, Qwantify, Ecosia, Yeti # Answer / voice / GEO Google-Extended, Google-Read-Aloud, Applebot, # Applebot-Extended, Amazonbot, # binglocalbusiness, Yandex-Local # Social preview ...... facebookexternalhit, facebot, Twitterbot, # LinkedInBot, Slackbot, TelegramBot, WhatsApp, # Pinterestbot # AI / LLM / LLMO ..... GPTBot, OAI-SearchBot, ChatGPT-User, # ClaudeBot, Claude-Web, anthropic-ai, # PerplexityBot, Perplexity-User, Gemini, # Google-Extended, meta-externalagent, # meta-externalfetcher, Copilot, cohere-ai, # MistralAI, YouBot, Bytespider, CCBot, Diffbot # ----------------------------------------------------------------- # ----------------------------------------------------------------- # 3) BLOCKED — commercial SEO / backlink scrapers # These honor robots.txt. Blocking them conserves server + Varnish # resources and keeps your backlink profile private. They are NOT # search engines, so this does not affect search visibility. # Consecutive User-agent lines share the single rule below (RFC 9309). # ----------------------------------------------------------------- User-agent: AhrefsBot User-agent: SemrushBot User-agent: SemrushBot-SA User-agent: MJ12bot User-agent: DotBot User-agent: BLEXBot User-agent: PetalBot User-agent: DataForSeoBot User-agent: serpstatbot User-agent: SEOkicks User-agent: GrapeshotCrawler User-agent: MegaIndex User-agent: ZoominfoBot Disallow: / # ----------------------------------------------------------------- # 4) SITEMAP # One canonical XML sitemap lists all public URLs. Keeping its # dates current lets recrawls pick up changes quickly. # (Abusive/malicious scanners ignore robots.txt entirely — they # are stopped at the server by .htaccess + firewall, not here.) # ----------------------------------------------------------------- Sitemap: https://www.schooloffreelancing.com/sitemap.xml