Descripció
When a known AI crawler (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and others) requests one of your posts or pages, this plugin returns clean Markdown of the main content on the same URL instead of the full theme HTML — no header, footer, sidebar or scripts. Regular visitors keep seeing your normal site.
It also generates llms.txt and llms-full.txt — a sitemap-style index of your content for AI tools — served at /llms.txt and /llms-full.txt. These are kept in sync as real files at your site root as a fallback for hosts that block the virtual route before WordPress runs, so the files stay available even there.
What it does
- Detects AI crawlers by User-Agent (the list is editable in the settings).
- Serves clean Markdown on the same URL with
Content-Type: text/markdown,Vary: User-AgentandX-Robots-Tag: noindex. - Works with any public post type — including WooCommerce products — plus a Markdown index for category, tag and custom-taxonomy archives.
- Configurable YAML front matter (title, url, date, author, tags, canonical) with per-field toggles.
- Include your own custom fields / ACF values in the front matter.
- Adds machine-readable Schema.org JSON-LD (title, dates, author, cover image, publisher).
- Generates
/llms.txtand/llms-full.txt, with an on-disk fallback file for hosts that block the virtual route. - Caches the converted Markdown in post meta for fast responses; the cache is cleared automatically when a post is edited.
- Plays nice with page caches: sets
DONOTCACHEPAGE(WP Rocket, W3 Total Cache, WP Super Cache, LiteSpeed) so bots are never served a stale HTML cache. - Exclude content with a per-post «Hide from AI bots» checkbox or by category/tag slug.
- «View Markdown» link in the editor and post list — preview exactly what crawlers receive.
Privacy and external services
The Markdown/llms.txt generation itself does not connect to any external service, send any data off your site, or load any remote scripts — all content processing happens on your own server.
This plugin uses the Freemius SDK for licensing. If you choose to enable premium features with a license key, the plugin connects to the Freemius API (freemius.com) to validate your license and enable the paid functionality. This connection is opt-in: it is only made if you enter a license key, and no license or plan data is required to use the free feature set described above. See the Freemius privacy policy for details on what limited data is shared with Freemius during opt-in and licensing.
Good to know
- No required .htaccess edits and no Composer — a built-in, zero-dependency HTMLMarkdown converter is included. An optional, one-click .htaccess fix is available if your host blocks
/llms.txt(see Settings AI Bots & Markdown). - Does not interfere with SEO plugins, caches, or WooCommerce shop/cart/checkout pages.
- Private, password-protected and unpublished content is never served to bots — access rights are not bypassed.
Pro version
A Pro version is available with bot IP verification (anti-spoofing), bot-visit analytics with CSV export, a one-click self-test, and priority support. See the plugin’s settings page for details. The free version is fully functional on its own and is not time-limited.
Captures
Instal·lació
- Upload the
adg-markdown-for-ai-crawlersfolder to/wp-content/plugins/, or install the plugin through the WordPress «Plugins» screen. - Activate the plugin.
- Go to Settings AI Bots & Markdown to configure it.
- On Nginx, if
/llms.txtreturns a 404, make sure your config falls through toindex.php(the standardtry_files $uri $uri/ /index.php?$args;block covers this). On Apache it works out of the box.
PMF
-
Will private or draft pages be exposed to bots?
-
No. Markdown is served only for published, non-password-protected, non-private posts of the allowed types. Access rights are never bypassed.
-
Does it send my content to any external server?
-
No. Your content is never sent anywhere — detection is by User-Agent, and all conversion happens locally. The plugin does contact the Freemius API (freemius.com) for license validation, but only if you activate a Pro license key; this never involves your site’s content.
-
Does it work with page builders like Elementor?
-
Yes. Content is taken through
the_content, so shortcodes and builder output are rendered before conversion. Testing on your specific setup is recommended. -
A bot still receives HTML instead of Markdown. Why?
-
Most often the request is blocked at the server/CDN level (HTTP 403) or served from a page cache before the plugin runs. Make sure your cache honors
Vary: User-Agentor excludes bot user-agents. -
My llms.txt file returns a 404 even though the plugin is active and enabled.
-
On Newfold-family hosting (Bluehost, HostGator, iPage and similar cPanel providers), a server-level block in
.htaccess(# BEGIN NFD EPC) may prevent/llms.txtrequests from reaching WordPress. This is not a plugin bug — the server intercepts.txtfile requests before PHP. Since 2.2.0 the plugin also keeps a realllms.txtfile at your site root as a fallback, which resolves this on many hosts automatically; if it’s still blocked, Settings AI Bots & Markdown offers a «Try automatic fix» button (Apache) or the exact manual line to add: open.htaccess, find the# BEGIN NFD EPCblock, locate the line withRewriteCond %{REQUEST_URI} !(robots\.txt|...), and addllms\.txt|llms-full\.txtto the exception list:RewriteCond %{REQUEST_URI} !(robots\.txt|llms\.txt|llms-full\.txt|...). Save and re-test using Settings AI Bots & Markdown «Re-check now».
Ressenyes
No hi ha ressenyes per a aquesta extensió.
Col·laboradors i desenvolupadors
«ADG Markdown for AI Crawlers» és programari de codi obert. La següent gent ha col·laborat en aquesta extensió.
Col·laboradors
Traduïu «ADG Markdown for AI Crawlers» a la vostra llengua.
Interessats en el desenvolupament?
Navegueu pel codi, baixeu-vos el repositori SVN, o subscriviu-vos al registre de desenvolupament per fisl de subscripció RSS.
Registre de canvis
2.2.0
- New: /llms.txt and /llms-full.txt are now also written as real files at the site root, as a fallback for hosts that block the virtual route before WordPress runs — the site keeps serving them even where that block can’t be fixed from within WordPress.
- New: «Try automatic fix» button applies a self-contained .htaccess exception ahead of a blocking host rule (Apache only), with an automatic backup of the previous file.
- Improved: self-test now recognizes WP Engine, Kinsta, Pantheon and SiteGround in addition to Newfold, with tailored guidance per host.
- Improved: regenerating llms.txt now also purges WP Rocket, LiteSpeed Cache and W3 Total Cache if active, so a page cache can’t keep serving a stale copy.
- Fixed: the self-test no longer reports a false «blocked» when /llms.txt is being served correctly by the new on-disk fallback file.
2.1.1
- Improved: the /llms.txt self-test check now compares against /robots.txt to reliably distinguish a server-level block (Newfold/Bluehost-family hosting) from a plain rewrite-rule issue, with a «Copy fix» button for the exact .htaccess line.
2.1.0
- New: automatic detection of server-level blocking of /llms.txt (common on Bluehost/HostGator/iPage and other Newfold-family hosting via their «.htaccess NFD EPC» block) — an admin notice appears on Settings AI Bots & Markdown with a «Re-check now» button and step-by-step manual fix instructions, including clearing the host’s page cache.
2.0.0
- Single codebase with a Pro license (Freemius): one plugin, Pro features unlocked by a license key.
- Pro: training-vs-citation access policy, robots.txt management, bot IP verification (anti-spoofing), bot-visit analytics + CSV, AI referral tracking with a by-source chart, token-savings estimate, one-click self-test, AI Readiness Score, and a weekly email digest.
1.2.0
- New: Pro feature previews on the settings screen — the training-vs-citation access policy, robots.txt management and the AI analytics/citation dashboard are shown as locked teasers with a link to the Pro version.
1.1.0
- New: configurable YAML front matter (title, url, date, author, tags, canonical) with per-field toggles.
- New: include custom fields / ACF values in the Markdown front matter.
- New: Markdown index for category, tag and custom-taxonomy archives.
- New: WooCommerce products (and any public post type) can be served as Markdown and listed in llms.txt.
- New: exclude content by category/tag slug, in addition to the per-post checkbox.
- New: «View Markdown» preview link in the post editor and post list.
- Improved: page-cache compatibility — sets DONOTCACHEPAGE for WP Rocket, W3 Total Cache, WP Super Cache and LiteSpeed.
- Fixed: list item text (simple <li> entries) was dropped from the Markdown output.
1.0.0
- Initial release: content negotiation (Markdown for AI bots on the same URL), Schema.org/JSON-LD front matter, post-level Markdown cache, virtual llms.txt / llms-full.txt, and a «Hide from AI bots» metabox.