feat: combined SEO/GEO/AEO + IndexNow auto-submission + build fix (#827)

* aeo update

Signed-off-by: Neha Gupta <gneha21[at]yahoo>

* fix: remove unused FAQSchema component

FAQSchema.jsx was never imported anywhere — dead code with no matching
visible FAQ content, risking structured-data spam if accidentally used.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: slayerjain <shubhamkjain@outlook.com>

* feat: GEO improvements — answer nuggets, schema upgrade, cross-linking, sitemap

- Add 40-60 word answer nuggets after H2s in what-is-keploy, how-keploy-works, features docs
- Rephrase doc headings as questions for AI extraction
- Upgrade Product schema to SoftwareApplication with featureList and keywords
- Add sameAs entity disambiguation links to Organization schema
- Add blog + landing cross-refs to llms.txt
- Fix canonicalBase to match actual domain (keploy.io)
- Add missing plugin package.json for docusaurus-tailwindcss-loader
- Configure sitemap filename

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: slayerjain <shubhamkjain@outlook.com>

* fix: resolve all 8 copilot review comments on PR #818

static/robots.txt:
- Remove per-bot Allow groups that bypassed default Disallow rules.
  All crawlers now use single User-agent:* block. Only Bytespider blocked.

how-keploy-works.md:
- Fix description: "replay in production-like sandboxed environments"
  (not "replay as sandboxes")
- Fix body: "capture socket-level application traffic" instead of
  "intercept network packets" (packets are L3, sockets are L4+)

what-is-a-keploy-ebpf.md:
- Fix description: "capture socket-level network I/O via kernel hooks"
  instead of "intercept network packets at the socket level"

what-is-keploy.md:
- Fix: "kernel socket layer (syscall-level network I/O)" instead of
  "kernel network layer" to avoid L3/OSI confusion

docusaurus.config.js:
- Fix softwareHelp URL: https://keploy.io/docs/ (matches actual
  url + baseUrl config) instead of https://docs.keploy.io

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: slayerjain <shubhamkjain@outlook.com>

* fix: resolve 2 copilot review comments (round 2)

static/robots.txt:
- Move Bytespider block before User-agent:* to ensure first-match-wins
  crawlers don't skip the specific block

docusaurus.config.js:
- Change operatingSystem from ["Linux","macOS","Windows"] to "Linux"
  since macOS/Windows require Lima/WSL/Docker (not native support)
- Add clarification in description about macOS/Windows requirements

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: slayerjain <shubhamkjain@outlook.com>

* feat: Phase 2 GEO/AEO — enrich docs schemas and llms.txt

docusaurus.config.js:
- Add AggregateRating to SoftwareApplication (Gartner 4.6/5, 11 ratings)
- Add foundingDate, knowsAbout, award to Organization schema
- Add Gartner, G2, Capterra review URLs to sameAs

static/llms.txt:
- Rewrite from 44 to 72 lines with comprehensive coverage
- Add awards, quickstart guides by language, cloud integrations,
  contributing section, and expanded core concepts

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: slayerjain <shubhamkjain@outlook.com>

* fix: resolve 8 copilot review comments on PR #819

static/llms.txt:
- Fix 7 broken URLs to match actual Docusaurus doc routes
  (use frontmatter id-based slugs, not file paths)

docusaurus.config.js:
- Change AggregateRating values from strings to numbers
  for Google Rich Results validation compliance

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: slayerjain <shubhamkjain@outlook.com>

* fix: resolve 7 copilot comments on PR #819 (round 2)

static/llms.txt:
- Fix eBPF doc URL to match page id (what-is-keploy-ebpf)
- Fix Supported Languages description to match actual page content
- Add doc links to Use Cases section for discoverability
- Remove hardcoded metrics (stars/installs) that go stale
- Replace Slack invite URL with stable contribution guide link

docusaurus.config.js:
- Change foundingDate to ISO-8601 format (2021-01-01)
- Remove aggregateRating — no visible on-page rating to match

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: slayerjain <shubhamkjain@outlook.com>

* fix: resolve 3 copilot comments on PR #819 (round 3)

static/llms.txt:
- Fix quickstart descriptions to match actual sample app content
- Fix Rust description (GraphQL+MongoDB, not Actix Web)
- Rename "Slack Community" to "Community & Contributing" since
  link points to contribution guide, not Slack directly

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: slayerjain <shubhamkjain@outlook.com>

* feat: Phase 3 SEO — glossary titles, freshness signals, noindex old versions

Task 68 — Rewrite 10 glossary page titles for CTR optimization:
- "Idempotency" → "What is Idempotency in REST APIs? Complete Guide"
- "Black Box Testing" → "What is Black Box Testing? Types & Examples"
- Plus 8 more glossary pages with question-format titles
- Added 140-155 char meta descriptions to all 10 pages

Task 22 — Enable showLastUpdateTime and showLastUpdateAuthor:
- Freshness signals for AI citation and search ranking

Task 4 — Noindex deprecated doc versions (v1.0.0, v2.0.0):
- Added Disallow rules in robots.txt
- Added X-Robots-Tag noindex headers in netlify.toml
- Prevents duplicate content from old versions consuming crawl budget

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: slayerjain <shubhamkjain@outlook.com>

* fix: GSC audit — descriptions, redirects, titles, breadcrumbs, headers

- Added descriptions to 51 docs pages missing them in v4.0.0
- Standardized keyword: -> keywords: across 37 files
- Fixed 16 duplicate page titles to be unique
- Removed breadcrumb microdata, keeping only JSON-LD
- Added OG image and standardized Organization logo
- Added security headers to netlify.toml
- Added soft 404 redirects (vercel.json + _redirects + client-redirect)
- Added Docusaurus client redirect for /keploy-explained/faq

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Neha Gupta <gneha21@yahoo.in>

* fix: align descriptions and H1s with frontmatter titles per Copilot review

Update generic "What is API testing" descriptions in test-run-reports.md
and test-generate.md to match their actual page titles. Sync in-page H1
headings with frontmatter titles in installation.md, unit-testing-faq.md,
and integration-testing-faq.md.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Neha Gupta <gneha21@yahoo.in>

* fix: align H1 headings with frontmatter titles per Copilot review

For 5 pages, changed the in-page H1 to match the frontmatter title
exactly (unit-test-architecture, api-test-architecture,
integrations-vscode, integrations-pr-agent, test-generate).
For 2 pages with embedded InstallationGuide components
(server/installation, quickstart/samples-node-mongo), added
hide_title: true to prevent duplicate H1 rendering.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Neha Gupta <gneha21@yahoo.in>

* fix: add 'linux' and 'env' to Vale vocabulary

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Neha Gupta <gneha21@yahoo.in>

* ci: add IndexNow submission for changed docs after deploy

Submits only the docs URLs whose markdown files changed in the push
to Bing IndexNow after S3 deploy. Skips if no docs content changed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Neha Gupta <gneha21@yahoo.in>

* fix: remove webpackbar override causing ProgressPlugin build failure

webpackbar 6.0.1 passes invalid options (name, color, reporters) to
webpack 5.97.1's ProgressPlugin. Removing it from dependencies and
overrides lets Docusaurus use its own compatible version.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Neha Gupta <gneha21@yahoo.in>

* fix: address all 9 Copilot review comments on IndexNow step

- Upgrade actions/checkout@v2 → v4 (comment 1)
- Use built sitemap instead of git diff file mapping (comments 2-6)
  Avoids unreliable path→URL mapping due to frontmatter id/slug,
  versioned doc prefixes, and index page edge cases
- Move IndexNow key to secret with hardcoded fallback (comments 7-8)
- Replace ::warning:: with descriptive log + response body (comment 9)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Neha Gupta <gneha21@yahoo.in>

* fix: allow crawling of docs v1.0.0 and v2.0.0

Remove Disallow rules from robots.txt and noindex headers from
netlify.toml for old doc versions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Neha Gupta <gneha21@yahoo.in>

* fix: make IndexNow step truly non-blocking

Remove set -e and add continue-on-error so grep/jq/curl failures
don't block the deploy pipeline.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Neha Gupta <gneha21@yahoo.in>

* fix: correct quickstart URLs in llms.txt, remove URL cap in IndexNow

- Fix Language Guides URLs to use actual frontmatter ids (golang-filter,
  java-springboot-filter, javascript-filter, python-filter)
- Remove head -100 limit — submit all sitemap URLs to IndexNow

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Neha Gupta <gneha21@yahoo.in>

* fix: use case-insensitive patterns for linux/env in Vale vocabulary

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Neha Gupta <gneha21@yahoo.in>

---------

Signed-off-by: Neha Gupta <gneha21[at]yahoo>
Signed-off-by: slayerjain <shubhamkjain@outlook.com>
Signed-off-by: Neha Gupta <gneha21@yahoo.in>
Co-authored-by: Neha Gupta <gneha21[at]yahoo>
Co-authored-by: slayerjain <shubhamkjain@outlook.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: 3 people

.github/workflows/main.yml

@@ -21,7 +21,7 @@ jobs:
     # Steps represent a sequence of tasks that will be executed as part of the job
     steps:
       # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v4
 
       - name: Set Node.js 22.x
         uses: actions/setup-node@v3
@@ -52,4 +52,47 @@ jobs:
           invalidation: /docs*
           delete-removed: true
           cache: "public, max-age:86400"
-          private: true    
+          private: true
+
+      - name: Submit docs URLs to IndexNow
+        env:
+          INDEXNOW_KEY: ${{ secrets.INDEXNOW_KEY }}
+        continue-on-error: true
+        run: |
+          # Use the built sitemap as the source of truth for URLs.
+          SITEMAP="build/docs/sitemap.xml"
+          if [ ! -f "$SITEMAP" ]; then
+            echo "Sitemap not found at $SITEMAP, skipping IndexNow"
+            exit 0
+          fi
+
+          URLS=$(grep -oP '<loc>\K[^<]+' "$SITEMAP" || true)
+
+          if [ -z "$URLS" ]; then
+            echo "No URLs found in sitemap, skipping"
+            exit 0
+          fi
+
+          URL_COUNT=$(echo "$URLS" | wc -l)
+          echo "Submitting $URL_COUNT docs URLs to IndexNow"
+
+          KEY="${INDEXNOW_KEY:-9ded60b43b8b40578588fc42b04c3e5d}"
+          JSON_URLS=$(echo "$URLS" | jq -R -s 'split("\n") | map(select(length > 0))' || true)
+
+          if [ -z "$JSON_URLS" ]; then
+            echo "Failed to build JSON URL list, skipping"
+            exit 0
+          fi
+
+          RESPONSE_FILE=$(mktemp)
+          HTTP_CODE=$(curl -s -o "$RESPONSE_FILE" -w "%{http_code}" -X POST "https://api.indexnow.org/indexnow" \
+            -H "Content-Type: application/json" \
+            -d "{\"host\":\"keploy.io\",\"key\":\"$KEY\",\"urlList\":$JSON_URLS}" || echo "000")
+
+          echo "IndexNow response: $HTTP_CODE"
+          if [ "$HTTP_CODE" -ge 400 ] || [ "$HTTP_CODE" = "000" ]; then
+            echo "IndexNow submission failed (HTTP $HTTP_CODE) — will not block deployment."
+            echo "Verify the key file is hosted at https://keploy.io/${KEY}.txt"
+            cat "$RESPONSE_FILE" 2>/dev/null || true
+          fi
+          rm -f "$RESPONSE_FILE"

netlify.toml

@@ -9,16 +9,6 @@
 ## Note: if you are looking for Redirects
 # they have been moved to /static/_redirects to make it more manageable - swyx
 
-[[headers]]
-  for = "/docs/1.0.0/*"
-  [headers.values]
-    X-Robots-Tag = "noindex, nofollow"
-
-[[headers]]
-  for = "/docs/2.0.0/*"
-  [headers.values]
-    X-Robots-Tag = "noindex, nofollow"
-
 # Security headers for all pages
 [[headers]]
   for = "/*"

package.json

@@ -25,7 +25,6 @@
     "@docusaurus/core": "3.9.2",
     "@docusaurus/plugin-sitemap": "3.9.2",
     "@docusaurus/preset-classic": "3.9.2",
-    "webpackbar": "6.0.1",
     "@mdx-js/react": "^3.0.0",
     "ajv": "^6.12.6",
     "ajv-keywords": "^3.5.2",
@@ -76,7 +75,6 @@
     "node": ">=18.0"
   },
   "overrides": {
-    "webpackbar": "6.0.1",
     "webpack": "5.97.1"
   }
 }

static/llms.txt

@@ -4,6 +4,7 @@
 
 For comprehensive details, see: /docs/llms-full.txt
 
+
 ## About Keploy
 Keploy (founded 2021) is a CNCF Landscape project and open-source testing platform with a growing open-source community and widely used VS Code extension. It uses eBPF kernel hooks to capture real API traffic and replay it as auto-generated tests with dependency mocks and stubs — no code instrumentation required. Keploy handles non-deterministic data (timestamps, IDs) via time-freezing and AI noise detection, eliminating flaky tests.
 
@@ -35,6 +36,13 @@ Keploy (founded 2021) is a CNCF Landscape project and open-source testing platfo
 - [C# / .NET](https://keploy.io/docs/quickstart/samples-csharp/): .NET + PostgreSQL sample
 - [Code Coverage](https://keploy.io/docs/quickstart/code-coverage/): Measure coverage of Keploy-generated tests
 
+
+## Language Guides
+- [Go Quick Start](https://keploy.io/docs/quickstart/golang-filter/): Gin, Echo, Fiber, Chi
+- [Java Quick Start](https://keploy.io/docs/quickstart/java-springboot-filter/): Spring Boot, Quarkus, Micronaut
+- [Node.js Quick Start](https://keploy.io/docs/quickstart/javascript-filter/): Express, Fastify, NestJS
+- [Python Quick Start](https://keploy.io/docs/quickstart/python-filter/): Django, Flask, FastAPI
+
 ## CI/CD Integration
 - [GitHub Actions](https://keploy.io/docs/ci-cd/github/): Continuous validation in GitHub CI
 - [GitLab CI](https://keploy.io/docs/ci-cd/gitlab/): Continuous validation in GitLab CI
@@ -54,6 +62,7 @@ Keploy (founded 2021) is a CNCF Landscape project and open-source testing platfo
 - [VS Code Extension](https://keploy.io/docs/keploy-explained/integrations-vscode/): IDE integration for test generation
 - [PR Agent Integration](https://keploy.io/docs/keploy-explained/integrations-pr-agent/): Automated test suggestions on pull requests
 
+
 ## FAQ
 - [Unit Testing FAQ](https://keploy.io/docs/keploy-explained/unit-testing-faq/): Auto-generated unit tests, mock generation, coverage
 - [API Testing FAQ](https://keploy.io/docs/keploy-explained/api-testing-faq/): Traffic-based API test generation, regression detection

vale_styles/config/vocabularies/Base/accept.txt

@@ -103,3 +103,5 @@ total_amount
 created_at
 updated_at
 shipping_address_id
+[Ll]inux
+[Ee]nv