> ## Documentation Index
> Fetch the complete documentation index at: https://user-guides.demand-genius.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Content Intelligence troubleshooting

> Pages missing from the Library, an unreadable sitemap, crawlers blocked by Shopify or a CDN, stuck classification and empty KPI cells.

## Content not found

A page you expect is not in the Library.

<Steps>
  <Step title="Check the sitemap">
    The crawler discovers pages from your `sitemap.xml`. Open `https://yourdomain.com/sitemap.xml` in a browser. If the page is not listed there, or the sitemap does not load, that is the cause. Fix the sitemap in your CMS, then click **Update** on the Library toolbar.
  </Step>

  <Step title="Check the page is reachable">
    Pages behind a login, or blocked by `robots.txt`, cannot be read. If the page returns an error, the Page Status column shows 404 or 503.
  </Step>

  <Step title="Add the page by hand">
    If it is a one-off, such as a gated asset or a landing page not in the sitemap, use **Add content** on the toolbar. Choose the **Domain**, enter the **Path** or the full **Content URL** and a **Title**. The page is crawled and tagged like any other.

    <Frame>
      <img src="https://mintcdn.com/demand-genius/bH-VFrWRSoO5Gb3h/images/Image-11-09-2026-at-10.22.png?fit=max&auto=format&n=bH-VFrWRSoO5Gb3h&q=85&s=f59bd8b0e15fa25b51cfe24294cd7674" alt="Image 11 09 2026 At 10 22" width="2296" height="1023" data-path="images/Image-11-09-2026-at-10.22.png" />
    </Frame>
  </Step>

  <Step title="Flag it for manual lookup">
    If there is no sitemap and the site is large, contact Demand-Genius. We can look at the site structure and agree how to build the library. We can usually find a way, even if it's doing a manual scrape and uploading via the backend. 
  </Step>
</Steps>

## Sitemap unreadable during sign-up

The sign-up flow reads your sitemap before creating the workspace. If it cannot, you see **Sitemap unreadable** with three options: **Talk to us** books a call, **Try another URL** lets you correct the domain, and an email link pre-fills a message to [hello@demand-genius.com](mailto:hello@demand-genius.com). Most AI crawlers rely on the sitemap too, so fixing it helps beyond Demand-Genius.

## Shopify and other sites that block crawlers

Some platforms return errors to unknown crawlers or throttle them at the edge. Shopify storefronts are the common case: the crawl runs for a while, then pages start coming back as 503. The fix is to attach custom HTTP headers that the crawler sends only when fetching your site.

<Steps>
  <Step title="Agree the headers with your platform">
    Shopify and most CDNs let you allow a crawler by a specific User-Agent string or a custom header and value. Get the exact header name and value from whoever manages your storefront or CDN.
  </Step>

  <Step title="Open Settings > Crawler & Sitemap">
    <Frame>
      <img src="https://mintcdn.com/demand-genius/bH-VFrWRSoO5Gb3h/images/Image-11-09-2026-at-10.23.png?fit=max&auto=format&n=bH-VFrWRSoO5Gb3h&q=85&s=711b70e728a93426727b09674c52f190" alt="Image 11 09 2026 At 10 23" width="2305" height="1353" data-path="images/Image-11-09-2026-at-10.23.png" />
    </Frame>
  </Step>

  <Step title="Add the headers">
    Under **Live-site crawler headers**, enter each **Header name** and **Value**. A custom `User-Agent` replaces the default one, which is `Mozilla/5.0 (compatible; SitemapBot/1.0)`. Up to 20 headers can be set; names may contain letters, digits and hyphens only; values are capped at 1024 characters. Hop-by-hop headers and `Cookie` are not allowed.

    If your site is a Shopify site, which is where we most often see this, [instructions to configure your headers in Shopify are here.](https://help.shopify.com/en/manual/promoting-marketing/seo/crawling-your-store) 
  </Step>

  <Step title="Save changes">
    Headers are replaced wholesale on save, so an empty editor clears them. Then click **Update** on the Library to re-crawl.
  </Step>
</Steps>

For a competitor directory, the same headers editor sits in the **Add Content Library** and **Edit Directory** dialogs and applies to that directory only.

## Classification is stuck or wrong

* **A spinner never resolves.** Check **Settings > Jobs**. A Sitemap Scan or Directory Classification job should be listed as running. If nothing is running, use **Re-run classification** from the row action bar; if a job is running but stale, **Stop classification** and re-run.
* **A tag is wrong.** Edit it inline. Manual edits are kept.
* **A page shows Error status or is not Scrapable.** The page returned an error or blocked the crawler when fetched. Check the Explanation column, fix the page or its access, then re-run that row.
* **Bulk re-classification failed on a custom column.** An older issue where unexpected tag values failed the whole page has been fixed; a bad value now costs only that column. Re-run.

## KPI cells are empty

| What you see                        | What it means                                                       | What to do                                             |
| ----------------------------------- | ------------------------------------------------------------------- | ------------------------------------------------------ |
| **Off** chip                        | KPI scoring is not enabled for the workspace                        | Ask Demand-Genius to enable it                         |
| Dash                                | Page not yet scored                                                 | Wait for the nightly sweep, or use the per-page Re-run |
| Blank signal inside an expanded KPI | The model returned an invalid label twice                           | Stored as unscored by design; re-run later             |
| **Too many requests** on Re-run     | Per-page cooldown (five minutes) or hourly limit (thirty pages) hit | Wait the time shown                                    |

## Where to look first

**Settings > Jobs** lists recent background jobs by type: Sitemap Scan, Directory Classification, Category Classification (Smart Columns), Content Intelligence Scoring, and the AI Search Intelligence runs. A job with **Budget limit reached** hit the workspace's daily AI budget and will resume on the next pass.