Content not found
A page you expect is not in the Library.1
Check the sitemap
The crawler discovers pages from your
sitemap.xml. Open https://yourdomain.com/sitemap.xml in a browser. If the page is not listed there, or the sitemap does not load, that is the cause. Fix the sitemap in your CMS, then click Update on the Library toolbar.2
Check the page is reachable
Pages behind a login, or blocked by
robots.txt, cannot be read. If the page returns an error, the Page Status column shows 404 or 503.3
Add the page by hand
If it is a one-off, such as a gated asset or a landing page not in the sitemap, use Add content on the toolbar. Choose the Domain, enter the Path or the full Content URL and a Title. The page is crawled and tagged like any other.
4
Flag it for manual lookup
If there is no sitemap and the site is large, contact Demand-Genius. We can look at the site structure and agree how to build the library. We can usually find a way, even if it’s doing a manual scrape and uploading via the backend.
Sitemap unreadable during sign-up
The sign-up flow reads your sitemap before creating the workspace. If it cannot, you see Sitemap unreadable with three options: Talk to us books a call, Try another URL lets you correct the domain, and an email link pre-fills a message to hello@demand-genius.com. Most AI crawlers rely on the sitemap too, so fixing it helps beyond Demand-Genius.Shopify and other sites that block crawlers
Some platforms return errors to unknown crawlers or throttle them at the edge. Shopify storefronts are the common case: the crawl runs for a while, then pages start coming back as 503. The fix is to attach custom HTTP headers that the crawler sends only when fetching your site.1
Agree the headers with your platform
Shopify and most CDNs let you allow a crawler by a specific User-Agent string or a custom header and value. Get the exact header name and value from whoever manages your storefront or CDN.
2
Open Settings > Crawler & Sitemap
3
Add the headers
Under Live-site crawler headers, enter each Header name and Value. A custom
User-Agent replaces the default one, which is Mozilla/5.0 (compatible; SitemapBot/1.0). Up to 20 headers can be set; names may contain letters, digits and hyphens only; values are capped at 1024 characters. Hop-by-hop headers and Cookie are not allowed.If your site is a Shopify site, which is where we most often see this, instructions to configure your headers in Shopify are here. 4
Save changes
Headers are replaced wholesale on save, so an empty editor clears them. Then click Update on the Library to re-crawl.
Classification is stuck or wrong
- A spinner never resolves. Check Settings > Jobs. A Sitemap Scan or Directory Classification job should be listed as running. If nothing is running, use Re-run classification from the row action bar; if a job is running but stale, Stop classification and re-run.
- A tag is wrong. Edit it inline. Manual edits are kept.
- A page shows Error status or is not Scrapable. The page returned an error or blocked the crawler when fetched. Check the Explanation column, fix the page or its access, then re-run that row.
- Bulk re-classification failed on a custom column. An older issue where unexpected tag values failed the whole page has been fixed; a bad value now costs only that column. Re-run.