Skip to main content
Your main site is the default library. A directory is an additional library, typically a competitor’s site, crawled and tagged the same way so you can compare content side by side and run Smart Columns across both. Each directory has its own update schedule and its own crawler headers.

​
Add a directory

1

Open the Library and click Add directory

The scope picker at the top of the Library lists Main Site and any directories. The plus button next to it opens Add Content Library.
Image 11 09 2026 At 10 08 1
2

Enter the URL and a Display Name

The domain of the site to crawl, and a friendly name such as “Main Competitor”.
Image 11 09 2026 At 10 08
3

Choose the Context Type

How this directory relates to your main site. Competitor is the option offered today.
4

Set Auto-Update

Turn auto-update on to re-crawl the directory on a schedule, then pick the Auto-Update Day. Turn it off to crawl only when you click Update.
5

Add crawler headers if the site needs them (optional)

Some sites (e.g. Shopify) block or throttle crawlers. Add HTTP headers here that are sent only when crawling this directory. This is only possible for owned sub-brands, not for competitor sites. See Troubleshooting for when this is needed.
6

Save

The first crawl starts. The directory appears as a tab in the scope picker; pages fill in as they are processed.

​
Switch between libraries

Use the scope picker to view Main Site, a single directory, or several at once. Smart Columns have an Apply To control that uses the same list.

​
Content update schedule

Click Update on the toolbar to open the Content Update Schedule popover for the current library.
  • Auto-Update on a specific day of the week.
  • Last Update, Status and URLs Found for the most recent crawl. Status is success, partial failure, error or no new content.
  • Next Update when the next automatic crawl is due.
  • Update now runs a crawl immediately. Manual updates have a cooldown, seven days by default, so the button is unavailable until the cooldown passes.
The same popover exists for the main site.

​
Edit or delete a directory

From the directory’s tab, choose Edit to change the display name, context type, schedule or headers. The URL cannot be changed; add a new directory instead. Choose Delete and confirm to remove the directory and its pages from the Library.

​
Limits

The number of directories a workspace can hold is set by Demand-Genius per customer. If the Add button is unavailable, ask.

​
Troubleshooting

  • The directory shows no pages after the first crawl. Check the site has a readable sitemap and is not blocking crawlers. See Troubleshooting.
  • Status shows partial failure. Some pages returned errors. Open the directory and sort by Page Status to find them.
  • Update is disabled. The manual cooldown has not passed since the last run, or a crawl is already in progress.