> ## Documentation Index
> Fetch the complete documentation index at: https://user-guides.demand-genius.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Adding and managing directories

> Add a competitor site as its own content library, choose its update day, attach crawler headers, and manage the update schedule for your main site.

Your main site is the default library. A directory is an additional library, typically a competitor's site, crawled and tagged the same way so you can compare content side by side and run Smart Columns across both. Each directory has its own update schedule and its own crawler headers.

## Add a directory

<Steps>
  <Step title="Open the Library and click Add directory">
    The scope picker at the top of the Library lists **Main Site** and any directories. The plus button next to it opens **Add Content Library**.

    <Frame>
      <img src="https://mintcdn.com/demand-genius/bH-VFrWRSoO5Gb3h/images/Image-11-09-2026-at-10.08-1.png?fit=max&auto=format&n=bH-VFrWRSoO5Gb3h&q=85&s=c28b021caa9a5fcde6f12c38b756647a" alt="Image 11 09 2026 At 10 08 1" width="2304" height="1347" data-path="images/Image-11-09-2026-at-10.08-1.png" />
    </Frame>
  </Step>

  <Step title="Enter the URL and a Display Name">
    The domain of the site to crawl, and a friendly name such as "Main Competitor".

    <Frame>
      <img src="https://mintcdn.com/demand-genius/bH-VFrWRSoO5Gb3h/images/Image-11-09-2026-at-10.08.png?fit=max&auto=format&n=bH-VFrWRSoO5Gb3h&q=85&s=960f2ecca889c80e98769e2c047f0319" alt="Image 11 09 2026 At 10 08" width="2311" height="1355" data-path="images/Image-11-09-2026-at-10.08.png" />
    </Frame>
  </Step>

  <Step title="Choose the Context Type">
    How this directory relates to your main site. **Competitor** is the option offered today.
  </Step>

  <Step title="Set Auto-Update">
    Turn auto-update on to re-crawl the directory on a schedule, then pick the **Auto-Update Day**. Turn it off to crawl only when you click Update.
  </Step>

  <Step title="Add crawler headers if the site needs them (optional)">
    Some sites (e.g. Shopify) block or throttle crawlers. Add HTTP headers here that are sent only when crawling this directory. This is only possible for owned sub-brands, not for competitor sites. 

    See [Troubleshooting](/content-intelligence/troubleshooting) for when this is needed.
  </Step>

  <Step title="Save">
    The first crawl starts. The directory appears as a tab in the scope picker; pages fill in as they are processed.
  </Step>
</Steps>

## Switch between libraries

Use the scope picker to view **Main Site**, a single directory, or several at once. Smart Columns have an **Apply To** control that uses the same list.

## Content update schedule

Click **Update** on the toolbar to open the **Content Update Schedule** popover for the current library.

* **Auto-Update** on a specific day of the week.
* **Last Update**, **Status** and **URLs Found** for the most recent crawl. Status is success, partial failure, error or no new content.
* **Next Update** when the next automatic crawl is due.
* **Update now** runs a crawl immediately. Manual updates have a cooldown, seven days by default, so the button is unavailable until the cooldown passes.

The same popover exists for the main site.

## Edit or delete a directory

From the directory's tab, choose **Edit** to change the display name, context type, schedule or headers. The URL cannot be changed; add a new directory instead. Choose **Delete** and confirm to remove the directory and its pages from the Library.

## Limits

The number of directories a workspace can hold is set by Demand-Genius per customer. If the Add button is unavailable, ask.

## Troubleshooting

* **The directory shows no pages after the first crawl.** Check the site has a readable sitemap and is not blocking crawlers. See [Troubleshooting](/content-intelligence/troubleshooting).
* **Status shows partial failure.** Some pages returned errors. Open the directory and sort by Page Status to find them.
* **Update is disabled.** The manual cooldown has not passed since the last run, or a crawl is already in progress.