Skip to main content
Crawl your website to automatically train your AI agent on your web content.

​
How Website Crawling Works

  1. Enter a URL to start crawling
  2. The crawler visits the page and follows internal links
  3. Text content is extracted from each page
  4. Extracted content becomes training data

​
Adding a Website

  1. Go to Knowledge Base > Websites
  2. Enter the starting URL (e.g., https://example.com)
  3. Click Crawl
  4. Wait for crawling to complete
  5. Review discovered pages
  6. Click Retrain to update your agent

​
Viewing Crawled Pages

After crawling completes, you can:
  • See all discovered URLs
  • View extracted content per page
  • Check character count
  • Remove individual pages

​
Updating Website Content

When your website content changes:
  1. Go to Knowledge Base > Websites
  2. Click Re-crawl on the website
  3. New content will be extracted
  4. Click Retrain to update the agent
Re-crawling replaces previous content. Deleted pages will be removed from training data.

​
What Gets Extracted

​
Best Practices

  • Start with important pages - Begin with product pages, FAQs, about us
  • Avoid duplication - Don’t crawl the same content from multiple URLs
  • Keep content current - Re-crawl after major website updates
  • Review extracted content - Verify the right information was captured

​
Troubleshooting

  • Ensure pages are linked from the starting URL
  • Check if pages require authentication
  • Verify robots.txt isn’t blocking the crawler
  • Some pages may have complex layouts
  • Dynamic content (JavaScript-loaded) may not be captured
  • Consider adding content as a document instead
  • Large sites take longer to crawl
  • Consider starting with a specific section (e.g., /docs/)
  • Check your page limit settings