...
Privacy Settings: Choose between “Private” or “Public” toggle. A “Public” knowledge base is accessible to all users, while a “Private” knowledge base is only accessible to the owner.
Bypass Settings: Decide whether to “Comply” or “Bypass” websites that normally block crawlers.
Comply: The crawler will respect website settings that block crawling and won’t be able to crawl blocked pages.
Bypass: The crawler attempts to access and crawl even those websites that have anti-crawling measures. (*bypass of the blocked pages consumes 50 more tokens per page)
...
Setting Limits for Crawling
...