Help Center | tawk.to | How to whitelist tawk.to's AI Assist crawler bot

How to whitelist tawk.to's AI Assist crawler bot

AI Assist can learn about your products and services by crawling and gathering information from your website. To do this, our crawler bot needs access to your pages.

Some security services, like Cloudflare, may block our bot from accessing your website. If you’re facing issues getting AI Assist to ingest information from your website, it may be due to such restrictions.

Note: Our crawler bot can only access content rendered on the server. Your pages may not be visible to our bot if your website relies heavily on client-side rendering.

Whitelist our user agent

To allow the AI Assist crawler bot to access your pages, whitelist this user agent: Tawktobot-AIAssistant/1.0

An example using Cloudflare

Here’s an example of whitelisting our user agent on Cloudflare:

1. Log into your Cloudflare account and select Security from the left menu.

2. Select WAF for the Web Application Firewall security settings.

3. Go to the Custom Rules tab, and click Create Rule.

4. Give the firewall rule a descriptive name such as “Whitelist AI Assist UserAgent“.

5. Set Field to User Agent.

6. Set Operator to equals.

7. Set Value to: Tawktobot-AIAssistant/1.0

8. Set Choose action to Skip.

9. Check all boxes under WAF components to skip.

10. Click More components to skip and select User Agent Blocking.

11. Double-check your details and click Deploy.

Now, AI Assist should be able to crawl your pages and learn about your products and services.

If you have feedback about this article, or if you need more help:

Click the green live chat icon
Schedule a call with us
Visit our community

Was this article helpful?

2 out of 5 liked this article

Still need help? Message Us