34

AI Assist

Last Update há um dia

AI Assist: 24/7 sales and customer service made easy Getting started with AI Assist Understanding billing for AI Assist Deploying multiple AI Agents across support channels How to train AI Assist to transfer chats to human agents

How to whitelist tawk.to's AI Assist crawler bot

tawk.to’s AI Assist uses a crawler bot to scan your website to understand your content and offer useful responses to your visitors. To make sure this bot works correctly, it needs to be allowed, or "whitelisted," in your site’s security settings or firewall.


This article explains how to whitelist tawk.to’s AI Assist crawler bot so it can access your website without being blocked.


Note: Our crawler bot can only access content rendered on the server. Your pages may not be visible to our bot if your website relies heavily on client-side rendering.

Why whitelist the AI Assist crawler bot?

The AI Assist crawler bot scans your site pages to understand your content better. This allows it to provide relevant answers quickly when visitors ask questions.


If your website or firewall blocks the crawler, the AI Assist won’t be able to gather content, and the suggestions may not work well.

Which user agent should you whitelist?

Make sure your firewall or security software allows requests from this user agent:

Tawktobot-AIAssistant/1.0

An example using Cloudflare

Here’s an example of whitelisting our user agent on Cloudflare:


1. Log into your Cloudflare account and select Security from the left menu.

2. Select WAF for the Web Application Firewall security settings.


3. Go to the Custom Rules tab, and click Create Rule.


4. Give the firewall rule a descriptive name such as “Whitelist AI Assist UserAgent“.


5. Set Field to User Agent.


6. Set Operator to equals.


7. Set Value to: Tawktobot-AIAssistant/1.0

8. Set Choose action to Skip.


9. Check all boxes under WAF components to skip.

10. Click More components to skip and select User Agent Blocking.


11. Double-check your details and click Deploy.

Now, AI Assist should be able to crawl your pages and learn about your products and services.

How to confirm the crawler bot is working

After whitelisting, the AI Assist will begin crawling your site within 24 to 48 hours. You can check your firewall or server logs to see if the crawler’s requests are coming through.

If you have feedback about this article, or if you need more help:

Was this article helpful?

4 out of 8 liked this article

Still need help? Message Us