Hi, are there some utilities/good practices in order to exclude crawlers from the experiments? For example, the possibility of defining a "global" rule (in the administration panel or through SDK)?

You can block crawlers, depending on the SDK - but nothing built in at the moment

I know this is not really related to what is being asked but made me wonder if there is a defined way to provide a custom robots.txt for the GrowthBook instance to use? The default install appears to return 404 for that.

Hmmmmm! I just joined this Slack to ask how we might do this. I saw that we can force a test based on an attribute. I feel it would be super handy if there was a 'bot' attribute. By chance would this be possible with the

browser

attribute? It's difficult to know what that encompasses. Is it user-agent or something more general?

Using a custom

isBot

attribute is probably a good way to do this. That way you can build bot detection into code rather than in targeting rules / regexes on browser attributes. Determining whether something is a bot or not is not always straightforward. There are some libraries like this one. There are also botlists by IP address that Google publishes. And some bots are cloaked by design.

ohhhh I see. I didn't realize we could create custom attributes. thanks!

in our case, when we detect that it is a bot, we set to true the option disable in the GrowthBook instance. in this way, the default value is always returned and we don't need to remember to set a custom rule in the features