Just stumbled upon growthbook. What is the easiest data source to get and running for testing some custom stuff, that handles near realtime data at medium scale. Preferably one that comes with all js tooling needed to reliably track events. 15m pageviews a month or so.

Mixpanel is probably the easiest

Tested them some years ago. Super expensive. I think we have a Matomo instance running somewhere

With sql based data sources, the complexity is similar

So basically what growth book does is to pull out all events and do the significance calculations and stuf on your end?

The data structure of matomo is quite strange

Basically just ran around looking for an ab testing api, without much gui other than reporting. Basically looking for two things without having to build everything. Testing a hierarchy of variations, so variation1 has some subtests for different elements. Which I guess should be ok to do by just prefixing tests. And testing sitewide and being able to combine results by some criteria. For example testing 3 types of headers on all products where one is "happy" one is "plain" etc, and then be able to segment those based on some product attribute to see if there are significance on specific attributes

You can use GrowthBook for just experiment reporting

But is it flexible enough ta calculate significance afterwards based on some type of segment?

ya, you can do that with one test if you want, and break the test down by a dimension

awesome!

Dimensions and segments are completely customizable