Hey, interested if you’re developing features for high level reporting of experiments (# run, failure rate etc etc), with tag filters would be good too. Some things in the dashboard management view, but for some reason we only see current month?

have you been running for more than 1 month?

Yes, running for almost year. The north stars are over longer period

We debated putting the win/failure rate - but we’re not sure that’s a metric that helps with a healthy experimentation culture

Interested re. the debate, I can hypothesise what was discussed. We likely want a discussion on whether our failure rate is high enough. i.e. reasoning over whether experiments are testing/proving or just measuring. (I assume the debate may have been the other way) Obvs this would go further down, to understand hypotheses better.

ya, we feel that a higher experimentation rate is more important than the win rate, and if you’re optimizing for win rate, you might become risk adverse with more controversial or potentially risky experiments

Yeh, thats what we want to foster (keeping risk and learning high). i.e. it has to be about learning, win rate too high means we likely do not have the balance of learning vs just proving our apriori. Hard to reason over without some data on it to then dive into the detail

oh interesting

Yeh my simplified version was win-rate, perhaps on metrics/areas (we tag teams, they own areas of product) Very limited additional thought, is how learning rate works in NN. i.e. about the step you make on the optimisation path for a metric. Likely over-complicated, but is kinda what we’re talking about. i.e. are you taking right steps to explore the space, to get to an optimum (which is harder for these metrics vs ML, but think the analogy holds)

That is cool- I think there maybe some ways to do this in the UI

hi @purple-cat-95990, about the specific "Experiments by month" issue where you're only seeing the current month, would you be able to do a screen share session? We can't seem to recreate the issue it on our end. Thanks!