Hi! I think I found a bug on the cloud version of growthbook or at least it behaves differently than local version. In the attached picture

Screenshot - cloud

and

Screenshot - local

you can see the data is exactly the same. In fact it’s taking from the same mixpanel production data store but the results are different. In cloud Chance to beat control is 50% and there is no Percent change graph. In local version it’s 36% with nice graph. In files

Screenshot - query cloud

and

Screenshot - query local

you can see that the query result data is identical. In files

Screenshot - metric cloud

and

Screenshot - metric local

you can see that the metric configuration is identical In files

Screenshot - experiment cloud

and

Screenshot - experiment local

you can see that the experiment configuration is identical When I export the transformed data that are going into the gbstats calculation it’s the same as in the query:

{
  "var_id_map": {
    "0": 0,
    "1": 1
  },
  "var_names": [
    "Small",
    "Big"
  ],
  "weights": [
    0.5,
    0.5
  ],
  "type": "count",
  "ignore_nulls": false,
  "inverse": false,
  "max_dimensions": 20,
  "rows": [
    {
      "users": 16,
      "count": 13,
      "mean": 5.769230769230769,
      "stddev": 7.148467787993196,
      "dimension": "",
      "variation": "0"
    },
    {
      "users": 10,
      "count": 8,
      "mean": 5,
      "stddev": 2.7386127875258306,
      "dimension": "",
      "variation": "1"
    }
  ]
}

And when I run it through jupiter notebook I get exactly the same results (chanceToWin: 0.36) as in local version (see `Screenshot - notebook`:

{
  "unknownVariations": [],
  "dimensions": [
    {
      "dimension": "",
      "srm": 0.23931654122149193,
      "variations": [
        {
          "cr": 4.6875,
          "value": 75,
          "users": 16,
          "stats": {
            "users": 16,
            "count": 13,
            "stddev": 6.803611336557586,
            "mean": 4.6875
          }
        },
        {
          "cr": 4,
          "value": 40,
          "users": 10,
          "expected": -0.1466666666666666,
          "chanceToWin": 0.3600440835131542,
          "uplift": {
            "dist": "lognormal",
            "mean": -0.15860503017663863,
            "stddev": 0.4426092654556529
          },
          "ci": [
            -0.6415991584084633,
            1.031741260830155
          ],
          "risk": [
            0.4933414277382564,
            1.1808414277382564
          ],
          "stats": {
            "users": 10,
            "count": 8,
            "stddev": 3.2058973436118907,
            "mean": 4
          }
        }
      ]
    }
  ]
}

I can’t debug the cloud version but my suspicion is that the calculation is for some reason different than on local version and I am trying to figure out why. Local commit:

16303dfba972b805bc3fcc5e6e6514cbf21a2209

Thanks for the detailed info. Our stats calculations for very small sample sizes is unreliable (as you can see in the Jupyter warning "probability of being negative..."). 11 days ago we pushed a change to skip the stats calculations when we're not able to provide accurate results - https://github.com/growthbook/growthbook/pull/411 So that's what is causing it to say 50% with no percent graph on the Cloud version. We need to improve this in the UI since it's not at all obvious what's going on What's weird is that your local commit version is after that PR was merged, so it should also be showing the same results as the Cloud version.

Thanks for the info. this could be caused by installing gbstats from pip rather than from local source. And you’re right. Information that there is not enough data is definitely better than showing 50% chance 👍