https://www.growthbook.io/ logo
#give-feedback
Title
# give-feedback
f

future-candle-10380

07/08/2022, 6:27 PM
Hi! I think I found a bug on the cloud version of growthbook or at least it behaves differently than local version. In the attached picture
Screenshot - cloud
and
Screenshot - local
you can see the data is exactly the same. In fact it’s taking from the same mixpanel production data store but the results are different. In cloud Chance to beat control is 50% and there is no Percent change graph. In local version it’s 36% with nice graph. In files
Screenshot - query cloud
and
Screenshot - query local
you can see that the query result data is identical. In files
Screenshot - metric cloud
and
Screenshot - metric local
you can see that the metric configuration is identical In files
Screenshot - experiment cloud
and
Screenshot - experiment local
you can see that the experiment configuration is identical When I export the transformed data that are going into the gbstats calculation it’s the same as in the query:
Copy code
{
  "var_id_map": {
    "0": 0,
    "1": 1
  },
  "var_names": [
    "Small",
    "Big"
  ],
  "weights": [
    0.5,
    0.5
  ],
  "type": "count",
  "ignore_nulls": false,
  "inverse": false,
  "max_dimensions": 20,
  "rows": [
    {
      "users": 16,
      "count": 13,
      "mean": 5.769230769230769,
      "stddev": 7.148467787993196,
      "dimension": "",
      "variation": "0"
    },
    {
      "users": 10,
      "count": 8,
      "mean": 5,
      "stddev": 2.7386127875258306,
      "dimension": "",
      "variation": "1"
    }
  ]
}
And when I run it through jupiter notebook I get exactly the same results (chanceToWin: 0.36) as in local version (see `Screenshot - notebook`:
Copy code
{
  "unknownVariations": [],
  "dimensions": [
    {
      "dimension": "",
      "srm": 0.23931654122149193,
      "variations": [
        {
          "cr": 4.6875,
          "value": 75,
          "users": 16,
          "stats": {
            "users": 16,
            "count": 13,
            "stddev": 6.803611336557586,
            "mean": 4.6875
          }
        },
        {
          "cr": 4,
          "value": 40,
          "users": 10,
          "expected": -0.1466666666666666,
          "chanceToWin": 0.3600440835131542,
          "uplift": {
            "dist": "lognormal",
            "mean": -0.15860503017663863,
            "stddev": 0.4426092654556529
          },
          "ci": [
            -0.6415991584084633,
            1.031741260830155
          ],
          "risk": [
            0.4933414277382564,
            1.1808414277382564
          ],
          "stats": {
            "users": 10,
            "count": 8,
            "stddev": 3.2058973436118907,
            "mean": 4
          }
        }
      ]
    }
  ]
}
I can’t debug the cloud version but my suspicion is that the calculation is for some reason different than on local version and I am trying to figure out why. Local commit:
16303dfba972b805bc3fcc5e6e6514cbf21a2209
f

future-teacher-7046

07/08/2022, 6:35 PM
Thanks for the detailed info. Our stats calculations for very small sample sizes is unreliable (as you can see in the Jupyter warning "probability of being negative..."). 11 days ago we pushed a change to skip the stats calculations when we're not able to provide accurate results - https://github.com/growthbook/growthbook/pull/411 So that's what is causing it to say 50% with no percent graph on the Cloud version. We need to improve this in the UI since it's not at all obvious what's going on What's weird is that your local commit version is after that PR was merged, so it should also be showing the same results as the Cloud version.
f

future-candle-10380

07/08/2022, 7:16 PM
Thanks for the info. this could be caused by installing gbstats from pip rather than from local source. And you’re right. Information that there is not enough data is definitely better than showing 50% chance 👍
12 Views