https://www.growthbook.io/ logo
Join Slack
Powered by
:bug: :question: Hey there, we just stopped an exp...
# experimentation
m
šŸ› ā“ Hey there, we just stopped an experiment and made Variation 1 auto-rollout. I saw the correct state on the feature flag and on the experiment. Both looked publish and correct. However, when we skipped the cache on init and then asked for if the flag was on, we got:
Copy code
on: false, source: 'defaultValue'
I would not expect to see the default of the feature flag, when the experiment rule should be serving true
āœ… 1
PS. The old revision is because we disabled the experiment rule and changed the default value instead to get the rollout we wanted.
h
However, when we skipped the cache on init and then asked for if the flag was on
Did you set a 
uniqueId
when you did this test?
Sorry for the slow response here (got lost in the holiday sauce!)
The temp rollout basically takes everyone that would get into the experiment and instead shows them one value. A unit only "should" get into the experiment if they have a 
uniqueId
set.
m
Sorry for the slow response here (got lost in the holiday sauce!)
No worries at all!
Did you set a 
uniqueId
when you did this test?
Yes, we have been running the code in prod for a bit and have been monitor against an A/A test of the same code. For some clarity, we have successfully used the rollout once before, but for some reason in this case we did not see what we expected. PS. I am not exactly sure if a rollout will have an experiment/experiment result when asking about a feature. I reached out mainly that I could repro at the time in a code sandbox ignoring our more complicated fanout (Cloudfront -> Vercel / Edge / NextJS) each of which has it's own caching layer. The code sandbox was simply init'ing GB and inspecting the getFeatures results
h
I am not exactly sure if a rollout will have an experiment/experiment result when asking about a feature.
I think it should.
Was the experiment still in the sdk payload when it was in the temp rollout state? I asked one of our eng and he agreed that it should function like an experiment but everyone will get the same value.
m
The logging shown above is us adding a custom logger, but we do print the entire result object. i.e. The image above shows no experiment running when the rollout was active
Was the experiment still in the sdk payload when it was in the temp rollout state?
Not that I saw at the time šŸ˜“
h
Got it. Understood.
I believe it should be from my understanding, let me follow up with the team here.
m
I can also poke at the source. I've read most of the JS/React code, but I have not looked at your server's code.
Basically our steps to repro: ā€¢ Feature flag - default off ā€¢ Add A/B test ā€¢ Stop experiment with B winning and Rollout B
h
fyi: I think an easier way to validate that this is or is not working would be to use the feature tester at the bottom of the feature page.
basically simulate a user who normally would have been in the experiment, change the experiment to a temp rollout, and validate if they still get the feature value corresponding to the rolled out variation
once its a temporary rollout, it's no longer an "experiment" in the way it presents in the SDK. it actually materializes as a forced rollout rule.
example
prior to the temporary rollout, i am bucketed normally
(trying to rule out whether this is an SDK-level problem, caching problem, feature flag/experiment setup problem, or a bug on our end)
m
Okay I found the issue: https://codesandbox.io/p/sandbox/happy-elgamal-55y52n Basically, due to limitations with Vercel's CDN caching we have been doing the computation of the Experiment's a user would see at the edge. This would allow us to only split our CDN caching based on the available number of experiment buckets. However, this limitation also forces us to not pass user specific attributes (user id, unique id, etc) from the edge down to the main app. Thus, when we rebuild a growthbook instance on the server we are not passing the uniqueId attribute and the result is:
Copy code
Skip rule because user not included in rollout {id: 'testing', rule: Object}
Use default value {id: 'testing', value: false}
We have been aware that our implementation is non-standard, at least according to your examples. But this was mainly so we can leverage the CDn cache as much as possible to limit our Vercel costs
h
yeah unfortunately BE/Edge experimentation on user attributes + cached responses doesn't often mix
m
I think the "bug" from my point of view is that I forced an experiment value in the code, but that code never is executed as the skip rule is hit first
I guess I could set a hardcoded uniqueId attribute on the server for all users. It would never be used since we'd force the feature for every user
h
I see. Side question: how do your tracking calls work when doing edge evaluation?
m
Can I send some files over directly to you?
h
sure
m
Send all of it over to you, but the main bit you are asking about is attached.
Untitled
I am unhappy that we had to mimic a lot of the internals of the JS source, but it does work great!
I ended up doing more testing and I am notice something. If I 
console.log(growthbook.evalFeature("testing"));
there is no experiment in the 
evalFeature
's result. My expectation was that there would be an experiment and result still. Re: https://growthbookusers.slack.com/archives/C07E4HA06MD/p1733338164984769?thread_ts=1732742454.275509&cid=C07E4HA06MD It is in the payload, but not returned by evalFeature in the same shape as a running experiment, which breaks the logic I've shown you
h
I believe I misspoke about it still being an experiment in the payload. It was based on a misunderstanding of how this works.
As Bryce says above it actually materializes as a forced rollout rule.
šŸ‘ 1
m
Thanks guys. I really appreciate all the help. I told Bryce that I likely will stew on our implementation and see what changes I can suggest that would allow our use case to work out of the box. Overall, we love the product and know our CDN caching and computing at the edge is not the use case the product was built for, but I'm sure others out there will want to try the same. Basically the only thing we fight are very small behavior differences where we made an assumption. ...Like in this case, our code assumes any experiment even one which is rolled out will have an experiment and result object after eval
anyways, nothing for you all to do unless you think it's reasonable to keep the experiment behavior on eval
h
We're actually speaking internally about ways we can either make edge/hybrid testing easier or be more prescriptive with better docs/examples. If you have any ideas, we'd love to hear them
m
more prescriptive
That would have helped a lot. We started the journey back in Feb. We were on the App router, but there was no example at the time. Once we determined we wanted to run at the edge for caching reasons it took a lot of trial and error to get here.
The main thing we do is pre-compute the buckets a user may see at the edge. Then if we ask about a feature, use the pre-computed value. On the server, this requires us to call 
setForcedVariations
to bypass the experiment logic on the server where we do not have the hash attribute available. This works as long as we run the experiment logic. Since the rollout's eval feature is not actually computed as an experiment, we are failing to set the variation. I would use 
getExperiments
and then just ask for the value for each. However that returns the experiments from the payload which is empty even when we do have experiment rules.
Copy code
{
    "status": 200,
    "features": {
        "product-overview-reorder-sections": {
            "defaultValue": false,
            "rules": [
                {
                    "coverage": 1,
                    "hashAttribute": "uniqueId",
                    "seed": "647ef094-a7c6-4dcc-adf4-d1c216286e6e",
                    "hashVersion": 2,
                    "variations": [
                        false,
                        true
                    ],
                    "weights": [
                        0.5,
                        0.5
                    ],
                    "key": "product-overview-reorder-sections",
                    "meta": [
                        {
                            "key": "0",
                            "name": "Description First"
                        },
                        {
                            "key": "1",
                            "name": "Compatibility First"
                        }
                    ],
                    "phase": "0",
                    "name": "Product Overview Reorder Sections"
                }
            ]
        },
        "testing": {
            "defaultValue": false,
            "rules": [
                {
                    "coverage": 1,
                    "hashAttribute": "uniqueId",
                    "seed": "3554bf39-d234-42a9-8a8f-93766d6203b8",
                    "hashVersion": 2,
                    "force": true
                }
            ]
        },
    "experiments": [],
    "dateUpdated": "2024-12-04T18:54:47.700Z"
}
h
fyi: getExperiments is for "auto experiments" (i.e. redirects and visual editor exps)
šŸ‘ 1
m
I figured that out eventually, but it is not obvious when you are first using the SDK
My ideas: ā€¢ something like 
getExperimentsFromPayload
or a 
computeExperimentsValues
which accounts for rollout experiments ā€¢ An example of computing the experiments at the edge and passing them into a Next.js app ā—¦ In our case we rewrite the url to add the computedExperiments in a path and then parse it serverside ā–ŖļøŽ This allows the CDN to cache server responses for each possible arrangement of values
šŸ‘€ 1
We are currently not handling forced experiments at the edge, hence the issue with the rollout
Even now an example of how you all would tackle the vercel caching and use the edge would go a long way
35 Views
Open in Slack
PreviousNext
Powered by