https://www.growthbook.io/ logo
#give-feedback
Title
# give-feedback
a

agreeable-elephant-38605

02/13/2024, 4:13 PM
Hey GrowthBook team - we noticed our mongodb deployment's CPU usage had gotten really high (I think this happened over a few months time), to the point of causing timeouts/aborted requests - I took a look at some of the mongo logs and it seemed to mostly be from queries that agenda makes to find/update jobs. It seems the
eventCreated
jobs weren't being cleaned up properly (or something along these lines):
Copy code
test> db.agendaJobs.find({name: 'eventCreated'}).count()
5057475
a majority of these had a
nextRunAt
of `null`:
Copy code
test> db.agendaJobs.find({name: 'eventCreated', nextRunAt: null}).count()
5056482
I ended up deleting these to see if it helped and our mongo CPU usage dropped pretty much back down to 0. I haven't dug into why so many are created or whether something should actually be cleaning them up, but just thought I'd share here in case this is a known issue. It's possible its related to our own setup, although I don't think we have any changes that would affect this job type specifically
f

future-teacher-7046

02/13/2024, 4:16 PM
Thanks, I think we might be missing some indexes on that collection. So those un-indexed queries were doing full scans which get really expensive as the size grows. In addition to adding missing indexes, we should be able to add a cronjob to clean up those old jobs pretty easily.
a

agreeable-elephant-38605

02/13/2024, 4:17 PM
my not-so-comfortable-with-node/typescript guess is that it's because
agenda.define
is used in the constructor of the `EventNotifier`: https://github.com/growthbook/growthbook/blob/1a9e129bfae401e62bde002fb4679b5dc9de956d/packages/back-end/src/events/notifiers/EventNotifier.ts#L[…]6 and in all usages of
perform()
a new EventNotifier is created
f

future-teacher-7046

02/13/2024, 4:20 PM
creating new docs for every event is the proper behavior since the event id is part of the unique key. We're just missing the cleanup part
a

agreeable-elephant-38605

02/13/2024, 4:20 PM
got it, that makes sense
f

future-teacher-7046

02/13/2024, 4:28 PM
Added a GitHub issue to track this - https://github.com/growthbook/growthbook/issues/2116
🙏 1
11 Views