On November 5, 2021 (a Friday of course), we’ve deployed an innocent looking gem updates. Minor versions of Ruby on Rails, Ruby Sentry client, Ruby Slack client, http libraries, Puma, Devise, OmniAuth Ruby client, Mongoid, and a few test gems.
However, something went very wrong.
Featured Content Ads
add advertising hereWe saw odd Stripe errors on Airbrake. Then, on our Stripe account, we saw this:
In less than 1h, we’ve created 474 new subscriptions for a total of $73,271.36.
We weren’t expecting that much business.
Our app for some reasons was creating new subscriptions from old accounts that was canceled or disabled a long time ago. We had no idea why but we decided to react proactively as fast as we can as the issue was still very ongoing.
Featured Content Ads
add advertising hereWe immediately rolled back the gem updates to their safe older versions, disabled our Stripe API keys to lock our own application out, and refunded everyone impacted:
After refunding everyone, we manually double checked the billing state of each account one by one and sent emails to apologize to each customer one by one; all 475 of them.
We still didn’t knew why.
Most of the updates seems inoffensive. And after deeper inspection, it was indeed minor gem updates on several small things. We worked on several hypotheses: caching issues, race conditions, or some thread safety issues as Puma changed their threading model in that update.
Featured Content Ads
add advertising hereHowever we’ve finally found it. One method of our billing logic particularly seemed a code smell (Obligatory disclaimer: I personally wrote this.):
This is part of a feature that seems good on paper: We offer to automatically renew your plan if you run out of searches on our service.
The goal of the renew_early_protected is avoid the case of someone submitting multiple searches at the exact same time getting his subscriptions renewed and charged several times.
Notice the or() method at line 47.
In Mongoid (The MongoDB driver for Ruby) 7.0.8 , or() meant filter documents that contain any of the argument conditions. In our case, filter users that have a specific id AND that has either never have a renewing_early_lock or their renewing_early_lock is less than 1 hour ago.
Here’s what it got computed to for a specific user:
computed_selector_in_mongoid_7_0_8 = {
"_id" => BSON::ObjectId('59af54094-----------64'), "$or" => [{
"renewing_early_lock_at" => {
"$lte" => 2022-01-07 20:14:53.44744 UTC
}
}, {
"renewing_early_lock_at" => nil
}]
}In Mongoid 7.3.3 , or() now means filter documents that contain any of the argument conditions OR any of previous method conditions! In our case filter users that have either a specific id OR that has never have a renewing_early_lock OR their renewing_early_lock is less than 1 hour ago.
Here’s what it looks like for the same user:
computed_selector_in_mongoid_7_3_3 = {
"$or" => [{
"_id" => BSON::ObjectId('59af54094-----------64')
}, {
"renewing_early_lock_at" => {
"$lte" => 2022-01-07 19:00:09.571034 UTC
}
}, {
"renewing_early_lock_at" => nil
}]
}Notice that the user ID selector got moved into an optional or()!
This ultimately means we were charging random customers new subscriptions instead of the actual customer who wanted to renew early:
Which is obviously awful.
We are deeply sorry if you were impacted by this issue. This is not acceptable.
Even if that Mongoid shouldn’t have changed existing methods behaviors between minor versions, my implementation was a true code smell as it was unclear what it did. This code should never have been deployed to production. But also and more importantly, the feature itself was a bad idea. Using our API to scrape search engine results shouldn’t trigger a renewal of a credit card subscription in the first place. And we’ll be removing that feature as soon as possible.
Sorry again to all of our users.