We almost broke Rainforest’s # 1 rule. The rule is “Don’t F*ck with the money!” and we take it VERY seriously.
Last year, we called an incident over a $6.34 billing error.
Because if a system is capable of a $6.34 billing error, it’s also capable of a much larger error. And our goal is zero errors.
Fast-forward to a few days ago.
Something hiccuped and we got awfully close to moving the same money twice – like groundhog day.
In this case, it wasn’t a huge sum of money, but it could have been.
Most payment providers would not have caught it.
And payment providers who show complacence over a small billing error? They would not have caught it. Because the same causal mechanisms are responsible for small mistakes and big ones.
But we have remarkably robust controls on our systems.
We developed and continuously improved these controls over the course of years.
And our controls caught the potential duplicate so we were able to stop it before it happened.
On-call engineers were paged and we had eyes on screens within a minute or two. This could have been a really big deal. But in the end, it wasn’t.
To be clear – this wasn’t luck.
It was a direct result of the countless times we dug into the root cause of a tiny anomaly to make sure it wouldn’t happen again, because we don’t accept anything less than 100% accuracy, 100% of the time.
And if a payment provider lets small mistakes slide, you have to wonder what else they’re going to miss.
