[Intermittent Case] RDS Connection

This articles is created to document a case that very seldom happen in my career. Hope this article can help you easier to solve this case in future.

At this article written, i working at a law media company at Indonesia called Hukumonline.com. In here, we have a unique db infrastructure which we have a very big RDS that contain a lot of DB. So in here, we separate a write RDS (Master) and read-only RDS like below to separate a heavy write and read process.

• Master_RDS
• Read_Replica_RDS

1.0 Reference RDS structure for this article

In last year, we just finish migrate all core system from other type of system from Non Kubernetes (Rancher v1, VM, DO ) to Kubernetes (AWS EKS). And in that week, we have do not found any issue as our traffic at end of year is lower than other months. Our traffic start increase to normal season as the new year start. And this case happen with a first alert that say our site down (but we can access the site), and more alert happen to system that access Read Replica RDS but once again we can access the site when that alert show at our slack.

1.1 This one of alert is sent by uptime robot every 5 minute to check our site.

And we found out this alert happen in few days at super random time. As we try to check the our other monitoring tools, we found a few issue that happen frequently. And as we try to dig more info from it, we found out it's a RDS connection error.

1.2 Error from Sentry

After a deep check by the EM of the system, he said it's working well in its local and our staging site. This case piqued my interest to jump in to help.

We try to check the current RDS monitoring and check it's rds connection graphs.

1.3 This is the graph of connection every minute in last 1 week.

And we try to check which rds user open a lot of connection in that moment.

1.4 We found the culprit of this issue which the one of core system for articles.

In this moment, the goal is we need to decrease the usage database connection of zapdos and articuno (in this case) as much as possible.

In this step, we decide to check more detail in each session and found out the node js mysql library create pooling connection on every request and pool connection release is buggy.

After that, we decide to fix the bug in the mysql library with a patch in the database pool creation and pool connection release and success fully working. But lately, we found out in another mysql forking called mysql2 there is a new PR that same as we do and we use the new library.

So now, our database connection become around 250 connection in a period of time.

I think that's all for today. Thanks all.