[Part 1] Url Shortener System Design
This articles is documentation of what i am learning about url shortener system design from a youtube video and some research in a night and decide to make this system design become a real simple application that robust and can run very fast that i write (include to learn about) in Go and Fiber Framework.
In a night few days ago, i decide to learn more depth Go and fiber framework. But i do not know why, i see a system design video that explain this system very detail and very clear specially in high scale traffics. You can view the video in below bookmark.
URL shortener system design | tinyurl system design | bitly system design
So i think you already know what is Url Shortener. If you do not know, you may refer to Tiny Url and Bitly.
So the basic important of url shortener is that it's will shorten the url to be more less. I will use youtube as an example. This is the url of the video above in full link → https://www.youtube.com/watch?v=JQDHz72OA3c&t=177s&ab_channel=TechDummiesNarendraL And we can short to https://youtu.be/JQDHz72OA3c .
So in the video there is several check point that i think must be think →
- Data Capacity
- Url Shortener Logic
- Database Platform
- Distributed Scale
Data Capacity
Let's do some assumption.
If we do an assumption that twitter have 300 Million Active User per month, and 10% of them is use shortener services, it's will create around 30 Million Short Url per Month
Let's do assumption again that the model of shortener url will be:
| column_name | column_data_size | Sample Data | | --- | --- | --- | | longurl | 2k (2048 Chars) | https://www.youtube.com/watch?v=JQDHz72OA3c&t=177s | | shorturl | 17 Bytes (7 Chars) | https://youtu.be/JQDHz72OA3c | | created_at | 7 Bytes | Epoch datetime | | expired_at | 7 Bytes | Epoch datetime |
With this assumption, 1 short url will create around 2.031 KB. And if we calculate it with 30 million per month short url, it's will become 60.7 GB/month → 0.7 TB /year → 3.6 TB/5 year and so on.
Url Shortener Logic
There is a several logic that described in the video:
MD5 Hash algorithm
In here, the logic we use MD5(long-url) and take 7 first digit and check if it's present in database. The problem with this approach is that you need to check to database if the short url is not duplicated or not which will become a single point of failure if the traffic is very high.
Counter based approach
In here, this approach will use a counter (for example → 1 ) as the counter and put that in the url shortener logic. The problem with this approach is that you need a counter ledger to maintain the counter.
Let's use counter based approach with 66 character. (In video, they use 62 character). Why i use 66 chars? it's will increase more combination. But the problem there is a reserved chars by browser.
URL Encoding | Google Maps Platform | Google Developers
Now lets combine with the logic from the video and it's become like this (You can view the code in my github too) →
url-shortener/encode.go at master · shamir92/url-shortener
Database Platform
In the video, they describe about RDMS vs No Sql based and Apache Zookeeper for storing the counter range. I will not describe about RDMS and NoSQl more detail. I will go to Apache Zookeper directly.
So in the video, they use Apache Zookeper for storing the counter range. For what is Apache Zookeper, i will add bookmark that explain for what is Apache Zookeeper.
ZooKeeper: Because coordinating distributed systems is a Zoo
Distributed Scale using Zookeeper
In distributed part of the video, the system design will be like this →
So this 4 checkpoint that need to be understand well and in the Part 2 articles, i will write what is the problem i face, what approach i use, and in what is the minimal result.
For now, you can see my alpha version in this github repo below.
GitHub - shamir92/url-shortener
References:
Working on something similar? Fastest over WhatsApp.
Message me on WhatsApp →