System Design Document: Shorl

The PDF version of this document can be found at Here

Overview:

Shorl is a url shortener website. A url shortener is a service that takes a long url from users and provides them with a shorter url which they can use and share with others. The short urls will redirect the users to the original long url when used.

Architecture Summery:

Front-end:

Templ is used for HTML templates, Tailwind for styling, Alpine.js for client side interactivity and Alpine’s Ajax plugin for rendering server responses.

Back-end:

The back-end is written in go. Sqlite is used for storing generated urls and users’ information.

Requirements:

Functional Requirements:

• URL Shortening: Users should be able to convert their long urls into short urls.
• Short Urls: Generated short urls should be unique and only include English letters and numbers.
• Authentication: Users who create an account should be able to view their urls analytics and delete their generated urls
• Analytics: The server should track the number of times each short url is used

Non-Functional Requirements:

• Security: The service must prevent users to create urls for malicious links. The server should keep user’s data safe and private and prevent attacks such as CSRF and session fixation.
• Low Latency: url redirection should be fast to create a good experience for the users.

* This project wasn’t built for production use, so requirements such as availability, durability and scalability weren’t factored into the design.

API:

POST /Shorten: Since this endpoint is designed to work with the form implemented in the website, the right header should be used if you want to use the endpoint outside of the website.

This endpoint is currently protected by the authentication middleware so the session token is expected to be included as well.

The response is in HTML form and is handled by alpine Ajax on the client side.

GET /{short url}: The user will be redirected to the corresponding long url upon opening using this endpoint. The redirection will use the 302 Found status code.(the details of redirection system will be addressed later)

Core Components:

URL Shortening:

1. The client sends a post request containing their long url to the server
2. A new short url is generated (short url generation will be discussed later)
3. The generated url will be stored in the database alongside its original url, the user id who created the url and the time of creation.

URL Generation:

The urls generated by our service should be unique and only include English letters and numbers so they could be easily shared and used.

1. A new uuid will be generated
2. the generated uuid will be converted to base 62(all English letters both uppercase and lowercase and numbers 0 to 9)
3. the first 5 letters of the base 62 uuid will be used as a new url
4. if the newly generated url is already in database (very unlikely but possible) a new url will be generated.

Redirecting:

1. The client opens a short url and visits the /{short url} page.
2. The server checks the urls table of the database to find the long url that belongs to the given short url
3. The server will response back with a redirection with the 302 Found status code.(the reason we are not using 301 status code can be found in the Design Decisions section)

* Since the redirection system should be as fast as possible, a caching layer will be implemented in the form of a Redis database that stores the frequently used urls.

Authentication:

The authentication is implemented via Oauth2 and session management. This way we don’t need to store sensitive data on our database. (currently users only can authenticate using their github accounts, but the implementation for other services such as google are the same)

1. The client clicks on the authentication button and goes to the page of the Oauth provider(in this case github)
2. The user signs into their github account.
3. The user will be redirected to our website and into our callback page. Github will provide a code in the url.
4. The server extracts the code from the url and sends a post request to the github server to receive an access token
5. The server then uses the given access token to retrieve user’s data from github API.
6. The server stores required user data (username, id, avatar image url)
7. A new session will be created for the user. The session id will be stored in a secure cookie. (if user had a session id before authenticating, we remove that session to prevent session fixation attacks)
8. The new session will be stored in our database.
9. The authentication middleware will check the session id provided by the client, if the session is valid and not expired, it will grant access to the user to create new short urls and view their dashboard.

Analytics:

Our server will keep track of the number of times each short url is used and store it in the database so users could access the analytics in their dashboard page.

* More information about the visitors of short urls could be stored as well (location, device type, time)

UI:

The HTML files are generated on the server with the help of templ templates. Styling is done with Tailwind and any client side interactivity can be implemented via Alpine.js. The HTML responses sent back from the server are handled by Alpine Ajax to prevent unnecessary page reloads. Tailwind generates a css file with only the required classes and no duplicates and Alpine and Alpine Ajax files are also light weight to help the loading time of our pages.

Design Decisions:

302 status code over 301:

3xx status codes are used for redirection. 301 is “Moved Permanently” and 302 is “Found” or in another word “Changed Temporarily”. The 301 status code tells the browser that the used url will always gets redirected to the url provided in the Location header. The browsers will cache 301 redirects. This will reduce the load on our servers. However, this will affect the accuracy of our analytics service.

Future Improvements:

Caching:

To make the redirections faster and provide a better experience for the users, we can implement a Redis database for the most frequently used urls and only query the database if the url doesn’t exists in the cache.

In-Memory Storage For Analytics:

Now that we introduced Redis to our tech stack, we could use it to store url analytics in a Redis database and periodically push its data to our main database to reduce the load on our database.

Optional Analytics:

We could give our users the option to disable analytics on their generated short urls. This way we could redirect with 301 status code and benefit from browser caching. By implementing this, we will also be able to redirect with 301 status code for the urls generated without having an account.

References and Resources:

• A video explaining Alpine Ajax and its philosophy by its creator
• Github Oauth Documentation
• Templ Documentation
• Tailwind Documentation
• Alpine Documentation
• Alpine Ajax Documentation
• Mozilla’s Documentation on HTTP status codes