DealTracker - Architecture
DealTracker's Architecture
DealTracker's architecture is designed in several parts, many of which are hosted at home in my homelab.
[Diagram]
1- The Data Access Layer
The goal of this part is to collect data from second-hand listings published on different marketplaces. This data is restructured and normalized, to allow equivalent processing regardless of the marketplace on which the listing was originally published.
2- The Enrichment Layer
When listings are collected, we need to enrich them with our Machine Learning system.
It's a process during which we:
- Download the images
- Send them to our Machine Learning system in charge of classification
- Perform any post-processing based on the textual content
- Calculate the listing's attractiveness (how good of a deal it is)
- Update it in the database
I wanted to make sure DealTracker could handle any volume. So I optimized this process by maximizing its processing throughput, and ensuring it was scalable.
Regarding processing throughput, the most important thing is to set up an asynchronous processing pipeline. Meaning that tasks are not processed sequentially, they are orchestrated by Taskiq (a Celery/Dramatiq equivalent with asyncio). The system's bottleneck is the PyTorch model, which uses 4GB of VRAM, and can process around 450 listings per minute on my RTX 3060 Ti.
Regarding the scalability of this pipeline, this is naturally enabled by taskiq, since tasks don't hold state, and are decoupled. It's possible to add hardware capacity to handle each of the tasks, which means the system is as capable as the hardware it's deployed on.
3- The Data Access Layer (The API)
Once listings are enriched, they're ready to be displayed on our website. For this, we need to set up an API, to make listing information available to clients. I naturally went with FastAPI (the most up-to-date Python web framework at the moment, asynchronous by nature and well-implemented).
I have few comments to make about this API, which is fairly classic overall. It retrieves data, caches it, and serves it on demand. Two parts seem particularly interesting:
Building the list of best deals:
What is a good deal? A good deal is a listing that is:
- Published recently
- Near you
- At a rock-bottom price
When a user asks to see good listings, these 3 criteria are taken into account and weighted to provide the most relevant list.
Geographical filtering:
A user on the DealTracker website has the ability to do geographical filtering, and say "what are the listings within X km radius around such and such city". When they do this filtering, it sends a request to our API, to return a filtered list of listings.
This geographical filtering of listings is done with geopandas, an enriched version of pandas dedicated to processing geographical information. After finding a list of vector geodata for each French municipality, I was able to implement the algorithm that, from the location sent by the user, finds all listings located within X km.
4- The Frontend
The goal of the Front-end is to provide an interface for users to access the results of our analysis. This interface must be:
- Fluid
- Ergonomic
- User-friendly
- Optimized for SEO
For this site, I ended up going with NextJS. It wasn't my first choice, I had initially decided to do it in Flutter web...
The design is done with antd, and I put real work into the hierarchy of relevant information. Of course, you can access the website at https://www.dealtracker.org