DealTracker - The Object Identification Problem

The True Nature of This Machine Learning Task#

The problem addressed by DealTracker, reliably identifying objects on the second-hand market, is a fine-grained image classification problem marked by extreme class imbalance. In other words, we're trying to identify images selling a specific object from an ocean of diverse and varied images published on the second-hand market. The visual characteristics used for differentiation are often very subtle (such as the arrangement of buttons on a speaker).

Statistically, when going through millions of images, there will always be some that make your model uncertain and get classified as images of interest. If it's even just 0.01% of the images, that's 100 false detections out of 1,000,000 images reviewed, while you might only have 50 that are correct.

How I Approached This Problem#

Training on a First Object#

As mentioned on the previous page, when I started trying to detect Aerons accurately, I ran into a problem: my model had a lot of false positives, which I wasn't sure I understood.

One of my first intuitions was that if I wasn't getting good results, it was because the model lacked training data. The few examples I had given it weren't enough to precisely define "what is the object" and "what isn't the object". So I needed more data, and of course the most relevant data to provide to the model is its own errors. Because these errors represent "holes" in the model's decision space, which need to be filled.

So at that point, my process was the following:

• Train my model
• Run inference on a series of images
• Take the results, and annotate them
• Retrain a model

At first, the model's errors were crude, and the images had nothing to do with the office chair I was trying to recognize. Then gradually, similarities began to appear. A curve, or a shape that looked a bit like the chair, which I systematically annotated in the "uninteresting image" category.

I also had to add "true positive" examples to avoid too much imbalance in my dataset, since I was adding a considerable number of false positives to the "uninteresting" class. To facilitate these annotations, I clustered all predicted images. It was quite fascinating, I got clusters of armrests, clusters of backrests... which I sorted by distance to the cluster center, so images most alike within the cluster were grouped together. I didn't include images that didn't show the object as a whole, because the model needed to learn to recognize the object as a whole, not its individual sub-parts.

And then eventually, I reached a stage where each training iteration only improved the model very little. At that point:

• The model was correct (not perfect, but most of its predictions were accurate)
• Its errors were very understandable, because the images showed great similarities with the target object

At that point, I decided to go further. I experimented with new hyperparameter values and loss functions. But most importantly, I looked for other models that might learn better on this task. So I analyzed a series of models on timm, looking for the best compromise between quality and learning speed / inference.

I ultimately decided to mix the results of different models, to take advantage of each one's strengths and weaknesses. So I applied a level of stacking, meaning I trained a meta-model to make the final prediction based on the results of the first models.

Adding Additional Objects#

At that point, I already had a working front-end, and I was very happy with my second-hand chair radar. What it was missing was more products.

New problems arose when reproducing the process for new objects:

How do you annotate an image that has two different objects on it?

In a multi-class classification, I have no method to handle the simultaneous detection of multiple objects of interest.

So what to do? If I classify the image as an image of interest, it's a slippery slope for my model, which must now learn to favor a chosen class in an arbitrary way in case of multiple detection. If I classify it as "undetected relevant object", that's not right either, because the objects are indeed present.

Ultimately, I decided to simply not annotate these examples, and classify them in the "uninteresting image" category during inference. Meaning that when the model hesitates between two classes, I default to "uninteresting image".

I chose not to include images showing sub-parts of the object, but at what point do you consider it a sub-part?

There, the rule is simple: for each object, you need to find true "class membership indicators". Precise visual characteristics that allow you to know whether the object truly belongs to the class or not.

For example, for Bose Soundlink Mini speakers, I determined that the class membership indicators are the buttons on top of the speaker, which must be clearly visible. Any image showing the speaker without showing these buttons is considered an uninteresting image.