Zabble Zero Mobile Tagging™ was designed to help customers get more data points, faster, with AI-assisted fullness and contaminant tracking. Our algorithms predict the fullness of a receptacle from a photo, and until recently, suggested contaminants based on tagging history. Now, our latest app version includes a much-requested new feature: Zabble Zero AI^TM has object detection to identify items from a photo and suggest relevant ones as contaminants. How well does the new AI work?
‍

How We Tested Zabble Zero AI^TM with Objection Detection‍

To evaluate the new model and algorithm prior to release, we had 5 testers, on multiple occasions, meticulously run tests to validate the outcome of the AI across 10 categories of items the AI has been trained to detect. ‍

The following process was followed:

1. A waste item was selected from one of the 10 categories.
2. The item was placed in a bin, along with other items.
3. In the Mobile Tagging™ app, an appropriate receptacle was chosen for a waste stream where the item would be considered a contaminant. For example, if testing a paper towel (compostable item), the tester could tag a recycling or a landfill bin.
4. The receptacle was scanned using the AI feature. The tester waited up to 2 seconds to see if a box appeared around the item. If it did, they noted the label. 
5. A photo was taken. On the next screen, the tester checked the suggested items list.
6. For each category, 3 different items were tested (e.g., for paper/cardboard, try paper, cardboard, and paperboard) and/or the same item was tested with a different orientation/position in the bin.

Strict Criteria for Positive Results

An accurate result (true positive) for that category had to meet each of the following criteria:

• A box was shown around item
• The box showed the correct label
• The suggested unacceptable item was shown in the list on the next screen

Here is an example of a true positive result when testing cans in the landfill stream:

We adjusted our algorithm based on initial testing, and then conducted a final round of testing to report the following results.

Final Results: Items Detected Correctly 73% of the Time

As shown in the table below, categories that were most accurately detected have higher true positives (TP) and fewer false negatives (FN). Aluminum cans had a perfect score! Food, plates, and bottles were successfully detected only about half the time in our test. 

Some categories had higher false positives, which means they were wrongly detected in the photo. For example, we saw 9 cases where paper/cardboard was detected but wasn’t actually in the photo. Overall, specificity was about 97.6%, which means our object detection model is good at knowing when items are NOT present in the photo.

The visual below shows which categories got mistaken for other items. For example, several different items (bottles, containers, cups, foil/plastic film, and food) were predicted as paper/cardboard at least once. On the other hand, even though our model wasn’t as good as detecting bottles or food, when it does detect them we can be pretty confident the prediction is correct - because no other item was wrongly labeled as a bottle or food in our test. 

‍

‍

Note: Confusion matrix has been normalized so that rows sum to 1.

‍More Use → Better Results

Objection detection in Zabble Zero AI™ will get smarter over time as we retrain our model with more images and different types of items. We’ll also add more categories and aim for higher accuracy at different distances.

Stay tuned for an update about how our latest AI performs in the field. And if you’re not already using Zabble Zero™, contact us for a demo.