Making food detection systems inclusive of global audiences.
We’ve created a report on how four widely used foundation image recognition models could better deal with images of food from people in 13 different countries across four continents.
Task
To better understand the needs and perspectives around food detection systems from people of different cultures.
Being able to detect food in images with Artificial Intelligence opens up a lot of potential value for food delivery services and social network services.
However, the enormous variety of foods across countries and cultures makes it difficult for data scientists to identify what users of these services find valuable in specific contexts.
We therefore set out to gain insights into where value can be created when tailoring foundation models* to a multi-ethnic production context of food in 13 countries across four continents**.
This work took place during the Corona virus pandemic, which made it difficult to meet people in real life, let alone in different countries. As a solution, we invited people from 13 different countries to share a digital meal with us over video chat. We simply requested them to prepare a meal they would otherwise also prepare.
During our meals, the participants told us about their meal, sent us pictures of their meal, and described the foods of their country. Through secondary research, we gained additional information on the food in each country.
Finally, we ran the collected images from each participants through the image recognition frameworks and analyzed the results.
Baggles and tomato soup from the USA (left), Sirene Po Shopski from Bulgaria (middle), rice and sweet & sour pork from Malaysia (right).
A number of insights on adding value to multi-ethnic food recognition systems were gained. A number of these recommendations are given below. We recommend data scientists working on food detection systems in international context to keep these in mind.
1. Address sensitive predictions
Alcohol and weapons are two sensitive topics when presenting predictions in multi-ethnic production contexts. Concerning alcohol, data scientists would do well to prioritize the performance of predictions on food items containing alcohol (such as a bottle of wine),. The current systems often performed poorly in this respect.
In multiple instances, dinner knifes were labelled as weapons. Though less critical, data scientists are invited to explore the degree to which their food detection systems need to label weaponry.
2. Create more specific and relevant labels
A lot of value can be added by providing more relevant labels for the current production context. The most impact would be gained by labeling and training for: spring rolls, bagels, bottle of wine, Cuy, Kaydos, crab cakes, among others.
3. Address cultural misrepresentations
On the other hand, some dishes are similar, but come from culturally sensitive contexts. My research indicates that negative attitudes could arise when predictions fail to acknowledge different cultural contexts. For instance:
- A Thali (Indian serving plate) in an image should not skew Yemeni food results towards Indian food results.
- A roti should be distinguishable from a tortilla or pita.
- Frittata should be distinguishable from a pizza.
- Minestrone should be distinguishable from a curry.
- Chopsticks should be as recognizable as forks and spoons.
- And more.
4. Address visual misrepresentations
Though completely different dishes, some look visually so similar they confuse the current recognition systems. A lot of user value is to be gained by improving performance between visually similar dishes, for instance:
- Spring rolls should be distinguishable from sausages.
- Rice, Kaydos and Crab cakes should be distinguishable from ice cream.
- Sliced melon should be distinguishable from bananas.
- Rice should be distinguishable from oatmeal.
- Cuy should be distinguishable from pizza.
- And more.
5. Do not harm people of certain religions or with certain diets with labels of meat
In many contexts, meat is a sensitive topic. Some people avoid certain meats because of religions beliefs, others because of medical conditions, and others still because of moral beliefs. Presenting faulty predictions about meat towards these audiences could be seriously detrimental to the value image recognition systems can offer. Based on the current investigation, we therefore recommend improving the following predictions:
- Cuy should not be recognized as duck and chicken meat.
- Carrot cake should not be recognized as steak, beef, or meat loaf.
- Mock meat should not be recognized as real meat.
- Chicken should not be recognized a s beef.
- And more.
6. Recognize the limits of image recognition systems
Image recognition technology is very exciting, but data scientists who want to create value for users in production need to make sure to recognize the technology’s limits. For instance:
- Can a system actually detect if a waffle has Kaya and Margarine on it based only on an image without further context?
- Can a system actually detect if a meal is gluten or sugar free simply based on an image without further context?
- And more.
We love tech, design and culture and get excited about getting AI adopted by a wide variety of audiences.