Being able to detect food in images with artificial intelligence opens up significant potential value for food delivery services and social networks. However, the enormous variety of foods across countries and cultures makes it difficult for data scientists to identify what users of these services actually find valuable in specific contexts.<\/p>\n\n\n\n
We set out to gain insights into where value can be created when tailoring foundation models to a multi-ethnic production context \u2014 covering food from 13 countries across four continents.<\/p>\n\n\n\n
This work took place during the COVID-19 pandemic, which made it impossible to meet participants in person. As a solution, we invited people from 13 different countries to share a digital meal with us over video chat \u2014 asking them to prepare a meal they would otherwise also make at home.<\/p>\n\n\n\n
During each meal, participants described their dish, shared images of it, and told us about the broader food culture of their country. Through secondary research, we gathered additional context on each country’s cuisine. We then ran all collected images through four widely used image recognition frameworks \u2014 Microsoft Azure, Google Vision, Amazon Rekognition, and IBM Watson \u2014 and analysed the results.<\/p>\n\n\n\n
Alcohol and weapons are two sensitive topics in multi-ethnic production contexts. The current systems often performed poorly on food items containing alcohol, such as a bottle of wine. Separately, dinner knives were repeatedly labelled as weapons \u2014 an edge case data scientists should actively explore and address.<\/p>\n\n\n\n
A lot of value can be added by providing more relevant labels for specific production contexts. The highest impact would come from training models to correctly identify dishes such as spring rolls, bagels, bottle of wine, Cuy, Kaydos, and crab cakes \u2014 items that appeared frequently across participants but were consistently misidentified.<\/p>\n\n\n\n
Some dishes are visually similar but come from culturally distinct contexts. Negative user experiences arise when predictions fail to acknowledge these differences. For example: a Thali should not skew Yemeni food results towards Indian food; a roti should be distinguishable from a tortilla or pita; and chopsticks should be as recognisable as forks and spoons.<\/p>\n\n\n\n
Some dishes are visually so similar that they confuse current recognition systems, despite being completely different foods. Spring rolls are misidentified as sausages; rice, Kaydos, and crab cakes as ice cream; sliced melon as bananas. Improving performance on these visually similar pairs would deliver significant user value.<\/p>\n\n\n\n
Meat is a sensitive topic across many contexts \u2014 for religious, medical, and moral reasons. Presenting faulty meat predictions to these audiences can be seriously detrimental. Among the most pressing issues: Cuy being recognised as duck or chicken; carrot cake being recognised as steak; and mock meat being labelled as real meat.<\/p>\n\n\n\n
Image recognition is powerful, but not unlimited. Data scientists building for production should be honest about what a system can and cannot reliably infer from an image alone. Can a system truly detect whether a waffle has Kaya and Margarine on it? Can it determine whether a meal is gluten-free or sugar-free? Acknowledging these limits is itself a form of responsible design.<\/p>\n\n\n\n
<\/p>","protected":false},"excerpt":{"rendered":"We onderzochten hoe vier veelgebruikte basis-modellen voor beeldherkenning beter kunnen omgaan met foto\u2019s van voedsel uit 13 landen op vier continenten.","protected":false},"featured_media":348,"template":"","meta":{"cs_client":"","cs_year":"","cs_engagement":"","cs_services":"","cs_technology":"","cs_stat_1_n":"","cs_stat_1_l":"","cs_stat_2_n":"","cs_stat_2_l":"","cs_stat_3_n":"","cs_stat_3_l":"","cs_filter_cat":"","footnotes":""},"case_study_category":[6,5],"class_list":["post-346","case_study","type-case_study","status-publish","has-post-thumbnail","hentry","case_study_category-product","case_study_category-vision"],"yoast_head":"\n