why
A friend of mine has struggled with high cholesterol, and eating out makes managing his diet especially difficult. Menus rarely say what’s in a dish, so tracking what you eat turns into a guessing game every time you eat out.
I built Roli to make that decision easier. But I also wanted to explore a broader engineering challenge: how do you build an AI application that produces useful answers when the underlying information is incomplete, uncertain, and scattered across different sources?
The project became an exercise in combining language models, structured data, statistical methods, and software engineering into something people can actually use.
how it works
- 01Read. A vision-language model pulls dishes from a menu photo, converting unstructured visual information into validated data.
- 02Classify. The model identifies each dish’s category and key attributes, which help narrow the search to comparable meals.
- 03Retrieve. A hybrid search engine combines keyword matching and semantic embeddings to find similar dishes across an index of more than 28,000 food and restaurant nutrition records.
- 04Estimate. Python calculates saturated fat and cholesterol estimates from the closest matches, using weighted statistics and an uncertainty range.
- 05Recommend. Dishes are ranked primarily by saturated fat, cholesterol breaks close ties, and the user receives a concise recommendation on the best options.
design principles
AGENT DESIGN
Use AI only where it adds value. An early, fully autonomous agent was slow, costly and mixed up its data. Now AI only reads the menu and sorts the dishes, while code does the math, so results are repeatable and traceable.
MODEL EVALUATION
Benchmark models before choosing one. I tested six open and closed models and several prompt versions against a hand-built answer key, comparing accuracy, speed and cost per photo before selecting Gemini Flash.
BENCHMARKING
Set success criteria before testing. I fixed the accuracy target in advance and tested on restaurant data the system had never seen. I also compared the results with simply asking an AI model for an estimate.
ARCHITECTURE
Choose the simplest design that works. My first version rebuilt each dish from online recipes. Replacing it with a search for similar restaurant dishes that already have published nutrition data made Roli faster and cheaper, and able to estimate every dish.
PRODUCTION DEPLOYMENT
Ship with cost guardrails. Roli is live on Telegram and the web for anyone to use. A daily spending cap holds even when many people send photos at once, and caching returns repeat photos in seconds.
what I measured
A final test on 100 restaurant dishes, scored on how often an estimate fell between half and double the published figure.
ACCURACY
63%
of Roli’s estimates landed within range
TARGET 60%
65%
of the time, two restaurant chains selling the same dish agree that closely
CEILING
Gemini, asked directly, was about as accurate, but it consistently overestimated and couldn’t show where its numbers came from. Most of Roli’s remaining error reflects real differences between kitchens.
LIVE APP
~35 s
for a new menu
< 6 s
for a cached photo
~2¢
model cost per menu