𝐂𝐥𝐨𝐮𝐝 𝐀𝐏𝐈𝐬 𝐚𝐫𝐞 𝐞𝐱𝐩𝐞𝐧𝐬𝐢𝐯𝐞. 𝐖𝐚𝐭𝐞𝐫𝐦𝐞𝐥𝐨𝐧 𝐢𝐬 𝐧𝐨𝐭. 🍉

Vision models grow in their capabilities while the execution demands way fewer resources.

Means: my sales slip scanner-project from two years ago needed an upgrade. Before I was using GPT-4 Vision via the @OpenAI API, which costs you control over your data and a few cents for every analysis.

Now, after intensive benchmarking across a range of 15 models (with 𝐎𝐥𝐥𝐚𝐦𝐚) and some surprising regressions (newer harness doesn’t mean the models execute faster; it can also mean they suddenly run into model load errors…), I put my money on 𝐐𝐰𝐞𝐧𝟑.𝟓:𝟒𝐁. A casual language model with a 𝐯𝐢𝐬𝐢𝐨𝐧 𝐞𝐧𝐜𝐨𝐝𝐞𝐫. My own benchmark over the selected models against the huge sample size of three receipts: all three were successfully evaluated. 😉

Then I built a 𝐡𝐨𝐭-𝐟𝐨𝐥𝐝𝐞𝐫-𝐜𝐨𝐧𝐜𝐞𝐩𝐭 around the script, and it works (thank you @Mr Zahorsky — without whom I would have never come across that idea). For instance: in less than ten seconds, 13 samples were processed, and you get the result sum plus a fancy HTML overview for comparison.

𝐃𝐚𝐭𝐚 𝐞𝐧𝐭𝐫𝐲 can be done for cheap: give the kids the smartphone, snap all receipts, throw out some watermelon slices 🍉😉

If you want to run the tool as well, check my GitHub: https://github.com/marcelpetrick/sales-slip-scanner-ng. As said: a local GPU is the only thing you need; 𝐧𝐨 𝐜𝐥𝐨𝐮𝐝, 𝐧𝐨 𝐀𝐏𝐈 𝐤𝐞𝐲𝐬, 𝐧𝐨 𝐜𝐫𝐞𝐝𝐢𝐭 𝐜𝐚𝐫𝐝. Everything is now self-hosted. You’ll also find the benchmark of the vision models there, including the results.

𝐓𝐡𝐞 𝐛𝐢𝐠 𝐪𝐮𝐞𝐬𝐭𝐢𝐨𝐧 𝐢𝐬: 𝐰𝐡𝐚𝐭 𝐬𝐡𝐨𝐮𝐥𝐝 𝐈 𝐚𝐮𝐭𝐨𝐦𝐚𝐭𝐞 𝐧𝐞𝐱𝐭 𝐰𝐢𝐭𝐡 𝐚 𝐥𝐨𝐜𝐚𝐥 𝐯𝐢𝐬𝐢𝐨𝐧 𝐦𝐨𝐝𝐞𝐥? 𝐖𝐡𝐚𝐭 𝐰𝐨𝐮𝐥𝐝 𝐲𝐨𝐮 𝐝𝐨?

#neverstoplearning