Do we notice when AI is wrong, or do we settle for quantity?

Evaluating the quality of AI’s work is not as easy as it is with other products. If we buy apples that look gorgeous, but when we cut them open, we find mold inside, we take them back. We can simply tell the difference thanks to what we already know: mold = bad.

Recently I have worked on several projects evaluating the quality of AI outputs. I kept seeing poor results that users accepted without question, either because they didn’t notice or because, over time, they had stopped being critical.

This made me think that I needed to build, starting from myself, tools to monitor and quantify what AI does. There was also a personal reason: I was worried about my mood after hours of dealing with AI errors.

So, I used Python to build an app that tracks the moments when I “fought” with AI’s quality.

The app is called “Semaforo” (Italian for traffic light) because, like the device that manages traffic, it tells me where the red light is so that I know when to stop.

I turned the data into a dashboard, and here is a preview (or click on the image below).

The data in short

I have a more complete future variant in mind for my app, but first I analyzed the data from the first version. The app is written in Python; I did the analysis in R and used R Quarto for the dashboard.

Between 17 August and 19 October 2026 (about nine weeks), Semaforo logged 94 rows. After cleaning, they became 63 incidents in 18 sessions. Claude had 34 incidents and Gemini 27; two involved both tools or ChatGPT. I often used Claude and Gemini in the same session, so the per-tool session counts (13 and 8) add up to more than 18.

With 63 incidents, we can read everything below as patterns to watch, not as conclusions.

I wanted to know the following:

1. When do the “incidents” happen?

Mostly early. Of the 55 incidents with a reliable timer, 36 (65%) happened in the first 90 minutes, and the median first incident came after 26 minutes.

Most “stuck” episodes (incidents less than 30 minutes apart) ended within 30 minutes. Two lasted about 100 minutes, one with Gemini and one with Claude, and together they make up nearly half of all estimated stuck time.

2. Which problems are the most frequent?

Wrong output, the AI being unable to solve the problem, and slow or overlong answers lead the list. One incident can have several themes:

• Wrong output: 16 incidents (25%)
• AI cannot solve it: 14 (22%)
• Too slow or too long: 12 (19%)
• Ignores instructions: 12 (19%)
• Unrequested actions: 10 (16%)
• Overcomplicated answers: 7; lost context: 6; made-up content: 3

3. Where Claude and Gemini differ

They fail in different ways, and I won’t name a winner. Per session, Gemini had more incidents (3.4 vs 2.6), and a higher share of “cannot solve”, ignored instructions, unrequested actions and wrong output. Claude had a higher share of “too slow or too long” and overcomplicated answers.

The “switched tool” notes look higher for Claude.

4. Which problems occur at the same time?

No strong links, but problems tend to pile up. The small dot at the top is “Made up”. Made-up content is the problem people talk about most, but in my notes it appeared in only 3 incidents, and mostly on its own. Wrong output, ignored instructions and slow answers were far more common.

Quick guide: every dot is a kind of problem, and its size shows how many incidents had it. When two problems appeared in the same incident, I connected them with a line, and the thicker the line, the more often it happened. Where the dots sit on the page does not matter.

5. Analysis of my angry comments: what do I complain about?

Mostly about code, and wrong instructions for Quarto. The most frequent words in my notes are “code” (15), “Quarto” (11) and “instructions” (9). The most frequent negated phrases are “cannot solve” (3), “not understand” (2) and “not respect” (2). My signature complaint: “1 zillion attempts”.

What I learned

The data taught me some things to improve about the Semaforo app and something about AI.

The first version of the app saved only the red clicks, but the next version will log every event, positive and negative, and will add mood and frustration ratings during use.

In my notes, much of the frustration came from the AI forgetting information, or ignoring instructions that had worked earlier in the same session or in previous ones.

For instance, I do not want the AI to write long blocks of code: I want only the chunks I need, and I make the edits myself. I repeated this request endless times, and the AI kept writing “books” of code.

What is next

The next step is to turn Semaforo into an app that other people can use, and to publish the results here on my blog, so that we will have a larger sample.

I would like your input: Do you ever feel frustrated with AI? Would you use an app like this? What would you want it to track? Leave a comment and let me know.