Lately, the PostgreSQL’s bug-report mailing list has been receiving a large number of AI-generated bug reports. Often, these come in flurries of 10-15 bugs, each from the same email address.
These bug reports are of, shall we say, highly variable quality.
Just looking through a few of the recent flurries, they include:
- One which was clearly as-designed (a manual corruption of data caused PostgreSQL to throw an error… what else would it do?).
- At least two which were being discused already on -hackers.
- One that was a duplicate of a previous report by the same person.
- At least three which have already been patched in version 19.
- A couple that, while there was some kind of arguable bug, the diagnosis was incorrect.
- At least one where the diagnosis was more or less correct, but the patch offered would have not fixed the bug (in fact, would have made the problem worse).
- And one or two others that may be bugs in the strict sense of the term, but the cases were so edge and the consequences so small that it was barely worth the effort to read the bug report, let alone fix it.
This took me about 90 minutes, which was probably about 10x as much time as it took the human beings involved to run the AI, and cut and paste the results into emails.
This wasn’t a good use of my time, except for the purposes of writing this post.
What these batches have in common is that the human involved didn’t really expend any effort at all in examining the report: they ran the AI, it said there was a bug and here’s the fix, and sent it off.
The problem is that an AI can generate bug reports far faster than human beings can triage them. The number of people who can do triage for PostgreSQL issues is not large, and their time is very limited. Transferring the work on determining if the AI is right or not onto them is at minimum rude.
Worse, the submitters of these bugs will get a reputation as being high noise and low signal, and their bug reports will start getting ignored… even a very serious one that might really need to be fixed.
As open source has taken over the world, we’ve created a new social currency where contributing to projects has become a very valuable commodity, especially high-profile projects like PostgreSQL. Unfortunately, this has also created a motivation to throw anything at the wall and hope it sticks so you get a mention in release notes.
Please don’t do that.
I am definitely not against AI. I use it as a coding and writing assist tool, and it is very helpful. It gets things wrong all the time, though, and you need to be very vigilant, because when it is wrong, it sounds just as confident as when it right.
By all means, use AI to find bugs. But you need to be the one reporting the bug, and be prepared to explain it, justify it, and stand behind it. AI bug reports injected into the pipeline without human intervention are worse than useless.