In my open source work on Apache DataFusion, I’ve seen this growing trend of people finding an open issue and having an AI agent author the complete solution with no real input from the author. We’ve had plenty of discussion about this in the different project mailing lists, issue boards, and during our meetings.

I’m going to try making this post actually helpful, so if you want skip my venting just scroll down to the section “How to work differently.”

This post was written entirely by a human.

Venting

I see the majority of these PRs are from people who have very little experience. Many of the github users have almost no history and many even explicitly state how they’re students. I suspect these people are trying to be helpful, but they simply are not. Here’s the thing:

I could have dropped that issue into an agent just as easily.

The biggest problem we have in these projects is getting reviewer time. It’s been the bottleneck the entire time I’ve been involved in DataFusion. Most of use are extremely busy people. Today I had 3 different PRs all addressing the same problem and all 3 were just an AI dump. So now I’m spending my time trying to figure out if there was any human quality checking what those agents did or if it was just blindly posted.

Agents frequently do the wrong thing! I’ve got access to some of the best models on the market and I frequently have to correct the agent when it hallucinates about the solution or solves the wrong problem. I wrote a blog post about solving the right problem before, and I feel like it’s gotten a hundred times worse in the LLM era.

So these people are trying to be helpful or are trying to get some credentials about their contribution, but instead they’re just putting even more work on the maintainers to wade through their “contributions.”

How to work differently

I know we are in the era of the agent coders. I am using them, and so is almost everyone else. I’m going to try suggestions based on how I actually work on problems myself these days.

Try to understand the problem before asking an agent to just solve it. Read the issue. See if you, as a human, understand why it isn’t working as it should. The key word in that last sentence is why. It’s important to not just reproduce the problem but to identify the root cause of the problem.

Make sure the agent solved the right problem. It’s entirely possible the issue was written incorrectly! Especially issues that were written by agents, because it’s original evaluation may be wrong. I have taken to having agents open issues for follow on work as I go along, and when I come back to these I do not trust the issues that were written on my own behalf. I know when I come back I may have a different view, and I also know the original analysis may be wrong.

If the agent solved the problem wrong or solved the wrong problem, don’t just squash all of your commit history. Let that show in the PR. That is a strong indicator to the reviewers that this was something that you put effort into. When I see a single commit PR, that’s usually and indicator that the author did just post agent output without evaluation.

Show your work. – Every math school teacher I ever had

Rewrite the PR description in human readable prose. This is a huge indicator that you not only understand what the agent did, but it greatly improves the experience for the reviewer. There may come a time when we cannot tell the difference between agent prose and human, but that’s not the case right now. Writing the PR description yourself will usually make the review by others much more easy to understand.

I give a pass on the question of the PR description to people who have been active members of the community – basically people with write access to the repository. I look at these as someone I’ve already built the trust that they’ve reviewed the work appropriately.

Lastly, I find it incredibly helpful for new contributors to add comments in the code review. I myself do this when I want to point out to reviewers where they should focus their effort.

Closing thoughts

Throwing a raw issue into an agent and posting the result is not helpful.

Actually, it’s hurtful because it’s making more work for the code maintainers to try to evaluate if they should even spend their own time looking at the contribution.

If you want to really contribute, try to understand the issue and demonstrate that you’ve put real effort into making sure the agent wrote the right thing. Make sure it solved the real problem.

When you understand the problem and the solution, it will put you in a better position to understand the code base and to make more meaningful changes in the future.