PR Guard: Utilising GitHub Actions to Promote Responsible AI Usage

What is PR Guard

PR Guard is a first attempt at using AI to tackle the issue of PR proliferation! To be exact, PR Guard aims to prevent PRs in which people are submitting code they don’t truly understand.

It’s not an effort to detect AI generated code, instead it’s an effort to push the authors of PRs to understand what they are contribiting, all the hwile taking off the onus on reviewers to understand AI assisted PRs.

It works by taking the diff of a PR, passing it to an LLM, the LLM generates 3 questions and posts these as a comment on the PR. The author must then answer the questions, the answers being passed back to the LLM which determines if the author understands the diff or not - if they do it goes green if not then another comment is posted with the LLMs reason for not marking it as a pass.

My Takeaways + Future Work

It remains to be seen how this will work in a large codebase, where the diff impacts not only the file it is in but completely different parts of the codebase. Also, the ability of an LLM to understand complex code was drawn into question - how would you test this? Also, the obvious issue that someone can just use an LLM to answer the questions, although my response would be to assert PR Guard depends heavily on developer culture.

For future work, it’d be worth testing PR Guard on large codebases and including specific parts of the codebase (perhaps based on dependencies of code file where diff is). Also, more work on getting an LLM to understand a codebase and figuring out what that means, perhaps adding an interpreter here would be useful such that the LLM can run and play around with the code.