Feb 22, 2024
The advent of AI technologies, especially with the release of ChatGPT by Open AI in November 2022, has significantly impacted the software development landscape.
At Mibex Software, our experience of designing and selling developer productivity apps meant that we were especially keen on exploring use cases for AI, in particular for Code Review processes. We wanted to challenge the ability of AI to overcome traditional problems of the code review process such as subjective feedback, inconsistency and time consumption.
In this context, we've worked on and identified several use cases where AI can make a substantial difference and created custom implementations relying on OpenAI’s API. Although the bulk of the demand and appeal for using AI in software development still stems from code generation like GitHub Copilot, code review use cases also appeal to most actors of the industry. Leading code-hosting platforms GitHub, GitLab and Bitbucket have indeed also explored AI features for code reviews.
GitHub launched a research project called Copilot for Pull Requests that includes several ideas of use cases, some of which have or will soon find their way into Copilot or other GitHub products.
GitLab has been rolling out AI features within GitLab Duo and is testing new ones in beta and even experimental mode.
Bitbucket has so far only included limited generative AI features through Atlassian Intelligence but mentioned working on Code Review with AI features that have not yet been announced.
For the use cases we explored, we will provide a short summary of our experience and what GitHub, GitLab and Bitbucket have already released or plan to work on.
1. Automating PR Descriptions:
AI tools can generate concise summaries, thereby enhancing understanding and efficiency.
Our experience:
The AI is great at summarizing and generating clear and helpful content from text input. As such, we have observed great results, in particular for the generation of Pull Request titles and descriptions based on the code diff and commit titles and other context like Jira issue descriptions. Also, generating visuals based on the PR content like Gitmoji's works well with AI and helps PR reviewers to understand what a PR is about.
How GitHub, GitLab and Bitbucket are approaching it:
For GitHub, the first AI feature for Pull Requests that made it into GitHub Copilot features is the generation of Pull Request Summaries, showing that the confidence in the usability of AI for this use case is quite high.
GitLab is testing two features relating to this use case: merge request template population to fill in your merge request template description section and summary of merge request change to provide you with a description of the changes found from the code diff.
For Bitbucket, the currently available AI feature in beta lets you use generative AI for descriptions or summaries but you need to pass the context (commits, code diffs) manually.
2. Code Improvement Suggestions:
AI can detect security vulnerabilities and suggest code improvements, broadening the scope of reviews.
Our experience:
We found that having the AI review the code of a Pull Request can be helpful, in particular when providing line-specific suggestions that you can easily accept or reject. However, as you are limited in the context you can provide, the AI lacks the full picture to truly assess the code and its impact. In terms of security analysis, we found in particular a high possibility of false positive or incomplete fixes suggestions. In the current state of things, while using the AI for this use case could be a nice addition to your existing processes, the suggestions need to be thoroughly reviewed and absolutely do not replace static code analysis tools.
How GitHub, GitLab and Bitbucket are approaching it:
During the 'Copilot for Pull Request' project, GitHub investigated the potential of using AI to review Pull Requests, but this feature was not made publicly available. However, in November 2023, GitHub launched a beta version of the code scanning autofix feature. This enhancement to GitHub's scanning capabilities allows for the automatic review of vulnerabilities and provides suggestions for improvements in JavaScript or TypeScript code. This development indicates GitHub's ongoing exploration of AI-assisted code review, with more advancements likely on the horizon.
GitLab also worked on vulnerability scanning as well with beta features already available allowing for vulnerability explanation and vulnerability resolution.
Bitbucket does not have built-in code scanning features yet, you can however get suggestions if you copy the code to review and ask Atlassian Intelligence for suggestions through a comment (as demonstrated at the end of this video).
3. Categorization of PRs:
AI can classify pull requests based on various criterias, helping reviewers to see what they should focus on in their review (e.g. PRs with "security-relevant" label). This can also improve the organization of PRs, i.e. which team should review a PR.
Our experience:
We were able to get good results for this use case, providing the AI with a list of possible labels and the criterias to be applied.
How GitHub, GitLab and Bitbucket are approaching it:
No platform has yet addressed this particular use case.
Other interesting use cases explored right now are:
- Code Explanation: the AI breaks down code snippets or files and explain it to facilitate the review and understanding. A feature called walkthrough can be expected to be included in GitHub Copilot Enterprise and GitLab Duo has a feature called Code Explanation.
- Test Cases Generation: the AI provides test cases based on its understanding of code snippets to help achieve higher test coverage and code quality. This is already available in IDEs for Copilot and GitLab Duo and this use case is also explored within the Pull Request workflow as mentioned by Github.
- Discussions Summary: the AI sums up the discussions that have happened between reviewers in the Pull Request to get a clear recap of talking points and decisions made. See GitLab experimentation on this.
- Issues Resolution: the AI analyzes issues and proposes ways to fix them by generating new code. This is a feature explored by GitHub as initially explained here and elaborated on in Copilot Workspace.
The result of our experiments showed us that AI can definitely be a nice help in your code review process by addressing certain use cases with satisfying results. As long as you remember that AI is a helper with its limitations and need for oversight, the value added it can provide is definitely real.
At Mibex, we are rolling out AI features for these use cases through our app Code Review Assistant that you can already check out.
At the same time, we are eagerly monitoring the fast development of platform native features and their marketplace apps. A lot of companies have great use cases ideas for code reviews with AI and innovative ways of implementing them.
This is an exciting time and we strongly believe that our code review processes will be made more efficient by the use of AI in a matter of months, allowing for humans to maximize the value added of manual reviews.
Stay tuned for more insights and updates as we delve deeper into the potential of AI in software development.