Thank you! This is definitely a design choice we believe strongly in.
Jazzberry starts by viewing the patch associated with the PR, but can choose to view any file in the repo as it searches for bugs. All the models in use are off-the-shelf for now, but we have been thinking about training a model from the beginning, stay tuned!
We largely agree, we don't think pure LLM-based approaches are sufficient. Having an LLM automatically orchestrate tools, like a software fuzzer, is something we've been thinking about for a while and we view incorporating code execution as the first step.
We think that LLMs are able to capture semantic bugs that traditional software testing cannot find in a hands-off way, and ideas from both worlds will be needed for a truly holistic bug finder.
We are laser focused on bug finding, and aren't targeting general code review, like comments on code style and variable names. We also run your code as part of bug finding instead of only having an LLM inspect it
Jazzberry starts by viewing the patch associated with the PR, but can choose to view any file in the repo as it searches for bugs. All the models in use are off-the-shelf for now, but we have been thinking about training a model from the beginning, stay tuned!