Improving diff success rate

[djex]

I'm creating this discussion as a means to document my findings and to open up discussion with others on this. All ideas, suggestions and contributions are welcome.

I have found that there are a few key issues AI chat models have when generating proper diff patch formatting which are:

• Line counts are often wrong on diff patches especially when diff patches are large. This is a common issue with AI models as they can not count very well currently.
• AI models have issues with producing exact white space and line breaks.

I have found that it is likely the AI will generate proper code that will compile and runs in a diff patch however the items listed above will prevent the patch from being applied if one or more are present.

I have been working on improving the success rate of apply patches generated by AI models and will be posting my results here.

Line Counts

• Using the --recount flag with git apply usually will allow a patch with incorrect line counts to apply.
• Idea: Provide the line counts of each file when submitting the initial prompt. This might help but will use more tokens and it is best to use as little tokens as possible for more accurate responses.

White Space and Line Breaks

• White spaces will be removed or added in a patch which will differ from the original code. This may only be solvable with a custom diff patch processor.
  For example:
  Original code: var test_string = some_variable + ' ' + some_other_var
  Patch code: test_string=some_variable + ' ' +some_other_var
• There is an issue I found where the AI will produce a line like + \n which should be +\n. This will cause git apply to fail even when --whitespace=fix --ignore-whitespace are passed to git apply. If we do a bit of pre-processing on the generated patch before passing it to git apply we can clean up issues like this.

function clean_up_git_patch(patchContent: string): string 
{
  const lines = patchContent.split('\n');
  let output = "";
 
  for (let i = 0; i < lines.length; i++) 
  {
    const line = lines[i];

    let cleanedLine = "";

    // Clean up all trailing whitespace to prevent git apply from failing
    // Fixes issue: "+       /n" casuing trailing whitespace error -> "+\n"
    cleanedLine = line.trimEnd();
    
    output += cleanedLine + '\n';
  }

  // Discard a single newline keeping a single newline to make git apply happy
  return output.slice(0, -1);
}

Using wiggle to apply rejected patches or as a replacement for git apply

• Using --reject with git apply will produce .rej files of any rejected patches that it could not apply. Then using wiggle --replace to apply the rejected patches works but on large patches it can fail.
• Using wiggle as a replacement for git apply. This may work better than solution below. Not sure yet. . Wiggle always wants line endings that match the original code so normalizing the diff patch to \n causes issues with wiggle. Decided to go for the custom patch processor approach.
• wiggle is currently only available on Linux. I have compiled a copy for Windows and am currently testing it.

Writing a custom diff patch processor [Currently testing]

Update: I have implemented a custom patch processor and so far it's working quite well (much better than using git apply or wiggle). Still testing and will update.

This may end up being the final answer as it will allow for better control over cleaning up, parsing and applying the patches when there are formatting issues with the generated patch.

An idea would be to normalize a copy of both the original code and diff patch and then treat the diff as a search and replace ignoring the generated patch line counts. Something like this might work?

• Remove all white space from each line
• Remove all empty newlines
• Iterate through patch treating - as a search and + as a replace. Special logic would be needed for lines starting with a space " "
• Use an algorithm like levenshtein distance to calculate the similarity between the search text and code lines in the original code. Not needed unless the diff patch is really badly malformed and then in that case we shouldn't even bother to fix it.
• Would need some way to track original line indexes for replacing code when a match is found.

[robertpiosik]

I love this comprehensive overview! What do you think should be our action points on this?

[djex]

Thanks. I have just finished up a custom diff patch processor and am testing it currently. I have it integrated into CwC and it's working well but I'm still testing to make sure it's 100% then I'll make a pull request for review if you would like.

[djex]

#250