Percentage of code written (PCW) by Cody

In order to track Cody's impact, we measure the percentage of new code inserted into users’ editors that is generated by Cody. You can find this data in Sourcegraph Analytics.

What it measures

This metric demonstrates how much Cody is contributing to your development process—providing both real productivity gains and improved product velocity. But it's just one of a many metrics we offer to measure Cody's impact on your development workflow. For more, refer to the ROI page in your Sourcegraph Analytics portal.

How it’s measured

To calculate this metric we measure:

  1. Inserted code that Cody generates via:
  • Chat or prompt responses that are inserted into the editor
  • Accepted autocompletions
  • Suggestions/fixes
  1. Total new code inserted (which includes both user-generated and cody-generated code)

We then create a simple ratio:

(Inserted code that Cody generates ÷ Total new code inserted) * 100 = Percentage of code written by Cody

Note that components are measured in characters of code.

Technical details

When calculating this metric, we try to include only new code being written by the user or by Cody. This means we exclude insertions from, for example, pasting in large chunks of existing code from a different file, or pulling in changes via git pull.

It’s simple to identify which code is generated by Cody, but much more difficult to tell the difference between user-written code and code added to the file from one of the above methods given the data available from VS Code, JetBrains, and other editors. To account for this, we use a variety of heuristics to make a best-effort calculation. Here are a few specific specific heuristics:

  • We exclude large, rapid changes - users can type a few characters in 15ms, but if the insertion is much bigger, it's a sign that it's coming from a different source
  • We exclude disjoint insertions - insertions made in multiple places at once, which are likely to come from third party sources
  • We exclude any insertions made in an inactive file/editor window
  • We exclude any insertions into “stale” code - code the user hasn’t touched in the last 5 seconds