scip-python: a precise Python indexer
We are excited to announce the release of scip-python
, a new SCIP indexer for Python built on top of the excellent Pyright type checker.
What does precise Python mean?
This represents a significant advance in Python navigation for Sourcegraph. Previously, we only supported search-based navigation for Python, which is convenient because it works out of the box and returns correct results for many cases, BUT is limited by:
- not understanding your dependencies
- not understanding type hints, and
- suffering from false negatives with variable names that appear frequently in different contexts.
Precise Python addresses all of those, giving you equivalent code navigation quality to what you get with the Python language server in VS Code or PyCharm (and of course Neovim đ).
Some of the key features of scip-python
include:
- Correctness: scip-python re-uses the type checking and analysis of
pyright
. - Cross-repository navigation: scip-python is designed to support navigating between your codebase and python libraries.
- Find Implementations scip-python takes steps to support advanced code navigation features, such as Find Implementations of Classes and Class Methods.
Correctness
pyright
supports a very impressive list of features. scip-python
was implemented with very few changes (primarily just exposing some internal functions) to the pyright
library, which allows scip-python
to take full advantage of the type checking and semantic analysis available in the pyright
library -- including new Python features as they are supported.
Cross-repository navigation
When scip-python
is run in a Python environment with your dependencies available it will generate stable references to your dependencies. For example, if you were using the PyYAML
packageâs dump
function for the 6.0 release, scip-python
would generate a symbol value of:
> scip-python python PyYAML 6.0 yaml/dump().
Assuming that you have also indexed either the PyYAML repo or the Python package repo for PyYAML 6.0, then when clicking on the identifier dump
in this situation:
Sourcegraph will take you to the definition in the corresponding PyYAML repository.
Find Implementations
There are two primary supported cases right now for âFind Implementationsâ via scip-python in Sourcegraph.
The first case is to examine when and where some âBaseClassâ is used as a parent class for another class. Instead of looking for references (which can show instantiation, isinstance
checks and more) the user may view the âImplementationsâ in Sourcegraph. This will only show child class definitions in Sourcegraph.
For a concrete example, see the rich package, where weâre viewing the MarkdownElement
.
The references of MarkdownElement are shown below:
However, if you were looking to find out what classes inherit from this element, we can use the Implementations pane
to view that information:
Additionally, if any child classes implement or override a particular class method they can be found using the âImplementationsâ panel in Sourcegraph. For a class method, it will only show child classes that have an explicit implementation in their definition, not wherever the function is called.
Get started with scip-python
Use the scip-python index
command to index a Python codebase.
For more information, please see the scip-python repository.
Projects ready to explore on Sourcegraph.com
We are currently indexing the following projects with scip-python
:
Feel free to navigate around in the code on Sourcegraph and leave any issues that you find on our repo.
Performance
The indexing performance is varied across codebases, particularly depending on the amount of dependencies and their usage across the project (scip-python will attempt to gain any possible hover documentation and references to these dependencies while indexing). Below is a snapshot of several different repos and their corresponding indexing time (including environment and dependency analysis).
Project | Lines of code | scip-python index(es) | Index loc/s |
---|---|---|---|
Django | 101218 | 39 | 2595 |
Rich | 18923 | 10 | 1892 |
Pre-commit | 4801 | 5 | 960 |
Scip-python is available today
Scip-python is available publicly today. If youâre a Sourcegraph user, you can implement scip-python to enable precise code navigation for your Python repositories.
If youâre not a Sourcegraph user but are interested in trying it out for navigating Python code (or a number of other languages), you can install it locally for free, or reach out to our team for a demo.