Metadata-Version: 2.4
Name: sage-pycharm-stubgen
Version: 0.9.15
Summary: Generate PyCharm-friendly type stubs from an installed SageMath package
Author: starnotes-xj
License-Expression: GPL-3.0-only
Project-URL: Homepage, https://github.com/starnotes-xj/sage-pycharm-stubgen
Project-URL: Repository, https://github.com/starnotes-xj/sage-pycharm-stubgen
Project-URL: Issues, https://github.com/starnotes-xj/sage-pycharm-stubgen/issues
Keywords: sagemath,pycharm,type-stubs,cython,wsl
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Software Development :: Code Generators
Classifier: Typing :: Stubs Only
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: stubgen-pyx<0.3,>=0.2.18
Dynamic: license-file

# Sage PyCharm Stubgen

[English](README.md) | [简体中文](README.zh-CN.md)

> Writing crypto Sage scripts? Stop switching to the source for every method
> name — completion and docs are right in the editor.

![stubgen demo](docs/demo.gif)

Generate and install static type stubs for SageMath so that PyCharm, Pyright,
Jedi, and other Python-aware editors can understand dynamic Sage APIs such as
`Mod`, `GF`, `PolynomialRing`, `matrix`, and `vector`.

**No PyCharm plugin is required for the type information itself.** The
generated `.pyi` files are installed next to the Sage runtime modules in the
active Python environment, which makes them visible to PyCharm's WSL
interpreter indexer without adding a Sources Root to every project.

IDE *features* built on top of the stubs — first-class `.sage` files and
**Sage postfix completion** (`expr.ZZ`, `.factor`, `.b2i`, ...) — live in the
companion [sage-ide-support](https://github.com/starnotes-xj/sage-ide-support)
JetBrains plugin, not in this package (see
[Configure PyCharm](#configure-pycharm)).

Tested with SageMath 10.9 on Python 3.13 in WSL. The generator discovers the
installed Sage version at runtime instead of relying on a fixed symbol list.

## Quick start

Inside the Sage environment (WSL / native / Docker), run:

```bash
python -m pip install sage-pycharm-stubgen
python -m sage_pycharm_stubgen --install --runtime-audit
```

`--install` also builds the Sage API index sidecar and installs it into every
JetBrains PyCharm configuration directory found on the machine (a WSL run
discovers the Windows-side config through `/mnt/c/Users/*`; `--sidecar-dir`
overrides the destination, `--no-sidecar` skips this step).

Then point PyCharm at that Sage interpreter (WSL SDK): completion and
Ctrl+Q work in any project without extra Sources Roots, because the stubs
sit next to the runtime modules.  For first-class `.sage` files (generator
sugar, implicit `sage.all` namespace, run via the `sage` command) and Sage
**postfix completion** (`expr.ZZ` / `.factor` / `.b2i` ...), also install the
[sage-ide-support](https://github.com/starnotes-xj/sage-ide-support) plugin.
Optional full-Chinese docs: `sage-pycharm-stubgen translate-docs --apply-only`.

## What it fixes

SageMath exposes many objects dynamically and ships a large amount of compiled
Cython code. A static analyzer may therefore report errors such as:

```text
Cannot find reference 'Mod' in 'all'
Cannot find reference 'load' in 'persist'
```

It may also fail to complete methods returned by factories:

```python
from sage.all import Mod

x = Mod(5, 29)
x.sqrt(all=True)
```

This project:

- converts installed `.pyx` files, with matching `.pxd` declarations when
  possible, into `.pyi` files using
  [`stubgen-pyx`](https://github.com/jon-edward/stubgen-pyx);
- with `--include-py` (0.8.3+), also renders every pure-Python module
  (`*.py`) into a complete `.pyi` — signatures and docstrings kept, function
  bodies stripped — so element chains through pure-Python functions
  (`for q in prime_divisors(p - 1)` → `q.nbits()`) are typed too; package
  inits, tests, dynamic distribution shims and the `all.py` re-export files
  are skipped automatically;
- generates explicit exports for the dynamic `sage.all` module;
- infers importable return types for dynamic factories from the active Sage
  environment;
- falls back from Cython source parsing to conservative runtime reflection when
  an individual extension cannot be parsed;
- bridges base classes that stubgen-pyx drops for old-style Parent classes so
  inherited members (`__getitem__`, `_first_ngens`, ...) stay reachable;
- declares Sage-specific members that static analysis cannot discover
  (`FiniteField.characteristic`, `Integer` arithmetic operators,
  `CategoryObject._first_ngens`);
- enriches every stub with docstrings so PyCharm's Quick Documentation
  (Ctrl+Q) explains what a function returns -- including curated Chinese
  docs with verified examples for CTF-critical APIs such as `GF`,
  `from_integer`/`to_integer`, `log`, `discrete_log`, `CRT`, and `xgcd`;
- validates every generated stub before installation;
- preserves existing Sage-owned or user-owned `.pyi` files;
- tracks its own files in a manifest so updates and uninstallations only touch
  files owned by this tool, refuses downgrade installs from older tool
  versions, and prefers source-declared factory return annotations (Sage
  upstream PRs #42670/#42672) over runtime probing.

## Requirements

- SageMath installed in a Python/Conda environment
- Python 3.11 or newer
- PyCharm configured to use that same interpreter (WSL is supported)
- For first-class `.sage` files in PyCharm (dedicated file type, Sage sugar
  parsing, implicit `sage.all` namespace) and Sage postfix completion
  (`expr.ZZ` / `.b2i` ...): the companion
  [sage-ide-support](https://github.com/starnotes-xj/sage-ide-support)
  plugin (PyCharm 2026.1+), which consumes the stubs installed here as its
  type-information data layer

Run all installation commands with Sage's Python interpreter. A normal Windows
Python installation cannot inspect a Sage environment installed in WSL.

The Chinese curated docs, the finite-field element-class return annotations,
and the downgrade protection ship since **0.7.0**; **0.8.0** adds the
opt-in machine-translation layer (a bundled shared cache plus the
`translate-docs` command below).  **0.8.3** adds pure-Python module stubs
(`--include-py`): with it, `sage-pycharm-stubgen --include-py --install`
also types every function defined in pure-Python modules (`prime_divisors`,
`divisors`, ...), not only the compiled ones.

**Prefer English docs?** Run `sage-pycharm-stubgen --doc-language en
--install`: the curated Chinese docstrings are skipped (the original
English source docs stay) while every language-neutral type annotation is
still applied.

## Install

Inside the Sage environment:

```bash
conda activate sage
python -m pip install sage-pycharm-stubgen
sage-pycharm-stubgen --include-py --install
```

To install the current development version directly from GitHub instead, use:

```bash
python -m pip install "git+https://github.com/starnotes-xj/sage-pycharm-stubgen.git"
```

`--install` performs generation, strict validation, and installation in one
command. The generated build tree is stored at
`<current-environment>/sage_typings`, while the stubs used by the IDE are
placed beside the installed Sage modules, for example:

```text
<environment>/lib/pythonX.Y/site-packages/sage/all.pyi
<environment>/lib/pythonX.Y/site-packages/sage/misc/persist.pyi
```

The tool never overwrites `.py`, `.pyx`, or compiled extension files. It also
does not overwrite pre-existing `.pyi` files that it does not own.

## Configure PyCharm

1. Select the same WSL/Conda Python interpreter used above.
2. Remove any manually added `sage_typings` Sources Root from the project.
3. Refresh the interpreter package list.
4. If an old error remains cached, use **File → Invalidate Caches / Restart**.

The stubs then apply to every project using that interpreter. No custom
PyCharm plugin is needed for `.py` files. For first-class `.sage` files (own
file type, sugar-aware parsing, implicit `sage.all` namespace, typed
`F.<a> = GF(2^8)` completion) and Sage postfix completion (`expr.ZZ`,
`.factor`, `.b2i`, ... — configurable in **Settings → Editor → Postfix
Completion**), install the companion
[sage-ide-support](https://github.com/starnotes-xj/sage-ide-support) plugin;
alternatively, see [Preparsing Sage syntax](#preparsing-sage-syntax) for
converting Sage sugar to plain Python.

Press Ctrl+Q on any Sage function name to see its description, return type,
and examples (Chinese docs for 700+ common APIs); after a first install or
upgrade, use **File → Invalidate Caches / Restart** if the docs do not show
up.

## Configure VS Code

1. Install the official **WSL**, **Python**, and **Pylance** extensions.
2. Open the project in a WSL window, for example by running `code .` in Ubuntu.
3. Select the same Sage Python interpreter used to install the stubs.
4. Open a Python file and request completion after `x.` in the example above.

The generated stubs have been verified with Pyright 1.1.411: it resolves `x`
as `IntegerMod_abstract`, exposes the `sqrt` signature, and resolves
`sage.misc.persist.load` without errors. VS Code users do not need to install
the Pyright CLI separately when using Pylance.

Associating `*.sage` with the Python language can provide ordinary Python API
completion, but Pylance does not understand Sage preparser-only syntax such as
`R.<x> = PolynomialRing(...)`. Convert such files with the
[`preparse` command](#preparsing-sage-syntax) first.

## Update and uninstall

Run the same command after upgrading SageMath:

```bash
sage-pycharm-stubgen --install
```

Remove only the stubs owned by this tool:

```bash
sage-pycharm-stubgen --uninstall
```

If the environment already contains stale third-party `.pyi` files on the
same paths (older stub generators leave them behind), the installer
preserves them by default.  Take them over explicitly with
`--install --overwrite-unowned`; each replaced file is backed up as
`<name>.pyi.sps-bak` and restored by `--uninstall`.

The install manifest records the generator version.  Running `--install`
with an **older** tool version than the one that produced the current
stubs is refused — overwriting would silently lose the newer tool's
documentation enrichment (curated Chinese docs and return annotations).
Upgrade the tool (`pip install -U sage-pycharm-stubgen`) or pass
`--force` to bypass.  The curated Chinese doc table is also shipped next
to the installed stubs (`.sage-pycharm-stubgen-curated-docs.json`) as an
audit/recovery artifact.

## Preparsing Sage syntax

Sage's preparser sugar (`R.<x> = GF(2)[]`, `F.<a> = GF(2^8, ...)`, `^` as
power, `e^(-1)`) is only expanded in `.sage` files. A `.py` file using that
syntax is invalid Python, so PyCharm cannot parse it at all, let alone index
it. Convert a file to plain Python:

```bash
sage-pycharm-stubgen preparse test.py
```

The file is rewritten in place (atomically, keeping a
`test.py.preparse-backup` copy of the original). The conversion expands
generator declarations, powers, and numeric literals exactly as Sage itself
would, and inserts `from sage.all import *` when Sage symbols are used
without it — `.py` files do not get the implicit namespace injection that
`.sage` files receive from the Sage command.

Combined with the generated stubs, static analysis then resolves the
converted file end to end: `F` is typed as `FiniteField`, `a` and `x` come
from `_first_ngens`, and `from_integer`, `to_integer`, `polynomial`,
`characteristic`, and friends all complete. Verified with Pyright: zero
errors on a converted AES finite-field exercise.

Options:

- `--check` — only report files that still need conversion; exit code 1 if
  any do (useful in scripts and CI).
- `--output DIR` — write converted copies into `DIR` instead of rewriting in
  place.
- `--no-backup` — do not keep a `.preparse-backup` copy.

Several files can be converted at once:

```bash
sage-pycharm-stubgen preparse a.py b.py c.py
```

## Documentation enrichment

PyCharm's Quick Documentation reads the docstring *body* of a stub function,
so the generator repairs and fills docstrings during generation from three
sources, in priority order:

1. **Curated docs** (`supplemental_docs.py`) -- Chinese explanations with
   verified `sage:` examples and precise return annotations for 700+
   CTF-critical APIs (finite fields, polynomial rings, modular arithmetic,
   elliptic curves, matrices, number-theory tools), every example executed
   against the installed Sage before being written.  Regenerate the file
   with `python tools/build_supplemental_docs.py <research-output.json>`
   and merge new research with
   `python tools/merge_supplemental_docs.py <research-output.json>`.
2. **Source docstrings** -- extracted from the installed `.pyx` sources,
   including `cpdef`/`cdef` functions whose Cython return types upgrade
   `-> Any` where the stub's imports allow it.
3. **Runtime docstrings** -- the live Sage environment is imported module by
   module and `inspect.getdoc` fills what the sources leave out (inherited
   and decorator-built docstrings).  This import sweep takes minutes;
   disable it with `--no-runtime-docs`.

The pass also moves docstrings that stubgen-pyx emits as standalone string
statements *after* `def ...: ...` into the function body, which is the only
placement PyCharm's stub indexer associates with the function.

### Machine-translating the remaining docs (opt-in)

The curated layer covers the CTF-critical APIs; everything else keeps its
English source docstring.  To additionally translate **all** remaining
English docstrings into Chinese:

```bash
sage-pycharm-stubgen translate-docs --apply-only
```

This is an **explicit, opt-in command** — it never runs automatically
during `--install`, so English-only users are unaffected unless they invoke
it.  It applies every translation from two sources, merged together:

1. the **shared cache shipped inside the package** (the accumulated batch
   translation of the whole Sage API, ~12k docstrings — instant, no
   network needed; disable with `--no-bundled`), and
2. your own persistent cache (`~/.sage-pycharm-stubgen/translations.json`),
   which wins over the shipped one where they overlap.

Docstrings missing from both caches can be translated on demand with a
live backend (`--backend google`, or `--backend baidu` with
`BAIDU_APPID`/`BAIDU_API_KEY` env vars; `--workers N` parallelizes the
Baidu LLM requests).  Runs are resumable after interruptions, and
re-applying the cache is instant:

```bash
sage-pycharm-stubgen translate-docs
sage-pycharm-stubgen translate-docs --apply-only
```

Machine translations are readable but unverified; the curated entries
remain the only layer whose examples are executed against Sage.

## Runtime audit

`--runtime-audit` reconciles the generated stubs against the live Sage
runtime after enrichment and merge, and appends declarations for public
members the source scan missed.  Resolution tiers for a stub class, first
hit wins: module namespace, `_with_category`/`_with_realization` dynamic
variants, ancestor package namespaces, same-module nested classes
(stubgen flattens Sage's inner element classes), then a `sys.modules` deep
scan restricted to exact `__module__` matches.  Signature tiers: real
`inspect` signatures, then `sage_getdef`/docstring/`.pyx`-source recovery,
then a `(*args, **kwargs)` placeholder.  Members already declared on the
class or any stub base are skipped (no flattened `Any` shadows an ancestor's
precise return), and `class X: ...` inline bodies are expanded when the
runtime class has members to add.

Skip taxonomy after all tiers (measured on Sage 10.9): optional-dependency
modules, inline classes whose runtime has nothing addable, and flattened
nested helpers with no direct usage path — these keep their converted
content and are intentionally not audited.

## Conformance check

The curated layer patches return types that upstream Sage does not
declare.  As upstream annotations land (the factory-annotation PRs), those
patches become checkable claims:

```bash
sage-pycharm-stubgen conformance
```

For every curated return fix it reports whether the installed Sage source
declares the same type (`ok`), disagrees (`conflict` — either the data
layer is stale or the upstream annotation is wrong, both actionable), or
declares nothing (`unannotated` — the patch is still needed).  `--json`
emits one object per finding for automation.  `--runtime` additionally
executes each entry's doctest examples against the live Sage runtime and
reports declared types that do not fit the observed result.

## Duplicate-declaration merge and product lint

Several independent writers decorate a generated stub (the converter,
curated `declare` entries, the arithmetic enhancers, the alias
promoter).  Since 0.9.0 the pipeline ends with two safety passes over the
final artifact, so one writer can no longer silently fight another:

- **Merge** (`sage_pycharm_stubgen.merge`) — every name in every class is
  declared exactly once.  An `@overload` group wins over any plain `def`
  or assignment with the same name; among plain defs the best-annotated
  one wins; a `def` wins over a same-named assignment.  This is the fix
  for the 0.8.x artifacts where `Integer.__mul__`/`__pow__` were declared
  twice and the trailing plain `def` shadowed the overload set, typing
  `ZZ(5) * QQ(...)` as `Integer`.
- **Lint** (`sage_pycharm_stubgen.lint`) — reports structural errors on
  the final tree: `overload-shadow`, `duplicate-member`,
  `member-shadowed-by-assignment`, and warnings for annotations that
  reference names the stub neither imports nor declares, plus curated
  entries that did not land.  `--strict` and `--install` refuse to ship
  a tree with lint errors; every finding is listed in
  `generation-report.json`.

## Verifying what is actually installed

```bash
sage-pycharm-stubgen verify
```

Recomputes the SHA-256 of every installed stub file against the
installation manifest and reports `MISSING`, `MODIFIED` and `UNOWNED`
`.pyi` files.  The manifest (format 2) records per-file hashes plus the
generator's git commit, so "the version says 0.8.4 but the logic
differs" becomes a checkable claim instead of archaeology.

## Advanced generation

To generate a small test subset without installing it:
```bash
sage-pycharm-stubgen \
  --pattern 'rings/finite_rings/integer_mod.pyx' \
  --output ./sage_typings_test
```

The output contains a `generation-report.json` and the generated `sage/` stub
tree. Factory inference details are written to
`sage/factory-inference.json`.

## Limitations

No static stub generator can promise one perfectly precise type for every
possible dynamic Python call. A function may choose its return type from
argument values, plugins, files, network state, or classes created at runtime.
This tool instead requires every detected factory to have an importable static
return type during strict installation. Factories with several implementations
use a common importable base class whose members are valid for the observed
implementations.

## Companion projects

- [sage-ide-support](https://github.com/starnotes-xj/sage-ide-support) — a
  PyCharm plugin giving `.sage` files first-class support: a dedicated "Sage"
  file type (Python dialect), transparent Sage generator-sugar parsing
  (`F.<a> = GF(2^8, ...)`), the runtime-injected `sage.all` namespace for
  static analysis, run configurations (native/WSL/Docker), live templates,
  and Sage postfix completion (`expr.ZZ` / `.factor` / `.euler_phi` / `.b2i`
  ... with configurable templates).  It consumes the `.pyi` stubs generated
  and installed by this tool; all type information and curated documentation
  live here, in the stub data layer.
- [JetBrains/intellij-community PR #3614](https://github.com/JetBrains/intellij-community/pull/3614) —
  a generic `typeInformationGenerator` extension point that lets PyCharm
  discover, invoke and refresh this engine for Sage interpreters
  (local/WSL/Docker).
- [sagemath/sage PR #42670](https://github.com/sagemath/sage/pull/42670) and
  [#42672](https://github.com/sagemath/sage/pull/42672) — upstream Sage
  annotations for the `GF`/`Zmod` factories and the `FiniteField`
  element-returning methods; once merged, this engine prefers those
  source-declared types over its runtime probing.

## Development

```bash
python -m pip install -e .
python -m unittest discover -s tests -v
python -m compileall -q src tests
```

## License

[GPL-3.0](LICENSE). Derivative works must remain open under the same
license, and the original copyright notice must be preserved.
