Stubfiles placed alongside are ignored ? Neovim user

[wilgri]

Hi,

I don't manage to make pyright see .pyi stubfiles placed alongside their corresponding module in my codebase.

For example, if I have module_a.py and module_a.pyi in the same directory, pyright returns reportUnknownVariableType in module_a.py (in strict mode)

I've installed pyright in a python virtual env using pip. At the root directory of my project, I launch nvim after my venv has been activated

Below :

• my pyright config written in lsp/pyright.lua of the nvim config directory. It is the same given by nvim-lspconfig repo.
• my pyrightconfig.json.

I'm confused because python doc states :

the types bundled with the package SHOULD be used (be they in .pyi type stub files or inline in .py files)

... and Pycharm IDE understands this configuration by default. Also the numpy codebase seems to use this stubfiles arrangement.

What am I missing ? Do I understand things correctly ?

.config/nvim/lsp/pyright.lua

-- see : https://github.com/neovim/nvim-lspconfig/blob/master/lsp/pyright.lua

return {
	-- pyright is installed in a python venv, nvim must be launched after the correct venv has been activated
	cmd = { "pyright-langserver", "--stdio" },
	filetypes = { "python" },
	root_markers = {
		"pyproject.toml",
		"setup.py",
		"setup.cfg",
		"requirements.txt",
		"Pipfile",
		"pyrightconfig.json",
		".git",
	},
	settings = {
		python = {
			analysis = {
				autoSearchPaths = true,
				diagnosticMode = "openFilesOnly",
				useLibraryCodeForTypes = true,
			},
		},
	},
}

pyrightconfig.json

{
	"typeCheckingMode" : "strict",
	"python" : {
		"pythonPath" : "<path_to_my_venv_folder>/bin/python",
	},
}

[erictraut]

You're not using stub files correctly here. A stub file is a stand-in for a ".py" file in cases where the ".py" file is unannotated or the implementation is in a language other than Python. A stub file is used by a type checker when the module is imported. For example, if you have a stub file named module_a.pyi and another module has the statement import module_a, the type checker will use the stub rather than the ".py" file for type information. The stub file is not used to type check the contents of the ".py" file itself, since it doesn't import itself.

If your intent is to type check module_a.py, you should include type annotations directly in that file rather than creating a separate stub file.

[wilgri]

Thanks for your attention.

Your answer is confusing me... At least, I'm now pretty confident that it does not come from Neovim (good point to start with)

I've read again PEP 484 - stub files and PEP 561 entirely.

In PEP 484 it is said :

The easiest form of stub file storage and distribution is to put them alongside Python modules in the same directory. This makes them easy to find by both programmers and the tools.

PEP 561 says :

For package maintainers wishing to ship stub files containing all of their type information, it is preferred that the *.pyi stubs are alongside the corresponding *.py files.

Why do I have to care about which module is imported by whom ? Do you suggest that I have to mix inline and stub-files typing ? My current understanding makes me think that it is contradictory to the above PEP's specifications, no ?

To be clearer, if I have this toy project structure :

toy-package
└── pyproject.toml
└── pyrightconfig.toml
└── toy-package
    └── py.typed # eventually with partial\n
    └── __init__.py # from .module_a import f; __all__ = ["f"]
    └── __init__.pyi # from .module_a import f; __all__ = ["f"]
    └── module_a.py
    └── module_a.pyi

module_a.py

def f(x): # here pyright complains in strict mode and does not see typing in module_a.pyi
    return x + 1

module_a.pyi

def f(x : int) -> int: ...

my understanding of PEP 561 is that I can distribute toy-package with its typing and, when I'm working on the codebase, type checkers like Pyright might look at all typing put alongside.

There is definitely something that I've missed because Mypy does not seem to see module_a.pyi either...

I really don't want to do inline typing as on a bigger codebase I think it makes the code less readable, especially when I need to introduce type var or structural typing... and typing is, for me, a gradual work that I do in parallel. I would love to keep things separated.

Have you an idea of what is confusing me and, if possible, what can I do to configure my environment so that Pyright works as I expect ?

[erictraut]

There are two scenarios here:

Scenario 1: You want to type check a codebase that imports symbols from the toy-package library, but the toy-package library is either written in a language other than Python or is written in untyped Python. Stub files are designed for this scenario. You can create stub files and include them either within toy-package or as a separate "stub library". Consumers of toy-package can then benefit from type checking when they import symbols from the library. When a type checker sees a import statement like from toy_package.foo import bar, it will look for a toy_package/foo.pyi stub file before looking for a toy_package/foo.py file. If the stub file exists, it will use the stub definition for symbol bar rather than looking in the corresponding ".py" file.

Scenario 2: You want to type check the Python code in toy-package itself. In this case, stub files are not the correct tool. They will not help you here. Type checkers do not somehow "combine" the type information in a ".pyi" file with the type information in a ".py" file. If your goal is to run a type checker on the ".py" files in toy-package, you will need to add type annotations to the toy-package ".py" files. You can add type annotations gradually, but you need to add them to ".py" files themselves. Stub files will not help you here.

It sounds like you're talking about scenario 2, so stub files will not work here. As much as you may dislike inlined type annotations, that's the right tool for the job.

[wilgri]

Oh I finally get it ! Thank you for disambiguating the role of stub-files. I did not understand that it was meant to be only "consumed" by other project type checking. I'm definitely in scenario 2.

Would you have the time for one last question ?
For a pure Python project, in which case one would only want to distribute stub-files for third parties ? If one has taken the time to write stub-files, why not doing inline typing directly and get the benefits of type checking as a development helper ? Likewise, how can someone validate the "correctness" of his stub-files if his type checker can't use them ? If I look at Numpy codebase, I only see stub-files. Is it because it's not a pure Python project and the presence of C code makes inline typing difficult to work ? Is it for another more complex reason ?

[erictraut]

For pure Python projects, stub libraries are typically used in cases where the maintainer is not interested in adding or maintaining type information. It allows other members of the community to augment the library with a separately-maintained set of stubs. You'll find many such third-party stub libraries in the typeshed project.

Inlined type annotations are generally preferable for pure Python project. They benefit both library consumers and maintainers because they eliminate versioning, compatibility, discoverability, and distribution challenges inherent with stubs. Plus, with inlined type annotations, the library code itself can be statically type checked. Any regressions resulting from normal library maintenance (bug fixes, additions, etc.) can be caught quickly in a CI script.

For more details, refer to this documentation.

Libraries like numpy are a bit different because they are written (largely) in languages other than Python, so they need to use stubs for type information.

[wilgri]

Got it. Thank you so much for the time you have invested in answering my questions! It's very helpful.