<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
If a try-catch block guards the names, we don't raise warnings. During
this change, I discovered that some of the replacement types were
missed. Thus, I extend the fix to types other than AutoImport as well
## Test Plan
<!-- How was it tested? -->
Test fixtures are added and updated.
## Summary
Updates the `--python` flag to accept Python executables in virtual
environments. Notably, we do not query the executable and it _must_ be
in a canonical location in a virtual environment. This is pretty naive,
but solves for the trivial case of `ty check --python .venv/bin/python3`
which will be a common mistake (and `ty check --python $(which python)`)
I explored this while trying to understand Python discovery in ty in
service of https://github.com/astral-sh/ty/issues/272, I'm not attached
to it, but figure it's worth sharing.
As an alternative, we can add more variants to the
`SearchPathValidationError` and just improve the _error_ message, i.e.,
by hinting that this looks like a virtual environment and suggesting the
concrete alternative path they should provide. We'll probably want to do
that for some other cases anyway (e.g., `3.13` as described in the
linked issue)
This functionality is also briefly mentioned in
https://github.com/astral-sh/ty/issues/193
Closes https://github.com/astral-sh/ty/issues/318
## Test Plan
e.g.,
```
uv run ty check --python .venv/bin/python3
```
needs test coverage still
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
The existing implementation of RUF060 (InEmptyCollection) is not
recursive, meaning that although set([]) results in an empty collection,
the existing code fails it because set is taking an argument.
The updated implementation allows set and frozenset to take empty
collection as positional argument (which results in empty
set/frozenset).
## Test Plan
Added test cases for recursive cases + updated snapshot (see RUF060.py).
---------
Co-authored-by: Marcus Näslund <marcus.naslund@kognity.com>
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
Fixes#17776.
This PR also handles all other `PTH*` rules that don't support file
descriptors.
## Test Plan
<!-- How was it tested? -->
Update existing tests.
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
We can only guarantee the safety of the autofix for number literals, all
other cases may change the runtime behaviour of the program or introduce
a syntax error. For the cases reported in the issue that would result in
a syntax error, I disabled the autofix.
Follow-up of #17661.
Fixes#16472.
<!-- What's the purpose of the change? What does it do, and why? -->
## Test Plan
Snapshot tests.
<!-- How was it tested? -->
Function literals have an optional specialization, which is applied to
the parameter/return type annotations lazily when the function's
signature is requested. We were previously only applying this
specialization to the final overload of an overloaded function.
This manifested most visibly for `list.__add__`, which has an overloaded
definition in the typeshed:
b398b83631/crates/ty_vendored/vendor/typeshed/stdlib/builtins.pyi (L1069-L1072)
Closes https://github.com/astral-sh/ty/issues/314
## Summary
I found this bug while working on #18041. The following code leads to
infinite recursion.
```python
from ty_extensions import is_disjoint_from, static_assert, TypeOf
class C:
@property
def prop(self) -> int:
return 1
static_assert(not is_disjoint_from(int, TypeOf[C.prop]))
```
The cause is a trivial missing binding in `is_disjoint_from`. This PR
fixes the bug and adds a test case (this is a simple fix and may not
require a new test case?).
## Test Plan
A new test case is added to
`mdtest/type_properties/is_disjoint_from.md`.
## Summary
Added version 3.14 to the script generating the `known_stdlib.rs` file.
Rebuilt the known stdlibs with latest version (2025.5.10) of [stdlibs
Python lib](https://pypi.org/project/stdlibs/) (which added support for
3.14.0b1).
_Note: Python 3.14 is now in [feature
freeze](https://peps.python.org/pep-0745/) so the modules in stdlib
should be stable._
_See also: #15506_
## Test Plan
The following command has been run. Using for tests the `compression`
module which been introduced with Python 3.14.
```sh
ruff check --no-cache --select I001 --target-version py314 --fix
```
With ruff 0.11.9:
```python
import base64
import datetime
import compression
print(base64, compression, datetime)
```
With this PR:
```python
import base64
import compression
import datetime
print(base64, compression, datetime)
```
## Summary
`KnownClass::Range`, `KnownInstanceType::Any` and `ClassBase::any()` are
no longer used or useful: all our tests pass with them removed.
`KnownModule::Abc` _is_ now used outside of tests, however, so I removed
the `#[allow(dead_code)]` branch above that variant.
## Test Plan
`cargo test -p ty_python_semantic`
Following #17991, removes some of
https://github.com/astral-sh/ruff/pull/17222 which is no longer strictly
necessary. I don't actually think it's that ugly to have around? no
strong feelings on retaining it or not.
Follow-up to https://github.com/astral-sh/ruff/pull/17991 ensuring we do
not allow detection of system environments when the origin is
`VIRTUAL_ENV` or a discovered `.venv` directory — i.e., those always
require a `pyvenv.cfg` file.
Adds test coverage for https://github.com/astral-sh/ruff/pull/17991,
which includes some minor refactoring of the virtual environment test
infrastructure.
I tried to minimize stylistic changes, but there are still a few because
I was a little confused by the setup. I could see this evolving more in
the future, as I don't think the existing model can capture all the test
coverage I'm looking for.
This adds basic support for non-virtual Python environments by accepting
a directory without a `pyvenv.cfg` which allows existing, subsequent
site-packages discovery logic to succeed. We can do better here in the
long-term, by adding more eager validation (for error messages) and
parsing the Python version from the discovered site-packages directory
(which isn't relevant yet, because we don't use the discovered Python
version from virtual environments as the default `--python-version` yet
either).
Related
- https://github.com/astral-sh/ty/issues/265
- https://github.com/astral-sh/ty/issues/193
You can review this commit by commit if it makes you happy.
I tested this manually; I think refactoring the test setup is going to
be a bit more invasive so I'll stack it on top (see
https://github.com/astral-sh/ruff/pull/17996).
```
❯ uv run ty check --python /Users/zb/.local/share/uv/python/cpython-3.10.17-macos-aarch64-none/ -vv example
2025-05-09 12:06:33.685911 DEBUG Version: 0.0.0-alpha.7 (f9c4c8999 2025-05-08)
2025-05-09 12:06:33.685987 DEBUG Architecture: aarch64, OS: macos, case-sensitive: case-insensitive
2025-05-09 12:06:33.686002 DEBUG Searching for a project in '/Users/zb/workspace/ty'
2025-05-09 12:06:33.686123 DEBUG Resolving requires-python constraint: `>=3.8`
2025-05-09 12:06:33.686129 DEBUG Resolved requires-python constraint to: 3.8
2025-05-09 12:06:33.686142 DEBUG Project without `tool.ty` section: '/Users/zb/workspace/ty'
2025-05-09 12:06:33.686147 DEBUG Searching for a user-level configuration at `/Users/zb/.config/ty/ty.toml`
2025-05-09 12:06:33.686156 INFO Defaulting to python-platform `darwin`
2025-05-09 12:06:33.68636 INFO Python version: Python 3.8, platform: darwin
2025-05-09 12:06:33.686375 DEBUG Adding first-party search path '/Users/zb/workspace/ty'
2025-05-09 12:06:33.68638 DEBUG Using vendored stdlib
2025-05-09 12:06:33.686634 DEBUG Discovering site-packages paths from sys-prefix `/Users/zb/.local/share/uv/python/cpython-3.10.17-macos-aarch64-none` (`--python` argument')
2025-05-09 12:06:33.686667 DEBUG Attempting to parse virtual environment metadata at '/Users/zb/.local/share/uv/python/cpython-3.10.17-macos-aarch64-none/pyvenv.cfg'
2025-05-09 12:06:33.686671 DEBUG Searching for site-packages directory in `sys.prefix` path `/Users/zb/.local/share/uv/python/cpython-3.10.17-macos-aarch64-none`
2025-05-09 12:06:33.686702 DEBUG Resolved site-packages directories for this environment are: ["/Users/zb/.local/share/uv/python/cpython-3.10.17-macos-aarch64-none/lib/python3.10/site-packages"]
2025-05-09 12:06:33.686706 DEBUG Adding site-packages search path '/Users/zb/.local/share/uv/python/cpython-3.10.17-macos-aarch64-none/lib/python3.10/site-packages'
...
❯ uv run ty check --python /tmp -vv example
2025-05-09 15:36:10.819416 DEBUG Version: 0.0.0-alpha.7 (f9c4c8999 2025-05-08)
2025-05-09 15:36:10.819708 DEBUG Architecture: aarch64, OS: macos, case-sensitive: case-insensitive
2025-05-09 15:36:10.820118 DEBUG Searching for a project in '/Users/zb/workspace/ty'
2025-05-09 15:36:10.821652 DEBUG Resolving requires-python constraint: `>=3.8`
2025-05-09 15:36:10.821667 DEBUG Resolved requires-python constraint to: 3.8
2025-05-09 15:36:10.8217 DEBUG Project without `tool.ty` section: '/Users/zb/workspace/ty'
2025-05-09 15:36:10.821888 DEBUG Searching for a user-level configuration at `/Users/zb/.config/ty/ty.toml`
2025-05-09 15:36:10.822072 INFO Defaulting to python-platform `darwin`
2025-05-09 15:36:10.822439 INFO Python version: Python 3.8, platform: darwin
2025-05-09 15:36:10.822773 DEBUG Adding first-party search path '/Users/zb/workspace/ty'
2025-05-09 15:36:10.822929 DEBUG Using vendored stdlib
2025-05-09 15:36:10.829872 DEBUG Discovering site-packages paths from sys-prefix `/tmp` (`--python` argument')
2025-05-09 15:36:10.829911 DEBUG Attempting to parse virtual environment metadata at '/private/tmp/pyvenv.cfg'
2025-05-09 15:36:10.829917 DEBUG Searching for site-packages directory in `sys.prefix` path `/private/tmp`
ty failed
Cause: Invalid search path settings
Cause: Failed to discover the site-packages directory: Failed to search the `lib` directory of the Python installation at `sys.prefix` path `/private/tmp` for `site-packages`
```
## Summary
Suppress false positives for uses of PEP-695 `ParamSpec` in `Callable`
annotations:
```py
from typing_extensions import Callable
def f[**P](c: Callable[P, int]):
pass
```
addresses a comment here:
https://github.com/astral-sh/ty/issues/157#issuecomment-2859284721
## Test Plan
Adapted Markdown tests
Summary
--
This should resolve the formatter ecosystem errors we've been seeing
lately. https://github.com/mesonbuild/meson-python/pull/728 added the
links, which I think are intentionally broken for testing purposes.
Test Plan
--
Ecosystem check on this PR
Re: #17526
## Summary
Add integration test for semantic syntax for `IrrefutableCasePattern`,
`SingleStarredAssignment`, `WriteToDebug`, and `InvalidExpression`.
## Notes
- Following @ntBre's suggestion, I will keep the test coming in batches
like this over the next few days in separate PRs to keep the review load
per PR manageable while also not spamming too many.
- I did not add a test for `del __debug__` which is one of the examples
in `crates/ruff_python_parser/src/semantic_errors.rs:1051`.
For python version `<= 3.8` there is no error and for `>=3.9` the error
is not `WriteToDebug` but `SyntaxError: cannot delete __debug__ on
Python 3.9 (syntax was removed in 3.9)`.
- The `blacken-docs` bypass is necessary because otherwise the test does
not pass pre-commit checks; but we want to check for this faulty syntax.
<!-- What's the purpose of the change? What does it do, and why? -->
## Test Plan
This is a test.
Summary
--
This was suggested on Discord, I hope this is roughly what we had in
mind. I took the message from the ty README, but I'm more than happy to
update it. Otherwise I just tried to mimic the appearance of the `ruff
analyze graph` warning (although I'm realizing now the whole text is
bold for ruff).
Test Plan
--
New warnings in the CLI tests. I thought this might be undesirable but
it looks like uv did the same thing
(https://github.com/astral-sh/uv/pull/6166).
This makes one very simple change: we report all call binding
errors from each union variant.
This does result in duplicate-seeming diagnostics. For example,
when two union variants are invalid for the same reason.
## Summary
Adds a simple progress bar for the `ty check` CLI command. The style is
taken from uv, and like uv the bar is always shown - for smaller
projects it is fast enough that it isn't noticeable. We could
alternatively hide it completely based on some heuristic for the number
of files, or only show it after some amount of time.
I also disabled it when `--watch` is passed, cancelling inflight checks
was leading to zombie progress bars. I think we can fix this by using
[`MultiProgress`](https://docs.rs/indicatif/latest/indicatif/struct.MultiProgress.html)
and managing all the bars globally, but I left that out for now.
Resolves https://github.com/astral-sh/ty/issues/98.
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
* `airflow.models.Connection` → `airflow.sdk.Connection`
* `airflow.models.Variable` → `airflow.sdk.Variable`
## Test Plan
<!-- How was it tested? -->
The test fixtures has been updated (see the first commit for easier
review)
This does a deeper removal of the `lint:` prefix by removing the
`DiagnosticId::as_str` method and replacing it with `as_concise_str`. We
remove the associated error type and simplify the `Display` impl for
`DiagnosticId` as well.
This turned out to catch a `lint:` that was still in the diagnostic
output: the part that says why a lint is enabled.
We just set the ID on the `Message` and it just does what we want in
this case. I think I didn't do this originally because I was trying to
preserve the existing rendering? I'm not sure. I might have just missed
this method.
In a subsequent commit, we're going to start using `annotate-snippets`'s
functionality for diagnostic IDs in the rendering. As part of doing
that, I wanted to remove this special casing of an empty message. I did
that independently to see what, if anything, would change. (The changes
look fine to me. They'll be tweaked again in the next commit along with
a bunch of others.)
## Summary
Use a self-reference "marker" ~~and fixpoint iteration~~ to solve the
stack overflow problems with recursive protocols. This is not pretty and
somewhat tedious, but seems to work fine. Much better than all my
fixpoint-iteration attempts anyway.
closes https://github.com/astral-sh/ty/issues/93
## Test Plan
New Markdown tests.
## Summary
Add cycle handling for `try_metaclass` and `pep695_generic_context`
queries, as well as adjusting the cycle handling for `try_mro` to ensure
that it short-circuits on cycles and won't grow MROs indefinitely.
This reduces the number of failing fuzzer seeds from 68 to 17. The
latter count includes fuzzer seeds 120, 160, and 335, all of which
previously panicked but now either hang or are very slow; I've
temporarily skipped those seeds in the fuzzer until I can dig into that
slowness further.
This also allows us to move some more ecosystem projects from `bad.txt`
to `good.txt`, which I've done in
https://github.com/astral-sh/ruff/pull/17903
## Test Plan
Added mdtests.
@AlexWaygood pointed out that the `SliceLiteral` type variant was
originally created to handle slices before we had generics.
https://github.com/astral-sh/ruff/pull/17927#discussion_r2078115787
Now that we _do_ have generics, we can use a specialization of the
`slice` builtin type for slice literals.
This depends on https://github.com/astral-sh/ruff/pull/17956, since we
need to make sure that all typevar defaults are fully substituted when
specializing `slice`.
It's possible for a typevar to list another typevar as its default
value:
```py
class C[T, U = T]: ...
```
When specializing this class, if a type isn't provided for `U`, we would
previously use the default as-is, leaving an unspecialized `T` typevar
in the specialization. Instead, we want to use what `T` is mapped to as
the type of `U`.
```py
reveal_type(C()) # revealed: C[Unknown, Unknown]
reveal_type(C[int]()) # revealed: C[int, int]
reveal_type(C[int, str]()) # revealed: C[int, str]
```
This is especially important for the `slice` built-in type.
## Summary
This PR is a first step toward integration of the new `Diagnostic` type
into ruff. There are two main changes:
- A new `UnifiedFile` enum wrapping `File` for red-knot and a
`SourceFile` for ruff
- ruff's `Message::SyntaxError` variant is now a `Diagnostic` instead of
a `SyntaxErrorMessage`
The second of these changes was mostly just a proof of concept for the
first, and it went pretty smoothly. Converting `DiagnosticMessage`s will
be most of the work in replacing `Message` entirely.
## Test Plan
Existing tests, which show no changes.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
Co-authored-by: Micha Reiser <micha@reiser.io>
Summary
--
This PR resolves both the typing-related and syntax error TODOs added in
#17563 by tracking a set of `global` bindings for each scope. As
discussed below, we avoid the additional AST traversal from ruff by
collecting `Name`s from `global` statements while building the semantic
index and emit a syntax error if the `Name` is already bound in the
current scope at the point of the `global` statement. This has the
downside of separating the error from the `SemanticSyntaxChecker`, but I
plan to explore using this approach in the `SemanticSyntaxChecker`
itself as a follow-up. It seems like this may be a better approach for
ruff as well.
Test Plan
--
Updated all of the related mdtests to remove the TODOs (and add quotes I
forgot on the messages).
There is one remaining TODO, but it requires `nonlocal` support, which
isn't even incorporated into the `SemanticSyntaxChecker` yet.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
Fixes: astral-sh/ty#159
This PR adds support for using `Self` in methods.
When the type of an annotation is `TypingSelf` it is converted to a type
var based on:
https://typing.python.org/en/latest/spec/generics.html#self
I just skipped Protocols because it had more problems and the tests was
not useful.
Also I need to create a follow up PR that implicitly assumes `self`
argument has type `Self`.
In order to infer the type in the `in_type_expression` method I needed
to have scope id and semantic index available. I used the idea from
[this PR](https://github.com/astral-sh/ruff/pull/17589/files) to pass
additional context to this method.
Also I think in all places that `in_type_expression` is called we need
to have this context because `Self` can be there so I didn't split the
method into one version with context and one without.
## Test Plan
Added new tests from spec.
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
Co-authored-by: Carl Meyer <carl@astral.sh>
#17897 added variance handling for legacy typevars — but they were only
being considered when checking generic aliases of the same class:
```py
class A: ...
class B(A): ...
class C[T]: ...
static_assert(is_subtype_of(C[B], C[A]))
```
and not for generic subclasses:
```py
class D[U](C[U]): ...
static_assert(is_subtype_of(D[B], C[A]))
```
Now we check those too!
Closes https://github.com/astral-sh/ty/issues/101
Fixes#17867
## Summary
The CPython parser does not allow generator expressions which are the
sole arguments in an argument list to have a trailing comma.
With this change, we start flagging such instances.
## Test Plan
Added new inline tests.
## Summary
We now expect the client to send initialization options to opt-in to
experimental (but LSP-standardized) features, like completion support.
Specifically, the client should set `"experimental.completions.enable":
true`.
Closes https://github.com/astral-sh/ty/issues/74.
## Summary
This PR adds support for the `__all__` module variable.
Reference spec:
https://typing.python.org/en/latest/spec/distributing.html#library-interface-public-and-private-symbols
This PR adds a new `dunder_all_names` query that returns a set of
`Name`s defined in the `__all__` variable of the given `File`. The query
works by implementing the `StatementVisitor` and collects all the names
by recognizing the supported idioms as mentioned in the spec. Any idiom
that's not recognized are ignored.
The current implementation is minimum to what's required for us to
remove all the false positives that this is causing. Refer to the
"Follow-ups" section below to see what we can do next. I'll a open
separate issue to keep track of them.
Closes: astral-sh/ty#106Closes: astral-sh/ty#199
### Follow-ups
* Diagnostics:
* Add warning diagnostics for unrecognized `__all__` idioms, `__all__`
containing non-string element
* Add an error diagnostic for elements that are present in `__all__` but
not defined in the module. This could lead to runtime error
* Maybe we should return `<type>` instead of `Unknown | <type>` for
`module.__all__`. For example:
https://playknot.ruff.rs/2a6fe5d7-4e16-45b1-8ec3-d79f2d4ca894
* Mark a symbol that's mentioned in `__all__` as used otherwise it could
raise (possibly in the future) "unused-name" diagnostic
Supporting diagnostics will require that we update the return type of
the query to be something other than `Option<FxHashSet<Name>>`,
something that behaves like a result and provides a way to check whether
a name exists in `__all__`, loop over elements in `__all__`, loop over
the invalid elements, etc.
## Ecosystem analysis
The following are the maximum amount of diagnostics **removed** in the
ecosystem:
* "Type <module '...'> has no attribute ..."
* `collections.abc` - 14
* `numpy` - 35534
* `numpy.ma` - 296
* `numpy.char` - 37
* `numpy.testing` - 175
* `hashlib` - 311
* `scipy.fft` - 2
* `scipy.stats` - 38
* "Module '...' has no member ..."
* `collections.abc` - 85
* `numpy` - 508
* `numpy.testing` - 741
* `hashlib` - 36
* `scipy.stats` - 68
* `scipy.interpolate` - 7
* `scipy.signal` - 5
The following modules have dynamic `__all__` definition, so `ty` assumes
that `__all__` doesn't exists in that module:
* `scipy.stats`
(95a5d6ea8b/scipy/stats/__init__.py (L665))
* `scipy.interpolate`
(95a5d6ea8b/scipy/interpolate/__init__.py (L221))
* `scipy.signal` (indirectly via
95a5d6ea8b/scipy/signal/_signal_api.py (L30))
* `numpy.testing`
(de784cd6ee/numpy/testing/__init__.py (L16-L18))
~There's this one category of **false positives** that have been added:~
Fixed the false positives by also ignoring `__all__` from a module that
uses unrecognized idioms.
<details><summary>Details about the false postivie:</summary>
<p>
The `scipy.stats` module has dynamic `__all__` and it imports a bunch of
symbols via star imports. Some of those modules have a mix of valid and
invalid `__all__` idioms. For example, in
95a5d6ea8b/scipy/stats/distributions.py (L18-L24),
2 out of 4 `__all__` idioms are invalid but currently `ty` recognizes
two of them and says that the module has a `__all__` with 5 values. This
leads to around **2055** newly added false positives of the form:
```
Type <module 'scipy.stats'> has no attribute ...
```
I think the fix here is to completely ignore `__all__`, not only if
there are invalid elements in it, but also if there are unrecognized
idioms used in the module.
</p>
</details>
## Test Plan
Add a bunch of test cases using the new `ty_extensions.dunder_all_names`
function to extract a module's `__all__` names.
Update various test cases to remove false positives around `*` imports
and re-export convention.
Add new test cases for named import behavior as `*` imports covers all
of it already (thanks Alex!).
## Summary
Fixes#17541
Before this change, in the case of overloaded functions,
`@dataclass_transform` was detected only when applied to the
implementation, not the overloads.
However, the spec also allows this decorator to be applied to any of the
overloads as well.
With this PR, we start handling `@dataclass_transform`s applied to
overloads.
## Test Plan
Fixed existing TODOs in the test suite.
## Summary
This is sort of an anticlimactic resolution to #17863, but now that we
understand what the root cause for the stack overflows was, I think it's
fine to enable running on this project. See the linked ticket for the
full analysis.
closes#17863
## Test Plan
Ran lots of times locally and never observed a crash at worker thread
stack sizes > 8 MiB.
We now track the variance of each typevar, and obey the `covariant` and
`contravariant` parameters to the legacy `TypeVar` constructor. We still
don't yet infer variance for PEP-695 typevars or for the
`infer_variance` legacy constructor parameter.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
A recursive protocol like the following would previously lead to stack
overflows when attempting to create the union type for the `P | None`
member, because `UnionBuilder` checks if element types are fully static,
and the fully-static check on `P` would in turn list all members and
check whether all of them were fully static, leading to a cycle.
```py
from __future__ import annotations
from typing import Protocol
class P(Protocol):
parent: P | None
```
Here, we make the fully-static check on protocols a salsa query and add
fixpoint iteration, starting with `true` as the initial value (assume
that the recursive protocol is fully-static). If the recursive protocol
has any non-fully-static members, we still return `false` when
re-executing the query (see newly added tests).
closes#17861
## Test Plan
Added regression test
## Summary
Resolves#15502.
`ty generate-shell-completion` now works in a similar manner to `ruff
generate-shell-completion`.
## Test Plan
Manually:
<details>
```shell
$ cargo run --package ty generate-shell-completion nushell
module completions {
# An extremely fast Python type checker.
export extern ty [
--help(-h) # Print help
--version(-V) # Print version
]
# ...
}
export use completions *
```
</details>
@AlexWaygood discovered that even though we've been propagating
specializations to _parent_ base classes correctly, we haven't been
passing them on to _grandparent_ base classes:
https://github.com/astral-sh/ruff/pull/17832#issuecomment-2854360969
```py
class Bar[T]:
x: T
class Baz[T](Bar[T]): ...
class Spam[T](Baz[T]): ...
reveal_type(Spam[int]().x) # revealed: `T`, but should be `int`
```
This PR updates the MRO machinery to apply the current specialization
when starting to iterate the MRO of each base class.
## Summary
This PR partially addresses #16418 via the following:
- `LinterSettings::unresolved_python_version` is now a `TargetVersion`,
which is a thin wrapper around an `Option<PythonVersion>`
- `Checker::target_version` now calls `TargetVersion::linter_version`
internally, which in turn uses `unwrap_or_default` to preserve the
current default behavior
- Calls to the parser now call `TargetVersion::parser_version`, which
calls `unwrap_or_else(PythonVersion::latest)`
- The `Checker`'s implementation of
`SemanticSyntaxContext::python_version` also uses
`TargetVersion::parser_version` to use `PythonVersion::latest` for
semantic errors
In short, all lint rule behavior should be unchanged, but we default to
the latest Python version for the new syntax errors, which should
minimize confusing version-related syntax errors for users without a
version configured.
## Test Plan
Existing tests, which showed no changes (except for printing default
settings).
## Summary
Introducing a new rule based on discussions in #15732 and #15729 that
checks for unnecessary in with empty collections.
I called it in_empty_collection and gave the rule number RUF060.
Rule is in preview group.
e.g.,
```
❯ uv run -q -- ty -V
ty 0.0.0-alpha.4 (08881edba 2025-05-05)
❯ uv run -q -- ty --version
ty 0.0.0-alpha.4 (08881edba 2025-05-05)
```
Previously, this just displayed `ty 0.0.0` because it didn't use our
custom version implementation. We no longer have a short version —
matching the interface in uv. We could add a variant for it, if it seems
important to people. However, I think we found it more confusing than
not over there and didn't get any complaints about the change.
Closes https://github.com/astral-sh/ty/issues/54
Extends https://github.com/astral-sh/ruff/pull/17866, using
`dist-workspace.toml` as a source of truth for versions to enable
version retrieval in distributions that are not Git repositories (i.e.,
Python source distributions and source tarballs consumed by Linux
distros).
I retain the Git tag lookup from
https://github.com/astral-sh/ruff/pull/17866 as a fallback — it seems
harmless, but we could drop it to simplify things here.
I confirmed this works from the repository as well as Python source and
binary distributions:
```
❯ uv run --refresh-package ty --reinstall-package ty -q -- ty version
ty 0.0.1-alpha.1+5 (2eadc9e61 2025-05-05)
❯ uv build
...
❯ uvx --from ty@dist/ty-0.0.0a1.tar.gz --no-cache -q -- ty version
ty 0.0.1-alpha.1
❯ uvx --from ty@dist/ty-0.0.0a1-py3-none-macosx_11_0_arm64.whl -q -- ty version
ty 0.0.1-alpha.1
```
Requires https://github.com/astral-sh/ty/pull/36
cc @Gankra and @MichaReiser for review.
Currently, `ty version` pulls its information from the Ruff repository —
but we want this to pull from the repository in the directory _above_
when Ruff is a submodule.
I tested this in the `ty` repository after tagging an arbitrary commit:
```
❯ uv run --refresh-package ty --reinstall-package ty ty version
Built ty @ file:///Users/zb/workspace/ty
Uninstalled 1 package in 2ms
Installed 1 package in 1ms
ty 0.0.0+3 (34253b1d4 2025-05-05)
```
We also use the last Git tag as the source of truth for the version,
instead of the crate version. However, we'll need a way to set the
version for releases still, as the tag is published _after_ the build.
We can either tag early (without pushing the tag to the remote), or add
another environment variable. (**Note, this approach is changed in a
follow-up. See https://github.com/astral-sh/ruff/pull/17868**)
From this repository, the version will be `unknown`:
```
❯ cargo run -q --bin ty -- version
ty unknown
```
We could add special handling like... `ty unknown (ruff@...)` but I see
that as a secondary goal.
Closes https://github.com/astral-sh/ty/issues/5
The reviewer situation in this repository is unhinged, cc @Gankra and
@MichaReiser for review.
## Summary
This fixes some false positives that showed up in the primer diff for
https://github.com/astral-sh/ruff/pull/17832
## Test Plan
new mdtests added that fail with false-positive diagnostics on `main`
## Summary
This PR fixes#17595.
## Test Plan
New test cases are added to `mdtest/narrow/conditionals/nested.md`.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
If a typevar is declared as having a default, we shouldn't require a
type to be specified for that typevar when explicitly specializing a
generic class:
```py
class WithDefault[T, U = int]: ...
reveal_type(WithDefault[str]()) # revealed: WithDefault[str, int]
```
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Fixes
https://github.com/astral-sh/ruff/pull/17832#issuecomment-2851224968. We
had a comment that we did not need to apply specializations to generic
aliases, or to the bound `self` of a bound method, because they were
already specialized. But they might be specialized with a type variable,
which _does_ need to be specialized, in the case of a "multi-step"
specialization, such as:
```py
class LinkedList[T]: ...
class C[U]:
def method(self) -> LinkedList[U]:
return LinkedList[U]()
```
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
## Summary
closes#17472
This is obviously just a band-aid solution to this problem (in that you
can always make your [pathological
inputs](28994edd82/sympy/polys/numberfields/resolvent_lookup.py)
bigger and it will still crash), but I think this is not an unreasonable
change — even if we add more sophisticated solutions later. I tried
using `stacker` as suggested by @MichaReiser, and it works. But it's
unclear where exactly would be the right place to put it, and even for
the `sympy` problem, we would need to add it both in the semantic index
builder AST traversal and in type inference. Increasing the default
stack size for worker threads, as proposed here, doesn't solve the
underlying problem (that there is a hard limit), but it is more
universal in the sense that it is not specific to large binary-operator
expression chains.
To determine a reasonable stack size, I created files that look like
*right associative*:
```py
from typing import reveal_type
total = (1 + (1 + (1 + (1 + (… + 1)))))
reveal_type(total)
```
*left associative*
```py
from typing import reveal_type
total = 1 + 1 + 1 + 1 + … + 1
reveal_type(total)
```
with a variable amount of operands (`N`). I then chose the stack size
large enough to still be able to handle cases that existing type
checkers can not:
```
right
N = 20: mypy takes ~ 1min
N = 350: pyright crashes with a stack overflow (mypy fails with "too many nested parentheses")
N = 800: ty(main) infers Literal[800] instantly
N = 1000: ty(main) crashes with "thread '<unknown>' has overflowed its stack"
N = 7000: ty(this branch) infers Literal[7000] instantly
N = 8000+: ty(this branch) crashes
left
N = 300: pyright emits "Maximum parse depth exceeded; break expression into smaller sub-expressions"
total is inferred as Unknown
N = 5500: mypy crashes with "INTERNAL ERROR"
N = 2500: ty(main) infers Literal[2500] instantly
N = 3000: ty(main) crashes with "thread '<unknown>' has overflowed its stack"
N = 22000: ty(this branch) infers Literal[22000] instantly
N = 23000+: ty(this branch) crashes
```
## Test Plan
New regression test.
This fixes cycle panics in several ecosystem projects (moved to
`good.txt` in a following PR
https://github.com/astral-sh/ruff/pull/17834 because our mypy-primer job
doesn't handle it well if we move projects to `good.txt` in the same PR
that fixes `ty` to handle them), as well as in the minimal case in the
added mdtest. It also fixes a number of panicking fuzzer seeds. It
doesn't appear to cause any regression in any ecosystem project or any
fuzzer seed.
The PR add the fix safety section for rule `RUF013`
(https://github.com/astral-sh/ruff/issues/15584 )
The fix was introduced here #4831
The rule as a lot of False Negative (as it is explained in the docs of
the rule).
The main reason because the fix is unsafe is that it could change code
generation tools behaviour, as in the example here:
```python
def generate_api_docs(func):
hints = get_type_hints(func)
for param, hint in hints.items():
if is_optional_type(hint):
print(f"Parameter '{param}' is optional")
else:
print(f"Parameter '{param}' is required")
# Before fix
def create_user(name: str, roles: list[str] = None):
pass
# After fix
def create_user(name: str, roles: Optional[list[str]] = None):
pass
# Generated docs would change from "roles is required" to "roles is optional"
```
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
Re: #17526
## Summary
Add test fixtures for `AwaitOutsideAsync` and
`AsyncComprehensionOutsideAsyncFunction` errors.
<!-- What's the purpose of the change? What does it do, and why? -->
## Test Plan
This is a test.
<!-- How was it tested? -->
Re: #17526
## Summary
Add integration tests for Python Semantic Syntax for
`InvalidStarExpression`, `DuplicateMatchKey`, and
`DuplicateMatchClassAttribute`.
## Note
- Red knot integration tests for `DuplicateMatchKey` exist already in
line 89-101.
<!-- What's the purpose of the change? What does it do, and why? -->
## Test Plan
This is a test.
<!-- How was it tested? -->
When attempting to determine whether `import foo.bar.baz` is a known
first-party import relative to [user-provided source
paths](https://docs.astral.sh/ruff/settings/#src), when `preview` is
enabled we now check that `SRC/foo/bar/baz` is a directory or
`SRC/foo/bar/baz.py` or `SRC/foo/bar/baz.pyi` exist.
Previously, we just checked the analogous thing for `SRC/foo`, but this
can be misleading in situations with disjoint namespace packages that
share a common base name (e.g. we may be working inside the namespace
package `foo.buzz` and importing `foo.bar` from elsewhere).
Supersedes #12987Closes#12984
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
Fixes#17798
<!-- What's the purpose of the change? What does it do, and why? -->
## Test Plan
Snapshot tests
<!-- How was it tested? -->
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
Skip attribute check in try catch block (`AIR301`)
## Test Plan
<!-- How was it tested? -->
update
`crates/ruff_linter/resources/test/fixtures/airflow/AIR301_names_try.py`
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
Remove `airflow.utils.dag_parsing_context.get_parsing_context` from
AIR301 as it has been moved to AIR311
## Test Plan
<!-- How was it tested? -->
the test fixture was updated in the previous PR
## Summary
When entering an `infer_expression_types` cycle from
`TypeInferenceBuilder::infer_standalone_expression`, we might get back a
`TypeInference::cycle_fallback(…)` that doesn't actually contain any new
types, but instead it contains a `cycle_fallback_type` which is set to
`Some(Type::Never)`. When calling `self.extend(…)`, we therefore don't
really pull in a type for the expression we're interested in. This
caused us to panic if we tried to call `self.expression_type(…)` after
`self.extend(…)`.
The proposed fix here is to retrieve that type from the nested
`TypeInferenceBuilder` directly, which will correctly fall back to
`cycle_fallback_type`.
## Details
I minimized the second example from #17792 a bit further and used this
example for debugging:
```py
from __future__ import annotations
class C: ...
def f(arg: C):
pass
x, _ = f(1)
assert x
```
This is self-referential because when we check the assignment statement
`x, _ = f(1)`, we need to look up the signature of `f`. Since evaluation
of annotations is deferred, we look up the public type of `C` for the
`arg` parameter. The public use of `C` is visibility-constraint by "`x`"
via the `assert` statement. While evaluating this constraint, we need to
look up the type of `x`, which in turn leads us back to the `x, _ =
f(1)` definition.
The reason why this only showed up in the relatively peculiar case with
unpack assignments is the code here:
78b4c3ccf1/crates/ty_python_semantic/src/types/infer.rs (L2709-L2718)
For a non-unpack assignment like `x = f(1)`, we would not try to infer
the right-hand side eagerly. Instead, we would enter a
`infer_definition_types` cycle that handles the situation correctly. For
unpack assignments, however, we try to infer the type of `value`
(`f(1)`) and therefore enter the cycle via `standalone_expression_type
=> infer_expression_type`.
closes#17792
## Test Plan
* New regression test
* Made sure that we can now run successfully on scipy => see #17850
This PR updates the semantic model for Python 3.14 by essentially
equating "run using Python 3.14" with "uses `from __future__ import
annotations`".
While this is not technically correct under the hood, it appears to be
correct for the purposes of our semantic model. That is: from the point
of view of deciding when to parse, bind, etc. annotations, these two
contexts behave the same. More generally these contexts behave the same
unless you are performing some kind of introspection like the following:
Without future import:
```pycon
>>> from annotationlib import get_annotations,Format
>>> def foo()->Bar:...
...
>>> get_annotations(foo,format=Format.FORWARDREF)
{'return': ForwardRef('Bar')}
>>> get_annotations(foo,format=Format.STRING)
{'return': 'Bar'}
>>> get_annotations(foo,format=Format.VALUE)
Traceback (most recent call last):
[...]
NameError: name 'Bar' is not defined
>>> get_annotations(foo)
Traceback (most recent call last):
[...]
NameError: name 'Bar' is not defined
```
With future import:
```
>>> from __future__ import annotations
>>> from annotationlib import get_annotations,Format
>>> def foo()->Bar:...
...
>>> get_annotations(foo,format=Format.FORWARDREF)
{'return': 'Bar'}
>>> get_annotations(foo,format=Format.STRING)
{'return': 'Bar'}
>>> get_annotations(foo,format=Format.VALUE)
{'return': 'Bar'}
>>> get_annotations(foo)
{'return': 'Bar'}
```
(Note: the result of the last call to `get_annotations` in these
examples relies on the fact that, as of this writing, the default value
for `format` is `Format.VALUE`).
If one day we support lint rules targeting code that introspects using
the new `annotationlib`, then it is possible we will need to revisit our
approximation.
Closes#15100
## Summary
Currently red-knot does not understand `Foo` and `Bar` here as being
equivalent:
```py
from typing import Protocol
class A: ...
class B: ...
class C: ...
class Foo(Protocol):
x: A | B | C
class Bar(Protocol):
x: B | A | C
```
Nor does it understand `A | B | Foo` as being equivalent to `Bar | B |
A`. This PR fixes that.
## Test Plan
new mdtest assertions added that fail on `main`
## Summary
Currently this assertion fails on `main`, because we do not synthesize a
`__call__` attribute for Callable types:
```py
from typing import Protocol, Callable
from knot_extensions import static_assert, is_assignable_to
class Foo(Protocol):
def __call__(self, x: int, /) -> str: ...
static_assert(is_assignable_to(Callable[[int], str], Foo))
```
This PR fixes that.
See previous discussion about this in
https://github.com/astral-sh/ruff/pull/16493#discussion_r1985098508 and
https://github.com/astral-sh/ruff/pull/17682#issuecomment-2839527750
## Test Plan
Existing mdtests updated; a couple of new ones added.
This adds support for legacy generic classes, which use a
`typing.Generic` base class, or which inherit from another generic class
that has been specialized with legacy typevars.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
Quick follow-on to #17788. If there is no bound `self` parameter, we can
reuse the existing `CallArgument{,Type}s`, and we can use a straight
`Vec` instead of a `VecDeque`.
## Summary
Remove mutability in parameter types for a few functions such as
`with_self` and `try_call`. I tried the `Rc`-approach with cheap cloning
[suggest
here](https://github.com/astral-sh/ruff/pull/17733#discussion_r2068722860)
first, but it turns out we need a whole stack of prepended arguments
(there can be [both `self` *and*
`cls`](3cf44e401a/crates/red_knot_python_semantic/resources/mdtest/call/constructor.md (L113))),
and we would need the same construct not just for `CallArguments` but
also for `CallArgumentTypes`. At that point we're cloning `VecDeque`s
anyway, so the overhead of cloning the whole `VecDeque` with all
arguments didn't seem to justify the additional code complexity.
## Benchmarks
Benchmarks on tomllib, black, jinja, isort seem neutral.
## Summary
Add the ability to detect instance attribute assignments in class
methods that are generic.
This does not address the code duplication mentioned in #16928. I can
open a ticket for this after this has been merged.
closes#16928
## Test Plan
Added regression test.
This PR does the wiring necessary to respond to completion requests from
LSP clients.
As far as the actual completion results go, they are nearly about the
dumbest and simplest thing we can do: we simply return a de-duplicated
list of all identifiers from the current module.
Summary
--
Fixes#16598 by adding the `--python` flag to `ruff analyze graph`,
which adds a `PythonPath` to the `SearchPathSettings` for module
resolution. For the [albatross-virtual-workspace] example from the uv
repo, this updates the output from the initial issue:
```shell
> ruff analyze graph packages/albatross
{
"packages/albatross/check_installed_albatross.py": [
"packages/albatross/src/albatross/__init__.py"
],
"packages/albatross/src/albatross/__init__.py": []
}
```
To include both the the workspace `bird_feeder` import _and_ the
third-party `tqdm` import in the output:
```shell
> myruff analyze graph packages/albatross --python .venv
{
"packages/albatross/check_installed_albatross.py": [
"packages/albatross/src/albatross/__init__.py"
],
"packages/albatross/src/albatross/__init__.py": [
".venv/lib/python3.12/site-packages/tqdm/__init__.py",
"packages/bird-feeder/src/bird_feeder/__init__.py"
]
}
```
Note the hash in the uv link! I was temporarily very confused why my
local tests were showing an `iniconfig` import instead of `tqdm` until I
realized that the example has been updated on the uv main branch, which
I had locally.
Test Plan
--
A new integration test with a stripped down venv based on the
`albatross` example.
[albatross-virtual-workspace]:
aa629c4a54/scripts/workspaces/albatross-virtual-workspace
## Summary
Adds preliminary support for `NamedTuple`s, including:
* No false positives when constructing a `NamedTuple` object
* Correct signature for the synthesized `__new__` method, i.e. proper
checking of constructor calls
* A patched MRO (`NamedTuple` => `tuple`), mainly to make type inference
of named attributes possible, but also to better reflect the runtime
MRO.
All of this works:
```py
from typing import NamedTuple
class Person(NamedTuple):
id: int
name: str
age: int | None = None
alice = Person(1, "Alice", 42)
alice = Person(id=1, name="Alice", age=42)
reveal_type(alice.id) # revealed: int
reveal_type(alice.name) # revealed: str
reveal_type(alice.age) # revealed: int | None
# error: [missing-argument]
Person(3)
# error: [too-many-positional-arguments]
Person(3, "Eve", 99, "extra")
# error: [invalid-argument-type]
Person(id="3", name="Eve")
```
Not included:
* type inference for index-based access.
* support for the functional `MyTuple = NamedTuple("MyTuple", […])`
syntax
## Test Plan
New Markdown tests
## Ecosystem analysis
```
Diagnostic Analysis Report
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┓
┃ Diagnostic ID ┃ Severity ┃ Removed ┃ Added ┃ Net Change ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━┩
│ lint:call-non-callable │ error │ 0 │ 3 │ +3 │
│ lint:call-possibly-unbound-method │ warning │ 0 │ 4 │ +4 │
│ lint:invalid-argument-type │ error │ 0 │ 72 │ +72 │
│ lint:invalid-context-manager │ error │ 0 │ 2 │ +2 │
│ lint:invalid-return-type │ error │ 0 │ 2 │ +2 │
│ lint:missing-argument │ error │ 0 │ 46 │ +46 │
│ lint:no-matching-overload │ error │ 19121 │ 0 │ -19121 │
│ lint:not-iterable │ error │ 0 │ 6 │ +6 │
│ lint:possibly-unbound-attribute │ warning │ 13 │ 32 │ +19 │
│ lint:redundant-cast │ warning │ 0 │ 1 │ +1 │
│ lint:unresolved-attribute │ error │ 0 │ 10 │ +10 │
│ lint:unsupported-operator │ error │ 3 │ 9 │ +6 │
│ lint:unused-ignore-comment │ warning │ 15 │ 4 │ -11 │
├───────────────────────────────────┼──────────┼─────────┼───────┼────────────┤
│ TOTAL │ │ 19152 │ 191 │ -18961 │
└───────────────────────────────────┴──────────┴─────────┴───────┴────────────┘
Analysis complete. Found 13 unique diagnostic IDs.
Total diagnostics removed: 19152
Total diagnostics added: 191
Net change: -18961
```
I uploaded the ecosystem full diff (ignoring the 19k
`no-matching-overload` diagnostics)
[here](https://shark.fish/diff-namedtuple.html).
* There are some new `missing-argument` false positives which come from
the fact that named tuples are often created using unpacking as in
`MyNamedTuple(*fields)`, which we do not understand yet.
* There are some new `unresolved-attribute` false positives, because
methods like `_replace` are not available.
* Lots of the `invalid-argument-type` diagnostics look like true
positives
---------
Co-authored-by: Douglas Creager <dcreager@dcreager.net>
## Summary
This PR updates the existing overload matching methods to return an
iterator of all the matched overloads instead.
This would be useful once the overload call evaluation algorithm is
implemented which should provide an accurate picture of all the matched
overloads. The return type would then be picked from either the only
matched overload or the first overload from the ones that are matched.
In an earlier version of this PR, it tried to check if using an
intersection of return types from the matched overload would help reduce
the false positives but that's not enough. [This
comment](https://github.com/astral-sh/ruff/pull/17618#issuecomment-2842891696)
keep the ecosystem analysis for that change for prosperity.
> [!NOTE]
>
> The best way to review this PR is by hiding the whitespace changes
because there are two instances where a large match expression is
indented to be inside a loop over matching overlods
>
> <img width="1207" alt="Screenshot 2025-04-28 at 15 12 16"
src="https://github.com/user-attachments/assets/e06cbfa4-04fa-435f-84ef-4e5c3c5626d1"
/>
## Test Plan
Make sure existing test cases are unaffected and no ecosystem changes.
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
This is not yet fixing anything as the names are not changed, but it
lays down the foundation for fixing.
## Test Plan
<!-- How was it tested? -->
the existing test fixture should already cover this change
## Summary
Part of #17412
Starred expressions cannot be used as values in assignment expressions.
Add a new semantic syntax error to catch such instances.
Note that we already have
`ParseErrorType::InvalidStarredExpressionUsage` to catch some starred
expression errors during parsing, but that does not cover top level
assignment expressions.
## Test Plan
- Added new inline tests for the new rule
- Found some examples marked as "valid" in existing tests (`_ = *data`),
which are not really valid (per this new rule) and updated them
- There was an existing inline test - `assign_stmt_invalid_value_expr`
which had instances of `*` expression which would be deemed invalid by
this new rule. Converted these to tuples, so that they do not trigger
this new rule.
## Summary
Model the lookup of `__new__` without going through
`Type::try_call_dunder`. The `__new__` method is only looked up on the
constructed type itself, not on the meta-type.
This now removes ~930 false positives across the ecosystem (vs 255 for
https://github.com/astral-sh/ruff/pull/17662). It introduces 30 new
false positives related to the construction of enums via something like
`Color = enum.Enum("Color", ["RED", "GREEN"])`. This is expected,
because we don't handle custom metaclass `__call__` methods. The fact
that we previously didn't emit diagnostics there was a coincidence (we
incorrectly called `EnumMeta.__new__`, and since we don't fully
understand its signature, that happened to work with `str`, `list`
arguments).
closes#17462
## Test Plan
Regression test
## Summary
Part of #15383.
As per the spec
(https://typing.python.org/en/latest/spec/overload.html#invalid-overload-definitions):
For `@staticmethod` and `@classmethod`:
> If one overload signature is decorated with `@staticmethod` or
`@classmethod`, all overload signatures must be similarly decorated. The
implementation, if present, must also have a consistent decorator. Type
checkers should report an error if these conditions are not met.
For `@final` and `@override`:
> If a `@final` or `@override` decorator is supplied for a function with
overloads, the decorator should be applied only to the overload
implementation if it is present. If an overload implementation isn’t
present (for example, in a stub file), the `@final` or `@override`
decorator should be applied only to the first overload. Type checkers
should enforce these rules and generate an error when they are violated.
If a `@final` or `@override` decorator follows these rules, a type
checker should treat the decorator as if it is present on all overloads.
## Test Plan
Update existing tests; add snapshots.
## Summary
As mentioned in the spec
(https://typing.python.org/en/latest/spec/overload.html#invalid-overload-definitions),
part of #15383:
> The `@overload`-decorated definitions must be followed by an overload
implementation, which does not include an `@overload` decorator. Type
checkers should report an error or warning if an implementation is
missing. Overload definitions within stub files, protocols, and on
abstract methods within abstract base classes are exempt from this
check.
## Test Plan
Remove TODOs from the test; create one diagnostic snapshot.
Re: #17526
## Summary
Adds tests to red knot and `linter.rs` for the semantic syntax.
Specifically add tests for `ReboundComprehensionVariable`,
`DuplicateTypeParameter`, and `MultipleCaseAssignment`.
Refactor the `test_async_comprehension_in_sync_comprehension` →
`test_semantic_error` to be more general for all semantic syntax test
cases.
## Test Plan
This is a test.
## Question
I'm happy to contribute more tests the coming days.
Should that happen here or should we merge this PR such that the
refactor `test_async_comprehension_in_sync_comprehension` →
`test_semantic_error` is available on main and others can chime in, too?
## Summary
Part of #15383, this PR adds the core infrastructure to check for
invalid overloads and adds a diagnostic to raise if there are < 2
overloads for a given definition.
### Design notes
The requirements to check the overloads are:
* Requires `FunctionType` which has the `to_overloaded` method
* The `FunctionType` **should** be for the function that is either the
implementation or the last overload if the implementation doesn't exists
* Avoid checking any `FunctionType` that are part of an overload chain
* Consider visibility constraints
This required a couple of iteration to make sure all of the above
requirements are fulfilled.
#### 1. Use a set to deduplicate
The logic would first collect all the `FunctionType` that are part of
the overload chain except for the implementation or the last overload if
the implementation doesn't exists. Then, when iterating over all the
function declarations within the scope, we'd avoid checking these
functions. But, this approach would fail to consider visibility
constraints as certain overloads _can_ be behind a version check. Those
aren't part of the overload chain but those aren't a separate overload
chain either.
<details><summary>Implementation:</summary>
<p>
```rs
fn check_overloaded_functions(&mut self) {
let function_definitions = || {
self.types
.declarations
.iter()
.filter_map(|(definition, ty)| {
// Filter out function literals that result from anything other than a function
// definition e.g., imports.
if let DefinitionKind::Function(function) = definition.kind(self.db()) {
ty.inner_type()
.into_function_literal()
.map(|ty| (ty, definition.symbol(self.db()), function.node()))
} else {
None
}
})
};
// A set of all the functions that are part of an overloaded function definition except for
// the implementation function and the last overload in case the implementation doesn't
// exists. This allows us to collect all the function definitions that needs to be skipped
// when checking for invalid overload usages.
let mut overloads: HashSet<FunctionType<'db>> = HashSet::default();
for (function, _) in function_definitions() {
let Some(overloaded) = function.to_overloaded(self.db()) else {
continue;
};
if overloaded.implementation.is_some() {
overloads.extend(overloaded.overloads.iter().copied());
} else if let Some((_, previous_overloads)) = overloaded.overloads.split_last() {
overloads.extend(previous_overloads.iter().copied());
}
}
for (function, function_node) in function_definitions() {
let Some(overloaded) = function.to_overloaded(self.db()) else {
continue;
};
if overloads.contains(&function) {
continue;
}
// At this point, the `function` variable is either the implementation function or the
// last overloaded function if the implementation doesn't exists.
if overloaded.overloads.len() < 2 {
if let Some(builder) = self
.context
.report_lint(&INVALID_OVERLOAD, &function_node.name)
{
let mut diagnostic = builder.into_diagnostic(format_args!(
"Function `{}` requires at least two overloads",
&function_node.name
));
if let Some(first_overload) = overloaded.overloads.first() {
diagnostic.annotate(
self.context
.secondary(first_overload.focus_range(self.db()))
.message(format_args!("Only one overload defined here")),
);
}
}
}
}
}
```
</p>
</details>
#### 2. Define a `predecessor` query
The `predecessor` query would return the previous `FunctionType` for the
given `FunctionType` i.e., the current logic would be extracted to be a
query instead. This could then be used to make sure that we're checking
the entire overload chain once. The way this would've been implemented
is to have a `to_overloaded` implementation which would take the root of
the overload chain instead of the leaf. But, this would require updates
to the use-def map to somehow be able to return the _following_
functions for a given definition.
#### 3. Create a successor link
This is what Pyrefly uses, we'd create a forward link between two
functions that are involved in an overload chain. This means that for a
given function, we can get the successor function. This could be used to
find the _leaf_ of the overload chain which can then be used with the
`to_overloaded` method to get the entire overload chain. But, this would
also require updating the use-def map to be able to "see" the
_following_ function.
### Implementation
This leads us to the final implementation that this PR implements which
is to consider the overloaded functions using:
* Collect all the **function symbols** that are defined **and** called
within the same file. This could potentially be an overloaded function
* Use the public bindings to get the leaf of the overload chain and use
that to get the entire overload chain via `to_overloaded` and perform
the check
This has a limitation that in case a function redefines an overload,
then that overload will not be checked. For example:
```py
from typing import overload
@overload
def f() -> None: ...
@overload
def f(x: int) -> int: ...
# The above overload will not be checked as the below function with the same name
# shadows it
def f(*args: int) -> int: ...
```
## Test Plan
Update existing mdtest and add snapshot diagnostics.
## Summary
@sharkdp and I realised in our 1:1 this morning that our control flow
for `assert` statements isn't quite accurate at the moment. Namely, for
something like this:
```py
def _(x: int | None):
assert x is None, reveal_type(x)
```
we currently reveal `None` for `x` here, but this is incorrect. In
actual fact, the `msg` expression of an `assert` statement (the
expression after the comma) will only be evaluated if the test (`x is
None`) evaluates to `False`. As such, we should be adding a constraint
of `~None` to `x` in the `msg` expression, which should simplify the
inferred type of `x` to `int` in that context (`(int | None) & ~None` ->
`int`).
## Test Plan
Mdtests added.
---------
Co-authored-by: David Peter <mail@david-peter.de>
## Summary
We were previously recording wrong reachability constraints for negative
branches. Instead of `[cond] AND (NOT [True])` below, we were recording
`[cond] AND (NOT ([cond] AND [True]))`, i.e. we were negating not just
the last predicate, but the `AND`-ed reachability constraint from last
clause. With this fix, we now record the correct constraints for the
example from #17723:
```py
def _(cond: bool):
if cond:
# reachability: [cond]
if True:
# reachability: [cond] AND [True]
pass
else:
# reachability: [cond] AND (NOT [True])
x
```
closes#17723
## Test Plan
* Regression test.
* Verified the ecosystem changes
## Summary
Part of #15383, this PR adds `is_equivalent_to` support for overloaded
callables.
This is mainly done by delegating it to the subtyping check in that two
types A and B are considered equivalent if A is a subtype of B and B is
a subtype of A.
## Test Plan
Add test cases for overloaded callables in `is_equivalent_to.md`
## Summary
Includes minor changes to the semantic type inference to help detect the
return type of function call.
Fixes#17691
## Test Plan
Snapshot tests
## Summary
Subtyping was already modeled, but assignability also needs an explicit
branch. Removes 921 ecosystem false positives.
## Test Plan
New Markdown tests.
We are currently representing type variables using a `KnownInstance`
variant, which wraps a `TypeVarInstance` that contains the information
about the typevar (name, bounds, constraints, default type). We were
previously only constructing that type for PEP 695 typevars. This PR
constructs that type for legacy typevars as well.
It also detects functions that are generic because they use legacy
typevars in their parameter list. With the existing logic for inferring
specializations of function calls (#17301), that means that we are
correctly detecting that the definition of `reveal_type` in the typeshed
is generic, and inferring the correct specialization of `_T` for each
call site.
This does not yet handle legacy generic classes; that will come in a
follow-on PR.
A small PR that just updates the various settings/configurations to
allow Python 3.14. At the moment selecting that target version will
have no impact compared to Python 3.13 - except that a warning
is emitted if the user does so with `preview` disabled.
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
Apply auto fixes to cases where the names have changed in Airflow 3 in
AIR302 and split the huge test cases into different test cases based on
proivder
## Test Plan
<!-- How was it tested? -->
the test cases has been split into multiple for easier checking
Summary
--
This PR resolves https://github.com/astral-sh/ruff/issues/9761 by adding
a linter configuration option to disable
`typing_extensions` imports. As mentioned [here], it would be ideal if
we could
detect whether or not `typing_extensions` is available as a dependency
automatically, but this seems like a much easier fix in the meantime.
The default for the new option, `typing-extensions`, is `true`,
preserving the current behavior. Setting it to `false` will bail out of
the new
`Checker::typing_importer` method, which has been refactored from the
`Checker::import_from_typing` method in
https://github.com/astral-sh/ruff/pull/17340),
with `None`, which is then handled specially by each rule that calls it.
I considered some alternatives to a config option, such as checking if
`typing_extensions` has been imported or checking for a `TYPE_CHECKING`
block we could use, but I think defaulting to allowing
`typing_extensions` imports and allowing the user to disable this with
an option is both simple to implement and pretty intuitive.
[here]:
https://github.com/astral-sh/ruff/issues/9761#issuecomment-2790492853
Test Plan
--
New linter tests exercising several combinations of Python versions and
the new config option for PYI019. I also added tests for the other
affected rules, but only in the case where the new config option is
enabled. The rules' existing tests also cover the default case.
This is done in what appears to be the same way as Ruff: we get the CWD,
strip the prefix from the path if possible, and use that. If stripping
the prefix fails, then we print the full path as-is.
Fixes#17233
## Summary
Removes ~850 diagnostics related to assignability of callable types,
where the callable-being-assigned-to has a "Todo signature", which
should probably accept any left hand side callable/signature.
This PR promotes the fix applicability of [readlines-in-for
(FURB129)](https://docs.astral.sh/ruff/rules/readlines-in-for/#readlines-in-for-furb129)
to always safe.
In the original PR (https://github.com/astral-sh/ruff/pull/9880), the
author marked the rule as unsafe because Ruff's type inference couldn't
quite guarantee that we had an `IOBase` object in hand. Some false
positives were recorded in the test fixture. However, before the PR was
merged, Charlie added the necessary type inference and the false
positives went away.
According to the [Python
documentation](https://docs.python.org/3/library/io.html#io.IOBase), I
believe this fix is safe for any proper implementation of `IOBase`:
>[IOBase](https://docs.python.org/3/library/io.html#io.IOBase) (and its
subclasses) supports the iterator protocol, meaning that an
[IOBase](https://docs.python.org/3/library/io.html#io.IOBase) object can
be iterated over yielding the lines in a stream. Lines are defined
slightly differently depending on whether the stream is a binary stream
(yielding bytes), or a text stream (yielding character strings). See
[readline()](https://docs.python.org/3/library/io.html#io.IOBase.readline)
below.
and then in the [documentation for
`readlines`](https://docs.python.org/3/library/io.html#io.IOBase.readlines):
>Read and return a list of lines from the stream. hint can be specified
to control the number of lines read: no more lines will be read if the
total size (in bytes/characters) of all lines so far exceeds hint. [...]
>Note that it’s already possible to iterate on file objects using for
line in file: ... without calling file.readlines().
I believe that a careful reading of our [versioning
policy](https://docs.astral.sh/ruff/versioning/#version-changes)
requires that this change be deferred to a minor release - but please
correct me if I'm wrong!
This PR collects all behavior gated under preview into a new module
`ruff_linter::preview` that exposes functions like
`is_my_new_feature_enabled` - just as is done in the formatter crate.
## Summary
Do not emit errors when defining `TypedDict`s:
```py
from typing_extensions import TypedDict
# No error here
class Person(TypedDict):
name: str
age: int | None
# No error for this alternative syntax
Message = TypedDict("Message", {"id": int, "content": str})
```
## Ecosystem analysis
* Removes ~ 450 false positives for `TypedDict` definitions.
* Changes a few diagnostic messages.
* Adds a few (< 10) false positives, for example:
```diff
+ error[lint:unresolved-attribute]
/tmp/mypy_primer/projects/hydra-zen/src/hydra_zen/structured_configs/_utils.py:262:5:
Type `Literal[DataclassOptions]` has no attribute `__required_keys__`
+ error[lint:unresolved-attribute]
/tmp/mypy_primer/projects/hydra-zen/src/hydra_zen/structured_configs/_utils.py:262:42:
Type `Literal[DataclassOptions]` has no attribute `__optional_keys__`
```
* New true positive
4f8263cd7f/corporate/lib/remote_billing_util.py (L155-L157)
```diff
+ error[lint:invalid-assignment]
/tmp/mypy_primer/projects/zulip/corporate/lib/remote_billing_util.py:155:5:
Object of type `RemoteBillingIdentityDict | LegacyServerIdentityDict |
None` is not assignable to `LegacyServerIdentityDict | None`
```
## Test Plan
New Markdown tests
The PR add the `fix safety` section for rule `RUF027` (#15584 ).
Actually, I have an example of a false positive. Should I include it in
the` fix safety` section?
---------
Co-authored-by: Dylan <dylwil3@gmail.com>
The PR add the fix safety section for rule `FLY002` (#15584 )
The motivation for the content of the fix safety section is given by the
following example
```python
foo = 1
bar = [2, 3]
try:
result_join = " ".join((foo, bar))
print(f"Join result: {result_join}")
except TypeError as e:
print(f"Join error: {e}")
```
which print `Join error: sequence item 0: expected str instance, int
found`
But after the fix is applied, we have
```python
foo = 1
bar = [2, 3]
try:
result_join = f"{foo} {bar}"
print(f"Join result: {result_join}")
except TypeError as e:
print(f"Join error: {e}")
```
which print `Join result: 1 [2, 3]`
---------
Co-authored-by: dylwil3 <dylwil3@gmail.com>
## Summary
This PR add the `fix safety` section for rule `ASYNC116` in
`long_sleep_not_forever.rs` for #15584
---------
Co-authored-by: dylwil3 <dylwil3@gmail.com>
## Summary
I remember we discussed about adding this as a property tests so here I
am.
## Test Plan
```console
❯ QUICKCHECK_TESTS=10000000 cargo test --locked --release --package red_knot_python_semantic -- --ignored types::property_tests::stable::bottom_callable_is_subtype_of_all_fully_static_callable
Finished `release` profile [optimized] target(s) in 0.10s
Running unittests src/lib.rs (target/release/deps/red_knot_python_semantic-e41596ca2dbd0e98)
running 1 test
test types::property_tests::stable::bottom_callable_is_subtype_of_all_fully_static_callable ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 233 filtered out; finished in 30.91s
```
As discussed today, this is needed to handle legacy generic classes
without having to infer the types of the class's explicit bases eagerly
at class construction time. Pulling this out into a separate PR so
there's a smaller diff to review.
This also makes our representation of generic classes and functions more
consistent — before, we had separate Rust types and enum variants for
generic/non-generic classes, but a single type for generic functions.
Now we each a single (respective) type for each.
There were very few places we were differentiation between generic and
non-generic _class literals_, and these are handled now by calling the
(salsa cached) `generic_context` _accessor function_.
Note that _`ClassType`_ is still an enum with distinct variants for
non-generic classes and specialized generic classes.
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
Add "airflow.operators.python.get_current_context" →
"airflow.sdk.get_current_context" rule
## Test Plan
<!-- How was it tested? -->
the test fixture has been updated accordingly
## Summary
Even though the original suggestion works, they've been removed in later
version and is no longer the best practices.
e.g., many sql realted operators have been removed and are now suggested
to use SQLExecuteQueryOperator instead
## Test Plan
The existing test fixtures have been updated
Previously, we could iterate over files in an unspecified order (via
`HashSet` iteration) and we could accumulate diagnostics from files in
an unspecified order (via parallelism).
Here, we change the status quo so that diagnostics collected from files
are sorted after checking is complete. For now, we sort by severity
(with higher severity diagnostics appearing first) and then by
diagnostic ID to give a stable ordering.
I'm not sure if this is the best ordering.
## Summary
After https://github.com/astral-sh/ruff/pull/17620 (which this PR is
based on), I was looking at other call sites of `Type::into_class_type`,
and I began to feel that _all_ of them were currently buggy due to
silently skipping unspecialized generic class literal types (though in
some cases the bug hadn't shown up yet because we don't understand
legacy generic classes from typeshed), and in every case they would be
better off if an unspecialized generic class literal were implicitly
specialized with the default specialization (which is the usual Python
typing semantics for an unspecialized reference to a generic class),
instead of silently skipped.
So I changed the method to implicitly apply the default specialization,
and added a test that previously failed for detecting metaclasses on an
unspecialized generic base.
I also renamed the method to `to_class_type`, because I feel we have a
strong naming convention where `Type::into_foo` is always a trivial
`const fn` that simply returns `Some()` if the type is of variant `Foo`
and `None` otherwise. Even the existing method (with it handling both
`GenericAlias` and `ClassLiteral`, and distinguishing kinds of
`ClassLiteral`) was stretching this convention, and the new version
definitely breaks that envelope.
## Test Plan
Added a test that failed before this PR.
## Summary
The `ClassLiteralType::inheritance_cycle` method is intended to detect
inheritance cycles that would result in cyclic MROs, emit a diagnostic,
and skip actually trying to create the cyclic MRO, falling back to an
"error" MRO instead with just `Unknown` and `object`.
This method didn't work properly for generic classes. It used
`fully_static_explicit_bases`, which filter-maps `explicit_bases` over
`Type::into_class_type`, which returns `None` for an unspecialized
generic class literal. So in a case like `class C[T](C): ...`, because
the explicit base is an unspecialized generic, we just skipped it, and
failed to detect the class as cyclically defined.
Instead, iterate directly over all `explicit_bases`, and explicitly
handle both the specialized (`GenericAlias`) and unspecialized
(`ClassLiteral`) cases, so that we check all bases and correctly detect
cyclic inheritance.
## Test Plan
Added mdtests.
Summary
--
While going through the syntax errors in [this comment], I was surprised
to see the error `name 'x' is assigned to before global declaration`,
which corresponds to [load-before-global-declaration (PLE0118)] and has
also been reimplemented as a syntax error (#17135). However, it looks
like neither of the implementations consider `global` declarations in
the top-level module scope, which is a syntax error in CPython:
```python
# try.py
x = None
global x
```
```shell
> python -m compileall -f try.py
Compiling 'try.py'...
*** File "try.py", line 2
global x
^^^^^^^^
SyntaxError: name 'x' is assigned to before global declaration
```
I'm not sure this is the best or most elegant solution, but it was a
quick fix that passed all of our tests.
Test Plan
--
New PLE0118 test case.
[this comment]:
https://github.com/astral-sh/ruff/issues/7633#issuecomment-1740424031
[load-before-global-declaration (PLE0118)]:
https://docs.astral.sh/ruff/rules/load-before-global-declaration/#load-before-global-declaration-ple0118
## Summary
Tracked structs have some issues with fixpoint iteration in Salsa, and
there's not actually any need for this to be tracked, it should be
interned like most of our type structs.
The removed comment was probably never correct (in that we could have
disambiguated sufficiently), and is definitely not relevant now that
`TypeVarInstance` also holds its `Definition`.
## Test Plan
Existing tests.
## Summary
This PR adds special-casing for `@final` and `@override` decorator for a
similar reason as https://github.com/astral-sh/ruff/pull/17591 to
support the invalid overload check.
Both `final` and `override` are identity functions which can be removed
once `TypeVar` support is added.
## Summary
As promised, this just adds a TODO comment to document something we
discussed today that should probably be improved at some point, but
isn't a priority right now (since it's an issue that in practice would
only affect generic classes with both `__init__` and `__new__` methods,
where some typevar is bound to `Unknown` in one and to some other type
in another.)
## Summary
Part of #17412
Add a new compile-time syntax error for detecting `nonlocal`
declarations at a module level.
## Test Plan
- Added new inline tests for the syntax error
- Updated existing tests for `nonlocal` statement parsing to be inside a
function scope
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
## Summary
While adding semantic error support to red-knot, I noticed duplicate
diagnostics for code like this:
```py
# error: [invalid-syntax] "cannot use an asynchronous comprehension outside of an asynchronous function on Python 3.9 (syntax was added in 3.11)"
# error: [invalid-syntax] "`asynchronous comprehension` outside of an asynchronous function"
[reveal_type(x) async for x in AsyncIterable()]
```
Beyond the duplication, the first error message doesn't make much sense
because this syntax is _not_ allowed on Python 3.11 either.
To fix this, this PR renames the
`async-comprehension-outside-async-function` semantic syntax error to
`async-comprehension-in-sync-comprehension` and fixes the rule to avoid
applying outside of sync comprehensions at all.
## Test Plan
New linter test demonstrating the false positive. The mdtests from my red-knot
PR also reflect this change.
## Summary
This PR updates the `to_overloaded` method to use an iterative approach
instead of a recursive one.
Refer to
https://github.com/astral-sh/ruff/pull/17585#discussion_r2056804587 for
context.
The main benefit here is that it avoids calling the `to_overloaded`
function in a recursive manner which is a salsa query. So, this is a bit
hand wavy but we should also see less memory used because the cache will
only contain a single entry which should be the entire overload chain.
Previously, the recursive approach would mean that each of the function
involved in an overload chain would have a cache entry. This reduce in
memory shouldn't be too much and I haven't looked at the actual data for
it.
## Test Plan
Existing test cases should pass.
This mostly only improves things for incorrect arguments and for an
incorrect return type. It doesn't do much to improve the case where
`__bool__` isn't callable and leaves the union/other cases untouched
completely.
I picked this one because, at first glance, this _looked_ like a lower
hanging fruit. The conceptual improvement here is pretty
straight-forward: add annotations for relevant data. But it took me a
bit to figure out how to connect all of the pieces.
I wanted to use this method in other places, so I moved it
to what appears to be a God-type. I also made it slightly
more versatile: callers can ask for the entire parameter list
by omitting a specific parameter index.
## Summary
Historically we have avoided narrowing on `==` tests because in many
cases it's unsound, since subclasses of a type could compare equal to
who-knows-what. But there are a lot of types (literals and unions of
them, as well as some known instances like `None` -- single-valued
types) whose `__eq__` behavior we know, and which we can safely narrow
away based on equality comparisons.
This PR implements equality narrowing in the cases where it is sound.
The most elegant way to do this (and the way that is most in-line with
our approach up until now) would be to introduce new Type variants
`NeverEqualTo[...]` and `AlwaysEqualTo[...]`, and then implement all
type relations for those variants, narrow by intersection, and let union
and intersection simplification sort it all out. This is analogous to
our existing handling for `AlwaysFalse` and `AlwaysTrue`.
But I'm reluctant to add new `Type` variants for this, mostly because
they could end up un-simplified in some types and make types even more
complex. So let's try this approach, where we handle more of the
narrowing logic as a special case.
## Test Plan
Updated and added tests.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
Co-authored-by: Carl Meyer <carl@oddbird.net>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
## Summary
Another follow-up to the unions-of-large-literals optimization. Restore
the behavior that e.g. `Literal[""] | ~Literal[""]` collapses to
`object`.
## Test Plan
Added mdtests.
## Summary
This is required because otherwise the inferred type is not going to be
`Type::FunctionLiteral` but a todo type because we don't recognize
`TypeVar` yet:
```py
_FuncT = TypeVar("_FuncT", bound=Callable[..., Any])
def abstractmethod(funcobj: _FuncT) -> _FuncT: ...
```
This is mainly required to raise diagnostic when only some (and not all)
`@overload`-ed functions are decorated with `@abstractmethod`.
## Summary
This PR adds a new method `FunctionType::to_overloaded` which converts a
`FunctionType` into an `OverloadedFunction` which contains all the
`@overload`-ed `FunctionType` and the implementation `FunctionType` if
it exists.
There's a big caveat here (it's the way overloads work) which is that
this method can only "see" all the overloads that comes _before_ itself.
Consider the following example:
```py
from typing import overload
@overload
def foo() -> None: ...
@overload
def foo(x: int) -> int: ...
def foo(x: int | None) -> int | None:
return x
```
Here, when the `to_overloaded` method is invoked on the
1. first `foo` definition, it would only contain a single overload which
is itself and no implementation.
2. second `foo` definition, it would contain both overloads and still no
implementation
3. third `foo` definition, it would contain both overloads and the
implementation which is itself
### Usages
This method will be used in the logic for checking invalid overload
usages. It can also be used for #17541.
## Test Plan
Make sure that existing tests pass.
## Summary
This is a first step toward `global` support in red-knot (#15385). I
went through all the matches for `global` in the `mypy/test-data`
directory, but I didn't find anything too interesting that wasn't
already covered by @carljm's suggestions on Discord. I still pulled in a
couple of cases for a little extra variety. I also included a section
from the
[PLE0118](https://docs.astral.sh/ruff/rules/load-before-global-declaration/)
tests in ruff that will become syntax errors once #17463 is merged and
we handle `global` statements.
I don't think I figured out how to use `@Todo` properly, so please let
me know if I need to fix that. I hope this is a good start to the test
suite otherwise.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
Status
--
This is a pretty minor change, but it was breaking a red-knot mdtest
until #17463 landed. Now this should close#11934 as the last syntax
error being tracked there!
Summary
--
Moves `Parser::validate_parameters` to
`SemanticSyntaxChecker::duplicate_parameter_name`.
Test Plan
--
Existing tests, with `## Errors` replaced with `## Semantic Syntax
Errors`.
We now handle generic constructor methods on generic classes correctly:
```py
class C[T]:
def __init__[S](self, t: T, s: S): ...
x = C(1, "str")
```
Here, constructing `C` requires us to infer a specialization for the
generic contexts of `C` and `__init__` at the same time.
At first I thought I would need to track the full stack of nested
generic contexts here (since the `[S]` context is nested within the
`[T]` context). But I think this is the only way that we might need to
specialize more than one generic context at once — in all other cases, a
containing generic context must be specialized before we get to a nested
one, and so we can just special-case this.
While we're here, we also construct the generic context for a generic
function lazily, when its signature is accessed, instead of eagerly when
inferring the function body.
## Summary
Model assignability of class instances with a `__call__` method to
`Callable` types. This should solve some false positives related to
`functools.partial` (yes, 1098 fewer diagnostics!).
Reference:
https://github.com/astral-sh/ruff/issues/17343#issuecomment-2824618483
## Test Plan
New Markdown tests.
## Summary
Many symbols in typeshed are defined without being declared. For
example:
```pyi
# builtins:
IOError = OSError
# types
LambdaType = FunctionType
NotImplementedType = _NotImplementedType
# typing
Text = str
# random
uniform = _inst.uniform
# optparse
make_option = Option
# all over the place:
_T = TypeVar("_T")
```
Here, we introduce a change that skips widening the public type of these
symbols (by unioning with `Unknown`).
fixes#17032
## Ecosystem analysis
This is difficult to analyze in detail, but I went over most changes and
it looks very favorable to me overall. The diff on the overall numbers
is:
```
errors: 1287 -> 859 (reduction by 428)
warnings: 45 -> 59 (increase by 14)
```
### Removed false positives
`invalid-base` examples:
```diff
- error[lint:invalid-base] /tmp/mypy_primer/projects/pip/src/pip/_vendor/rich/console.py:548:27: Invalid class base with type `Unknown | Literal[_local]` (all bases must be a class, `Any`, `Unknown` or `Todo`)
- error[lint:invalid-base] /tmp/mypy_primer/projects/tornado/tornado/iostream.py:84:25: Invalid class base with type `Unknown | Literal[OSError]` (all bases must be a class, `Any`, `Unknown` or `Todo`)
- error[lint:invalid-base] /tmp/mypy_primer/projects/mitmproxy/test/conftest.py:35:40: Invalid class base with type `Unknown | Literal[_UnixDefaultEventLoopPolicy]` (all bases must be a class, `Any`, `Unknown` or `Todo`)
```
`invalid-exception-caught` examples:
```diff
- error[lint:invalid-exception-caught] /tmp/mypy_primer/projects/cloud-init/cloudinit/cmd/status.py:334:16: Cannot catch object of type `Literal[ProcessExecutionError]` in an exception handler (must be a `BaseException` subclass or a tuple of `BaseException` subclasses)
- error[lint:invalid-exception-caught] /tmp/mypy_primer/projects/jinja/src/jinja2/loaders.py:537:16: Cannot catch object of type `Literal[TemplateNotFound]` in an exception handler (must be a `BaseException` subclass or a tuple of `BaseException` subclasses)
```
`unresolved-reference` examples
7a0265d36e/cloudinit/handlers/jinja_template.py (L120-L123)
(we now understand the `isinstance` narrowing)
```diff
- error[lint:unresolved-attribute] /tmp/mypy_primer/projects/cloud-init/cloudinit/handlers/jinja_template.py:123:16: Type `Exception` has no attribute `errno`
```
`unknown-argument` examples
https://github.com/hauntsaninja/boostedblob/blob/master/boostedblob/request.py#L53
```diff
- error[lint:unknown-argument] /tmp/mypy_primer/projects/boostedblob/boostedblob/request.py:53:17: Argument `connect` does not match any known parameter of bound method `__init__`
```
`unknown-argument`
There are a lot of `__init__`-related changes because we now understand
[`@attr.s`](3d42a6978a/src/attr/__init__.pyi (L387))
as a `@dataclass_transform` annotated symbol. For example:
```diff
- error[lint:unknown-argument] /tmp/mypy_primer/projects/attrs/tests/test_hooks.py:72:18: Argument `x` does not match any known parameter of bound method `__init__`
```
### New false positives
This can happen if a symbol that previously was inferred as `X |
Unknown` was assigned-to, but we don't yet understand the assignability
to `X`:
https://github.com/strawberry-graphql/strawberry/blob/main/strawberry/exceptions/handler.py#L90
```diff
+ error[lint:invalid-assignment] /tmp/mypy_primer/projects/strawberry/strawberry/exceptions/handler.py:90:9: Object of type `def strawberry_threading_exception_handler(args: tuple[type[BaseException], BaseException | None, TracebackType | None, Thread | None]) -> None` is not assignable to attribute `excepthook` of type `(_ExceptHookArgs, /) -> Any`
```
### New true positives
6bbb5519fe/tests/tracer/test_span.py (L714)
```diff
+ error[lint:invalid-argument-type] /tmp/mypy_primer/projects/dd-trace-py/tests/tracer/test_span.py:714:33: Argument to this function is incorrect: Expected `str`, found `Literal[b"\xf0\x9f\xa4\x94"]`
```
### Changed diagnostics
A lot of changed diagnostics because we now show `@Todo(Support for
`typing.TypeVar` instances in type expressions)` instead of `Unknown`
for all kinds of symbols that used a `_T = TypeVar("_T")` as a type. One
prominent example is the `list.__getitem__` method:
`builtins.pyi`:
```pyi
_T = TypeVar("_T") # previously `TypeVar | Unknown`, now just `TypeVar`
# …
class list(MutableSequence[_T]):
# …
@overload
def __getitem__(self, i: SupportsIndex, /) -> _T: ...
# …
```
which causes this change in diagnostics:
```py
xs = [1, 2]
reveal_type(xs[0]) # previously `Unknown`, now `@Todo(Support for `typing.TypeVar` instances in type expressions)`
```
## Test Plan
Updated Markdown tests
## Summary
Apply auto fixes to cases where the names have changed in Airflow 3
## Test Plan
Add `AIR301_names_fix.py` and `AIR301_provider_names_fix.py` test fixtures
This pull request fixes https://github.com/astral-sh/ruff/issues/17014
changes this
```python
from __future__ import annotations
flag1 = True
flag2 = True
if flag1 == True or flag2 == True:
pass
if flag1 == False and flag2 == False:
pass
flag3 = True
if flag1 == flag3 and (flag2 == False or flag3 == True): # Should become: if flag1==flag3 and (not flag2 or flag3)
pass
if flag1 == True and (flag2 == False or not flag3 == True): # Should become: if flag1 and (not flag2 or not flag3)
pass
if flag1 != True and (flag2 != False or not flag3 == True): # Should become: if not flag1 and (flag2 or not flag3)
pass
flag = True
while flag == True: # Should become: while flag
flag = False
flag = True
x = 5
if flag == True and x > 0: # Should become: if flag and x > 0
print("ok")
flag = True
result = "yes" if flag == True else "no" # Should become: result = "yes" if flag else "no"
x = flag == True < 5
x = (flag == True) == False < 5
```
to this
```python
from __future__ import annotations
flag1 = True
flag2 = True
if flag1 or flag2:
pass
if not flag1 and not flag2:
pass
flag3 = True
if flag1 == flag3 and (not flag2 or flag3): # Should become: if flag1 == flag3 and (not flag2 or flag3)
pass
if flag1 and (not flag2 or not flag3): # Should become: if flag1 and (not flag2 or not flag3)
pass
if not flag1 and (flag2 or not flag3): # Should become: if not flag1 and (flag2 or not flag3)
pass
flag = True
while flag: # Should become: while flag
flag = False
flag = True
x = 5
if flag and x > 0: # Should become: if flag and x > 0
print("ok")
flag = True
result = "yes" if flag else "no" # Should become: result = "yes" if flag else "no"
x = flag is True < 5
x = (flag) is False < 5
```
---------
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
Summary
--
This PR extends semantic syntax error detection to red-knot. The main
changes here are:
1. Adding `SemanticSyntaxChecker` and `Vec<SemanticSyntaxError>` fields
to the `SemanticIndexBuilder`
2. Calling `SemanticSyntaxChecker::visit_stmt` and `visit_expr` in the
`SemanticIndexBuilder`'s `visit_stmt` and `visit_expr` methods
3. Implementing `SemanticSyntaxContext` for `SemanticIndexBuilder`
4. Adding new mdtests to test the context implementation and show
diagnostics
(3) is definitely the trickiest and required (I think) a minor addition
to the `SemanticIndexBuilder`. I tried to look around for existing code
performing the necessary checks, but I definitely could have missed
something or misused the existing code even when I found it.
There's still one TODO around `global` statement handling. I don't think
there's an existing way to look this up, but I'm happy to work on that
here or in a separate PR. This currently only affects detection of one
error (`LoadBeforeGlobalDeclaration` or
[PLE0118](https://docs.astral.sh/ruff/rules/load-before-global-declaration/)
in ruff), so it's not too big of a problem even if we leave the TODO.
Test Plan
--
New mdtests, as well as new errors for existing mdtests
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
## Summary
Allow (instances of) subclasses of `Any` and `Unknown` to be assignable
to (instances of) other classes, unless they are final. This allows us
to get rid of ~1000 false positives, mostly when mock-objects like
`unittest.mock.MagicMock` are assigned to various targets.
## Test Plan
Adapted and new Markdown tests.
## Summary
mypy_primer changes included here:
ebaa9fd27b..4c22d192a4
- Add strawberry as a `good.txt` project (was previously included in our
fork)
- Print Red Knot compilation errors to stderr (thanks @MichaReiser)
## Summary
We currently emit a diagnostic for code like the following:
```py
from typing import Any
# error: Invalid class base with type `GenericAlias` (all bases must be a class, `Any`, `Unknown` or `Todo`)
class C(tuple[Any, ...]): ...
```
The changeset here silences this diagnostic by recognizing instances of
`GenericAlias` in `ClassBase::try_from_type`, and inferring a `@Todo`
type for them. This is a change in preparation for #17557, because `C`
previously had `Unknown` in its MRO …
```py
reveal_type(C.__mro__) # tuple[Literal[C], Unknown, Literal[object]]
```
… which would cause us to think that `C` is assignable to everything.
The changeset also removes some false positive `invalid-base`
diagnostics across the ecosystem.
## Test Plan
Updated Markdown tests.
## Summary
Add parentheses to multi-element intersections, when displayed in a
context that's otherwise potentially ambiguous.
## Test Plan
Update mdtest files
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
#17451 was incomplete. `AlwaysFalsy` and `AlwaysTruthy` are not the only
two types that are super-types of some literals (of a given kind) and
not others. That set also includes intersections containing
`AlwaysTruthy` or `AlwaysFalsy`, and intersections containing literal
types of the same kind. Cover these cases as well.
Fixes#17478.
## Test Plan
Added mdtests.
`QUICKCHECK_TESTS=1000000 cargo test -p red_knot_python_semantic --
--ignored types::property_tests::stable` failed on both
`all_fully_static_type_pairs_are_subtypes_of_their_union` and
`all_type_pairs_are_assignable_to_their_union` prior to this PR, passes
after it.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
I gave up trying to do this one lint at a time and just (mostly)
mechanically translated this entire file in one go.
Generally the messages stay the same (with most moving from an
annotation message to the diagnostic's main message). I added a couple
of `info` sub-diagnostics where it seemed to be the obvious intent.
This finishes the migration for the `INVALID_ASSIGNMENT` lint.
Notice how I'm steadily losing steam in terms of actually improving the
diagnostics. This change is more mechanical, because taking the time to
revamp every diagnostic is a ton of effort. Probably future migrations
will be similar unless there are easy pickings.
We mostly keep things the same here, but the message has been moved from
the annotation to the diagnostic's top-line message. I think this is
perhaps a little worse, but some bigger improvements could be made here.
Indeed, we could perhaps even add a "fix" here.
This moves all INVALID_ASSIGNMENT lints related to unpacking over to the new
diagnostic model.
While we're here, we improve the diagnostic a bit by adding a secondary
annotation covering where the value is. We also split apart the original
singular message into one message for the diagnostic and the "expected
versus got" into annotation messages.
This tests the diagnostic rendering of a case that wasn't previously
covered by snapshots: when unpacking fails because there are too few
values, but where the left hand side can tolerate "N or more." In the
code, this is a distinct diagnostic, so we capture it here.
(Sorry about the diff here, but it made sense to rename the other
sections and that changes the name of the snapshot file.)
I believe this was an artifact of an older iteration of the diagnostic
reporting API. But this is strictly not necessary now, and indeed, might
even be annoying. It is okay, but perhaps looks a little odd, to do
`builder.into_diagnostic("...")` if you don't want to add anything else
to the diagnostic.
I suspect this will be used pretty frequently (I wanted it
immediately). And more practically, this avoids needing to
import `Annotation` to create it.
## Summary
A switch from 16 to 32 cores reduces the `mypy_primer` CI time from
3.5-4 min to 2.5-3 min. There's also a 64-core runner, but the 4 min ->
3 min change when doubling the cores once does suggest that it doesn't
parallelize *this* well.
## Summary
I ran red-knot on every project in mypy-primer. I moved every project
where red-knot ran to completion (fast enough, and mypy-primer could
handle its output) into `good.txt`, so it will run in our CI.
The remaining projects I left listed in `bad.txt`, with a comment
summarizing the failure mode (a few don't fail, they are just slow -- on
a debug build, at least -- or output too many diagnostics for
mypy-primer to handle.)
We will now run CI on 109 projects; 34 are left in `bad.txt`.
## Test Plan
CI on this PR!
---------
Co-authored-by: David Peter <mail@david-peter.de>
## Summary
Takes the `good.txt` changes from #17474, and removes the following
projects:
- arrow (not part of mypy_primer upstream)
- freqtrade, hydpy, ibis, pandera, xarray (saw panics locally, all
related to try_metaclass cycles)
Increases the mypy_primer CI run time to ~4 min.
## Test Plan
Three successful CI runs.
## Summary
`mypy_primer` is not deterministic (we pin `mypy_primer` itself, but
projects change over time and we just pull in the latest version). We've
also seen occasional panics being caught in `mypy_primer` runs, so this
is trying to make these CI failures more helpful.
## Summary
* Add initial support for `typing.dataclass_transform`
* Support decorating a function decorator with `@dataclass_transform(…)`
(used by `attrs`, `strawberry`)
* Support decorating a metaclass with `@dataclass_transform(…)` (used by
`pydantic`, but doesn't work yet, because we don't seem to model
`__new__` calls correctly?)
* *No* support yet for decorating base classes with
`@dataclass_transform(…)`. I haven't figured out how this even supposed
to work. And haven't seen it being used.
* Add `strawberry` as an ecosystem project, as it makes heavy use of
`@dataclass_transform`
## Test Plan
New Markdown tests
This is an implementation of the discussion from #16719.
This change will allow list function calls to be replaced with
comprehensions:
```python
result = list()
for i in range(3):
result.append(i + 1)
# becomes
result = [i + 1 for i in range(3)]
```
I added a new test to `PERF401.py` to verify that this fix will now work
for `list()`.
## Summary
This PR is a follow-up to #16852.
Instance variables bound in comprehensions are recorded, allowing type
inference to work correctly.
This required adding support for unpacking in comprehension which
resolves https://github.com/astral-sh/ruff/issues/15369.
## Test Plan
One TODO in `mdtest/attributes.md` is now resolved, and some new test
cases are added.
---------
Co-authored-by: Dhruv Manilawala <dhruvmanila@gmail.com>
## Summary
If two types are gradually-equivalent, that means they share the same
set of possible materializations. There's no need to keep two such types
in the same union or intersection; we should simplify them.
Fixes https://github.com/astral-sh/ruff/issues/17465
The one downside here is that now we will simplify e.g. `Unknown |
Todo(...)` to just `Unknown`, if `Unknown` was added to the union first.
This is correct from a type perspective (they are equivalent types), but
it can mean we lose visibility into part of the cause for the type
inferring as unknown. I think this is OK, but if we think it's important
to avoid this, I can add a special case to try to preserve `Todo` over
`Unknown`, if we see them both in the same union or intersection.
## Test Plan
Added and updated mdtests.
## Summary
The long line of projects in `mypy_primer.yaml` is hard to work with
when adding projects or checking whether they are currently run. Use a
one-per-line text file instead.
## Test Plan
Ecosystem check on this PR.
## Summary
add fix safety section to replace_stdout_stderr and
super_call_with_parameters, for #15584
I checked the behavior and found that these two files could only
potentially delete the appended comments, so I submitted them as a PR.
The PR fixes#16457 .
Specifically, `FURB161` is marked safe, but the rule generates safe
fixes only in specific cases. Therefore, we attempt to mark the fix as
unsafe when we are not in one of these cases.
For instances, the fix is marked as aunsafe just in case of strings (as
pointed out in the issue). Let me know if I should change something.
---------
Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
## Summary
Member lookup can be cyclic, with type inference of implicit members. A
sample case is shown in the added mdtest.
There's no clear way to handle such cases other than to fixpoint-iterate
the cycle.
Fixes#17457.
## Test Plan
Added test.
## Summary
This change adds an auto-fix for manual dict comprehensions. It also
copies many of the improvements from #13919 (and associated PRs fixing
issues with it), and moves some of the utility functions from
`manual_list_comprehension.rs` into a separate `helpers.rs` to be used
in both.
## Test Plan
I added a preview test case to showcase the new fix and added a test
case in `PERF403.py` to make sure lines with semicolons function. I
didn't yet make similar tests to the ones I added earlier to
`PERF401.py`, but the logic is the same, so it might be good to add
those to make sure they work.
You can now use subscript expressions in a type expression to explicitly
specialize generic classes, just like you could already do in value
expressions.
This still does not implement bidirectional checking, so a type
annotation on an assignment does not influence how we infer a
specialization for a (not explicitly specialized) constructor call. You
might get an `invalid-assignment` error if (a) we cannot infer a class
specialization from the constructor call (in which case you end up e.g.
trying to assign `C[Unknown]` to `C[int]`) or if (b) we can infer a
specialization, but it doesn't match the annotation.
Closes https://github.com/astral-sh/ruff/issues/17432
## Summary
There was some narrowing constraints not covered from the previous PR
```py
def _(x: object):
if (type(y := x)) is bool:
reveal_type(y) # revealed: bool
```
Also, refactored a bit
## Test Plan
Update type_api.md
In #17403 I added a comment asserting that all same-kind literal types
share all the same super-types. This is true, with two notable
exceptions: the types `AlwaysTruthy` and `AlwaysFalsy`. These two types
are super-types of some literal types within a given kind and not
others: `Literal[0]`, `Literal[""]`, and `Literal[b""]` inhabit
`AlwaysFalsy`, while other literals inhabit `AlwaysTruthy`.
This PR updates the literal-unions optimization to handle these types
correctly.
Fixes https://github.com/astral-sh/ruff/issues/17447
Verified locally that `QUICKCHECK_TESTS=100000 cargo test -p
red_knot_python_semantic -- --ignored types::property_tests::stable` now
passes again.
## Summary
Fixes#17147.
This was landed in #17149 and then reverted in #17335 because it caused
cycle panics in checking pybind11. #17456 fixed the cause of that panic.
## Test Plan
Add new narrow/assert.md test file
Co-authored-by: Matthew Mckee <matthewmckee04@yahoo.co.uk>
## Summary
We were over-conflating the conditions for deferred name resolution.
`from __future__ import annotations` defers annotations, but not class
bases. In stub files, class bases are also deferred. Modeling this
correctly also reduces likelihood of cycles in Python files using `from
__future__ import annotations` (since deferred resolution is inherently
cycle-prone). The same cycles are still possible in `.pyi` files, but
much less likely, since typically there isn't anything in a `pyi` file
that would cause an early return from a scope, or otherwise cause
visibility constraints to persist to end of scope. Usually there is only
code at module global scope and class scope, which can't have `return`
statements, and `raise` or `assert` statements in a stub file would be
very strange. (Technically according to the spec we'd be within our
rights to just forbid a whole bunch of syntax outright in a stub file,
but I kinda like minimizing unnecessary differences between the handling
of Python files and stub files.)
## Test Plan
Added mdtests.
## Summary
Part of #15383, this PR adds support for overloaded callables.
Typing spec: https://typing.python.org/en/latest/spec/overload.html
Specifically, it does the following:
1. Update the `FunctionType::signature` method to return signatures from
a possibly overloaded callable using a new `FunctionSignature` enum
2. Update `CallableType` to accommodate overloaded callable by updating
the inner type to `Box<[Signature]>`
3. Update the relation methods on `CallableType` with logic specific to
overloads
4. Update the display of callable type to display a list of signatures
enclosed by parenthesis
5. Update `CallableTypeOf` special form to recognize overloaded callable
6. Update subtyping, assignability and fully static check to account for
callables (equivalence is planned to be done as a follow-up)
For (2), it is required to be done in this PR because otherwise I'd need
to add some workaround for `into_callable_type` and I though it would be
best to include it in here.
For (2), another possible design would be convert `CallableType` in an
enum with two variants `CallableType::Single` and
`CallableType::Overload` but I decided to go with `Box<[Signature]>` for
now to (a) mirror it to be equivalent to `overload` field on
`CallableSignature` and (b) to avoid any refactor in this PR. This could
be done in a follow-up to better split the two kind of callables.
### Design
There were two main candidates on how to represent the overloaded
definition:
1. To include it in the existing infrastructure which is what this PR is
doing by recognizing all the signatures within the
`FunctionType::signature` method
2. To create a new `Overload` type variant
<details><summary>For context, this is what I had in mind with the new
type variant:</summary>
<p>
```rs
pub enum Type {
FunctionLiteral(FunctionType),
Overload(OverloadType),
BoundMethod(BoundMethodType),
...
}
pub struct OverloadType {
// FunctionLiteral or BoundMethod
overloads: Box<[Type]>,
// FunctionLiteral or BoundMethod
implementation: Option<Type>
}
pub struct BoundMethodType {
kind: BoundMethodKind,
self_instance: Type,
}
pub enum BoundMethodKind {
Function(FunctionType),
Overload(OverloadType),
}
```
</p>
</details>
The main reasons to choose (1) are the simplicity in the implementation,
reusing the existing infrastructure, avoiding any complications that the
new type variant has specifically around the different variants between
function and methods which would require the overload type to use `Type`
instead.
### Implementation
The core logic is how to collect all the overloaded functions. The way
this is done in this PR is by recording a **use** on the `Identifier`
node that represents the function name in the use-def map. This is then
used to fetch the previous symbol using the same name. This way the
signatures are going to be propagated from top to bottom (from first
overload to the final overload or the implementation) with each function
/ method. For example:
```py
from typing import overload
@overload
def foo(x: int) -> int: ...
@overload
def foo(x: str) -> str: ...
def foo(x: int | str) -> int | str:
return x
```
Here, each definition of `foo` knows about all the signatures that comes
before itself. So, the first overload would only see itself, the second
would see the first and itself and so on until the implementation or the
final overload.
This approach required some updates specifically recognizing
`Identifier` node to record the function use because it doesn't use
`ExprName`.
## Test Plan
Update existing test cases which were limited by the overload support
and add test cases for the following cases:
* Valid overloads as functions, methods, generics, version specific
* Invalid overloads as stated in
https://typing.python.org/en/latest/spec/overload.html#invalid-overload-definitions
(implementation will be done in a follow-up)
* Various relation: fully static, subtyping, and assignability (others
in a follow-up)
## Ecosystem changes
_WIP_
After going through the ecosystem changes (there are a lot!), here's
what I've found:
We need assignability check between a callable type and a class literal
because a lot of builtins are defined as classes in typeshed whose
constructor method is overloaded e.g., `map`, `sorted`, `list.sort`,
`max`, `min` with the `key` parameter, `collections.abc.defaultdict`,
etc. (https://github.com/astral-sh/ruff/issues/17343). This makes up
most of the ecosystem diff **roughly 70 diagnostics**. For example:
```py
from collections import defaultdict
# red-knot: No overload of bound method `__init__` matches arguments [lint:no-matching-overload]
defaultdict(int)
# red-knot: No overload of bound method `__init__` matches arguments [lint:no-matching-overload]
defaultdict(list)
class Foo:
def __init__(self, x: int):
self.x = x
# red-knot: No overload of function `__new__` matches arguments [lint:no-matching-overload]
map(Foo, ["a", "b", "c"])
```
Duplicate diagnostics in unpacking
(https://github.com/astral-sh/ruff/issues/16514) has **~16
diagnostics**.
Support for the `callable` builtin which requires `TypeIs` support. This
is **5 diagnostics**. For example:
```py
from typing import Any
def _(x: Any | None) -> None:
if callable(x):
# red-knot: `Any | None`
# Pyright: `(...) -> object`
# mypy: `Any`
# pyrefly: `(...) -> object`
reveal_type(x)
```
Narrowing on `assert` which has **11 diagnostics**. This is being worked
on in https://github.com/astral-sh/ruff/pull/17345. For example:
```py
import re
match = re.search("", "")
assert match
match.group() # error: [possibly-unbound-attribute]
```
Others:
* `Self`: 2
* Type aliases: 6
* Generics: 3
* Protocols: 13
* Unpacking in comprehension: 1
(https://github.com/astral-sh/ruff/pull/17396)
## Performance
Refer to
https://github.com/astral-sh/ruff/pull/17366#issuecomment-2814053046.
## Summary
Add more narrowing analysis for match statements:
* add narrowing constraints from guard expressions
* add negated constraints from previous predicates and guards to
subsequent cases
This PR doesn't address that guards can mutate your subject, and so
theoretically invalidate some of these narrowing constraints that you've
previously accumulated. Some prior art on this issue [here][mutable
guards].
[mutable guards]:
https://www.irif.fr/~scherer/research/mutable-patterns/mutable-patterns-mlworkshop2024-abstract.pdf
## Test Plan
Add some new tests, and update some existing ones
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
Fixes#14866Fixes#17437
## Test Plan
Update mdtests in `narrow/`
## Summary
Prevent overcommit by using max 4 threads as intended.
Unintuitively, `.max()` returns the maximum value of `self` and the
argument (not limiting to the argument). To limit the value to 4, one
needs to use `.min()`.
https://doc.rust-lang.org/std/cmp/trait.Ord.html#method.max
## Summary
This PR extends version-related syntax error detection to red-knot. The
main changes here are:
1. Passing `ParseOptions` specifying a `PythonVersion` to parser calls
2. Adding a `python_version` method to the `Db` trait to make this
possible
3. Converting `UnsupportedSyntaxError`s to `Diagnostic`s
4. Updating existing mdtests to avoid unrelated syntax errors
My initial draft of (1) and (2) in #16090 instead tried passing a
`PythonVersion` down to every parser call, but @MichaReiser suggested
the `Db` approach instead
[here](https://github.com/astral-sh/ruff/pull/16090#discussion_r1969198407),
and I think it turned out much nicer.
All of the new `python_version` methods look like this:
```rust
fn python_version(&self) -> ruff_python_ast::PythonVersion {
Program::get(self).python_version(self)
}
```
with the exception of the `TestDb` in `ruff_db`, which hard-codes
`PythonVersion::latest()`.
## Test Plan
Existing mdtests, plus a new mdtest to see at least one of the new
diagnostics.
add fix safety section to docs for #15584, I'm new to ruff and not sure
if the content of this PR is correct, but I hope it can be helpful.
---------
Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
- \[`flake8-bandit`\] Mark tuples of string literals as trusted input in `S603` ([#17801](https://github.com/astral-sh/ruff/pull/17801))
- \[`isort`\] Check full module path against project root(s) when categorizing first-party imports ([#16565](https://github.com/astral-sh/ruff/pull/16565))
- \[`ruff`\] Add new rule `in-empty-collection` (`RUF060`) ([#16480](https://github.com/astral-sh/ruff/pull/16480))
### Bug fixes
- Fix missing `combine` call for `lint.typing-extensions` setting ([#17823](https://github.com/astral-sh/ruff/pull/17823))
- \[`flake8-async`\] Fix module name in `ASYNC110`, `ASYNC115`, and `ASYNC116` fixes ([#17774](https://github.com/astral-sh/ruff/pull/17774))
- \[`pyupgrade`\] Add spaces between tokens as necessary to avoid syntax errors in `UP018` autofix ([#17648](https://github.com/astral-sh/ruff/pull/17648))
- \[`refurb`\] Fix false positive for float and complex numbers in `FURB116` ([#17661](https://github.com/astral-sh/ruff/pull/17661))
- [parser] Flag single unparenthesized generator expr with trailing comma in arguments. ([#17893](https://github.com/astral-sh/ruff/pull/17893))
### Documentation
- Add instructions on how to upgrade to a newer Rust version ([#17928](https://github.com/astral-sh/ruff/pull/17928))
- Update code of conduct email address ([#17875](https://github.com/astral-sh/ruff/pull/17875))
- Add fix safety sections to `PLC2801`, `PLR1722`, and `RUF013` ([#17825](https://github.com/astral-sh/ruff/pull/17825), [#17826](https://github.com/astral-sh/ruff/pull/17826), [#17759](https://github.com/astral-sh/ruff/pull/17759))
- Add link to `check-typed-exception` from `S110` and `S112` ([#17786](https://github.com/astral-sh/ruff/pull/17786))
### Other changes
- Allow passing a virtual environment to `ruff analyze graph` ([#17743](https://github.com/astral-sh/ruff/pull/17743))
## 0.11.8
### Preview features
- \[`airflow`\] Apply auto fixes to cases where the names have changed in Airflow 3 (`AIR302`, `AIR311`) ([#17553](https://github.com/astral-sh/ruff/pull/17553), [#17570](https://github.com/astral-sh/ruff/pull/17570), [#17571](https://github.com/astral-sh/ruff/pull/17571))
- \[`flake8-pyi`\] Ensure `Literal[None,] | Literal[None,]` is not autofixed to `None | None` (`PYI061`) ([#17659](https://github.com/astral-sh/ruff/pull/17659))
- \[`flake8-use-pathlib`\] Avoid suggesting `Path.iterdir()` for `os.listdir` with file descriptor (`PTH208`) ([#17715](https://github.com/astral-sh/ruff/pull/17715))
- \[`flake8-use-pathlib`\] Fix `PTH104` false positive when `rename` is passed a file descriptor ([#17712](https://github.com/astral-sh/ruff/pull/17712))
- \[`flake8-use-pathlib`\] Fix `PTH116` false positive when `stat` is passed a file descriptor ([#17709](https://github.com/astral-sh/ruff/pull/17709))
- \[`flake8-use-pathlib`\] Fix `PTH123` false positive when `open` is passed a file descriptor from a function call ([#17705](https://github.com/astral-sh/ruff/pull/17705))
- \[`pycodestyle`\] Fix duplicated diagnostic in `E712` ([#17651](https://github.com/astral-sh/ruff/pull/17651))
- \[`pylint`\] Detect `global` declarations in module scope (`PLE0118`) ([#17411](https://github.com/astral-sh/ruff/pull/17411))
- [syntax-errors] Make `async-comprehension-in-sync-comprehension` more specific ([#17460](https://github.com/astral-sh/ruff/pull/17460))
### Configuration
- Add option to disable `typing_extensions` imports ([#17611](https://github.com/astral-sh/ruff/pull/17611))
### Documentation
- Fix example syntax for the `lint.pydocstyle.ignore-var-parameters` option ([#17740](https://github.com/astral-sh/ruff/pull/17740))
- Add Python 3.14 to configuration options ([#17647](https://github.com/astral-sh/ruff/pull/17647))
- Make syntax error for unparenthesized except tuples version specific to before 3.14 ([#17660](https://github.com/astral-sh/ruff/pull/17660))
## 0.11.7
### Preview features
- \[`airflow`\] Apply auto fixes to cases where the names have changed in Airflow 3 (`AIR301`) ([#17355](https://github.com/astral-sh/ruff/pull/17355))
- \[`perflint`\] Implement fix for `manual-dict-comprehension` (`PERF403`) ([#16719](https://github.com/astral-sh/ruff/pull/16719))
- [syntax-errors] Make duplicate parameter names a semantic error ([#17131](https://github.com/astral-sh/ruff/pull/17131))
- \[`pylint`\] make fix unsafe if delete comments (`PLR1730`) ([#17459](https://github.com/astral-sh/ruff/pull/17459))
### Documentation
- Add fix safety sections to docs for several rules ([#17410](https://github.com/astral-sh/ruff/pull/17410),[#17440](https://github.com/astral-sh/ruff/pull/17440),[#17441](https://github.com/astral-sh/ruff/pull/17441),[#17443](https://github.com/astral-sh/ruff/pull/17443),[#17444](https://github.com/astral-sh/ruff/pull/17444))
If the class's `__bases__` cause an exception to be raised at runtime and therefore the class
creation to fail, we infer the class's `__mro__` as being `[<class>, Unknown, object]`:
```py
# error: [inconsistent-mro] "Cannot create a consistent method resolution order (MRO) for class `Foo` with bases list `[<class 'object'>, <class 'int'>]`"
// Using `Printed::into_code` requires adding `ruff_formatter` as a direct dependency, and I suspect that Rust can optimize the closure away regardless.
When [preview](https://docs.astral.sh/ruff/preview/) is enabled,
the criterion for determining whether an import is first-party
is stricter, which could affect the suggested fix. See [this FAQ section](https://docs.astral.sh/ruff/faq/#how-does-ruff-determine-which-of-my-imports-are-first-party-third-party-etc) for more details.
// (Lint(LintName("invalid-argument-type")), Some("/src/tomllib/_parser.py"), Some(8224..8254), "Argument to function `skip_until` is incorrect", Error),
// (Lint(LintName("invalid-argument-type")), Some("/src/tomllib/_parser.py"), Some(16914..16948), "Argument to function `skip_until` is incorrect", Error),
// (Lint(LintName("invalid-argument-type")), Some("/src/tomllib/_parser.py"), Some(17319..17363), "Argument to function `skip_until` is incorrect", Error),
// ]
//right: [
// (Lint(LintName("invalid-argument-type")), Some("/src/tomllib/_parser.py"), Some(8224..8254), "Argument to this function is incorrect", Error),
// (Lint(LintName("invalid-argument-type")), Some("/src/tomllib/_parser.py"), Some(16914..16948), "Argument to this function is incorrect", Error),
// (Lint(LintName("invalid-argument-type")), Some("/src/tomllib/_parser.py"), Some(17319..17363), "Argument to this function is incorrect", Error),
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
