Cyclic0007/ruff
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[ty] handle recursive type inference properly (#20566)

Browse Source 
## Summary

Derived from #17371

Fixes astral-sh/ty#256
Fixes https://github.com/astral-sh/ty/issues/1415
Fixes https://github.com/astral-sh/ty/issues/1433
Fixes https://github.com/astral-sh/ty/issues/1524

Properly handles any kind of recursive inference and prevents panics.

---

Let me explain techniques for converging fixed-point iterations during
recursive type inference.
There are two types of type inference that naively don't converge
(causing salsa to panic): divergent type inference and oscillating type
inference.

### Divergent type inference

Divergent type inference occurs when eagerly expanding a recursive type.
A typical example is this:

```python
class C:
    def f(self, other: "C"):
        self.x = (other.x, 1)

reveal_type(C().x) # revealed: Unknown | tuple[Unknown | tuple[Unknown | tuple[..., Literal[1]], Literal[1]], Literal[1]]
```

To solve this problem, we have already introduced `Divergent` types
(https://github.com/astral-sh/ruff/pull/20312). `Divergent` types are
treated as a kind of dynamic type [^1].

```python
Unknown | tuple[Unknown | tuple[Unknown | tuple[..., Literal[1]], Literal[1]], Literal[1]]
=> Unknown | tuple[Divergent, Literal[1]]
```

When a query function that returns a type enters a cycle, it sets
`Divergent` as the cycle initial value (instead of `Never`). Then, in
the cycle recovery function, it reduces the nesting of types containing
`Divergent` to converge.

```python
0th: Divergent
1st: Unknown | tuple[Divergent, Literal[1]]
2nd: Unknown | tuple[Unknown | tuple[Divergent, Literal[1]], Literal[1]]
=> Unknown | tuple[Divergent, Literal[1]]
```

Each cycle recovery function for each query should operate only on the
`Divergent` type originating from that query.
For this reason, while `Divergent` appears the same as `Any` to the
user, it internally carries some information: the location where the
cycle occurred. Previously, we roughly identified this by having the
scope where the cycle occurred, but with the update to salsa, functions
that create cycle initial values ​​can now receive a `salsa::Id`
(https://github.com/salsa-rs/salsa/pull/1012). This is an opaque ID that
uniquely identifies the cycle head (the query that is the starting point
for the fixed-point iteration). `Divergent` now has this `salsa::Id`.

### Oscillating type inference

Now, another thing to consider is oscillating type inference.
Oscillating type inference arises from the fact that monotonicity is
broken. Monotonicity here means that for a query function, if it enters
a cycle, the calculation must start from a "bottom value" and progress
towards the final result with each cycle. Monotonicity breaks down in
type systems that have features like overloading and overriding.

```python
class Base:
    def flip(self) -> "Sub":
        return Sub()

class Sub(Base):
    def flip(self) -> "Base":
        return Base()

class C:
    def __init__(self, x: Sub):
        self.x = x

    def replace_with(self, other: "C"):
        self.x = other.x.flip()

reveal_type(C(Sub()).x)
```

Naive fixed-point iteration results in `Divergent -> Sub -> Base -> Sub
-> ...`, which oscillates forever without diverging or converging. To
address this, the salsa API has been modified so that the cycle recovery
function receives the value of the previous cycle
(https://github.com/salsa-rs/salsa/pull/1012).
The cycle recovery function returns the union type of the current cycle
and the previous cycle. In the above example, the result type for each
cycle is `Divergent -> Sub -> Base (= Sub | Base) -> Base`, which
converges.

The final result of oscillating type inference does not contain
`Divergent` because `Divergent` that appears in a union type can be
removed, as is clear from the expansion. This simplification is
performed at the same time as nesting reduction.

```
T | Divergent = T | (T | (T | ...)) = T
```

[^1]: In theory, it may be possible to strictly treat types containing
`Divergent` types as recursive types, but we probably shouldn't go that
deep yet. (AFAIK, there are no PEPs that specify how to handle
implicitly recursive types that aren't named by type aliases)

## Performance analysis

A happy side effect of this PR is that we've observed widespread
performance improvements!
This is likely due to the removal of the `ITERATIONS_BEFORE_FALLBACK`
and max-specialization depth trick
(https://github.com/astral-sh/ty/issues/1433,
https://github.com/astral-sh/ty/issues/1415), which means we reach a
fixed point much sooner.

## Ecosystem analysis

The changes look good overall.
You may notice changes in the converged values ​​for recursive types,
this is because the way recursive types are normalized has been changed.
Previously, types containing `Divergent` types were normalized by
replacing them with the `Divergent` type itself, but in this PR, types
with a nesting level of 2 or more that contain `Divergent` types are
normalized by replacing them with a type with a nesting level of 1. This
means that information about the non-divergent parts of recursive types
is no longer lost.

```python
# previous
tuple[tuple[Divergent, int], int] => Divergent
# now
tuple[tuple[Divergent, int], int] => tuple[Divergent, int]
```

The false positive error introduced in this PR occurs in class
definitions with self-referential base classes, such as the one below.

```python
from typing_extensions import Generic, TypeVar

T = TypeVar("T")
U = TypeVar("U")

class Base2(Generic[T, U]): ...

# TODO: no error
# error: [unsupported-base] "Unsupported class base with type `<class 'Base2[Sub2, U@Sub2]'> | <class 'Base2[Sub2[Unknown], U@Sub2]'>`"
class Sub2(Base2["Sub2", U]): ...
```

This is due to the lack of support for unions of MROs, or because cyclic
legacy generic types are not inferred as generic types early in the
query cycle.

## Test Plan

All samples listed in astral-sh/ty#256 are tested and passed without any
panic!

## Acknowledgments

Thanks to @MichaReiser for working on bug fixes and improvements to
salsa for this PR. @carljm also contributed early on to the discussion
of the query convergence mechanism proposed in this PR.

---------

Co-authored-by: Carl Meyer <carl@astral.sh>
This commit is contained in:
Shunsuke Shibayama
2025-11-27 01:50:26 +09:00
committed by GitHub
parent adf4f1e3f4
commit 2c0c5ff4e7
38 changed files with 2223 additions and 556 deletions
Download Patch File Download Diff File Expand all files Collapse all files

26
crates/ty_python_semantic/resources/corpus/classliteral_decorators_cycle.py Normal file
View File

@@ -0,0 +1,26 @@
try:
type name_4 = name_1
finally:
from .. import name_3
try:
pass
except* 0:
pass
else:
def name_1() -> name_4:
pass
@name_1
def name_3():
pass
finally:
try:
pass
except* 0:
assert name_3
finally:
@name_3
class name_1:
pass

5
crates/ty_python_semantic/resources/corpus/cyclic_reassignment.py Normal file
View File

@@ -0,0 +1,5 @@
class foo[_: foo](object): ...
[_] = (foo,) = foo
def foo(): ...

20
crates/ty_python_semantic/resources/corpus/cyclic_type_alias.py Normal file
View File

@@ -0,0 +1,20 @@
name_3: Foo = 0
name_4 = 0
if _0:
type name_3 = name_5
type name_4 = name_3
_1: name_3
def name_1(_2: name_4):
pass
match 0:
case name_1._3:
pass
case 1:
type name_5 = name_4
case name_5:
pass
name_3 = name_5

64
crates/ty_python_semantic/resources/mdtest/attributes.md
View File

@@ -2383,6 +2383,34 @@ class B:
reveal_type(B().x) # revealed: Unknown | Literal[1]
reveal_type(A().x) # revealed: Unknown | Literal[1]
class Base:
def flip(self) -> "Sub":
return Sub()
class Sub(Base):
# error: [invalid-method-override]
def flip(self) -> "Base":
return Base()
class C2:
def __init__(self, x: Sub):
self.x = x
def replace_with(self, other: "C2"):
self.x = other.x.flip()
reveal_type(C2(Sub()).x) # revealed: Unknown | Base
class C3:
def __init__(self, x: Sub):
self.x = [x]
def replace_with(self, other: "C3"):
self.x = [self.x[0].flip()]
# TODO: should be `Unknown | list[Unknown | Sub] | list[Unknown | Base]`
reveal_type(C3(Sub()).x) # revealed: Unknown | list[Unknown | Sub] | list[Divergent]
```
And cycles between many attributes:
@@ -2432,6 +2460,30 @@ class ManyCycles:
reveal_type(self.x5) # revealed: Unknown | int
reveal_type(self.x6) # revealed: Unknown | int
reveal_type(self.x7) # revealed: Unknown | int
class ManyCycles2:
def __init__(self: "ManyCycles2"):
self.x1 = [0]
self.x2 = [1]
self.x3 = [1]
def f1(self: "ManyCycles2"):
# TODO: should be Unknown | list[Unknown | int] | list[Divergent]
reveal_type(self.x3) # revealed: Unknown | list[Unknown | int] | list[Divergent] | list[Divergent]
self.x1 = [self.x2] + [self.x3]
self.x2 = [self.x1] + [self.x3]
self.x3 = [self.x1] + [self.x2]
def f2(self: "ManyCycles2"):
self.x1 = self.x2 + self.x3
self.x2 = self.x1 + self.x3
self.x3 = self.x1 + self.x2
def f3(self: "ManyCycles2"):
self.x1 = self.x2 + self.x3
self.x2 = self.x1 + self.x3
self.x3 = self.x1 + self.x2
```
This case additionally tests our union/intersection simplification logic:
@@ -2611,12 +2663,18 @@ reveal_type(Answer.__members__) # revealed: MappingProxyType[str, Unknown]
## Divergent inferred implicit instance attribute types
If an implicit attribute is defined recursively and type inference diverges, the divergent part is
filled in with the dynamic type `Divergent`. Types containing `Divergent` can be seen as "cheap"
recursive types: they are not true recursive types based on recursive type theory, so no unfolding
is performed when you use them.
```py
class C:
def f(self, other: "C"):
self.x = (other.x, 1)
reveal_type(C().x) # revealed: Unknown | tuple[Divergent, Literal[1]]
reveal_type(C().x[0]) # revealed: Unknown | Divergent
```
This also works if the tuple is not constructed directly:
@@ -2655,11 +2713,11 @@ And it also works for homogeneous tuples:
def make_homogeneous_tuple(x: T) -> tuple[T, ...]:
return (x, x)
class E:
def f(self, other: "E"):
class F:
def f(self, other: "F"):
self.x = make_homogeneous_tuple(other.x)
reveal_type(E().x) # revealed: Unknown | tuple[Divergent, ...]
reveal_type(F().x) # revealed: Unknown | tuple[Divergent, ...]
```
## Attributes of standard library modules that aren't yet defined

49
crates/ty_python_semantic/resources/mdtest/cycle.md
View File

@@ -32,6 +32,55 @@ reveal_type(p.x) # revealed: Unknown | int
reveal_type(p.y) # revealed: Unknown | int
```
## Self-referential bare type alias
```toml
[environment]
python-version = "3.12" # typing.TypeAliasType
```
```py
from typing import Union, TypeAliasType, Sequence, Mapping
A = list["A" | None]
def f(x: A):
# TODO: should be `list[A | None]`?
reveal_type(x) # revealed: list[Divergent]
# TODO: should be `A | None`?
reveal_type(x[0]) # revealed: Divergent
JSONPrimitive = Union[str, int, float, bool, None]
JSONValue = TypeAliasType("JSONValue", 'Union[JSONPrimitive, Sequence["JSONValue"], Mapping[str, "JSONValue"]]')
```
## Self-referential legacy type variables
```py
from typing import Generic, TypeVar
B = TypeVar("B", bound="Base")
class Base(Generic[B]):
pass
T = TypeVar("T", bound="Foo[int]")
class Foo(Generic[T]): ...
```
## Self-referential PEP-695 type variables
```toml
[environment]
python-version = "3.12"
```
```py
class Node[T: "Node[int]"]:
pass
```
## Parameter default values
This is a regression test for <https://github.com/astral-sh/ty/issues/1402>. When a parameter has a

10
crates/ty_python_semantic/resources/mdtest/generics/legacy/classes.md
View File

@@ -732,6 +732,14 @@ class Base(Generic[T]): ...
class Sub(Base["Sub"]): ...
reveal_type(Sub) # revealed: <class 'Sub'>
U = TypeVar("U")
class Base2(Generic[T, U]): ...
# TODO: no error
# error: [unsupported-base] "Unsupported class base with type `<class 'Base2[Sub2, U@Sub2]'> | <class 'Base2[Sub2[Unknown], U@Sub2]'>`"
class Sub2(Base2["Sub2", U]): ...
```
#### Without string forward references
@@ -756,6 +764,8 @@ from typing_extensions import Generic, TypeVar
T = TypeVar("T")
# TODO: no error "Unsupported class base with type `<class 'list[Derived[T@Derived]]'> | <class 'list[@Todo]'>`"
# error: [unsupported-base]
class Derived(list[Derived[T]], Generic[T]): ...
```

2
crates/ty_python_semantic/resources/mdtest/import/cyclic.md
View File

@@ -38,7 +38,7 @@ See: <https://github.com/astral-sh/ty/issues/113>
from pkg.sub import A
# TODO: This should be `<class 'A'>`
reveal_type(A) # revealed: Never
reveal_type(A) # revealed: Divergent
```
`pkg/outer.py`:

31
crates/ty_python_semantic/resources/mdtest/pep613_type_aliases.md
View File

@@ -144,17 +144,46 @@ def _(x: IntOrStr):
## Cyclic
```py
from typing import TypeAlias
from typing import TypeAlias, TypeVar, Union
from types import UnionType
RecursiveTuple: TypeAlias = tuple[int | "RecursiveTuple", str]
def _(rec: RecursiveTuple):
# TODO should be `tuple[int | RecursiveTuple, str]`
reveal_type(rec) # revealed: tuple[Divergent, str]
RecursiveHomogeneousTuple: TypeAlias = tuple[int | "RecursiveHomogeneousTuple", ...]
def _(rec: RecursiveHomogeneousTuple):
# TODO should be `tuple[int | RecursiveHomogeneousTuple, ...]`
reveal_type(rec) # revealed: tuple[Divergent, ...]
ClassInfo: TypeAlias = type | UnionType | tuple["ClassInfo", ...]
reveal_type(ClassInfo) # revealed: types.UnionType
def my_isinstance(obj: object, classinfo: ClassInfo) -> bool:
# TODO should be `type | UnionType | tuple[ClassInfo, ...]`
reveal_type(classinfo) # revealed: type | UnionType | tuple[Divergent, ...]
return isinstance(obj, classinfo)
K = TypeVar("K")
V = TypeVar("V")
NestedDict: TypeAlias = dict[K, Union[V, "NestedDict[K, V]"]]
def _(nested: NestedDict[str, int]):
# TODO should be `dict[str, int | NestedDict[str, int]]`
reveal_type(nested) # revealed: @Todo(specialized generic alias in type expression)
my_isinstance(1, int)
my_isinstance(1, int | str)
my_isinstance(1, (int, str))
my_isinstance(1, (int, (str, float)))
my_isinstance(1, (int, (str | float)))
# error: [invalid-argument-type]
my_isinstance(1, 1)
# TODO should be an invalid-argument-type error
my_isinstance(1, (int, (str, 1)))
```
## Conditionally imported on Python < 3.10

66
crates/ty_python_semantic/resources/mdtest/pep695_type_aliases.md
View File

@@ -106,6 +106,29 @@ def _(flag: bool):
```py
type ListOrSet[T] = list[T] | set[T]
reveal_type(ListOrSet.__type_params__) # revealed: tuple[TypeVar | ParamSpec | TypeVarTuple, ...]
type Tuple1[T] = tuple[T]
def _(cond: bool):
Generic = ListOrSet if cond else Tuple1
def _(x: Generic[int]):
reveal_type(x) # revealed: list[int] | set[int] | tuple[int]
try:
class Foo[T]:
x: T
def foo(self) -> T:
return self.x
...
except Exception:
class Foo[T]:
x: T
def foo(self) -> T:
return self.x
def f(x: Foo[int]):
reveal_type(x.foo()) # revealed: int
```
## In unions and intersections
@@ -244,6 +267,47 @@ def f(x: IntOr, y: OrInt):
reveal_type(x) # revealed: Never
if not isinstance(y, int):
reveal_type(y) # revealed: Never
# error: [cyclic-type-alias-definition] "Cyclic definition of `Itself`"
type Itself = Itself
def foo(
# this is a very strange thing to do, but this is a regression test to ensure it doesn't panic
Itself: Itself,
):
x: Itself
reveal_type(Itself) # revealed: Divergent
# A type alias defined with invalid recursion behaves as a dynamic type.
foo(42)
foo("hello")
# error: [cyclic-type-alias-definition] "Cyclic definition of `A`"
type A = B
# error: [cyclic-type-alias-definition] "Cyclic definition of `B`"
type B = A
def bar(B: B):
x: B
reveal_type(B) # revealed: Divergent
# error: [cyclic-type-alias-definition] "Cyclic definition of `G`"
type G[T] = G[T]
# error: [cyclic-type-alias-definition] "Cyclic definition of `H`"
type H[T] = I[T]
# error: [cyclic-type-alias-definition] "Cyclic definition of `I`"
type I[T] = H[T]
# It's not possible to create an element of this type, but it's not an error for now
type DirectRecursiveList[T] = list[DirectRecursiveList[T]]
# TODO: this should probably be a cyclic-type-alias-definition error
type Foo[T] = list[T] | Bar[T]
type Bar[T] = int | Foo[T]
def _(x: Bar[int]):
# TODO: should be `int | list[int]`
reveal_type(x) # revealed: int | list[int] | Any
```
### With legacy generic
@@ -327,7 +391,7 @@ class C(P[T]):
pass
reveal_type(C[int]()) # revealed: C[int]
reveal_type(C()) # revealed: C[Divergent]
reveal_type(C()) # revealed: C[C[Divergent]]
```
### Union inside generic

5
crates/ty_python_semantic/resources/mdtest/regression/1377_iteration_count_mismatch.md
View File

@@ -2,10 +2,7 @@
Regression test for <https://github.com/astral-sh/ty/issues/1377>.
The code is an excerpt from <https://github.com/Gobot1234/steam.py> that is minimal enough to
trigger the iteration count mismatch bug in Salsa.
<!-- expect-panic: execute: too many cycle iterations -->
The code is an excerpt from <https://github.com/Gobot1234/steam.py>.
```toml
[environment]