4e6869d2c5bf4a7bd9aeddc3d12e417558ec4f94
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
David Peter
|
ae2cf91a36 |
[red-knot] Decorators and properties (#17017)
## Summary Add support for decorators on function as well as support for properties by adding special handling for `@property` and `@<name of property>.setter`/`.getter` decorators. closes https://github.com/astral-sh/ruff/issues/16987 ## Ecosystem results - ✔️ A lot of false positives are fixed by our new understanding of properties - 🔴 A bunch of new false positives (typically `possibly-unbound-attribute` or `invalid-argument-type`) occur because we currently do not perform type narrowing on attributes. And with the new understanding of properties, this becomes even more relevant. In many cases, the narrowing occurs through an assertion, so this is also something that we need to implement to get rid of these false positives. - 🔴 A few new false positives occur because we do not understand generics, and therefore some calls to custom setters fail. - 🔴 Similarly, some false positives occur because we do not understand protocols yet. - ✔️ Seems like a true positive to me. [The setter]( |
||
|
Douglas Creager
|
e17cd350b6 |
[red-knot] Support multiple overloads when binding parameters at call sites (#16568)
This updates the `Signature` and `CallBinding` machinery to support multiple overloads for a callable. This is currently only used for `KnownFunction`s that we special-case in our type inference code. It does **_not_** yet update the semantic index builder to handle `@overload` decorators and construct a multi-signature `Overloads` instance for real Python functions. While I was here, I updated many of the `try_call` special cases to use signatures (possibly overloaded ones now) and `bind_call` to check parameter lists. We still need some of the mutator methods on `OverloadBinding` for the special cases where we need to update return types based on some Rust code. |
||
|
David Peter
|
820a31af5d |
[red-knot] Attribute access and the descriptor protocol (#16416)
## Summary * Attributes/method are now properly looked up on metaclasses, when called on class objects * We properly distinguish between data descriptors and non-data descriptors (but we do not yet support them in store-context, i.e. `obj.data_descr = …`) * The descriptor protocol is now implemented in a single unified place for instances, classes and dunder-calls. Unions and possibly-unbound symbols are supported in all possible stages of the process by creating union types as results. * In general, the handling of "possibly-unbound" symbols has been improved in a lot of places: meta-class attributes, attributes, descriptors with possibly-unbound `__get__` methods, instance attributes, … * We keep track of type qualifiers in a lot more places. I anticipate that this will be useful if we import e.g. `Final` symbols from other modules (see relevant change to typing spec: https://github.com/python/typing/pull/1937). * Detection and special-casing of the `typing.Protocol` special form in order to avoid lots of changes in the test suite due to new `@Todo` types when looking up attributes on builtin types which have `Protocol` in their MRO. We previously looked up attributes in a wrong way, which is why this didn't come up before. closes #16367 closes #15966 ## Context The way attribute lookup in `Type::member` worked before was simply wrong (mostly my own fault). The whole instance-attribute lookup should probably never have been integrated into `Type::member`. And the `Type::static_member` function that I introduced in my last descriptor PR was the wrong abstraction. It's kind of fascinating how far this approach took us, but I am pretty confident that the new approach proposed here is what we need to model this correctly. There are three key pieces that are required to implement attribute lookups: - **`Type::class_member`**/**`Type::find_in_mro`**: The `Type::find_in_mro` method that can look up attributes on class bodies (and corresponding bases). This is a partial function on types, as it can not be called on instance types like`Type::Instance(…)` or `Type::IntLiteral(…)`. For this reason, we usually call it through `Type::class_member`, which is essentially just `type.to_meta_type().find_in_mro(…)` plus union/intersection handling. - **`Type::instance_member`**: This new function is basically the type-level equivalent to `obj.__dict__[name]` when called on `Type::Instance(…)`. We use this to discover instance attributes such as those that we see as declarations on class bodies or as (annotated) assignments to `self.attr` in methods of a class. - The implementation of the descriptor protocol. It works slightly different for instances and for class objects, but it can be described by the general framework: - Call `type.class_member("attribute")` to look up "attribute" in the MRO of the meta type of `type`. Call the resulting `Symbol` `meta_attr` (even if it's unbound). - Use `meta_attr.class_member("__get__")` to look up `__get__` on the *meta type* of `meta_attr`. Call it with `__get__(meta_attr, self, self.to_meta_type())`. If this fails (either the lookup or the call), just proceed with `meta_attr`. Otherwise, replace `meta_attr` in the following with the return type of `__get__`. In this step, we also probe if a `__set__` or `__delete__` method exists and store it in `meta_attr_kind` (can be either "data descriptor" or "normal attribute or non-data descriptor"). - Compute a `fallback` type. - For instances, we use `self.instance_member("attribute")` - For class objects, we use `class_attr = self.find_in_mro("attribute")`, and then try to invoke the descriptor protocol on `class_attr`, i.e. we look up `__get__` on the meta type of `class_attr` and call it with `__get__(class_attr, None, self)`. This additional invocation of the descriptor protocol on the fallback type is one major asymmetry in the otherwise universal descriptor protocol implementation. - Finally, we look at `meta_attr`, `meta_attr_kind` and `fallback`, and handle various cases of (possible) unboundness of these symbols. - If `meta_attr` is bound and a data descriptor, just return `meta_attr` - If `meta_attr` is not a data descriptor, and `fallback` is bound, just return `fallback` - If `meta_attr` is not a data descriptor, and `fallback` is unbound, return `meta_attr` - Return unions of these three possibilities for partially-bound symbols. This allows us to handle class objects and instances within the same framework. There is a minor additional detail where for instances, we do not allow the fallback type (the instance attribute) to completely shadow the non-data descriptor. We do this because we (currently) don't want to pretend that we can statically infer that an instance attribute is always set. Dunder method calls can also be embedded into this framework. The only thing that changes is that *there is no fallback type*. If a dunder method is called on an instance, we do not fall back to instance variables. If a dunder method is called on a class object, we only look it up on the meta class, never on the class itself. ## Test Plan New Markdown tests. |
||
|
David Peter
|
aac79e453a |
[red-knot] Better diagnostics for method calls (#16362)
## Summary
Add better error messages and additional spans for method calls. Can be
reviewed commit-by-commit.
before:
```
error: lint:invalid-argument-type
--> /home/shark/playground/test.py:6:10
|
5 | c = C()
6 | c.square("hello") # error: [invalid-argument-type]
| ^^^^^^^ Object of type `Literal["hello"]` cannot be assigned to parameter 2 (`x`); expected type `int`
7 |
8 | # import inspect
|
```
after:
```
error: lint:invalid-argument-type
--> /home/shark/playground/test.py:6:10
|
5 | c = C()
6 | c.square("hello") # error: [invalid-argument-type]
| ^^^^^^^ Object of type `Literal["hello"]` cannot be assigned to parameter 2 (`x`) of bound method `square`; expected type `int`
7 |
8 | # import inspect
|
::: /home/shark/playground/test.py:2:22
|
1 | class C:
2 | def square(self, x: int) -> int:
| ------ info: parameter declared in function definition here
3 | return x * x
|
```
## Test Plan
New snapshot test
|
||
|
David Peter
|
141ba253da |
[red-knot] Add support for @classmethods (#16305)
## Summary
Add support for `@classmethod`s.
```py
class C:
@classmethod
def f(cls, x: int) -> str:
return "a"
reveal_type(C.f(1)) # revealed: str
```
## Test Plan
New Markdown tests
|
||
|
David Peter
|
d2e034adcd |
[red-knot] Method calls and the descriptor protocol (#16121)
## Summary
This PR achieves the following:
* Add support for checking method calls, and inferring return types from
method calls. For example:
```py
reveal_type("abcde".find("abc")) # revealed: int
reveal_type("foo".encode(encoding="utf-8")) # revealed: bytes
"abcde".find(123) # error: [invalid-argument-type]
class C:
def f(self) -> int:
pass
reveal_type(C.f) # revealed: <function `f`>
reveal_type(C().f) # revealed: <bound method: `f` of `C`>
C.f() # error: [missing-argument]
reveal_type(C().f()) # revealed: int
```
* Implement the descriptor protocol, i.e. properly call the `__get__`
method when a descriptor object is accessed through a class object or an
instance of a class. For example:
```py
from typing import Literal
class Ten:
def __get__(self, instance: object, owner: type | None = None) ->
Literal[10]:
return 10
class C:
ten: Ten = Ten()
reveal_type(C.ten) # revealed: Literal[10]
reveal_type(C().ten) # revealed: Literal[10]
```
* Add support for member lookup on intersection types.
* Support type inference for `inspect.getattr_static(obj, attr)` calls.
This was mostly used as a debugging tool during development, but seems
more generally useful. It can be used to bypass the descriptor protocol.
For the example above:
```py
from inspect import getattr_static
reveal_type(getattr_static(C, "ten")) # revealed: Ten
```
* Add a new `Type::Callable(…)` variant with the following sub-variants:
* `Type::Callable(CallableType::BoundMethod(…))` — represents bound
method objects, e.g. `C().f` above
* `Type::Callable(CallableType::MethodWrapperDunderGet(…))` — represents
`f.__get__` where `f` is a function
* `Type::Callable(WrapperDescriptorDunderGet)` — represents
`FunctionType.__get__`
* Add new known classes:
* `types.MethodType`
* `types.MethodWrapperType`
* `types.WrapperDescriptorType`
* `builtins.range`
## Performance analysis
On this branch, we do more work. We need to do more call checking, since
we now check all method calls. We also need to do ~twice as many member
lookups, because we need to check if a `__get__` attribute exists on
accessed members.
A brief analysis on `tomllib` shows that we now call `Type::call` 1780
times, compared to 612 calls before.
## Limitations
* Data descriptors are not yet supported, i.e. we do not infer correct
types for descriptor attribute accesses in `Store` context and do not
check writes to descriptor attributes. I felt like this was something
that could be split out as a follow-up without risking a major
architectural change.
* We currently distinguish between `Type::member` (with descriptor
protocol) and `Type::static_member` (without descriptor protocol). The
former corresponds to `obj.attr`, the latter corresponds to
`getattr_static(obj, "attr")`. However, to model some details correctly,
we would also need to distinguish between a static member lookup *with*
and *without* instance variables. The lookup without instance variables
corresponds to `find_name_in_mro`
[here](https://docs.python.org/3/howto/descriptor.html#invocation-from-an-instance).
We currently approximate both using `member_static`, which leads to two
open TODOs. Changing this would be a larger refactoring of
`Type::own_instance_member`, so I chose to leave it out of this PR.
## Test Plan
* New `call/methods.md` test suite for method calls
* New tests in `descriptor_protocol.md`
* New `call/getattr_static.md` test suite for `inspect.getattr_static`
* Various updated tests
|