b623189560
## Summary
Closes: https://github.com/astral-sh/ty/issues/157
This PR adds support for the following capabilities involving a
`ParamSpec` type variable:
- Representing `P.args` and `P.kwargs` in the type system
- Matching against a callable containing `P` to create a type mapping
- Specializing `P` against the stored parameters
The value of a `ParamSpec` type variable is being represented using
`CallableType` with a `CallableTypeKind::ParamSpecValue` variant. This
`CallableTypeKind` is expanded from the existing `is_function_like`
boolean flag. An `enum` is used as these variants are mutually
exclusive.
For context, an initial iteration made an attempt to expand the
`Specialization` to use `TypeOrParameters` enum that represents that a
type variable can specialize into either a `Type` or `Parameters` but
that increased the complexity of the code as all downstream usages would
need to handle both the variants appropriately. Additionally, we'd have
also need to establish an invariant that a regular type variable always
maps to a `Type` while a paramspec type variable always maps to a
`Parameters`.
I've intentionally left out checking and raising diagnostics when the
`ParamSpec` type variable and it's components are not being used
correctly to avoid scope increase and it can easily be done as a
follow-up. This would also include the scoping rules which I don't think
a regular type variable implements either.
## Test Plan
Add new mdtest cases and update existing test cases.
Ran this branch on pyx, no new diagnostics.
### Ecosystem analysis
There's a case where in an annotated assignment like:
```py
type CustomType[P] = Callable[...]
def value[**P](...): ...
def another[**P](...):
target: CustomType[P] = value
```
The type of `value` is a callable and it has a paramspec that's bound to
`value`, `CustomType` is a type alias that's a callable and `P` that's
used in it's specialization is bound to `another`. Now, ty infers the
type of `target` same as `value` and does not use the declared type
`CustomType[P]`. [This is the
assignment](0980b9d9ab/src/async_utils/gen_transform.py (L108))
that I'm referring to which then leads to error in downstream usage.
Pyright and mypy does seem to use the declared type.
There are multiple diagnostics in `dd-trace-py` that requires support
for `cls`.
I'm seeing `Divergent` type for an example like which ~~I'm not sure
why, I'll look into it tomorrow~~ is because of a cycle as mentioned in
https://github.com/astral-sh/ty/issues/1729#issuecomment-3612279974:
```py
from typing import Callable
def decorator[**P](c: Callable[P, int]) -> Callable[P, str]: ...
@decorator
def func(a: int) -> int: ...
# ((a: int) -> str) | ((a: Divergent) -> str)
reveal_type(func)
```
I ~~need to look into why are the parameters not being specialized
through multiple decorators in the following code~~ think this is also
because of the cycle mentioned in
https://github.com/astral-sh/ty/issues/1729#issuecomment-3612279974 and
the fact that we don't support `staticmethod` properly:
```py
from contextlib import contextmanager
class Foo:
@staticmethod
@contextmanager
def method(x: int):
yield
foo = Foo()
# ty: Revealed type: `() -> _GeneratorContextManager[Unknown, None, None]` [revealed-type]
reveal_type(foo.method)
```
There's some issue related to `Protocol` that are generic over a
`ParamSpec` in `starlette` which might be related to
https://github.com/astral-sh/ty/issues/1635 but I'm not sure. Here's a
minimal example to reproduce:
<details><summary>Code snippet:</summary>
<p>
```py
from collections.abc import Awaitable, Callable, MutableMapping
from typing import Any, Callable, ParamSpec, Protocol
P = ParamSpec("P")
Scope = MutableMapping[str, Any]
Message = MutableMapping[str, Any]
Receive = Callable[[], Awaitable[Message]]
Send = Callable[[Message], Awaitable[None]]
ASGIApp = Callable[[Scope, Receive, Send], Awaitable[None]]
_Scope = Any
_Receive = Callable[[], Awaitable[Any]]
_Send = Callable[[Any], Awaitable[None]]
# Since `starlette.types.ASGIApp` type differs from `ASGIApplication` from `asgiref`
# we need to define a more permissive version of ASGIApp that doesn't cause type errors.
_ASGIApp = Callable[[_Scope, _Receive, _Send], Awaitable[None]]
class _MiddlewareFactory(Protocol[P]):
def __call__(
self, app: _ASGIApp, *args: P.args, **kwargs: P.kwargs
) -> _ASGIApp: ...
class Middleware:
def __init__(
self, factory: _MiddlewareFactory[P], *args: P.args, **kwargs: P.kwargs
) -> None:
self.factory = factory
self.args = args
self.kwargs = kwargs
class ServerErrorMiddleware:
def __init__(
self,
app: ASGIApp,
value: int | None = None,
flag: bool = False,
) -> None:
self.app = app
self.value = value
self.flag = flag
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None: ...
# ty: Argument to bound method `__init__` is incorrect: Expected `_MiddlewareFactory[(...)]`, found `<class 'ServerErrorMiddleware'>` [invalid-argument-type]
Middleware(ServerErrorMiddleware, value=500, flag=True)
```
</p>
</details>
### Conformance analysis
> ```diff
> -constructors_callable.py:36:13: info[revealed-type] Revealed type:
`(...) -> Unknown`
> +constructors_callable.py:36:13: info[revealed-type] Revealed type:
`(x: int) -> Unknown`
> ```
Requires return type inference i.e.,
https://github.com/astral-sh/ruff/pull/21551
> ```diff
> +constructors_callable.py:194:16: error[invalid-argument-type]
Argument is incorrect: Expected `list[T@__init__]`, found `list[Unknown
| str]`
> +constructors_callable.py:194:22: error[invalid-argument-type]
Argument is incorrect: Expected `list[T@__init__]`, found `list[Unknown
| str]`
> +constructors_callable.py:195:4: error[invalid-argument-type] Argument
is incorrect: Expected `list[T@__init__]`, found `list[Unknown | int]`
> +constructors_callable.py:195:9: error[invalid-argument-type] Argument
is incorrect: Expected `list[T@__init__]`, found `list[Unknown | str]`
> ```
I might need to look into why this is happening...
> ```diff
> +generics_defaults.py:79:1: error[type-assertion-failure] Type
`type[Class_ParamSpec[(str, int, /)]]` does not match asserted type
`<class 'Class_ParamSpec'>`
> ```
which is on the following code
```py
DefaultP = ParamSpec("DefaultP", default=[str, int])
class Class_ParamSpec(Generic[DefaultP]): ...
assert_type(Class_ParamSpec, type[Class_ParamSpec[str, int]])
```
It's occurring because there's no equivalence relationship defined
between `ClassLiteral` and `KnownInstanceType::TypeGenericAlias` which
is what these types are.
Everything else looks good to me!
10 KiB
10 KiB
Legacy ParamSpec
Definition
Valid
from typing import ParamSpec
P = ParamSpec("P")
reveal_type(type(P)) # revealed: <class 'ParamSpec'>
reveal_type(P) # revealed: typing.ParamSpec
reveal_type(P.__name__) # revealed: Literal["P"]
The paramspec name can also be provided as a keyword argument:
from typing import ParamSpec
P = ParamSpec(name="P")
reveal_type(P.__name__) # revealed: Literal["P"]
Must be directly assigned to a variable
from typing import ParamSpec
P = ParamSpec("P")
# error: [invalid-paramspec]
P1: ParamSpec = ParamSpec("P1")
# error: [invalid-paramspec]
tuple_with_typevar = ("foo", ParamSpec("W"))
reveal_type(tuple_with_typevar[1]) # revealed: ParamSpec
from typing_extensions import ParamSpec
T = ParamSpec("T")
# error: [invalid-paramspec]
P1: ParamSpec = ParamSpec("P1")
# error: [invalid-paramspec]
tuple_with_typevar = ("foo", ParamSpec("P2"))
reveal_type(tuple_with_typevar[1]) # revealed: ParamSpec
ParamSpec parameter must match variable name
from typing import ParamSpec
P1 = ParamSpec("P1")
# error: [invalid-paramspec]
P2 = ParamSpec("P3")
Accepts only a single name argument
The runtime should accept bounds and covariant and contravariant arguments in the declaration just as typing.TypeVar does, but for now we will defer the standardization of the semantics of those options to a later PEP.
from typing import ParamSpec
# error: [invalid-paramspec]
P1 = ParamSpec("P1", bound=int)
# error: [invalid-paramspec]
P2 = ParamSpec("P2", int, str)
# error: [invalid-paramspec]
P3 = ParamSpec("P3", covariant=True)
# error: [invalid-paramspec]
P4 = ParamSpec("P4", contravariant=True)
Defaults
[environment]
python-version = "3.13"
The default value for a ParamSpec can be either a list of types, ..., or another ParamSpec.
from typing import ParamSpec
P1 = ParamSpec("P1", default=[int, str])
P2 = ParamSpec("P2", default=...)
P3 = ParamSpec("P3", default=P2)
Other values are invalid.
# error: [invalid-paramspec]
P4 = ParamSpec("P4", default=int)
Forward references in stub files
Stubs natively support forward references, so patterns that would raise NameError at runtime are
allowed in stub files:
from typing_extensions import ParamSpec
P = ParamSpec("P", default=[A, B])
class A: ...
class B: ...
Validating ParamSpec usage
In type annotations, ParamSpec is only valid as the first element to Callable, the final element
to Concatenate, or as a type parameter to Protocol or Generic.
from typing import ParamSpec, Callable, Concatenate, Protocol, Generic
P = ParamSpec("P")
class ValidProtocol(Protocol[P]):
def method(self, c: Callable[P, int]) -> None: ...
class ValidGeneric(Generic[P]):
def method(self, c: Callable[P, int]) -> None: ...
def valid(
a1: Callable[P, int],
a2: Callable[Concatenate[int, P], int],
) -> None: ...
def invalid(
# TODO: error
a1: P,
# TODO: error
a2: list[P],
# TODO: error
a3: Callable[[P], int],
# TODO: error
a4: Callable[..., P],
# TODO: error
a5: Callable[Concatenate[P, ...], int],
) -> None: ...
Validating P.args and P.kwargs usage
The components of ParamSpec i.e., P.args and P.kwargs are only valid when used as the
annotated types of *args and **kwargs respectively.
from typing import Generic, Callable, ParamSpec
P = ParamSpec("P")
def foo1(c: Callable[P, int]) -> None:
def nested1(*args: P.args, **kwargs: P.kwargs) -> None: ...
def nested2(
# error: [invalid-type-form] "`P.kwargs` is valid only in `**kwargs` annotation: Did you mean `P.args`?"
*args: P.kwargs,
# error: [invalid-type-form] "`P.args` is valid only in `*args` annotation: Did you mean `P.kwargs`?"
**kwargs: P.args,
) -> None: ...
# TODO: error
def nested3(*args: P.args) -> None: ...
# TODO: error
def nested4(**kwargs: P.kwargs) -> None: ...
# TODO: error
def nested5(*args: P.args, x: int, **kwargs: P.kwargs) -> None: ...
# TODO: error
def bar1(*args: P.args, **kwargs: P.kwargs) -> None:
pass
class Foo1:
# TODO: error
def method(self, *args: P.args, **kwargs: P.kwargs) -> None: ...
And, they need to be used together.
def foo2(c: Callable[P, int]) -> None:
# TODO: error
def nested1(*args: P.args) -> None: ...
# TODO: error
def nested2(**kwargs: P.kwargs) -> None: ...
class Foo2:
# TODO: error
args: P.args
# TODO: error
kwargs: P.kwargs
The name of these parameters does not need to be args or kwargs, it's the annotated type to the
respective variadic parameter that matters.
class Foo3(Generic[P]):
def method1(self, *paramspec_args: P.args, **paramspec_kwargs: P.kwargs) -> None: ...
def method2(
self,
# error: [invalid-type-form] "`P.kwargs` is valid only in `**kwargs` annotation: Did you mean `P.args`?"
*paramspec_args: P.kwargs,
# error: [invalid-type-form] "`P.args` is valid only in `*args` annotation: Did you mean `P.kwargs`?"
**paramspec_kwargs: P.args,
) -> None: ...
Specializing generic classes explicitly
from typing import Any, Generic, ParamSpec, Callable, TypeVar
P1 = ParamSpec("P1")
P2 = ParamSpec("P2")
T1 = TypeVar("T1")
class OnlyParamSpec(Generic[P1]):
attr: Callable[P1, None]
class TwoParamSpec(Generic[P1, P2]):
attr1: Callable[P1, None]
attr2: Callable[P2, None]
class TypeVarAndParamSpec(Generic[T1, P1]):
attr: Callable[P1, T1]
Explicit specialization of a generic class involving ParamSpec is done by providing either a list
of types, ..., or another in-scope ParamSpec.
reveal_type(OnlyParamSpec[[int, str]]().attr) # revealed: (int, str, /) -> None
reveal_type(OnlyParamSpec[...]().attr) # revealed: (...) -> None
def func(c: Callable[P2, None]):
reveal_type(OnlyParamSpec[P2]().attr) # revealed: (**P2@func) -> None
# TODO: error: paramspec is unbound
reveal_type(OnlyParamSpec[P2]().attr) # revealed: (...) -> None
The square brackets can be omitted when ParamSpec is the only type variable
reveal_type(OnlyParamSpec[int, str]().attr) # revealed: (int, str, /) -> None
reveal_type(OnlyParamSpec[int,]().attr) # revealed: (int, /) -> None
# Even when there is only one element
reveal_type(OnlyParamSpec[Any]().attr) # revealed: (Any, /) -> None
reveal_type(OnlyParamSpec[object]().attr) # revealed: (object, /) -> None
reveal_type(OnlyParamSpec[int]().attr) # revealed: (int, /) -> None
But, they cannot be omitted when there are multiple type variables.
reveal_type(TypeVarAndParamSpec[int, [int, str]]().attr) # revealed: (int, str, /) -> int
reveal_type(TypeVarAndParamSpec[int, [str]]().attr) # revealed: (str, /) -> int
reveal_type(TypeVarAndParamSpec[int, ...]().attr) # revealed: (...) -> int
# TODO: We could still specialize for `T1` as the type is valid which would reveal `(...) -> int`
# TODO: error: paramspec is unbound
reveal_type(TypeVarAndParamSpec[int, P2]().attr) # revealed: (...) -> Unknown
# error: [invalid-type-arguments] "Type argument for `ParamSpec` must be either a list of types, `ParamSpec`, `Concatenate`, or `...`"
reveal_type(TypeVarAndParamSpec[int, int]().attr) # revealed: (...) -> Unknown
Nor can they be omitted when there are more than one ParamSpecs.
p = TwoParamSpec[[int, str], [int]]()
reveal_type(p.attr1) # revealed: (int, str, /) -> None
reveal_type(p.attr2) # revealed: (int, /) -> None
# error: [invalid-type-arguments]
# error: [invalid-type-arguments]
TwoParamSpec[int, str]
Specializing ParamSpec type variable using typing.Any isn't explicitly allowed by the spec but
both mypy and Pyright allow this and there are usages of this in the wild e.g.,
staticmethod[Any, Any].
reveal_type(TypeVarAndParamSpec[int, Any]().attr) # revealed: (...) -> int
Specialization when defaults are involved
[environment]
python-version = "3.13"
from typing import Any, Generic, ParamSpec, Callable, TypeVar
P = ParamSpec("P")
PList = ParamSpec("PList", default=[int, str])
PEllipsis = ParamSpec("PEllipsis", default=...)
PAnother = ParamSpec("PAnother", default=P)
PAnotherWithDefault = ParamSpec("PAnotherWithDefault", default=PList)
class ParamSpecWithDefault1(Generic[PList]):
attr: Callable[PList, None]
reveal_type(ParamSpecWithDefault1().attr) # revealed: (int, str, /) -> None
reveal_type(ParamSpecWithDefault1[[int]]().attr) # revealed: (int, /) -> None
class ParamSpecWithDefault2(Generic[PEllipsis]):
attr: Callable[PEllipsis, None]
reveal_type(ParamSpecWithDefault2().attr) # revealed: (...) -> None
reveal_type(ParamSpecWithDefault2[[int, str]]().attr) # revealed: (int, str, /) -> None
class ParamSpecWithDefault3(Generic[P, PAnother]):
attr1: Callable[P, None]
attr2: Callable[PAnother, None]
# `P` hasn't been specialized, so it defaults to `Unknown` gradual form
p1 = ParamSpecWithDefault3()
reveal_type(p1.attr1) # revealed: (...) -> None
reveal_type(p1.attr2) # revealed: (...) -> None
p2 = ParamSpecWithDefault3[[int, str]]()
reveal_type(p2.attr1) # revealed: (int, str, /) -> None
reveal_type(p2.attr2) # revealed: (int, str, /) -> None
p3 = ParamSpecWithDefault3[[int], [str]]()
reveal_type(p3.attr1) # revealed: (int, /) -> None
reveal_type(p3.attr2) # revealed: (str, /) -> None
class ParamSpecWithDefault4(Generic[PList, PAnotherWithDefault]):
attr1: Callable[PList, None]
attr2: Callable[PAnotherWithDefault, None]
p1 = ParamSpecWithDefault4()
reveal_type(p1.attr1) # revealed: (int, str, /) -> None
reveal_type(p1.attr2) # revealed: (int, str, /) -> None
p2 = ParamSpecWithDefault4[[int]]()
reveal_type(p2.attr1) # revealed: (int, /) -> None
reveal_type(p2.attr2) # revealed: (int, /) -> None
p3 = ParamSpecWithDefault4[[int], [str]]()
reveal_type(p3.attr1) # revealed: (int, /) -> None
reveal_type(p3.attr2) # revealed: (str, /) -> None
# TODO: error
# Un-ordered type variables as the default of `PAnother` is `P`
class ParamSpecWithDefault5(Generic[PAnother, P]):
attr: Callable[PAnother, None]
# TODO: error
# PAnother has default as P (another ParamSpec) which is not in scope
class ParamSpecWithDefault6(Generic[PAnother]):
attr: Callable[PAnother, None]
Semantics
The semantics of ParamSpec are described in
the PEP 695 ParamSpec document to avoid duplication unless there are
any behavior specific to the legacy ParamSpec implementation.