Wrong type inference when using a decorator returning a Protocol that uses a concatenated ParamSpec

[vxgmichel]

Bug Report

I found a really weird bug while writing a decorator that returns a Protocol that itself uses a concatenated ParamSpec. I managed to get to a small reproducible example:

P = ParamSpec("P")
T = TypeVar("T")
X = TypeVar("X")

class Wrapped(Protocol[P, T]):
    __call__: Callable[P, T]

def decorator(
    func: Callable[Concatenate[X, P], T],
) -> Wrapped[Concatenate[X, P], T]:
    # Do something with X
    return func

@decorator
def first(
    source: Iterable[T],
) -> T:
    return next(iter(source))

item = first(range(2))
reveal_type(item)
print(item.real)

Playground link

Expected Behavior

It works as expected using either one of those two decorators instead:

# Decorator without `Concatenate`
def decorator(
    func: Callable[P, T],
) -> Wrapped[P, T]:
    # Do something with X
    return func

# Decorator without `Protocol`
def decorator(
    func: Callable[Concatenate[X, P], T],
) -> Callable[Concatenate[X, P], T]:
    # Do something with X
    return func

In those two cases, mypy is happy and reports an int as expected:

note: Revealed type is "builtins.int"

Actual Behavior

Instead, mypy is confused and expects i to be a range:

note: Revealed type is "builtins.range"
error: "range" has no attribute "upper"  [attr-defined]

Your Environment

• Mypy version used: v1.4.1 and mypy 1.6.0+dev.d2022a0007c0eb176ccaf37a9aa54c958be7fb10 (compiled: no)
• Python version used: 3.10.4 and 3.11

[erictraut]

Edit: The following diagnosis is incorrect. Please see below.

Your code sample above is incorrect. The Wrapped class doesn't have a constructor, so there's no way to bind the type parameters at construction time. The constructed type therefore defaults to Wrapped[..., Any], which is why you're not seeing any errors when you use the other two decorator formulations.

Here's a formulation that works without type errors:

from typing import Callable, Generic, Iterable, ParamSpec, TypeVar


P = ParamSpec("P")
T = TypeVar("T", covariant=True)
X = TypeVar("X")


class Wrapped(Generic[P, T]):
    def __init__(self, cb: Callable[P, T]) -> None:
        self._cb = cb

    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> T:
        return self._cb(*args, **kwargs)


def decorator(func: Callable[P, T]) -> Wrapped[P, T]:
    return Wrapped(func)


@decorator
def first(
    source: Iterable[T],
) -> T:
    return next(iter(source))


reveal_type(first)
item = first(range(2))
reveal_type(item)
print(item.real)

[vxgmichel]

Thank you @erictraut for your response.

There are a couple of things I don't quite grasp about it.

The Wrapped class doesn't have a constructor, so there's no way to bind the type parameters at construction time.

I don't see how a constructor is necessary, since the Wrapped class only defines a protocol. As far as I understand, the following part:

) -> Wrapped[Concatenate[X, P], T]:

only means:

This function returns something that follows the `Wrapped` protocol parametrized with `Concatenate[X, P]` 
and `T`, i.e. something that you can call with `Concatenate[X, P]` and that returns a `T`.

The constructed type therefore defaults to Wrapped[..., Any], which is why you're not seeing any errors when you use the other two decorator formulations.

I don't think that's the case because the working examples actually print:

main.py:31: note: Revealed type is "builtins.int"

So mypy seems to be able to correctly infer the return type in those two cases.

I also created a third working example that illustrate that mypy does handle similar use cases correctly:

P = ParamSpec("P")
T = TypeVar("T", covariant=True)
A = TypeVar("A", contravariant=True)
X = TypeVar("X")

class Wrapped(Protocol[A, P, T]):
    def __call__(self, arg: A, /, *args: P.args, **kwargs: P.kwargs) -> T: ...

def decorator(
    func: Callable[Concatenate[X, P], T],
) -> Wrapped[X, P, T]:
    # Do something with X
    return func

Notice how it works around the problem I originally reported by adding a third type parameter to the protocol instead of incorporating it withing the param spec P. Here's the playground link

Those issues seem very tricky so I'm sorry if my report is not clear enough but the fact the mypy infer the wrong type in the initial example seems to be indicative of bug within mypy, although the approach used in the code sample might be off too.

[erictraut]

Apologies, my previous diagnosis was incorrect.

The issue with your first example is in the way you've defined __call__ in the protocol. A callback protocol should include a self parameter in the __call__ method.

class Wrapped(Protocol[P, T]):
    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> T:
        ...

[erictraut]

Note that if you make the above change, your original code sample still fails to type check. It appears that mypy isn't applying the decorator correctly. (Pyright also had a similar bug that I just fixed.)

Here's the updated code with comments that show where the bug occurs.

@hauntsaninja or @AlexWaygood, would you mind reopening this bug? Or I could create a new issue if you'd prefer.

from typing import Callable, Concatenate, Iterable, ParamSpec, Protocol, TypeVar

P = ParamSpec("P")
T = TypeVar("T", covariant=True)
X = TypeVar("X")

class Wrapped(Protocol[P, T]):
    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> T:
        ...

def decorator(func: Callable[Concatenate[X, P], T]) -> Wrapped[Concatenate[X, P], T]:
    return func

@decorator
def first(source: Iterable[T]) -> T:
    return next(iter(source))

# Should reveal "Wrapped[[Iterable[T]], T]"
# Mypy reveals "Wrapped[[X], T]"
reveal_type(first)
item = first(range(2))

# Should reveal "int"
# Mypy reveals "range"
reveal_type(item)
print(item.real)

Note: I haven't tried this with the latest code in the mypy master branch. Some of recent fixes from @ilevkivskyi may address this case.

[AlexWaygood]

Or I could create a new issue if you'd prefer.

That might be better, if you wouldn't mind? The conversation has become a bit confusing on this one now :)

[hauntsaninja]

It's fixed on master