[RFC] First class type support

[hynek]

This is kind of part of #214 but it stands for itself too.

Something like:

@attr.s(check_types=True)
class C:
    x = attr.ib(type=int)

ought to work.

Regardless of checking the type, the type information has to be accessible somewhere so we can have first-class deserialization support.

[yunstanford]

+1

[toumorokoshi]

This would be awesome! Is the goal to enable type checking within attrs itself? and to support types as listed in typing.py? There's a few caveats that we've discovered, diving in.

1 is there's no standardized type checking behavior for collections in typing. From my understanding, this is still an area that needs more work:

python/typing#136

Most people have been writing their own type validation libraries to compensate:

https://pypi.python.org/pypi/typeguard

All in all, I'm a huge fan of this direction. Just wondering what the choice is for type checker.

[hynek]

We don’t have concrete plans yet! Mostly because we want to make types & serialization first class concepts in attrs while doing this. :)

[hynek]

Wait, wrong ticket! :) Yeah, no still no idea yet!

[toumorokoshi]

I have a couple thoughts on this, if you're looking for ideas.

The type parameters should probably be equivalent to adding two behavior: validate and convert:

new_style = attr.ib(type=int)
old_style = attr.ib(validator=validators.instance_of(int), converter=int)

This works pretty well for primitive types. There's some nastiness with more complex types, which attrs probably shouldn't solve. For example, I'd expect to be able to do something like:

from datetime import daytime

@attr.s
class X:
    date = attr.ib(type=datetime)

# epoch
x = X(date=1092183940129)
# ISO 8601 string
x = X(date="2017-06-30T14:14:57Z")

but these would fail, because the datetime constructor takes none of these.

I would argue probably just not handling this case, and letting people override with whatever complex converters they prefer:

attr.ib(type=datetime, converter=my_complex_converter)

The other part is using or supporting typing modules. this will require some choices in behavior (e.g. validating types recursively, or just validating the top level collection for collection types). I prefer recursive validation and conversion, personally.

[hynek]

Yes, feel free to dump all your thoughts. The more input we get beforehand, the less mistakes we’re likely to make. :)

[Tinche]

OK, let's split the proposal into sub-features for easier commenting.

Create the type metadata

1. Adding a type parameter to attr.ib().

@attr.s
class C:
    x = attr.ib(type=int)

Presumably fields(C).x.type would be int then. I'm generally in favor of this. I already have a system like this in place for cattrs, except it uses metadata. I think this should be the backup though, for Pythons before 3.6. The 3.6 syntax reads much better for me.

@attr.s
class C:
    x: int = attr.ib()

So basically like #214, but with the type argument for older Pythons.

So what do we do with this data?

Static checks

My personal opinion is that static analysis is the future here. This means writing a MyPy plugin (which coincidentally I kinda promised to do, but I won't get around to until my main vacation at the end of July, probably), and maybe getting other vendors (flake8, pylint, pycharm...) in this space to cooperate too.

Static analysis has two major advantages. The first is that some types are impossible or exceedingly difficult to check for in runtime - think Iterator[int]. The other is performance. Generics in general are problematic.

@attr.s
class C:
    x: List[int] = attr.ib()

inst = C(list(range(1000)))  # Do what - iterate over the entire list?
inst.x.append('a string')  # No one will complain in the runtime here.

Even if you have simple type annotations, without generics, the isinstance calls are kinda heavy. IIRC for ints it's actually less expensive to use a convert=int than validator=instance_of(int), even though they aren't exactly the same.

This isn't me getting into static vs dynamic typing, just stating the fact that static types are a tool with advantages over doing it in runtime. We're getting them in Python and it's exciting!

We will still need validators since validators can check the values of types, and MyPy can't. So you will always need a validator to ensure your int is positive, for example.

Now, as I mentioned, this is probably the future. We can't really rely on this today since MyPy doesn't even support the easy cases of attrs classes.

Runtime checks

There is still value in runtime checking, of course. Not everyone will be able or willing to run MyPy over their code base, or maybe you want checks in the REPL, so us having a nice API for this would be good. This will require a little logic in our type validation, since not all types from typing can be used with isinstance.

The runtime type checks should be opt-in. I'm not sure if the opt-in should be on the class level (I feel like we'd need an opt-out at the attribute level) or at the attribute level, or both.

Let's experiment with some ideas for attribute-level opt-in.

from attr.validators import validate_type

def positive(_, __, val):
    if val <= 0:
        raise ValueError

@attr.s
class C: 
    x: int = attr.ib(validator=validate_type)  # Magic! Requires additional support in attrs.
    y: int = attr.ib(validator=[validate_type, positive])
    z: int = attr.ib(validator=validate_type(positive))

[toumorokoshi]

I agree that at least, introducing the type attribute and doing nothing but attaching it to the field would be great. It's a fantastic building block to build off of.

It would also be great to fold in the type definition into the attr.ib object, if possible, so

x = attr.ib(type=int)

and

x: int = attr.ib()

are functionally equal (if both are declared, attr.ib(type=foo) could take precedence).

I know there's an existing PR around attrs support for MyPy, so that'll handle the static checks at some point.

Personally, I'm more interested in the dynamic checks. I'd like to integrate it as a serializer for an API validation library I've been working on (http://transmute-core.readthedocs.io/en/latest/), and attrs has a great philosophy with performance as a feature.

Regarding the way validation is declared, validate_type seems syntactically the nicest. Although the complexities of type validation is definitely a larger ordeal. A lot of choices as @Tinche points out. validate_type could do shallow or deep validation, and that's control you may want to leave to the user.

Providing simple integrations with existing runtime type checking / conversion tooling might be better. e.g. typeguard or cattrs.

[hynek]

I'm drinking Aperol Spritz by Lake Como right now but briefly:

• Yes, I want .type as a building block and a dedicated validator that doesn't take further arguments. Everything just composes nicely. We may add a specific argument to attr.s that implicitly adds the validator to all fields. We'll see. As it should be obvious by now, I'm very anti runtime costs. :)
• I think declaring two types should blow up.

More when I'm back after EP. Although I may sprint on it.

[hynek]

Turns out I didn’t and went to San Marino instead!

---

Anyhow, so far we seem to have a consensus that:

• we add a new field to the Attribute class called type
• which can be set using x = attr.ib(type=int) if your life sucks and you can’t use Python 3.6
• using x: int = attr.ib() if life is wonderful.
• Additionally we’ll add a validator called check_type which will look at the type attribute and check.

Correct?

So far that seems rather straightforward.

I’d just like to make 100% sure that it won’t paint us in a corner re: deserialization? I’d like explicit comments from the two who built deserialization on top of attrs: @glyph and @Tinche

[Tinche]

I'm 100% on board with what you wrote FWIW.

[glyph]

FYI this is a very busy weekend for me but this is definitely in my queue to look at soon.