Add new moduleResolution option: `yarn-pnp`

[darthtrevino]

Search Terms

yarn pnp plug'n'play plug

Suggestion

Yarn has released a new feature for using modules without having a node_modules directory present: plug'n'play. Some community tooling is available for customizing the TypeScript compilerHost so that it can use plug'n'play modules, but this does not work for users of tsc.

My suggestion is to add a new "moduleResolution": "yarnpnp" option.

Use Cases / Examples

I want to be able to use yarn plug-n-play in my typescript projects. My projects are typically yarn monorepos where common libraries are basic node packages using tsc for compilation. This would allow the basic projects to continue using tsc, but with an alternative module resolution scheme.

Some community tooling has been created here for augmenting the CompileHost with a slightly updated moduleResolution strategy, but using it in basic projects would probably require a forked tsc.

https://github.com/arcanis/ts-pnp

Checklist

My suggestion meets these guidelines:

• This wouldn't be a breaking change in existing TypeScript / JavaScript code
• This wouldn't change the runtime behavior of existing JavaScript code
• This could be implemented without emitting different JS based on the types of the expressions
• This isn't a runtime feature (e.g. new expression-level syntax)

[weswigham]

I think we'd prefer if yarn and npm settled on a common format - since npm is working on tink which does the same thing with a .package-map.json. Regardless, we have a conventional stance right now to not execute external JS as part of a build (maybe that'll change in the future, but right now that's our standing philosophy), so something like this would, minimally, be blocked on yarnpkg/yarn#6388 for us.

Daniel has a much more detailed comment on the pnp rfc here

[darthtrevino]

That sounds perfectly reasonable, thanks @weswigham

[arcanis]

yarnpkg/yarn#6388 will likely not be acted upon (edit: it did). People wishing to generate the dependency tree in a specific format can easily do so using a postinstall step, as shown in the sample app. I'd advise against relying on that for tooling integration, though, since it puts an unnecessary burden on the user.

The most generic way for you to do this is to simply expose resolveModuleName within the tsconfig files (#18896), just like we did for ts-loader (TypeStrong/ts-loader#862). It would be a useful feature that would not only unlock PnP for tsc users, but also other use cases for other tools authors. I believe it also wouldn't conflict with your default policy of not executing external code, since it would require an explicit acknowledgment from the user.

[weswigham]

As I said - we've currently got no plans for a tsconfig.js (and I bring it up often) or a tsconfig managed plugin system. We have no current plans to expose any ways to execute 3rd party code during an invocation of the compiler, as we have a very strong incentive to retain complete control over the compiler's performance profile and the perception thereof (especially when in an editor). I definitely know how I'd like to expose the ability to do something like this, were we to accept it, since I designed an extensive plugin model for the compiler in the past (before it was declined).

Not to say we won't revisit it at some point, but similar suggestions have definitely been declined repeatedly in the past. A static format is much easier to deal with.

[arcanis]

I understand, I was simply informing you that waiting on yarnpkg/yarn#6388 would likely not be an option 🙂

Another thing I forgot to mention - in a world where tsc would have built-in support, the resolution name should be pnp, not yarn-pnp. Plug'n'Play is a generic specification designed in such a way that it can be implemented by different vendors, not only by Yarn.

[Phoenixmatrix]

I'd just love if resolveModuleNames was easy to override in tools such as tsc or VS Code, since yarn pnp isn't the only use case for this, and module bundlers being infinitely customizable means the only thing holding usage back is really the editor support (and if you want compatibility with the CLI compiler). The feature is basically "right there" but is hard to use without straight up forking.

There's always going to be a demand for custom ways of resolving modules. The node ecosystem is someone rigid in this regard (because of the node_modules convention being "untouchable"), but the community has done work around it (eg: Webpack custom resolvers and aliases). Now there's also pnp and tink. At HubSpot we have our own bpm. Pretty much all tools make this reasonably easy to work with (Jest, Webpack, stuff like eslint-plugin-imports, and so on. I think Flow now has that feature too?). The TS language service has resolveModuleNames, as mentioned above. If I understand well, there's even a custom resolution hook for ES6 modules in node now?

So the last bastions to make things friction-free are the language service consumers such as tsc and VSCode. If we have that, we're golden.

Another thing that comes to mind: right now because the rest of the toolchain allows customizations but the typescript ecosystem does not, there's additional friction in keeping stuff in sync: whatever I do in my webpack config, I have to replicate in my tsconfig. With better integration of custom resolvers, we could remove this friction by making them behave the same. I could have a custom resolution algorithm and have webpack and VSCode and Jest honor it without changing all the configs one by one. That would be huge. Awkwardly, this is trivially easy to do by forking the pieces involved, but that's not very scalable.

[sebinsua]

We have no current plans to expose any ways to execute 3rd party code during an invocation of the compiler, as we have a very strong incentive to retain complete control over the compiler's performance profile and the perception thereof (especially when in an editor).

I understand what you're saying but it would be nice if the TypeScript team could take a step back and look at the bigger picture here. It makes sense that you care more about the performance of TypeScript, but many of your users test their applications using CI systems and it matters to them how slow the installation of their application is just as much as the speed of the type-checking/compile process.

For example, installation of the application I'm currently consulting on takes ~2 minutes on CI. I would like to get that number down.

Is there anyway that TypeScript can support pnp without you supporting custom resolvers? If as @arcanis says, it's a "generic specification", I don't see why it shouldn't just be supported out-of-the-box.

I hope we can get this feature some way or other, otherwise it just seems like we are going to be permanently doomed to slower installation times.

[weswigham]

We can't support it directly as-is, but as @arcanis points out elsewhere, you can probably write a short postinstall script that converts the dynamic .pnp.js file into, say, the paths tsconfig option. I haven't attempted it myself yet, but I can't think of why that shouldn't work.

[Phoenixmatrix]

I've gone the dynamically generating tsconfig route for some of our stuff (we actually made a VS Code extension for this for our home grown resolution mechanism).

There's some awkward limitations there, such as nested dependencies when you have multiple version of the same package. (eg: package Foo depends on Bar@1 and Baz depends on Bar@2, and your app depends on Foo and Baz, you're out of luck as far as I know. I dont think tsconfig aliases can express this).

[arcanis]

@weswigham I'm wondering something - you might be aware that the Node project is working towards standardizing loaders. Doesn't it make your approach impracticable on the medium-long term?

Something to mention regarding the perfs - we actually noticed slightly better perfs in Flow when it uses PnP, which makes sense given that the PnP resolution removes a lot of stat calls.

[hybrist]

Some of the current thoughts around loaders involve having ways to properly sandbox them, e.g. not allowing completely arbitrary code but have clear boundaries and potentially allowing to run the hooks out of the main thread. Not saying that will happen but we are trying to take the security implications seriously.

[TobiasMelen]

I worked a bit on a script to run at postinstall and write dependency paths to a tsconfig as mentioned by @weswigham . There are issues making this solution very sub-optimal, even if Typescripts resolution is used for the type checker only.

1. The entire dependency tree needs to be flattened to a path list in tsconfig. With only direct dependencies, typescript won't be able to resolve types in nested dependencies. Since there's no way to tell what package is relevant for typescript this list gets very long and seems (I haven't benchmarked in any way, sorry) to slow down type checking in vs-code severely.

2. The tooling experience gets really wonky. Typescript will know, and hint, about all kinds of packages that pnp.js will not let you import since they're not direct dependencies. When auto-importing, typescript/vs-code will also translate module names to Yarn cache paths, which is not wanted.

3. @types/ packages have to be merged down in tsconfig/paths to their "real" package counterparts.
   I don't know what implications this has for the compilation process, but it sure feels like something that could cause problems.

Typescript would probably need a config property for overloading parts of the module tree in "node" resolution mode, in a way that can describe nested dependencies, to make a solution like this work reliably.