Feature: Load ESM over https

[viktor-ku]

Hi,

Just closed my PR to the Node.js core about this feature. I'd like to propose a loader that could do this thing.

Example

import sum from 'https://gitsomething.com/origin/v1.1.0/main.js'

console.log(sum(1, 1)) // 2

How I think it should work

1. Import over https reached
2. Convert this https path (without any filename) to the hash representation (e.g. 'https://gitsomething.com/origin/v1.1.0' to 70b56bcaff85f7836cc4688021cfbfe385550fbbeb6e5e801eb16bd9e66d3134)
3. Go to the main location with stored node modules over http (in PR it uses /tmp/ or the path specified by NODE_HTTP_MODULES env. Essential is that it is some base dir like node_modules)
4. Look for the folder with this hashed name or download the package
5. Look for the file matching the loading one (e.g. main.js in this case)
6. Load the file content as usual esm

What I don't know yet

How to resolve package dependencies

[xtuc]

I'm curious to know what do you think about allowing URL in Module specifier in WASM WebAssembly/esm-integration#11

[devsnek]

once again i think we need to do this the standardised way: html.spec.whatwg.org/multipage/webappapis.html#fetching-scripts

there are 1000000000000 different things that servers do with compression, transport security, content types, etc and node trying to figure all that out itself won't help anyone.

also why do you want to cache files?

[xtuc]

@devsnek Yes that's right, just wanted to bring that up because I think that's important to all agree on it.

also why do you want to cache files?

Sorry where's mentioned caching?

[devsnek]

@xtuc steps 3-5 are working on files cached in /tmp/ (which won't work on plenty of oses)

i'm unclear as to why source text is leaving node's memory

[benjamingr]

@devsnek presumably for the same reason ry caches files in deno - so it’s fast and consistent between loads and doesn’t delay startup time.

[devsnek]

@benjamingr wouldn't you just download the files in that case? i would expect an https import to hit the server every time.

[ljharb]

It must; in case there’s redirects or updated content. It might get a 304, of course.

[viktor-ku]

The reason behind caching is to avoid going to the internet if you already have this module downloaded. Why do you expect node to re-download module every time?

• If you specify url with certain version (see example) then you wouldn't need it to be re-fetched
• If you use master on un-versioned url then you will have this kind of issue, but then we can have a similar to ry idea with --reload command

[viktor-ku]

About tmp:

Guys, it is a draft. I am not gonna leave it as is. We simply need some convenient place to store this modules. Perhaps we can specify this place with flag or env (like in PR).

So you specify env variable and everything goes there.

[guybedford]

There’s nothing to stop this from being implemented in a loader already today. Basically a resolve hook can detect http requests, check if it is in the cache, downloading if necessary, then resolve to the file url of the stored cache entry through the resolve promise. Please do give it a go and share your findings, and just ask if you have any questions on the implementation or feedback on the loader hooks in the process too.

…

On Sat, 09 Jun 2018 at 17:38, Viktor Kuroljov ***@***.***> wrote: About *tmp*: Guys, it is a draft. I am not gonna leave it as is. We simply need some convenient place to store this modules. Perhaps we can specify this place with flag or env (like in PR). So you specify env variable and everything goes there. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#127 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAkiyt3iYvN4r9QRaiOnJrahoqNMk4aNks5t6-v8gaJpZM4UhUb5> .

[viktor-ku]

@guybedford I don't know how to create a loader at the moment. But I will start to research that

[ljharb]

A url is designed to have its contents cached, but urls always still hit the network (service workers aside). I would expect the same in node.

[viktor-ku]

@ljharb so you will use --reload a lot I guess ⌨️

[ljharb]

No, I’d expect that to be the default.