Import name projections don't support exports by interface name

[peterhuene]

Currently the projection rule for import names looks like:

projection    ::= '/' <label>
label         ::= <word>
                | <label> '-' <word>

This prevents projections for exports using the interfacename format, as these cannot be expressed simply as a label.

Say component foo:bar exports an instance with name bar:baz/qux (i.e. in interfacename format) and we wanted to import this instance from another component, but reference a possible implementation using an unlocked-dep import rather an import of an interface name. We're unable to express this because bar:baz/qux cannot be represented as a projection.

One possible solution:

Extend the projection rule as follows:

projection          ::= '/' <projection-segment>
projection-segment  ::= <label> | '<' <interfacename> '>' 

Or using some other delimiter to denote that the segment is an interface name and not a label.

[lukewagner]

Interesting question! Could one solution be to say that, if you are wanting to import the, let's say, wasi:http/incoming-handler interface of a component with package name ns:foo, you could generate an import:

(import "unlocked-dep=<ns:foo>" (instance
  (export "wasi:http/incoming-handler" (instance
    (export "handle" (func ...))
  ))
))

Hence, we're using nested instance types and subtyping to project out just the subset of the ns:foo/bar component that we want.

Do you think that could work? What's kindof neat about this approach is that is supports arbitrary subsets of a component's exports while still maintaining that they came from the same instance and share resource types.

[peterhuene]

That's the ideal solution at the component model level, but WIT cannot express that (i.e. an instance exporting an instance), which is what source bindings generation is based off of and that ultimately controls the component type one can build in the guest languages that use WIT for "componentization".

If we ditched WIT to describe the resulting component type, then we could encode a more complex import.

The way this is implemented currently is that every interface export of a dependency is treated as its own import of that interface, with the projection of unlocked-dep used to inform which export of the dependency is expected to satisfy the import, by default.

That's solely based off of limitations in WIT. It also makes for less than ideal bindings generation, as the individual imports don't group well.

[lukewagner]

Gotcha, makes sense. Since we haven't yet extended WIT to express locked/unlocked deps at all (mea culpa), perhaps the right fix here is to consider what the right new WIT syntax is for locked/unlocked deps and then see how this use case fits into the WIT syntax so that it can be encoded as a component-level import like I showed above. Does that sound right to you?

[peterhuene]

I think so? I don't think developers would author WIT with dependencies in it (at least by hand; tools that know about dependencies might insert these automatically), but if the intention is to always be able to see a (conforming*) component's type described by WIT and the component has dependencies, then we'll need a way to express those dependencies in WIT.

* obviously WIT supports only a subset of the component model, so the component can't express types outside of WIT representation

[peterhuene]

Spitballing:

world type-of-dep {
  export wasi:http/incoming-handler;
  export foo: func() -> string;
}

world my-components-type {
  unlocked-dep foo:bar@{>=1.0.0 <1.1.0}: type-of-dep;
  locked-dep bar:[email protected]: type-of-dep;
}

Or something like that (with syntax for all the information conveyable via unlocked-dep and locked-dep).

It would be nice if we could support this in bindings generation too, so bindings would get a foo::bar::<all exports of foo:bar here> rather than the piecemeal approach (have an import for each individual export) currently supported by cargo-component.

[lukewagner]

I don't think developers would author WIT with dependencies in it,

I think you're right that WIT-with-dependencies is not something most devs would ever want or need to author. We're probably already on the same page on this, but just to be explicit to for anyone else's benefit and to see if there's any mismatch: I'm assuming devs are using some build tool with a build-config (like cargo component with Cargo.toml or jco componentize with package.json) that allows them to express dependencies like normal package dependencies and then the build tool takes care of taking the explicit target world (like wasi:http/proxy) and enriching it with unlocked-dep instance-imports based on the build-config thereby synthesizing a new enriched world that is fed into wit-bindgen. In this context, the WIT is just an internal artifact passing between two tools, but it's perhaps useful for devs to be able to see it if they want (for the same reason a C/C++ dev occasionally wants to see the preprocessor's output) via some non-default build flag. (What's also nice about this "synthesize an enriched world" approach is that it should help handle tricky situations involving type-sharing of resource types.)

Spitballing: [...]

Cool, that looks pretty reasonable. Using unlocked-dep/locked-dep as the leading tokens for a new syntactic form makes sense (as they only make sense with imports and thus there's no import/export ambiguity). Just to check: in addition to allowing type-of-dep to reference a world, could it also reference an interface? And then, in the very-common case, would we expect to see unlocked-deps mostly referring to interfaces and locked-deps mostly referring to worlds?

Lastly, an idea: if we ever expect devs to manually author these worlds which have deps (which you could imagine happens when someone wants to do something advanced that is not supported by the build-tools), we could allow omitting the : type-of-dep and then the WIT tooling (using registry integration) could fetch the package and derive the type (interface by default for unlocked-dep and world by default for locked-dep). Maybe we don't do it first, but it'd be good to anticipate it if it makes sense.

[peterhuene]

Just to check: in addition to allowing type-of-dep to reference a world, could it also reference an interface? And then, in the very-common case, would we expect to see unlocked-deps mostly referring to interfaces and locked-deps mostly referring to worlds?

I'm confused by the distinction here. Wouldn't both unlocked-dep and locked-dep reference a world as the dependency would want to be able to export an interface (currently only expressible by a world)?

That said, probably moot as I think we would just not have a type specifier at all (see next).

, we could allow omitting the : type-of-dep and then the WIT tooling (using registry integration) could fetch the package and derive the type

Agreed, we can probably just remove the type specifier entirely and reference the dependency solely by package name. The WIT parser could just treat it like any other package reference and get the type of that package (resolved via filesystem or registry), rather than spell it all out. The key differentiator with this package reference, though, is that it would be to an implementation component and not another WIT package.

[lukewagner]

Great, agreed with the latter point. I do expect that the explicit : type-of-dep is useful in advanced cases and also probably makes sense as our first step (since it's lower-tech and what the registry-powered version is sugar for).

I'm confused by the distinction here. Wouldn't both unlocked-dep and locked-dep reference a world as the dependency would want to be able to export an interface (currently only expressible by a world)?

In the common unlocked-dep case, I think the raw output component we want imports an instance-type and thus has no knowledge or concern for the imports of the dependency. (It is the later dep-solve tool that emits the locked component that has to care about component types, but it's emitting locked-deps.) In the limit, though, I think both unlocked-deps and locked-deps have advanced cases that want to be able to refer to both instance- and component-types and thus both should be allowed (treating the type as a black box).

But maybe I'm missing something about the scenario here?

[peterhuene]

In the common unlocked-dep case, I think the raw output component we want imports an instance-type and thus has no knowledge or concern for the imports of the dependency.

Agreed. It would import an instance that itself may export instances, which is why I used a world to describe that manually above as we can't express that with an interface.

It would be expected that said world would not express any imports because ultimately it's representing an instance type and not a component type, but purely as a workaround for the limitation of interface in WIT.

[lukewagner]

Agreed. It would import an instance that itself may export instances, which is why I used a world to describe that manually above as we can't express that with an interface.

Ohhh, I see, thanks. I think we're missing a feature in WIT here to achieve expressive parity with the C-M which is: nested interfaces. In theory, we could just allow export some-interface inside interface (the same way as you can write it inside a world), and that would essentially mirror what you can do in instance-types, but maybe that looks wrong since interfaces don't otherwise "export" anything (reflecting the fact that interfaces can be imported just as well as exported). Thus, we'd have to come up with some reasonable syntax for an interface nesting another interface, but with that settled, I think it would allow you to express the exports of a component that itself exports interfaces as an interface.

Does that make sense?

[lukewagner]

Oh, and one addition to that: the reason to do via nested interfaces (vs. a world) is b/c importing an unlocked-dep via world should translate to an import of a component-type (sure, with no imports, but a component-with-no-imports is quite different from an instance, since the former is effectively a factory for constructing instances).

[peterhuene]

Does that make sense?

Definitely.

the reason to do via nested interfaces (vs. a world) is b/c importing an unlocked-dep via world should translate to an import of a component-type

So I think we may also want it translated to an import of an instance and not just an import of a component, depending on context; to me, the distinction is the intention of "I don't care how my dependency was instantiated" (instance) and "I'm the one instantiating my dependency" (component).

The output of a tool that is authoring a single component (e.g. cargo-component) would likely want to still import an instance (that itself exports instances) for an unlocked-dep; the unlocked-dep would encode the dependency information for a composition tool to use at a later time. It would likely be the output of a composition tool that would import a component via an unlocked-dep or locked-dep.

[lukewagner]

So I think we may also want it translated to an import of an instance and not just an import of a component, depending on context; to me, the distinction is the intention of "I don't care how my dependency was instantiated" (instance) and "I'm the one instantiating my dependency" (component).

Yes, I think you're right, these are the two intentions that the developer wants to express. My impression is that, when you mention a type directly in an import, you should get exactly that type. Rather, it's only when you mention a component implementation that we have the question of whether you want to import its full component-type or just its exported instance-type. I was thinking maybe we say that unlocked-dep defaults to instance-type (with some way to override) whereas locked-dep defaults to component-type (again, with a way to override), but I worry that will end up feeling ad hoc. A more regular scheme might be to say that if you write:

world w {
  unlocked-dep ns:foo;
  locked-dep ns:bar;
}

you're importing a component-type for both, and if you want an automatically-derived instance-type you write:

world w {
  unlocked-dep instance-of ns:foo;
  locked-dep instance-of ns:bar;
}

which is very explicit. Since mostly noone will write this by hand, perhaps this explicitness is good?