[Grace-core] Importing resources

[Michael Homer]

Hi,
I dug up my sketch from the last time I thought about this. This was
about as far as I got - there are some significant open questions that
linger. There is a three-sentence summary right at the end. Here's the
teaser: what if
  import "logo.png" as logo : Image
  import "about.txt" as aboutText : String
just did the right thing?

At the moment, you can import a Grace module with:
  import "path/to/module" as mod
to get a Grace object containing the top-level definitions from the
given module, bound to the name "mod". Given that this is "just" an
object, there's no reason it necessarily has to arise from Grace
source code. We already have the ability to import native code
conforming to the appropriate ABI, for example, which is how the "gtk"
and "dom" modules work.

One thing that would be interesting is if we could access other
resources through the same mechanism. In Grace, everything is an
object, so the representation of, say, an image, is naturally an
object. Because the implementation of objects is opaque, we don't know
what's in it or where it came from unless we wrote the object
constructor ourselves. We could access these through the import
mechanism.

The modules paper gestures at this a little, under the heading of
"External Data", but doesn't provide a particularly concrete
description of anything and points both at F# type providers and
importing text files with a particular prefix. An earlier version had
some more detail, and I implemented an "import hook" mechanism to deal
with that, but there were some practical problems wedging it in.
Import hooks relied on a prefix that took them outside the ordinary
import system, with "file://" used in the example but others envisaged
as well. Type providers remain an interesting area but I don't propose
to address them here.

Here is the system that I am semi-proposing. All existing working
import statements continue to have their current meaning. We instate a
syntactic restriction on the import paths of Grace modules: the given
import path may not contain a "." character (U+002e FULL STOP) after
the final occurrence of "/" (U+002f SOLIDUS). Remember that, when
importing a module, we do not include ".grace" in the name; this will
only prohibit importing from a path implying a source file name like
"foo.bar.grace".

If such a character does appear, this import statement is a "resource
import". A resource import still binds the given name to an object
identified by the given import path; the difference is that the
meaning of the import path may be interpreted by other code.

What code is that? It is determined by the part of the name following
the "." - in essence, the file extension. If I write this in my
program:
  import "my/tool/logo.png" as logo
then the "png" import handler is handed the path "my/tool/logo.png",
as a string, and returns an object to be bound to "logo". This object
can be anything the handler likes, but is probably some representation
of an image in this case. If I instead have:
  import "my/tool/licence.txt" as licence
then licence is likely to be a String.

How are these extensions mapped to handlers? Probably, a per-program
registry is defined, and user code can add entries to it (but see
below for potential issues). If I am using the "gtk" module, for
example, it may extend the registry with an entry mapping the "png"
extension to something creating a GTK+ image object. Other modules may
provide their own definitions. The system may predefine some handlers,
like "txt" as a String mapping, but these could be overridden.

What if we assume a Smalltalk-style interface, with a closed world,
and no "files"? Again, as the interpretation of the path is defined in
user code, it can be mapped according to whatever approach makes sense
in that system. The extensions are simply to identify an
interpretation in that case.

What advantages does this have? We can access resources, like images,
documentation, or sound files through a single consistent interface.
It fits entirely within the existing semantics of the language. It
also provides a logical location to store relevant resources adjacent
to the code that requires them, and even to distribute them in the
same way as code (perhaps through the package manager discussed
today). The meaning of 'import "logo.png" as logo' is likely to be
clear to the reader, even if they are unfamiliar with other code. The
disadvantage is that there is an added point of complexity in the
import system, and that a class of otherwise-valid filenames is
excluded from use as a module name.

There are some lingering issues with this:
1) This global registry may be subject to conflict if two different
third-party modules both want to hook onto the same extension. Some
way of resolving an ambiguity may be required.
2) A per-module registry might be required rather than per-program.
This is related to the previous issue and may resolve some of it.
3) Typing: given the runtime registry described here, assigning a
static type to the imported module is difficult and may be impossible
sometimes. We could require an explicit type annotation to be provided
in these cases, or use some sort of statically-determined registry
instead. In many ways gradual typing comes to the rescue here.
4) The registry introduces an ordering dependency for imports. If I
want to import a PNG resource as a GTK+ image, I need to have imported
the GTK+ module first. This may lead to errors sometimes. Note that
import order is already semantically important for ordinary modules.
5) Sensible interpretations may not map cleanly onto conventional file
extensions. Note that it is not necessarily the case that the "path"
given corresponds to an actual path - it simply identifies an
interpreter that can process the string - so a "false" extension could
be defined when required.
6) This system describes an essentially dynamic lookup process. In
Minigrace, this would map onto the same sort of lookup as happens for
dynamic modules now. Potentially it should statically include
resources instead; if all resources map directly onto concrete files
it could do this even with the dynamic registry. At the language
definition level this may be irrelevant, since there is no explicit
compile-time/run-time distinction, and no concept of static linking.
7) Maybe we don't want to make extensions semantically important (the
morally correct position). Another possibility is libmagic
autodetection, but that isn't very portable. MIME types are generally
available on the web and are the "right" way of identifying filetypes.
This approach would assume a one-to-one mapping onto some sort of
filesystem and from filetypes to interpretations, which may eliminate
other interesting or useful cases.

*****************
Short version
*****************
Proposed: That you can import resources using an import path with a
"." in it. The file extension maps to an object capable of
interpreting the path into an object to be bound to the "as" name. The
mapping from extension to interpreter is a per-program registry that
can be extended in user code.

A list of some potential issues is immediately above the line.
-Michael