Building a TypeScript Syntax Tree with source from another language (F#)

[alfonsogarciacaro]

Hi there! Since TypeScript is getting more and more wonderful features like tuples or union types, I wonder if it makes sense to retarget a project like FunScript (F# to JS compiler) to TypeScript and leverage all its power. @ctaggart suggested we could use the improved compiler services to create an AST directly instead of generating TS code.

From the repository I can see there's still some work to be done with the compiler services. But if you have time, I'd like to make a couple of questions to know what to expect of the upcoming version.

• Do you plan to provide some documentation/tutorials for the compiler services? If not possible, will the compiler API be close enough to Roslyn to use this knowledge when working with it?
• Do you think it will be feasible to build a TypeScript AST directly with the info from another language (F# in this case, with many similarities)? Or will it be better to generate the TypeScript code directly?

Thanks a lot in advance!

[mhegazy]

@DanielRosenwasser is putting together a walk through, it would be great to get your feedback about it early.

[alfonsogarciacaro]

Will do! Thanks a lot for your reply. If @DanielRosenwasser can give me a link to the draft, I'll start checking it right away :)

[ctaggart]

I am also interested in the draft. I spent time in June trying to access the TypeScript AST from TypeScript itself. I put the effort aside while the new language service was being developed. With access to the TypeScript AST, we can figure out how consume it from .NET as I mentioned here.

[DanielRosenwasser]

Hey guys, we're working on it and looking at the current constraints on such a task. I will keep you all posted.

[alfonsogarciacaro]

No hurry! Please take your time and tell us if we can help with anything. Thanks a lot for all your hard work 👍

[ctaggart]

I got the ball rolling with TypeScript 1.4 AST from Node.js.

[ctaggart]

Now I'm trying to generate TypeScript files and actively working on this right now for work. I posted this as a question on StackOverflow:

How do I use TypeScript 1.4 API to write a .ts file?

Am I on the right track? What do I need to do to have it populate the .text properties?

///<reference path='../TypeScript/built/local/typescript.d.ts' />

import ts = require('typescript');

export function main() {
    var m = <ts.ModuleDeclaration>ts.createNode(ts.SyntaxKind.ModuleDeclaration);
    var nm = <ts.Identifier>ts.createNode(ts.SyntaxKind.Identifier);
    nm.text = "Blah"
    m.name = nm;

    var sf = <ts.SourceFile>ts.createNode(ts.SyntaxKind.SourceFile);
    sf.statements = <ts.NodeArray<ts.ModuleElement>>[m];

    console.log("file: "+sf.text); // sf.text is undefined :(
}

main();

[DanielRosenwasser]

Hey guys, we have some documentation in flux here.

I've spoken with others on the team; there are difficulties associated with dynamically creating an AST and using our emitter. For one thing, there are times in which we look back at the source text for certain operations, so you need to have a valid TypeScript corpus to base some checking/emit off of. I don't think an AST-to-AST transformation is going to work as very ideally.

It honestly may be better to do a straight TS emit if what you're looking for is definition file emit, if you're looking to use it as a verification pass for an intermediate language, or if you're looking to take advantage of our downlevel emit functionality.

@ctaggart to populate .text properties, you simply assign them as you are doing. Our approach to reasoning is "write once" throughout different stages of the compiler. So for instance, the text field will be set once during parsing and never written again; flags for type checking are only set during type checking and never written to again.

[basarat]

suggestion: putting the wiki in a repo of MD files would help with people updating and sending PRs

[basarat]

The basic building block of the Abstract Syntax Tree (AST). In general node represent non-terminals in the language grammar; some terminals are kept in the tree such as identifiers and literals.

Just curious which terminals are excluded? Perhaps comments. I haven't reviewed the grammer (I know its in the specification).

[DanielRosenwasser]

@basarat Wiki suggestion isn't a bad idea, we'll consider it.

Terminals which have useful (i.e. semantically meaningful) textual content are kept. So for instance, anything that has a text property will have that. On the other hand, you don't need something like the EqualsGreaterThan (=>) token when you know you have an arrow function, so we don't hang on to it.

Off the top of my head, I can think of

SyntaxKind
NumericLiteral
StringLiteral
TemplateHeadLiteral
TemplateMiddleLiteral
TemplateTailLiteral
Identifier
NoSubstitutionTemplateLiteral

If you do need tokens beyond those on any given Node, they can be "rehydrated" on demand using the sourcetext by calling getChildren().

[ctaggart]

@DanielRosenwasser Thanks for the documentation!

My current attempt is failing with a Error: Object #<SourceFileObject> has no method 'getSyntacticDiagnostics'. My guess is that I need to create the SourceFile some other way or do some more processing to it. Any ideas?

https://github.com./ctaggart/TsAst/blob/c6db6f6b5d35cae81e2cb037280bb2914ab403ab/app.ts

///<reference path='../TypeScript/built/local/typescript.d.ts' />
import ts = require('typescript');

// https://github.com./Microsoft/TypeScript/wiki/Using-the-Compiler-API

function compile(sourceFile: ts.SourceFile) {
    var outputs = [];
    var compilerOptions: ts.CompilerOptions = {
        noImplicitAny: true, noEmitOnError: true,
        target: ts.ScriptTarget.Latest, module: ts.ModuleKind.CommonJS
    };
    var defaultCompilerHost = ts.createCompilerHost(compilerOptions);
    var compilerHost: ts.CompilerHost = {
        getSourceFile: function (filename, languageVersion, onError) {
            console.log("getSourcefile: " + filename);
            if (filename === "file.ts")
                //return ts.createSourceFile(filename, contents, compilerOptions.target);
                return sourceFile;
            return defaultCompilerHost.getSourceFile(filename, languageVersion, onError);
        },
        writeFile: function (name, text, writeByteOrderMark) {
            outputs.push({ name: name, text: text, writeByteOrderMark: writeByteOrderMark });
        },
        getDefaultLibFilename: defaultCompilerHost.getDefaultLibFilename,
        useCaseSensitiveFileNames: defaultCompilerHost.useCaseSensitiveFileNames,
        getCanonicalFileName: defaultCompilerHost.getCanonicalFileName,
        getCurrentDirectory: defaultCompilerHost.getCurrentDirectory,
        getNewLine: defaultCompilerHost.getNewLine
    };
    var program = ts.createProgram(["file.ts"], compilerOptions, compilerHost);
    var errors = program.getDiagnostics();
    if (!errors.length) {
        var checker = program.getTypeChecker(true);
        errors = checker.getDiagnostics();
        program.emitFiles();
    }
    return {
        outputs: outputs,
        errors: errors.map( e => e.file.filename + "(" + e.file.getLineAndCharacterFromPosition(e.start).line + "): " + e.messageText )
    };
}

export function main() {
    var m = <ts.ModuleDeclaration>ts.createNode(ts.SyntaxKind.ModuleDeclaration);
    var nm = <ts.Identifier>ts.createNode(ts.SyntaxKind.Identifier);
    nm.text = "Blah"
    m.name = nm;

    var sf = <ts.SourceFile>ts.createNode(ts.SyntaxKind.SourceFile);
    //sf.statements = <ts.NodeArray<ts.ModuleElement>>[m];

    var result = compile(sf);
    console.log(JSON.stringify(result));
}

main();

[DanielRosenwasser]

SourceFiles are a bit more involved than other Nodes - they are created in the createSourceFile function in parser.ts rather than simply with createNode.

[alfonsogarciacaro]

Thanks a lot for the documentation @DanielRosenwasser and for the work on the compiler services. They're looking great! Understood about emitting source files instead of building an AST directly. In any case, understanding the AST makes it easier to build an emitter and we can still use the compiler API to parse d.ts files for interacting with JS libraries in a type-safe way from F#.

Thanks again! I'm closing the issue now. Please reopen it if necessary.

[DanielRosenwasser]

Ah, understood - that's a use case I hadn't yet realized. Please feel free to ask us any further questions about it.

[asakusuma]

For future travelers, @DanielRosenwasser just added this example on how to build a source string from a synthetic AST: https://github.com./Microsoft/TypeScript/wiki/Using-the-Compiler-API#creating-and-printing-a-typescript-ast