TypeScript

github.com/ornew/pego/parsers/typescript parses TypeScript 5.9 and TSX as the parser of the TypeScript 5.9.3 compiler does, with a parser generated by PEGO from typescript.pego, a port of the compiler's src/compiler/parser.ts function by function. The AST is the compiler's: the types are named after its SyntaxKinds, their fields are in the order of its forEachChild, and the ranges are those of its getStart and getEnd. It depends only on the standard library.

go get github.com/ornew/pego/parsers/typescript

#Use

// Typed values with positions: a SourceFile of Statements, each node with its Span in the input.
f, err := typescript.ParseAST("export const answer: number = 6 * 7;")
decl := f.Statements[0].(*typescript.VariableStatement).DeclarationList.Declarations[0]
fmt.Println(typescript.AsNode(decl.Initializer).Range()) // 30 35: the BinaryExpression 6 * 7

// .tsx files, or by the name of the file, as ts.createSourceFile chooses.
f, err = typescript.ParseTSX("const el = <ul className=\"list\">{items.map((i) => <li key={i}>{i}</li>)}</ul>;")
f, err = typescript.ParseFile("main.mts", src) // by the name: "x.tsx" is TSX; a module's top level may await

// Walk the tree by kind, in the order of forEachChild.
f, err = typescript.ParseAST("fetch(url).then(log)")
typescript.Inspect(f, func(n typescript.ASTNode) bool {
	if call, ok := n.(*typescript.CallExpression); ok {
		fmt.Println(typescript.Kind(call.Expression), call.Start, call.End)
	}
	return true
})
// PropertyAccessExpression 0 20
// Identifier 0 10

// Syntax errors, and only checking.
_, err = typescript.ParseAST("const a = 1;\nconst b = ;")
var se *typescript.SyntaxError
if errors.As(err, &se) {
	fmt.Println(se.Line, se.Col, se.Message()) // 2 11 expected an expression
}
err = typescript.Recognize("interface A { x: number }") // nil

Positions are in code points by default; ParseAST(src, typescript.Bytes) counts bytes. A node's Span covers its tokens without the white space and comments before it, and the tree has the nodes the compiler's forEachChild visits: modifiers, decorators and type parameters are lists, tokens such as ?, !, ... and => are nodes (only where the compiler keeps them), and Identifier, Token, Modifier, KeywordTypeNode and the literals hold their Text. The fields of union types have interface types (Expression, TypeNode, ...); use a type switch, or AsNode to read the range and Kind for the kind of the compiler's node.

#Conformance

The parser is checked against the compiler itself, with differential tests. Each test case of tests/cases/compiler and tests/cases/conformance of the TypeScript repository (tag v5.9.3) is split into the files its // @filename directives name, as the compiler's test harness does, and each .ts, .mts, .cts and .tsx file is parsed by the compiler (ts.createSourceFile, run with node) and by ParseFile. The file agrees if

• the compiler reports no parse diagnostic and the parser accepts the file, and the kinds, ranges (UTF-16 offsets of getStart and getEnd) and numbers of children of all nodes of the two trees are equal, or
• the compiler reports parse diagnostics and the parser rejects the file (the positions of the errors are not compared).

14,799 of 14,804 files agree, 14,043 with the same tree and 756 rejected by both:

(The test cases hold 1,828 more files that are not .ts, .mts, .cts or .tsx, such as .js and .json; they are not compared.)

The five files that differ are listed in testdata/tsc-known-failures.list; the test fails on any other difference and on a listed file that no longer differs:

• compiler/amdModuleName2.ts, compiler/invalidReferenceSyntax1.ts and conformance/node/nodeModulesTripleSlashReferenceModeOverrideModeError.ts: the compiler reports diagnostics for triple-slash directives (two amd-module names, an unterminated reference path, a resolution-mode that is not require or import) while it processes pragmas. They are not syntax, and the parser accepts the files.
• conformance/classes/constructorDeclarations/quotedConstructors.ts: "\x63onstructor"() {} is the constructor of the class for the compiler, which compares the text of the name after decoding it; the parser reads a method (only constructor, "constructor" and 'constructor' spelled out are constructors).
• conformance/jsx/tsxOpeningClosingNames.tsx: the closing tag </A . B . C.D> has white space between the parts of its name, which the compiler accepts and the parser rejects.

Other tests, which need the compiler too:

• TestTypeScriptLib compares the trees of the 102 declaration files of the typescript package (lib/*.d.ts, from 1 KB to 1.8 MB), all of which agree.
• TestTSCMutations mutates test cases (a token deleted, duplicated, replaced by one of 98 tokens where the grammar decides, inserted, or swapped with the next) with a seeded generator, and compares acceptance and trees. On 150,000 mutations (seeds 1 to 5, 30,000 each) 32 differ: 17 are triple-slash pragmas, 6 the escaped constructor, 1 the JSX closing tag, 3 an await after the name of a class or a namespace in a module (export class B await {}: the compiler drops the diagnostic at the start of a statement that holds a top-level await in its reparse of the module, the parser reports it), and one each of interface I { [a, yield b]: number }, create<readonly>(x), typeof M.<T,>, import< <Promise<any>>("m") and an identifier escape after this (this\u{abcd}), which the compiler rejects and the parser accepts or the other way round.

The tests that need no external tool: TestGolden (the trees of testdata/*.txt, also checked on every backend of the engine by go test ./parsers), TestValid and TestInvalid (sources the parser must accept and reject), TestSpans, TestUnits, TestParseFile, TestErrors, TestKind, the Value tests and FuzzParse (go test -fuzz FuzzParse: ParseAST and Recognize agree on any input, and the ranges of a parsed file are valid; 2.8 million executions in 60 seconds without a failure).

#Running the differential tests

They are skipped unless two environment variables are set:

# The compiler: node and the typescript package, version 5.9.3
mkdir /tmp/ts && cd /tmp/ts && npm install typescript@5.9.3
# The test cases of the TypeScript repository at the same version
git clone --depth 1 --branch v5.9.3 https://github.com/microsoft/TypeScript /tmp/TypeScript

cd parsers/typescript
export PEGO_TYPESCRIPT=/tmp/ts/node_modules/typescript
export PEGO_TYPESCRIPT_TESTS=/tmp/TypeScript/tests/cases
go test -run 'TestTypeScriptSuite|TestTypeScriptLib|TestTSCMutations|TestTSCSnippets' -v .   # the table above is its output
go test -run TestTypeScriptSuite -tsc.v .                      # every difference, with the text around it
go test -run TestTypeScriptSuite -tsc.run 'classes/.*' .       # only the test cases whose path matches
go test -run TestTSCMutations -tsc.mutations 30000 -tsc.seed 2 .
go test -run TestTypeScriptSuite -tsc.update .                 # rewrite testdata/tsc-known-failures.list

testdata/tsc.js is the node program the tests talk to: it parses a source and writes the compiler's parse diagnostics and its tree.

#Grammar

typescript.pego has one rule or a few for each parse... function of the compiler's parser, with its name in a comment where it helps. The compiler's context flags (yield, await, in disallowed, decorator, no return type on arrow functions, conditional types disallowed, JSX) are bits of one variable, ctx, which rules test with predicates; one variable rather than seven because rules that read variables are memoized per value. The start rule main parses a .tsx file when the input starts with U+0000, which no TypeScript file can start with, and ParseTSX adds it and moves the positions back, so one generated parser (and one set of AST types) serves both; U+0001 after it reads the top level in the await context, which ParseFile does for modules.

Whitespace is skipped by the callers, not by the rules, so that every node spans exactly its tokens (ws skips white space and comments, sp the same without crossing a line break; where the compiler checks scanner.hasPrecedingLineBreak() the grammar uses sp, or nlAhead). The grammar is as lenient as the compiler's parser (see the package documentation), and the other differences from it are the five above.

#Performance

ParseAST on files of 220 KB to 6 MB, on an Apple M3 Max (go test -bench ., the fastest of five runs of ten iterations; the machine was not idle, so the figures are conservative), next to the compiler's parser (ts.createSourceFile on the same file in node 24 after a warm-up, with node testdata/tsc-bench.js $PEGO_TYPESCRIPT file..., which is not part of the Go tests):

The benchmarks of the synthetic source need nothing; the others read the typescript package from PEGO_TYPESCRIPT and are left out without it. The parser is 1.5 to 2.5 times slower than the compiler's hand-written parser on declaration files and 3 to 4.5 times on code (the synthetic source, dense in generics and types, is the slowest per byte). Recognize builds nothing but is slower than ParseAST, whose rules the generator compiles into direct code (see the code generation guide); Parse builds the generic tree.

The first version of the grammar, which followed the compiler's order of alternatives, took about 110 ms, 10 ms, 110 ms, 60 ms and 1.55 s for ParseAST on these files. The optimizations are commented in the grammar; what paid was testing the next character before trying a rule that fails at once and records what it expected (the operators of the Pratt loops, whitespace, modifiers, arrow functions), dispatching on the first character or word (primary expressions, types, statements), and parsing a union, intersection or conditional type or expression once instead of parsing its first operand, failing, and parsing it again from the memo. The number of rule calls on 300 KB of the bundle fell from 1.66 million to 0.54 million.

#Development

go generate ./parsers                 # regenerate parser.go after changing typescript.pego (from the repository root)
go test ./parsers                     # parser.go up to date; golden files on every backend of the engine
cd parsers/typescript && go test ./...   # the tests that need no compiler, and the differential tests if the variables are set
go test -fuzz FuzzParse               # fuzz the parser
go test -bench .                      # benchmarks