Python

github.com/ornew/pego/parsers/python parses Python 3.14 as CPython 3.14.0 parses it, the whole language, with a parser generated by PEGO from python.pego. It depends only on the standard library.

go get github.com/ornew/pego/parsers/python

#Use

// A typed AST modeled on Python's ast module, with positions.
src := "import os\n\ndef greet(name: str, *, loud=False) -> str:\n    return f'hello {name}'\n"
m, err := python.ParseModule(src)
for _, s := range m.Body {
	switch s := s.(type) {
	case *python.Import:
		fmt.Printf("import of %d module at %d-%d\n", len(s.Names), s.Start, s.End)
	case *python.FunctionDef:
		fmt.Printf("def %s with %d parameters and %d keyword-only\n", s.Name, len(s.Args.Args), len(s.Args.KwOnlyArgs))
	}
}
// import of 1 module at 0-9
// def greet with 1 parameters and 1 keyword-only

// The same text as Python's ast.dump(ast.parse(source)), or with positions, include_attributes=True.
fmt.Println(python.Dump(m))
fmt.Println(python.DumpWithPositions(m, src))

// Every node, in source order.
python.Inspect(m, func(n any) bool {
	if name, ok := n.(*python.Name); ok {
		fmt.Println(name.Id, name.Ctx)
	}
	return true
})

// The value of a constant, as Python computes it.
v, err := e.(*python.Constant).Value() // *big.Int, float64, complex128, string, []byte, bool, nil or python.Ellipsis

// Syntax errors.
_, err = python.ParseModule("class A:\npass\n")
var se *python.SyntaxError
if errors.As(err, &se) {
	fmt.Println(se.Line, se.Col, se.Message()) // 2 1 expected an indented block
}

The AST follows Python's ast module (FunctionDef, BinOp, Compare, MatchClass, TypeAlias, Interpolation, ...), with a Span in every node: positions are in code points by default (python.ParseModule(src, python.Bytes) counts bytes). It differs from Python's in that it is a tree of the source:

• identifiers are strings, and an absent one (Python's None) is ""; they are the source text (NormalizeName);
• ctx is the string "Load", "Store" or "Del"; operators are *Op holding their source text;
• constants and the literal parts of f-strings and t-strings are source text, decoded by the methods above; FormattedValue and Interpolation keep the source of their expression, = and conversion;
• Python's List, Match and Expr are ListExpr, MatchStmt and ExprStmt; booleans replace the ints of AnnAssign.simple and comprehension.is_async.

#Conformance

The reference is CPython 3.14.0 (ast.parse, mode exec): the same programs must be accepted and rejected, with the same ast.dump, and for the accepted ones the same positions of every node (DumpWithPositions against ast.dump(..., include_attributes=True)). The numbers below were measured with the commands in Development, against CPython 3.14.0 (python-build-standalone) and the CPython checkout at tag v3.14.0.

Every difference these tests found was fixed (the snippets alone found five rules the grammar had missed, and the mutations a line continuation before the end of the input). The positions are the same for all 28,362 sources that both accept (the files and the snippets of the table). A fuzz test (FuzzParse) checks that the parser never panics and that ParseAST, Recognize and the dumps agree.

The tests that need only Go (go test) run the vendored reference data in testdata: the CPython results, from ast.parse, for the 61,540 snippets and for 13 test files of Lib/test (see testdata/README.md). The others run CPython 3.14 and are skipped without it.

#Deviations

• Only accepted programs are compared. An error is a *SyntaxError with the position and what was expected, or a short message (invalid syntax, unexpected indent, expected an indented block, too many levels of indentation, and a few more). CPython reports specific messages from the second pass of its parser, which retries a failed parse with its invalid_* rules (Maybe you meant '==' or ':=' instead of '='?, Missing parentheses in call to 'print', '(' was never closed) and has classes of its own for tabs and indentation (TabError, IndentationError). None of that is reproduced, and error positions are not compared with CPython's.
• ParseAST does not make the checks that ParseModule makes (Check): the names of \N{...} escapes, the digit limit, future features, <> and !=, and identifiers that normalize to a constant. 10 of the 35,341 invalid snippets are rejected only by them, and ParseAST alone accepts them.
• The input is text, as for ast.parse(str): the encoding declaration (# coding: latin-1) and a byte order mark are not read, so decode the file first (a BOM left in the text is an error, as in ast.parse(str)). Invalid UTF-8 is read as U+FFFD, which is accepted in strings and comments and rejected elsewhere. Only mode exec is implemented, with no type comments, feature_version or optimize.
• Nesting. More than 200 levels of brackets and more than 99 levels of blocks are rejected, as in CPython. CPython also rejects very deep expressions: MemoryError: Parser stack overflowed for 3,000 nested lambda: or 3,000 ** (a**b**...) and 10,000 not (but not for 3,000 not), RecursionError in the compiler for a chain of 20,000 + or attributes. This parser accepts all of them: its limit is the generated parser's 100,000 rule calls (between 40,000 and 50,000 levels of not, lambda: or conditionals, and 99,978 unary operators -, +, ~ or ** in a chain; chains of binary operators other than ** are not nested), beyond which it fails with an error instead of exhausting the stack.
• Spans are not CPython's positions in a few places: the span of a binary operation, attribute, subscript or call does not include the parentheses of an operand at its edge, the span of a decorated definition starts at the first decorator, and that of a compound statement stops before a ; that ends its last line. DumpWithPositions prints CPython's, computed from the spans and the source, and agrees with CPython on the corpora above.
• Python warnings (SyntaxWarning for invalid escape sequences, is with a literal, and so on) are not reported.

#Grammar

python.pego works on characters: there is no separate tokenizer. The state of what CPython's tokenizer tracks is in two variables, lv (the bracket depth, whether the string being read is an f-string, its quote and rawness, and the nesting of format specs, packed in one number) and ind (the indentation of the current block with CPython's rules for tabs and the number of enclosing blocks); a rule that opens a bracket or a block defines them again, so returning restores them (INDENT and DEDENT without tokens). The header of the grammar describes it.

Choices that measurably help, each commented in the grammar:

• The state is packed into few variables: a rule is memoized once per value of the variables it reads, and the version with separate variables for the quote of an f-string, for the parameters of a lambda and for barry_as_FLUFL was 10 to 16% slower and allocated 25 to 30% more on the benchmarks below. The lambda has its own parameter rules, and <> is checked after the parse.
• The nine levels of binary operators, the unary operators and ** are one Pratt expression instead of a chain of rules (3 to 8% faster).
• Rules that must be tried in order are written so that the common case, an expression statement, is parsed once and an assignment is recognized by what follows; primary returns an atom without running the Pratt expression of attributes, calls and subscripts when none follows it.
• Constants and string parts are kept as source text and decoded on demand (Constant.Value), and the large integers of the grammar are computed with multiplications (the bytecode backends truncate integer literals of 2^31 and more).

ParseAST builds its values directly, since every value has a Go type of its own, which the generator compiles into direct code (see the code generation guide).

#Performance

Parsing the largest modules of the standard library in memory (go test -bench .; the files are in testdata/bench; Apple M3 Max, Go 1.27, median of 3 runs of 100 iterations):

CPython's ast.parse was timed outside the Go tests with python3.14 -I internal/refgen/astparse_time.py testdata/bench/*.py (median of 30 runs after 30 warm-ups). So ParseAST takes about 1.9 times and 1.5 times as long as CPython's hand-written C parser, which builds its tree with Python objects, from a grammar that has no tokenizer. There is no Python parser in the Go standard library to compare with. Recognize builds nothing but is slower than ParseAST, whose rules the generator compiles into direct code.

About a third of the time of ParseAST is the tracking of what was expected for error messages: with that tracking switched off in a copy of the generated parser, _pydecimal.py takes 17.3 ms instead of 25.8 ms. The cost is in the generated runtime (expect and mergeExpected: a linear search in sets of up to 40 expectations, repeated for every memoized call), not in the grammar.

#Development

go generate ./parsers                 # regenerate parser.go after changing python.pego (from the repository root)
go test ./parsers                     # parser.go up to date; golden files on every backend of the engine
cd parsers/python && go test ./...    # the golden files, the vendored reference data, the examples (Go alone)

# The comparisons with CPython: PEGO_PYTHON is CPython 3.14 (python3.14 on PATH by default); its standard library is the
# corpus. PEGO_CPYTHON_SRC is a checkout of CPython 3.14.0 (git clone --branch v3.14.0 https://github.com/python/cpython),
# whose Lib/test is a corpus too and the source of the snippets.
export PEGO_PYTHON=python3.14 PEGO_CPYTHON_SRC=/path/to/cpython
go test -v -run 'TestCPython' .       # the corpora: files, snippets, positions, mutations, line structures
PEGO_PYTHON_ONLY=test_grammar go test -run TestCPythonCorpus .       # the files whose path contains a string
PEGO_MUTANTS=400000 PEGO_MUTANTS_SEED=1 go test -v -run 'TestCPythonMutations|TestCPythonLines' .

go test -fuzz FuzzParse               # fuzz the parser
go test -bench .                      # benchmarks (PEGO_BENCH_DIR: another directory with _pydecimal.py and typing.py)
python3.14 -I internal/refgen/astparse_time.py testdata/bench/*.py   # CPython's ast.parse on the same files

The vendored reference data (testdata/cpython-*.jsonl.gz) is regenerated from a CPython checkout with PEGO_PYTHON=python3.14 internal/refgen/generate.sh /path/to/cpython (run from this directory), and the Unicode tables (tables.go, unicode.bin and the identifier classes of the grammar) from CPython's unicodedata with tools/gentables.py (Unicode 16.0.0).