PEGO
Package pego
import "github.com/ornew/pego"Generated from the source of this version. Also on pkg.go.dev.
Overview
Package pego is the public API of the PEGO parser framework.
A grammar is written in the PEGO grammar language (see ParseGrammar), built as an AST with the grammar package, or decoded from JSON. Compile turns it into a Parser, and Parser.Parse parses input with it.
Index
Constantsfunc GenerateGofunc GenerateTypeScriptfunc IsCompiledfunc ParseGrammartype Backendtype Documenttype Findingtype GenOptiontype LintChecktype LintOptiontype Locationtype MarshalOptiontype Nodetype ParseOptiontype ParseStatstype Parserfunc Compilefunc CompileSourcefunc LoadParserfunc (*Parser) Grammarfunc (*Parser) Marshalfunc (*Parser) MarshalBinaryfunc (*Parser) NewDocumentfunc (*Parser) Parsefunc (*Parser) ParseStreamfunc (*Parser) Startfunc (*Parser) WithStarttype Profiletype RuleProfiletype Severitytype SyntaxErrortype SyntaxErrorstype TraceEventtype TraceKindtype Unit
Constants
const (
// SeverityHint is a suggestion, usually about performance; the grammar is correct as written.
SeverityHint = lint.Hint
// SeverityWarning is a likely mistake, or an expression that has no effect.
SeverityWarning = lint.Warning
// SeverityError is a certain mistake: part of the grammar can never take effect.
SeverityError = lint.Error
)const (
// CodePoints counts Unicode code points. It is the default.
CodePoints = engine.CodePoints
// Bytes counts UTF-8 bytes.
Bytes = engine.Bytes
)const (
// DefaultBackend uses Closure when the grammar AST is available and Bytecode otherwise.
DefaultBackend = engine.Default
// Closure compiles parsing expressions into Go closures.
Closure = engine.Closure
// Bytecode runs language-independent bytecode (see spec/bytecode.md) on a VM that uses Go
// recursion for rule calls.
Bytecode = engine.Bytecode
// BytecodeIterative runs the bytecode on a VM that keeps rule calls on its own stack, so deeply
// nested input does not consume the Go stack.
BytecodeIterative = engine.BytecodeIterative
)const (
// TraceEnter is reported when a rule is called, before the memo is consulted.
TraceEnter = engine.TraceEnter
// TraceExit is reported when a rule call returns.
TraceExit = engine.TraceExit
)Functions
func GenerateGo source
func GenerateGo(g *grammar.Grammar, pkg, start string, opts ...GenOption) ([]byte, error)GenerateGo generates the source code of a standalone Go parser for g in package pkg. The generated code depends only on the standard library; its Parse(input) parses from the rule start. It does not support stream or incremental parsing.
func GenerateTypeScript source
func GenerateTypeScript(g *grammar.Grammar, start string, opts ...GenOption) ([]byte, error)GenerateTypeScript generates the source code of a standalone TypeScript parser for g: a single ES module without dependencies, whose parse(input) parses from the rule start and returns the same trees and errors as the engine. WithRecognize also generates recognize; WithTypes is not supported. Like GenerateGo, it does not support stream or incremental parsing.
func IsCompiled source
func IsCompiled(data []byte) boolIsCompiled reports whether data is a compiled grammar written by MarshalBinary or Marshal.
func ParseGrammar source
func ParseGrammar(src string) (*grammar.Grammar, error)ParseGrammar parses PEGO source code into a grammar AST.
Types
type Backend source
type Backend = engine.BackendBackend selects how a grammar is executed. All backends produce identical results.
The type it stands for is defined in an internal package as follows. source
type Backend intBackend is the backend used to run a parse.
func (Backend) String source
func (b Backend) String() stringtype Document source
type Document struct {
// contains filtered or unexported fields
}Document is a text that is edited repeatedly. Parsing after an edit reuses the results of the previous parse that the edit did not affect (incremental parsing), as needed by editors. A Document is not safe for concurrent use.
Reused results are shared with the trees returned by earlier parses, and the nodes after an edit are moved to their new positions in place: an earlier tree changes when Parse runs again, and is then a mix of nodes at new positions (those reused) and at old ones. Parse writes it, so it must not be read concurrently with Parse. Use Node.Clone to keep a tree as it was.
func (*Document) Edit source
func (d *Document) Edit(start, end int, text string) errorEdit replaces the range [start, end) of the text, in the document's position unit, with text. With byte positions, start and end must be on character boundaries.
func (*Document) Parse source
func (d *Document) Parse() (*Node, error)Parse parses the current text. Its results and errors are those of Parser.Parse.
func (*Document) Stats source
func (d *Document) Stats() ParseStatsStats returns the evaluation and reuse counts of the last Parse.
type Finding source
type Finding = lint.FindingFinding is a likely mistake that Lint found in a grammar: where it is (Pos, the zero Pos for a grammar without positions), how serious it is, the check that found it, the rule it is in, a message, and a suggested fix ("" if there is no single obvious one).
func Lint source
func Lint(g *grammar.Grammar, start string, opts ...LintOption) ([]Finding, error)Lint reports likely mistakes in the grammar g, whose start rule is start: rules the start rule never uses, alternatives and expressions that can never match or have no effect, captures that are never read, and grammar shapes that slow down incremental parsing. The findings are sorted by position. Comments of the form "// lint:ignore check reason" in the source suppress findings (see docs/guide/linting.md).
Lint returns an error, and no findings, if g does not compile, if start is not a rule of g, or if an option names an unknown check.
func WithRecognize source
func WithRecognize() GenOptionWithRecognize also generates a function Recognize, which checks the input against the start rule without building a tree, like Parse with RecognizeOnly.
func WithTypes source
func WithTypes() GenOptionWithTypes also generates a Go type for each type of the grammar and a function ParseAST, which returns the result of the start rule as values of those types instead of *Node.
func WithoutPackageDoc source
func WithoutPackageDoc() GenOptionWithoutPackageDoc leaves the package comment out of a generated Go parser, for a package that has its own documentation in another file.
type LintCheck source
type LintCheck = lint.CheckLintCheck describes a check of Lint: its name, the highest severity it reports, and what it looks for.
func DisableChecks source
func DisableChecks(names ...string) LintOptionDisableChecks turns off the named checks (see LintChecks).
type Location source
type Location = engine.LocationLocation is a position with its line and column.
The type it stands for is defined in an internal package as follows. source
type Location struct {
Pos int `json:"pos"`
Line int `json:"line"`
Col int `json:"col"`
}func (Location) String source
func (l Location) String() stringString returns line:col, or "position N" when the line is unknown (Line is 0: the position was in input that a stream parse had discarded; see TraceEvent.LineCol).
type MarshalOption source
type MarshalOption func(*engine.MarshalOptions)MarshalOption configures Marshal.
func WithoutAST source
func WithoutAST() MarshalOptionWithoutAST omits the grammar AST. The file becomes smaller and holds only the bytecode needed to run, but a parser loaded from it can only use the bytecode backends, and its Grammar method returns nil.
type Node source
type Node = engine.NodeNode is a node of a parse result. Nodes can be encoded to JSON with encoding/json.
The type it stands for is defined in an internal package as follows. source
type Node struct {
// Start and End are the range in the input, in the parse's position unit (End is exclusive).
Start, End int32
// Text is the text of a terminal.
Text string
// Children are the children of Seq, List, and Operator nodes. Omitted elements are nil.
Children []*Node
// Fields are the struct fields and captures. Each value is a *Node, int, string, bool, or nil.
Fields Fields
// contains filtered or unexported fields
}Node is a value produced by parsing. It is a CST node, a terminal, a list, or a struct built by an action. In JSON it is an object with the members type, rule (if any), start, end, text (if any), children (if any) and fields (if any).
func (*Node) Clone source
func (n *Node) Clone() *NodeClone returns a deep copy of the tree n. Subtrees shared within the tree stay shared in the copy. A Document moves the nodes it reuses after an edit, which changes trees returned by its earlier parses; a clone keeps a tree as it was.
func (*Node) Field source
func (n *Node) Field(name string) anyField returns the value of the field name.
func (*Node) IsTerminal source
func (n *Node) IsTerminal() boolIsTerminal reports whether the node is a terminal (Match or a terminal type).
func (*Node) MarshalJSON source
func (n *Node) MarshalJSON() ([]byte, error)MarshalJSON encodes the node as an object (see Node).
func (*Node) Rule source
func (n *Node) Rule() stringRule returns the name of the rule that produced the node, if it was produced by a rule without an action, or "".
func (*Node) String source
func (n *Node) String() stringString renders the node as an S-expression-like string, for tests and debugging.
"a" Match (terminal) Number"12" user-defined terminal type (Seq "a" "b") Seq ["a" "a"] List (Op L=1 R=2) struct x@rule node produced by the rule rule without an action
func (*Node) Type source
func (n *Node) Type() stringType returns the node's type name: a reserved type (Match, Seq, List, Operator, Error) for CST nodes, or otherwise the name of a type defined in the grammar.
type ParseOption source
type ParseOption func(*engine.ParseOptions)ParseOption configures a single parse.
func RecognizeOnly source
func RecognizeOnly() ParseOptionRecognizeOnly checks whether the input matches the grammar without building a tree. Parse then returns a nil node and the same syntax errors as a full parse. Actions are not evaluated, so runtime errors in actions are not reported; captures that predicates read are still built. Recognition is faster and allocates less than a full parse. It cannot be used with ParseStream or Document.
func WithMaxDepth source
func WithMaxDepth(n int) ParseOptionWithMaxDepth limits the nesting of rule calls to n; deeper nesting makes the parse fail with an error. The operand of a prefix operator and the right operand of an infix operator of a Pratt expression nest like rule calls. The default is 100,000 (10,000,000 for BytecodeIterative). The limit keeps runaway recursion from exhausting the stack or memory. The Closure and Bytecode backends nest rule calls on the goroutine stack, so raising the limit far above the default can exceed Go's maximum stack size, which aborts the program; use BytecodeIterative for deeper nesting. A call answered from the memo does not nest, so whether a parse that comes close to the limit succeeds can depend on memoization.
func WithProfile source
func WithProfile(prof *Profile) ParseOptionWithProfile adds the cost of the parse to prof. It traces the parse (see WithTrace), so the parse is several times slower, and the times in prof are meaningful relative to each other only. A nil prof is ignored.
func WithTrace source
func WithTrace(f func(TraceEvent)) ParseOptionWithTrace calls f at the start and at the end of every rule call of the parse, in order, so that the events nest like the calls (the depth of an event is in TraceEvent.Depth). It reports the calls the backend makes: a call skipped because the next character cannot start the rule (first-character dispatch) is not reported, and the backends do not skip the same calls. Tracing does not change the result, but it makes the parse several times slower. The event's methods (LineCol, Text, Failure) read the parser's state, so call them from f; its fields can be kept. With several WithTrace options, each function is called in order. A nil f is ignored.
Tracing works with every backend, with Parse, RecognizeOnly, ParseStream and Document (whose option applies to every Document.Parse), but not with generated Go parsers.
func WithUnit source
func WithUnit(u Unit) ParseOptionWithUnit selects the position unit. Matching always proceeds by code point, whatever the unit.
type ParseStats source
type ParseStats = engine.StatsParseStats counts rule evaluations and reused results in a Document.Parse.
The type it stands for is defined in an internal package as follows. source
type Stats struct {
Evaluated int // number of rule body evaluations
Reused int // number of memo results reused
}Stats counts rule evaluations and memo uses during a parse.
type Parser source
type Parser struct {
// contains filtered or unexported fields
}Parser is a compiled grammar. It is safe for concurrent use by multiple goroutines.
func Compile source
func Compile(g *grammar.Grammar, start string) (*Parser, error)Compile compiles the grammar g into a parser that starts parsing at the rule start.
func CompileSource source
func CompileSource(src, start string) (*Parser, error)CompileSource compiles PEGO source code into a parser that starts parsing at the rule start.
func LoadParser source
func LoadParser(data []byte) (*Parser, error)LoadParser loads a parser saved by MarshalBinary or Marshal. Corruption is detected by a checksum and structural validation, but types are not checked again: load only data from trusted sources, such as the output of pego compile or MarshalBinary.
func (*Parser) Grammar source
func (p *Parser) Grammar() *grammar.GrammarGrammar returns the parser's grammar. The grammar of a loaded parser has no source positions, and it is nil for a parser saved without the AST.
func (*Parser) Marshal source
func (p *Parser) Marshal(opts ...MarshalOption) ([]byte, error)Marshal encodes the parser like MarshalBinary, with options.
func (*Parser) MarshalBinary source
func (p *Parser) MarshalBinary() ([]byte, error)MarshalBinary encodes the parser (bytecode, grammar AST and start rule) in the .pegoc format. LoadParser restores it without parsing, analyzing, type checking or compiling the grammar again.
func (*Parser) NewDocument source
func (p *Parser) NewDocument(text string, opts ...ParseOption) (*Document, error)NewDocument creates a Document for text. opts select the position unit, which Edit also uses, and the backend. It returns an error if the backend is unavailable (Closure on a parser saved without the AST).
func (*Parser) Parse source
func (p *Parser) Parse(input string, opts ...ParseOption) (*Node, error)Parse parses the whole input and returns the value of the start rule. If the input does not match, it returns a *SyntaxError. If the parse recovered from errors with #recover, it returns both the node and the SyntaxErrors.
func (*Parser) ParseStream source
func (p *Parser) ParseStream(r io.Reader, emit func(*Node) error, opts ...ParseOption) errorParseStream parses input read from r and passes each element of the #stream repetition at the top level of the start rule to emit as soon as it matches. Emitted elements and the input before them are not retained, so the input never has to fit in memory. If emit returns an error, parsing stops and ParseStream returns that error. An element without a value (for example, one that is discarded with -) is passed as nil. As in any repetition, an element that matches without consuming input ends it.
func (*Parser) Start source
func (p *Parser) Start() stringStart returns the name of the parser's start rule.
func (*Parser) WithStart source
func (p *Parser) WithStart(name string) (*Parser, error)WithStart returns a parser that starts at the rule name. The compiled grammar is shared.
type Profile source
type Profile = engine.ProfileProfile accumulates the cost of each rule over the parses given WithProfile: calls, body evaluations, memo hits, matches and failures, the input consumed and the input examined by failed calls, repeated evaluations at a position, and time. Hints summarizes where to look. A Profile is not safe for concurrent use.
The type it stands for is defined in an internal package as follows. source
type Profile struct {
// Parses is the number of parses profiled.
Parses int `json:"parses"`
// Calls, Evals and MemoHits total the rules' counts.
Calls int `json:"calls"`
Evals int `json:"evals"`
MemoHits int `json:"memoHits"`
// Examined totals, over the parses, the end of the input the start rule examined.
Examined int `json:"examined"`
// Time totals the time of the parses' start rule calls (in nanoseconds in JSON).
Time time.Duration `json:"time"`
// Rules holds the rules called, in the order of their first call.
Rules []*RuleProfile `json:"rules"`
// contains filtered or unexported fields
}Profile accumulates the cost of each rule over the parses it is attached to (as their trace function, Profile.Trace). It is not safe for concurrent use.
func (*Profile) Hints source
func (pr *Profile) Hints() []stringHints returns observations about the profile that point at where a grammar does more work than it needs to, most important first. They are heuristics: each names what to look at, not what is wrong.
func (*Profile) Trace source
func (pr *Profile) Trace(e TraceEvent)Trace records the event e. Pass it as the trace function of the parses to profile.
type RuleProfile source
type RuleProfile = engine.RuleProfileRuleProfile is the cost of one rule in a Profile.
The type it stands for is defined in an internal package as follows. source
type RuleProfile struct {
Rule string `json:"rule"`
// Calls is the number of calls. Each call either evaluated the rule body or took the result from
// the memo (MemoHits); it matched (Matched) or failed (Failed).
Calls int `json:"calls"`
MemoHits int `json:"memoHits"`
Matched int `json:"matched"`
Failed int `json:"failed"`
// Evals is the number of body evaluations: one per call not answered by the memo, and one more
// for each step a left-recursive match grows.
Evals int `json:"evals"`
// Consumed totals the input matched by the calls that matched.
Consumed int `json:"consumed"`
// Wasted totals the input examined by the calls that evaluated the body and failed: the work
// that led to nothing. It is inclusive: a failed call counts what the calls it made examined.
// LongestFail is the most input a single failed call examined, at LongestFailAt.
Wasted int `json:"wasted"`
LongestFail int `json:"longestFail"`
LongestFailAt Location `json:"longestFailAt"`
// Repeats is the number of calls that evaluated the body at a position (and binding level)
// where an earlier call of the same parse had evaluated it already. MaxEvals is the most
// evaluating calls at one position, at MaxEvalsAt. A memoized rule is evaluated at most twice
// at a position (the memo keeps a result from the second call on); a rule that is not
// memoized is evaluated each time its callers are.
Repeats int `json:"repeats"`
MaxEvals int `json:"maxEvals"`
MaxEvalsAt Location `json:"maxEvalsAt"`
// Time is the time spent in the calls, nested calls included. A call within a call of the
// same rule counts once. SelfTime excludes the time of nested calls. Both include the
// overhead of measuring them, so they are meaningful relative to each other. JSON gives them in
// nanoseconds.
Time time.Duration `json:"time"`
SelfTime time.Duration `json:"selfTime"`
}RuleProfile is the cost of one rule. Calls of a rule's value-free twin count as calls of the rule.
type Severity source
type Severity = lint.SeveritySeverity tells how likely a Finding is to be a mistake.
type SyntaxError source
type SyntaxError = engine.SyntaxErrorSyntaxError reports that the input does not match the grammar.
The type it stands for is defined in an internal package as follows. source
type SyntaxError struct {
Pos int // position, in the parse's position unit
Line int // 1-based line
Col int // 1-based column, in the position unit of the parse
Expected []string // what was expected at that position
Messages []string // messages given by #error
}SyntaxError reports that the input did not match the grammar.
func (*SyntaxError) Error source
func (e *SyntaxError) Error() stringfunc (*SyntaxError) Message source
func (e *SyntaxError) Message() stringMessage returns the error description without the position.
type SyntaxErrors source
type SyntaxErrors = engine.SyntaxErrorsSyntaxErrors lists the syntax errors recovered from with #recover.
The type it stands for is defined in an internal package as follows. source
type SyntaxErrors []*SyntaxErrorSyntaxErrors is a list of syntax errors, including those recovered by error recovery.
func (SyntaxErrors) Error source
func (l SyntaxErrors) Error() stringtype TraceEvent source
type TraceEvent = engine.TraceEventTraceEvent describes the start (TraceEnter) or the end (TraceExit) of a rule call. See WithTrace.
The type it stands for is defined in an internal package as follows. source
type TraceEvent struct {
Kind TraceKind
// Rule is the name of the rule called.
Rule string
// Level is the binding level the rule was called with: for a Pratt rule called as e:level,
// the operators that bind no tighter than that level are left to the caller. It is 0 for
// calls without a level.
Level int
// Depth is the nesting depth of the call: 1 for the start rule.
Depth int
// Pos is the position at which the call starts.
Pos int
// Lookahead reports that the call is inside a lookahead (& or !), where failures record no
// expectations.
Lookahead bool
// Matched reports whether the rule matched.
Matched bool
// End is the position after the match (Pos if the rule failed).
End int
// Memo reports that the result came from the memo, without evaluating the rule body.
Memo bool
// Evals is the number of times this call evaluated the rule body: 0 for a result from the
// memo, 1 usually, and one more per step for a left-recursive rule whose match grows. Bodies of
// nested calls are not included.
Evals int
// Examined is the end of the input the call examined (exclusive): looking at a character to
// reject it counts, as do lookaheads.
Examined int
// contains filtered or unexported fields
}TraceEvent describes the start or the end of a rule call. Events nest: every TraceEnter is followed, after the events of the calls the rule makes, by its TraceExit, unless the parse is aborted: by a runtime error in an action, an error returned by a stream's emit function or reader, the nesting limit, or a panic in the trace function, which propagates out of the parse unchanged.
The methods LineCol, Text and Failure read the parser's state: call them only from the trace function, while it handles the event.
func (TraceEvent) Failure source
func (e TraceEvent) Failure() *SyntaxErrorFailure returns, for an exit event, the farthest failure recorded during the call: the position and what was expected there, as a parse that failed there would report it. It is nil when the call recorded no expectation (inside a lookahead, or when nothing failed). A call that matched can have one too: the input after its match was tried and rejected.
func (TraceEvent) LineCol source
func (e TraceEvent) LineCol(pos int) (line, col int)LineCol returns the 1-based line and column of the position pos. In a stream parse, it returns 0, 0 for a position (other than 0) in input that has been discarded or not read yet: keeping the line starts of discarded input would make the memory of a stream grow with its length.
func (TraceEvent) Recovered source
func (e TraceEvent) Recovered() []*SyntaxErrorRecovered returns, for an exit event of a call that matched, the syntax errors that #recover recovered from during the call, in nested calls too, or taken from the memo with the call's result. The expectations of a recovered error are not part of Failure: recovering removes them from the record of the call that contains the #recover.
func (TraceEvent) Text source
func (e TraceEvent) Text() stringText returns the input the call matched (empty for an enter event or a failure). Input that a stream parse has already discarded is not available.
type TraceKind source
type TraceKind = engine.TraceKindTraceKind tells whether a TraceEvent starts or ends a rule call.
The type it stands for is defined in an internal package as follows. source
type TraceKind uint8func (TraceKind) String source
func (k TraceKind) String() stringtype Unit source
type Unit = engine.UnitUnit is the unit of input positions: node Start and End, startPos and endPos in actions, syntax error positions and columns, and the len builtin.