CSV

github.com/ornew/pego/parsers/csv parses CSV as defined by RFC 4180, with a parser generated by PEGO from csv.pego. It depends only on the standard library.

go get github.com/ornew/pego/parsers/csv

#Use

// Records as strings, as encoding/csv's ReadAll returns them.
recs, err := csv.Records("name,note\r\nalice,\"likes \"\"quotes\"\", commas\"\r\n")
// [["name" "note"] ["alice" "likes \"quotes\", commas"]]

// A file with a header: every record must have as many fields as the header.
header, rows, err := csv.Table("id,name\n1,alice\n2,bob\n")
name := slices.Index(header, "name")

// Typed values with positions.
f, err := csv.ParseAST("a,\"b\"\"c\"\n\nd\n")
for _, r := range f.Records {
	for _, fl := range r.Fields {
		fmt.Println(fl.Start, fl.End, fl.Text, fl.Quoted(), fl.Value()) // 2 8 "b""c" true b"c
	}
}

// Only check.
ok := csv.Valid("a,\"b\" \n") // false: a space after the closing quote

// Syntax errors.
_, err = csv.Records("a,b\nc,d\"e\n")
var se *csv.SyntaxError
if errors.As(err, &se) {
	fmt.Println(se.Line, se.Col) // 2 4
}

Positions are in code points by default; csv.ParseAST(src, csv.Bytes) counts bytes. A record's span includes its line break; its fields end at the End of the last one. With code points, the Text of a field holds U+FFFD for each byte that is not part of valid UTF-8; with Bytes it keeps the bytes, and so do Records and Table.

The grammar works with the engine too: its #stream attribute hands records over one at a time in a stream parse (pego parse -g csv.pego -stream, or ParseStream in Go), for files larger than memory.

#Where RFC 4180 and practice differ

The RFC defines the syntax with an ABNF grammar that real files often stray from. The package accepts every file of the RFC's grammar, and in the points where the RFC and practice differ it follows encoding/csv with its default settings (except FieldsPerRecord), so that it can replace it, with two exceptions where encoding/csv loses information.

#Conformance

There is no official conformance suite for CSV. The tests check:

• 73 cases (edgeCases in csv_test.go): the examples of section 2 of RFC 4180, the cases of csv-spectrum (rewritten from their descriptions, not vendored), and the points in the table above, 13 of them rejected. Each has its expected records and is checked against encoding/csv (FieldsPerRecord -1), which gives the same result on all but 7, the differences in the table: a byte order mark (4 cases) and CRLF in quoted fields (3 cases);
• Records against encoding/csv on 2,000 files of random records written by encoding/csv's Writer with LF and with CRLF, and against the records written;
• the same comparison on arbitrary input in FuzzRecords (go test -fuzz FuzzRecords): both must accept the same inputs and return the same records once the byte order mark is removed and CRLFs in values are turned into LF (6 million inputs passed when it was written);
• the positions of ParseAST, the lines and columns of syntax errors, Table, and that Valid, Recognize, ParseAST and Records agree on every input.

All of them run with Go alone.

#Performance

On a file of 5,000 records (210 KB) with quoted fields that contain commas, doubled quotes and line breaks (go test -bench ., Apple M3 Max, minimum of 8 runs):

encoding/csv is a hand-written reader that builds nothing but strings; ParseAST takes about twice its time while recording the position of every field and record, and Records 2.5 times. Valid builds nothing but runs the general code of the generated parser (one method per expression), not the direct rules ParseAST runs. The comments in csv.pego give the measured effect of the grammar's choices.