Raise the scanner line limit

A Go program that reads a file line by line stops early on a very long line, and the loop ends without a visible error.

package main

import (
	"bufio"
	"fmt"
	"os"
)

func main() {
	sc := bufio.NewScanner(os.Stdin)
	if len(os.Args) > 1 {
		sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
	}
	lines := 0
	for sc.Scan() {
		lines++
	}
	fmt.Println("lines:", lines, "error:", sc.Err())
}

go build -o count main.go
(echo short; head -c 100000 /dev/zero | tr '\0' x; echo; echo last) | ./count
(echo short; head -c 100000 /dev/zero | tr '\0' x; echo; echo last) | ./count big

lines: 1 error: bufio.Scanner: token too long
lines: 3 error: <nil>

bufio.Scanner holds one token in a buffer that may grow up to 64 KB, the value of MaxScanTokenSize. A longer line stops the scan, Scan returns false and the reason is available only from Err, which many loops never read. Buffer supplies a larger limit, here 1 MB, and the second run reads all three lines. The behavior is in the bufio.Scanner documentation.

Buffer has to be called before the first Scan. The maximum is a limit, not an allocation, but one very long line still has to fit in memory. For input of unbounded line length use bufio.Reader.ReadString or ReadLine, which have no such cap. Always check sc.Err() after the loop.