Description

A Tokenizer is a simple wrapping of text/scanner.Scanner, configured for our purposes and made a TokenReader. It forms the lowest level, turning text from readers into tokens.

Tokenizer is referenced in 1 repository

github.com/golang/go