Description

A Tokenizer splits an input string into tokens, the usual behaviour being to map words to tokens.