A text segmentation program that scans text and determines if and when a series of characters can be recognized as a token.