> For the complete documentation index, see [llms.txt](https://amartyushov.gitbook.io/tech/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://amartyushov.gitbook.io/tech/programming-languages/java/io-nio/java-io-input-parsing/streamtokenizer.md).

# StreamTokenizer

The Java `StreamTokenizer` class (`java.io.StreamTokenizer`) can tokenize the characters read from a `Reader` into tokens. For instance, in the string "Mary had a little lamb" each word is a separate token.

When you are parsing files or computer languages it is normal to break the input into tokens, before further processing them. This process is also called "lexing" or "tokenizing".

Using a Java `StreamTokenizer` you can move through the tokens in the underlying `Reader`. You do so by calling the `nextToken()` method of the `StreamTokenizer` inside a loop. After each call to `nextToken()` the `StreamTokenizer` has several fields you can read to see what kind of token was read, it's value etc. These fields are:

| ttype | The type of token read (word, number, end of line)              |
| ----- | --------------------------------------------------------------- |
| sval  | The string value of the token, if the token was a string (word) |
| nval  | The number value of the token, if the token was a number.       |

### StreamTokenizer Example

Here is a simple Java `StreamTokenizer` example:

```
StreamTokenizer streamTokenizer = new StreamTokenizer(
        new StringReader("Mary had 1 little lamb..."));

while(streamTokenizer.nextToken() != StreamTokenizer.TT_EOF){

    if(streamTokenizer.ttype == StreamTokenizer.TT_WORD) {
        System.out.println(streamTokenizer.sval);
    } else if(streamTokenizer.ttype == StreamTokenizer.TT_NUMBER) {
        System.out.println(streamTokenizer.nval);
    } else if(streamTokenizer.ttype == StreamTokenizer.TT_EOL) {
        System.out.println();
    }

}
streamTokenizer.close();
```

The Java `StreamTokenizer` is capable of recognizing identifiers, numbers, quoted strings, and various comment styles. You can also specify what characters are to be interpreted as white space, comment begin, end etc. All these things are configured on the `StreamTokenizer` before you start parsing its contents. See the JavaDoc for more information about that.

### Closing a StreamTokenizer

When you are finished reading tokens from the `StreamTokenizer` you should remember to close it. Closing a `StreamTokenizer` will also close the `Reader` instance from which the `StreamTokenizer` is reading.

Closing a `StreamTokenizer` is done by calling its `close()` method. Here is how closing a `StreamTokenizer` looks:

```
streamTokenizer.close();
```

Unfortunately the Java StreamTokenizer does not implement the AutoCloseable interface, so you cannot close it with a [Java try-with-resources](https://jenkov.com/java-exception-handling/try-with-resources.html) construct. Instead, look to the [Java IO Exception Handling Tutorial](https://jenkov.com/tutorials/java-io/io-exception-handling.html) for more information about how to handle exceptions and close a StreamTokenizer correctly using a try-finally block.