String Tokenization What is String Tokenization? The StringTokenizer class Examples

What is String Tokenization? So far we have been reading our input one value at a time. Sometimes it is more natural to read a group of input at a time. For example, when reading records of students from a text file, it is natural to read a whole record at a time. "995432 Al-Suhaim Adil 3.5" The readLine() method of the Buffered Reader class can read a group of input as a single string object. The problem is, how do we break this string object into individual words known as tokens? "995432" "Al-Suhaim Adil” "3.5“ This process is what String tokenization is about.

The StringTokenizer Class The StringTokenizer class, of the java.util package, is used to break a string object into individual tokens. It has the following constructors: Constructor function StringTokenizer(String str) Creates a StringTokenizer object that uses white space characters as delimiters. StringTokenizer(String str, String delimiters) Creates a StringTokenizer object that uses the characters in delimiters as separators. StringTokenizer(String str,String delimiters,boolean returnTokens) Creates a StringTokenizer object that uses characters in delimiters as separators and treats separators as tokens.

StringTokenizer Methods The following are the main methods of the StringTokenizer class: Method function String nextToken() throws NoSuchElementException Returns the next token as a string from this StringTokenizer object. Throws an exception if there are no more tokens. int countTokens() Returns the count of tokens in this StringTokenizer object that are not yet processed by nextToken() -- initially all. boolean hasMoreTokens() Returns true if there are more tokens not yet processed by nextToken().

How to apply the methods To break a string into tokens, first, a StringTokenizer object is created. String myString = "I like Java very much"; StringTokenizer tokenizer = new StringTokenizer(myString); Then any of the following loops can be used to process the tokens: while(tokenizer.hasMoreTokens()){ String token = tokenizer.nextToken(); // process token } or int tokenCount = tokenizer.countTokens(); for(int k = 1; k <= tokenCount; k++){

Example 1 The following program reads grades from the keyboard and finds the average. The grades are read in one line. import java.io.*; import java.util.StringTokenizer; public class TokenizerExamplel{ public static void main(String[] args)throws IOException{ BufferedReader stdin = new BufferedReader(new InputStreamReader(System.in)); System.out.print("Enter grades in one line:"); String inputLine = stdin.readLine(); StringTokenizer tokenizer = new StringTokenizer(inputLine); int count = tokenizer.countTokens(); double sum = 0; while(tokenizer.hasMoreTokens()) sum += Double.parseDouble(tokenizer.nextToken()); System.out.println("\nThe average = "+ sum / count); }

Example 2 Output: Hi there do you like Java I very much This example shows how to use the second constructor of StringTokenizer class. It tokenizes the words in a string, such that the punctuation characters following the words are not appended to the resulting tokens. import java.util.StringTokenizer; public class TokenizerExample2{ public static void main(String[] args){ String inputLine = "Hi there, do you like Java? I do;very much."; StringTokenizer tokenizer = new StringTokenizer (inputLine, ",.?;:! \t\r\n"); while(tokenizer.hasMoreTokens()) System.out.println(tokenizer.nextToken()); } Output: Hi there do you like Java I very much

Example 3 This example shows how to use the third constructor of StringTokenizer class. It tokenizes an arithmetic expression based on the operators and returns both the operands and the operators as tokens. import java.util.StringTokenizer; public class TokenizerExample3{ public static void main(String[] args){ String inputLine = "(2+5)/(10-1)"; StringTokenizer tokenizer = new StringTokenizer(inputLine,“+—*/()",true); while(tokenizer.hasMoreTokens()) System.out.println(tokenizer.nextToken()); } Output: ( 2 + 5 ) / 10 - 1