Tokenizer in Linked List



Tokenizer in Linked List

The following is a program of one student who implemented successful the tokenizer. However, the class is not clean to be used by other classes. Then you will see a major modified version. Many unnecessary variables and structures are removed. Here are some general commonest:

1. The original program has a bug on the link tokenPre.

2. It does not handle the hidden token multiplication in the case of 125x or x125.

3. Attributes and constructors should be built to alleviate the burden on the main program. The main program suppose to be a testing or guideline for the class.

4. Many sections of codes are repeating. They should be moved to a method as a single copy so that it can be centralized for modification.

5. The constants for the token type 0-9 should be centralized as well. Instead using the values, implementers should used the constant statement to code them such as

CONST MULT = 4;

CONST PLUS = 2;

Two appendices are given:

A. A modified version of Tokenizer

B. The stack to be implemented in linked list.

Appendix A.

//Modified

//11-23-99

// source inpostfix.java

// infix order to postfix order

import java.io.*;

import java.lang.*;

import java.util.*;

class TokenAndType

{

// a pair as attributes

String token;

int type;

//constructor

public TokenAndType()

{

token = null;

type = -1; // means no type

} // end of contructor

public TokenAndType(String tokenName,int tokenType)

{

token = tokenName;

type = tokenType;

} // end of contructor

public void printTokenPair()

{

System.out.print("token pair: " + token + " " + type);

}

} // end of class

class NodePtr

{

// Attributes of any member of NodePtr

TokenAndType tokenPair;

NodePtr next;

// Constructor to insert the node at the top of inputNodePtr

public NodePtr(TokenAndType tokenPair, NodePtr next)

{

this.tokenPair = tokenPair;

this.next = next;

}

public NodePtr()

{

tokenPair = new TokenAndType();

next = null;

}

public NodePtr( String tokenName,int tokenType)

{

tokenPair = new TokenAndType(tokenName, tokenType);

next = null;

}

}

class Tokenizer

{

// attributes

NodePtr top = null; // this is the tokenStream in Linked list

// method

public void GetTokens()

{

NodePtr temp = null;

NodePtr last = null; // points to the last node

String token=""; // from string of characters, cut into 1 token

// at a time

InputStreamReader reader = new InputStreamReader (System.in);

BufferedReader console = new BufferedReader (reader);

System.out.print("please enter data> ");

String StrExp = null;

try

{

StrExp = console.readLine();

}

catch (IOException e)

{

System.out.println("we call an" + e);

System.exit(1);

}

char firstChar=' ';

if (StrExp.length()!=0)firstChar = StrExp.charAt(0);

while(StrExp!=null)

{

switch (firstChar)

{

case '.': case '0': case '1': case '2': case '3':

case '4': case '5': case '6': case '7': case '8': case '9':

do {

token=token+firstChar;

if (StrExp.length()==1){StrExp = null;}

else

{

StrExp=StrExp.substring(1);

firstChar = StrExp.charAt(0);

}

}while( ((firstChar >='0' && firstChar ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download