Lexical Analysis - GitHub Pages
嚜燉exical Analysis
? Recognize tokens and ignore white spaces,
comments
Generates token stream
? Error reporting
? Model using regular expressions
? Recognize using Finite State Automata1
Lexical Analysis
? Sentences consist of string of tokens (a
syntactic category)
For example, number, identifier, keyword,
string
? Sequences of characters in a token is a
lexeme
for example, 100.01, counter, const,
※How are you?§
? Rule of description is a pattern
for example, letter ( letter | digit )*
? Task: Identify Tokens and corresponding
Lexemes
2
Lexical Analysis
? Examples
? Construct constants: for example, convert a
number to token num and pass the value as its
attribute,
每 31 becomes
? Recognize keyword and identifiers
每 counter = counter + increment
becomes id = id + id
每 check that id here is not a keyword
? Discard whatever does not contribute to
parsing
每 white spaces (blanks, tabs, newlines) and
comments
3
Interface to other phases
Read
characters
Input
Push back
Extra
characters
Token
Lexical
Analyzer
Syntax
Analyzer
Ask for
token
? Why do we need Push back?
? Required due to look-ahead
for example, to recognize >= and >
? Typically implemented through a buffer
每 Keep input in a buffer
每 Move pointers over the input
4
Approaches to implementation
? Use assembly language
Most efficient but most difficult to implement
? Use high level languages like C
Efficient but difficult to implement
? Use tools like lex, flex
Easy to implement but not as efficient as the first
two cases
5
................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.
Related download
- lexical analysis github pages
- project 3 hannah choi data 8
- oracle to bigquery sql translation reference
- sql to hive cheat sheet cloudera
- i have a csv file and need to assign a data type to each
- reading and writing data with pandas
- bluebeam revu extreme script reference
- sympy symbolic computing in python
- comparing sas and python a coder s perspective