What is a Lexical Token?
A lexical token is a sequence of characters with a collective meaning in the grammar of programming languages. These tokens include Keyword, Identifier, Operator, Literal, and Punctuation.
For Example, the following are some lexical tokens:
- Keywords: int, String, long, etc.
- Identifier : x, y, i, j, num etc.
- Operators : +,-,*,/ etc.
- Literals: 108, 9, 12, 15 etc.
- Punctuations: , ; . etc
Lexical Analyser in C
In C, the lexical analysis phase is the first phase of the compilation process. In this step, the lexical analyzer (also known as the lexer) breaks the code into tokens, which are the smallest individual units in terms of programming. In the lexical analysis phase, we parse the input string, removing the whitespaces. It also helps in simplifying the subsequent stages. In this step, we do not check for the syntax of the code. Our main focus is to break down the code into small tokens.
To know more about lexical analysis, refer to the article – Introduction to Lexical Analysis