Question

A lexical analyzer uses the following token definitions
 𝑙𝑒𝑡𝑡𝑒𝑟→[𝐴−𝑍𝑎−𝑧]
 𝑑𝑖𝑔𝑖𝑡→[0 −9]
 𝑖𝑑→𝑙𝑒𝑡𝑡𝑒𝑟 (𝑙𝑒𝑡𝑡𝑒𝑟 | 𝑑𝑖𝑔𝑖𝑡)*
 𝑛𝑢𝑚𝑏𝑒𝑟→𝑑𝑖𝑔𝑖𝑡+
 𝑤𝑠→(𝑏𝑙𝑎𝑛𝑘 | 𝑡𝑎𝑏 | 𝑛𝑒𝑤𝑙𝑖𝑛𝑒)+
For the string given below,
𝑥1 23𝑚𝑚 78 𝑦 7𝑧 𝑧𝑧5 14𝐴 8𝐻 𝐴𝑎𝑌𝑐𝐷
the number of tokens (excluding 𝑤𝑠) that will be produced by the lexical analyzer
is __________. (answer in integer)

Hint:

An id token can absorb trailing digits after its first letter, but a number token made of leading digits cannot absorb a following letter, so a new token starts wherever digits are followed by a letter.

Answer 1

Correct Answer: 13

Step 1: The lexer uses maximal munch, so at each position it matches the longest possible token according to the given definitions: \(id \to letter\,(letter\,|\,digit)^*\) and \(number \to digit^+\).
Step 2: Break the input by whitespace and tokenize each chunk: \(x1\) starts with a letter, so the entire chunk matches one \(id\) token.
Step 3: \(23mm\) starts with digits, so \(23\) matches a \(number\) token, and the remaining \(mm\) matches a separate \(id\) token - 2 tokens.
Step 4: \(78\) is all digits, so it is a single \(number\) token.
Step 5: \(y\) is a single \(id\) token.
Step 6: \(7z\) splits into \(number\) \(7\) and \(id\) \(z\) - 2 tokens.
Step 7: \(zz5\) starts with a letter, so the whole chunk (letters followed by a digit) matches one \(id\) token.
Step 8: \(14A\) splits into \(number\) \(14\) and \(id\) \(A\) - 2 tokens.
Step 9: \(8H\) splits into \(number\) \(8\) and \(id\) \(H\) - 2 tokens.
Step 10: \(AaYcD\) starts with a letter and contains only letters, so it matches one \(id\) token.
Step 11: Adding up: \(1 + 2 + 1 + 1 + 2 + 1 + 2 + 2 + 1 = 13\) tokens excluding whitespace.
Final Answer: \[\boxed{13}\]