Virtual Terminals

Overview

This parameter was added in version 2.2 of the Builder.

Languages such as Python, do not use symbols to mark the start and end of a block of statements. Instead, the indentation of each statement is used to determine when  a block begins and ends. For Python, content of whitespace is important - or at least the position of a token rather than solely its classification by the tokenizer. If a program has an indent of 10 spaces, the grammar must contain a set of rules for statements at this level. The same is true for all other levels of indentation - requiring an infinite number of rules to parse.

Virtual terminals are design to resolve problems like this by allowing the developer to define terminals that will be create manually. Each of these terminals are entered into the Symbol Table, but will not be recognized by the Engine's tokenizer. In other words, these terminals only exist in the Symbol Table and will never be created by the tokenizer's Deterministic Finite Automata.

As a result, the developer must create these terminals at runtime and push them onto the front of the input queue. The actual semantics and method calls will vary greatly between different implementations of the Engine.

In the case of Python, tokens could be created to represent a increase or decrease in indentation. Of course, the developer can create special tokens for any number of reasons; not just this one alone.

Examples

The examples below define virtual terminals for programming languages that use indentation to delimit blocks. Virtual terminals are created to represent the increase and decrease of indentation. Note: There are no commas in the following declaration.

Virtual terminals, like all other terminals, do not need to have identifier-style names. The example below creates two virtual terminals called '==>' and '<=='.