Tokenizer rules file

#
Tokenizer rules file

The file is divided in three sections `<Macros>`, `<RegExps>` and `<Abbreviations>`. Each section is closed by `</Macros>`, `</RegExps>` and `</Abbreviations>` tags respectively.

The `<Macros>` section allows the user to define regexp macros that will be used later in the rules. Macros are defined with a name and a Perl regexp.
E.g. `ALPHA [A-Za-z]`

The `<RegExps>` section defines the tokenization rules. Previously defined macros may be referred to with their name in curly brackets.
E.g. `*ABREVIATIONS1 0 ((\{ALPHA\}+\.)+)(?!\.\.)`

Rules are regular expressions, and are applied in the order of definition. The first rule matching the beginning of the line is applied, a token is built, and the rest of the rules are ignored. The process is repeated until the line has been completely processed.

- The first field in the rule is the rule name. If it starts with a `*`, the RegExp will only produce a token if the match is found in abbreviation list (`<Abbreviations>` section).
- The second field in the rule is the substring to form the token/s with It may be 0 (the match of the whole expression) or any number from 1 to the number of substrings (up to 9). A token will be created for each substring from 1 to the specified value.
- The third field is the regexp to match against the input. line. Any Perl regexp convention may be used.

The `<Abbreviations>` section defines common abbreviations (one per line) that must not be separated of their following dot (e.g. `etc.`, `mrs.`). They must be lowercased.