This chapter describes the content of UNKNOWN_PROGRAM dictionary files.  These are normally divided into

 1. a prefix dictionary file (if needed),

 2. an infix dictionary file (if needed),

 3. an suffix dictionary file (if needed), and

 4. one or more root dictionary files.

With the -u command line option in conjunction with the \unified field in the dictionary code table file, the dictionary can be stored as one or more files containing entries of any type: prefix, infix, suffix, or root.

The following sections describe the different types of fields used in the different types of dictionary files.  Remember, the mapping from the actual field codes used in the dictionary files to the type codes that UNKNOWN_PROGRAM uses internally is controlled by the dictionary code table file (see Dictionary code table file).

Allomorph (internal code A)
===========================

Each dictionary entry must contain one or more allomorph fields.  Each of these contains one of the infix's allomorphs, that is, the string of characters by which the affix is represented in text and recognized by UNKNOWN_PROGRAM.

If an affix has multiple allomorphs, each one must be entered in its own allomorph field.  These fields should be ordered with those on which the strictest constraints have been imposed preceding those with less strict or no constraints.  The only exception to this is the use of indexed string classes to indicate reduplication.  (See lines 20 and 21 below.)

Properties, constraints, and comments may follow the allomorph string. Any properties must be listed before any constraints.  String, punctuation and morpheme environment constraints may be intermixed, but must come before any comments.  A complete BNF grammar of an allomorph field is given below.

      1a. <allomorph_field> ::= <allomorph>
      1b.                       <allomorph> <properties>
      1c.                       <allomorph> <constraints>
      1d.                       <allomorph> <properties> <constraints>
      1e.                       <allomorph> <comment>
      1f.                       <allomorph> <properties> <comment>
      1g.                       <allomorph> <constraints> <comment>
      1h.                       <allomorph> <properties> <constraints> <comment>

      2a. <allomorph>       ::= <literal>
      2b.                       <literal> { <literal> }
      2c.                       <redup_pattern>
      2d.                       <redup_pattern> { <literal> }

      3a. <properties>      ::= <literal>
      3b.                       <literal> <properties>

      4a. <constraints>     ::= <string_constraint>
      4b.                       <morph_constraint>
      4c.                       <punct_constraint>
      4d.                       <string_constraint> <constraints>
      4e.                       <morph_constraint> <constraints>
      4f.                       <punct_constraint> <constraints>

      5.  <comment>         ::= <comment_char> anything to the end of the line

      6a. <string_constraint> ::= / <envbar> <string_right>
      6b.                         / <string_left> <envbar>
      6c.                         / <string_left> <envbar> <string_right>

      7a. <string_left>       ::= <string_side>
      7b.                         <boundary>
      7c.                         <boundary> <string_side>
      7d.                         <string_side> # <string_side>
      7e.                         <boundary> <string_side> # <string_side>

      8a. <string_right>      ::= <string_side>
      8b.                         <boundary>
      8c.                         <string_side> <boundary>
      8d.                         <string_side> # <string_side>
      8e.                         <string_side> # <string_side> <boundary>

      9a. <string_side>       ::= <string_item>
      9b.                         <string_item> <string_side>
      9c.                         <string_item> ... <string_side>

     10a. <string_item>       ::= <string_piece>
     10b.                         ( <string_piece> )

     11a. <string_piece>      ::= ~ <string_piece>
     11b.                         <literal>
     11c.                         [ <literal> ]
     11d.                         [ <indexed_literal> ]

     12a. <morph_constraint>  ::= +/ <envbar> <morph_right>
     12b.                         +/ <morph_left> <envbar>
     12c.                         +/ <morph_left> <envbar> <morph_right>

     13a. <morph_left>        ::= <morph_side>
     13b.                         <boundary>
     13c.                         <boundary> <morph_side>
     13d.                         <morph_side> # <morph_side>
     13e.                         <boundary> <morph_side> # <morph_side>

     14a. <morph_right>       ::= <morph_side>
     14b.                         <boundary>
     14c.                         <morph_side> <boundary>
     14d.                         <morph_side> # <morph_side>
     14e.                         <morph_side> # <morph_side> <boundary>

     15a. <morph_side>        ::= <morph_item>
     15b.                         <morph_item> <morph_side>
     15c.                         <morph_item> ... <morph_side>

     16a. <morph_item>        ::= <morph_piece>
     16b.                         ( <morph_piece> )

     17a. <morph_piece>       ::= ~ <morph_piece>
     17b.                         <literal>
     17c.                         [ <literal> ]
     17d.                         { <literal> }

     18a. <punct_constraint>  ::= ./ <envbar> <punct_right>
     18b.                         ./ <punct_left> <envbar>
     18c.                         ./ <punct_left> <envbar> <punct_right>

     19a. <punct_left>        ::= <punct_side>
     19b.                         <boundary>
     19c.                         <boundary> <punct_side>

     20a. <punct_right>       ::= <punct_side>
     20b.                         <boundary>
     20c.                         <punct_side> <boundary>

     21a. <punct_side>        ::= <punct_item>
     21b.                         <punct_item> <punct_side>

     22a. <punct_item>        ::= <punct_piece>
     22b.                         ( <punct_piece> )

     23a. <punct_piece>       ::= ~ <punct_piece>
     23b.                         <literal>
     23c.                         [ <literal> ]

     24a. <envbar>            ::= _
     24b.                         ~_

     25a. <boundary>          ::= #
     25b.                         ~#

     26a. <redup_pattern>     ::= [ <indexed_literal> ]
     26b.                         <literal> [ <indexed_literal> ]
     26c.                         [ <indexed_literal> ] <literal>
     26d.                         [ <indexed_literal> ] <redup_pattern>
     26e.                         <redup_pattern> [ <indexed_literal> ]

     27.  <indexed_literal>   ::= <literal> ^ <number>

     28.  <literal>           ::= one or more contiguous characters

     29.  <comment_char>      ::= character defined by -c command
                                  line option, or | by default

     30.  <number>            ::= one or more contiguous digits (0-9)

Comments on selected BNF rules
..............................

2.
     The (first) literal string is a surface form representation of the morpheme.  The literal string enclosed in braces is a unique allomorph identification string.  (The identification string is a feature added to support LinguaLinks.  It is not stored unless the -b command line option is used.

3.
     Each literal string is an allomorph property defined by a \ap field in the analysis data file.

4.
     String, punctuation and morpheme constraints can be mixed together, but it is recommended that you group the string constraints together, the punctuation constraints together and the morpheme constraints together.

5.
     A comment begins with a specified character and ends with the end of the line.

7-8.
     Note that what can appear to the left of the environment bar is a mirror image of what can appear to the right.

7de.
8de.
     These should be avoided, and other means used to prune analyses based on adjacent words.

9c.
     An ellipsis (...) indicates a possible break in contiguity.

10b.
     Something enclosed in parentheses is optional.

11a.
     A tilde (~) reverses the desirability of an element, causing the constraint to fail if it is found rather than fail if it is not found.

11b.
     A literal is matched against the surface form of the word.

11c.
     A literal enclosed in square brackets must be the name of a string class defined by a \scl field in the analysis data file or the dictionary orthography change table file.

11d.
     The indexed literal enclosed in square brackets must match an indexed literal given as part of the reduplication allomorph pattern.  (See 2c, 2d, and 26.)

13-14.
     Note that what can appear to the left of the environment bar is a mirror image of what can appear to the right.

13de.
14de.
     These should be avoided, and other means used to prune analyses based on adjacent words.

15c.
     An ellipsis (...) indicates a possible break in contiguity.

16b.
     Something enclosed in parentheses is optional.

17a.
     A tilde (~) reverses the desirability of an element, causing the constraint to fail if it is found rather than fail if it is not found.

17b.
     A literal is a morphname from one of the dictionary files.

17c.
     A literal enclosed in square brackets must be the name of a morpheme class defined by a \mcl field in the analysis data file.

17d.
     A literal enclosed in curly braces must be one of the following (checked in this order):

      1. one of the keywords root, prefix, infix, or suffix

      2. a property name defined by an \ap or \mp field in the analyis data file

      3. a category name defined by a \ca field in the analysis data file

      4. a category class name defined by a \ccl field in the analysis data file

      5. a morpheme class name defined by a \mcl field in the analysis data file

19-20.
     Note that what can appear to the left of the environment bar is a mirror image of what can appear to the right.

22b.
     Something enclosed in parentheses is optional.

23a.
     A tilde (~) reverses the desirability of an element, causing the constraint to fail if it is found rather than fail if it is not found.

23b.
     A literal is a punctuation character.

     The punctuation characters can match punctuation characters either before or after the current word.  Unlike string constraints, punctuation constraints effectively ignore the position of the conditioned allomorph within the word.  All that matters are any punctuation characters immediately preceding or following the current word.  Further note that neither ellipsis nor cross word boundary conditions are allowed.

24.
     A tilde (~) attached to the environment bar inverts the sense of the constraint as a whole.

25b.
     The boundary marker preceded by a tilde (~#) indicates that it must not be a word boundary.

26-27.
     Although the BNF has spaces in it to improve readability, these two items cannot have embedded spaces in the dictionary file.

26.
     The reduplication allomorph pattern contains references to string classes and possibly literal strings.  The string class names are indexed to indicate identical shared values, either in the string environment constraint or in more than one location in the reduplication allomorph pattern itself.  Note: this has been implemented only for AMPLE at this point.

27.
     The literal (without the following index given by an ASCII caret (^) and a number) must be the name of a string class defined by a \scl field in the analysis data file or the dictionary orthography change table file.

28.
     The special characters used by environment constraints can be included in a literal only if they are immediately preceded by a backslash:

          \+  \/  \#  \~  \[  \]  \(  \)  \{  \}  \.  \_  \\

The allomorph field is used in all types of dictionary entries: prefix, infix, suffix, and root.

Category (internal code C)
==========================

Each dictionary entry must contain a category field.  If multiple category fields exist, then their contents are merged together.

For affix entries, this field must contain at least one category pair for the morpheme, but may contain any number of category pairs separated by spaces or tabs.  Each category pair consists of two category names separated by a slash (/).  The category names must have been defined by a \ca field in the analysis data file.  The first category is the from category, that is, the category of the unit to which this morpheme can be affixed.  The second category is the to category, that is, the category of the result after this morpheme has been affixed.

For root entries, this field contains one or more morphological categories as defined by a \ca field in the analysis data file. If multiple categories are listed, they should be separated by spaces or tabs.

The category field is used in all types of dictionary entries: prefix, infix, suffix, and root.

Elsewhere Allomorph (internal code E)
=====================================

The elsewhere allomorph field is used in all types of dictionary entries: prefix, infix, suffix, and root.

Infix location (internal code L)
================================

The infix location field serves to restrict where infixes may be found, and must be included in each infix dictionary entry.  Subject to the constraints imposed by the infix location field, UNKNOWN_PROGRAM searches the rest of the word for any occurrence of any allomorph string of the infix.  This makes infixes rather expensive, computationally, so they should be constrained as much as possible.

      1.  <infix_location> ::= <types> <constraints>

      2a. <types>          ::= <type>
      2b.                      <type> <types>

      3a. <constraints>    ::= <environment>
      3b.                      <environment> <constraints>

      4a. <environment>    ::= <marker> <leftside> <envbar> <rightside>
      4b.                      <marker> <leftside> <envbar>
      4c.                      <marker> <envbar> <rightside>

      5a. <leftside>       ::= <side>
      5b.                      <boundary>
      5c.                      <boundary> <side>

      6a. <rightside>      ::= <side>
      6b.                      <boundary>
      6c.                      <side> <boundary>

      7a. <side>           ::= <item>
      7b.                      <item> <side>
      7c.                      <item> ... <side>

      8a. <item>           ::= <piece>
      8b.                      ( <piece> )

      9a. <piece>          ::= ~ <piece>
      9b.                      <literal>
      9c.                      [ <literal> ]

     10a. <type>           ::= prefix
     10b.                      root
     10c.                      suffix

     11a. <marker>         ::= /
     11b.                      +/

     12a. <envbar>         ::= _
     12b.                      ~_

     13a. <boundary>       ::= #
     13b.                      ~#

     14.  <literal>        ::= one or more contiguous characters

Comments on selected BNF rules
..............................

2.
     The first part of the infix location field lists the type of morpheme in which the infix may be hidden.  This consists of one or more of the words prefix, root, or suffix.  If prefix is given, then UNKNOWN_PROGRAM looks for infixes after exhausting the possible prefixes at a given point in the word, and resumes looking for more prefixes after finding an infix.  Similarly, if root is given, then UNKNOWN_PROGRAM looks for infixes after running out of roots while parsing the word, and if it finds an infix, it looks for more roots.  Suffixes are treated the same way if suffix is given in the infix location field.

5.
     A boundary marker (#) on the left side of the environment bar refers to the place in the word which the parse has reached before looking for infixes, not to the beginning of the word.

6.
     A boundary marker (#) on the right side of the environment bar refers to the end of the word.

7c.
     An ellipsis (...) indicates a possible break in contiguity.

8b.
     Something enclosed in parentheses is optional.

9a.
     A tilde (~) reverses the desirability of an element, causing the constraint to fail if it is found rather than fail if it is not found.

11.
     A +/ is usually used for morpheme environment constraints, but may used for infix location environment constraints as well.

12.
     A tilde attached to the environment bar (~_) inverts the sense of the constraint as a whole.

13b.
     The boundary marker preceded by a tilde (~#) indicates that it must not be a word boundary.

14.
     The special characters used by environment constraints can be included in a literal only if they are immediately preceded by a backslash:

          \+  \/  \#  \~  \[  \]  \(  \)  \{  \}  \.  \_  \\

The infix location field is used only in infix dictionary entries.

Morphname (internal code M)
===========================

A morphname is an arbitrary name for a given morpheme.  Only the first word (string of contiguous nonspace characters) following the morphname field code is used as the morphname.  Morphnames must be less than 64 characters long.

A morphname serves two important functions:

 1. It identifies a morpheme in morpheme environment constraints, morpheme co-occurrence constraints, ad hoc pairs, and tests.

 2. It is the default morpheme identifier written to the output analysis file.

Generally, a morphname is an identifier of a morpheme and does not need to faithfully represent that morpheme's meaning or function.

If a dictionary entry has more than one morphname field, the morphname from the first one is used; the others cause an error message.  The morphname field is used in all types of dictionary entries: prefix, infix, suffix, and root.  The usage differs somewhat between affix and root dictionary entries, so these two types of morphnames are described separately.

Affix morphnames
----------------

Every affix dictionary entry must have a morphname field.  Users are strongly encouraged to observe the following suggestions in creating affix morphnames:

 1. Make each morphname unique.  If two morphemes have the same name, it is impossible to refer unambiguously to them.  The same morphname should not be used in different affix dictionaries (that is, in the prefix dictionary and in the suffix dictionary).

 2. Keep morphnames short.  This reduces the size of analysis files and makes text glossing more aesthetically pleasing.  For example, for a verbal person marker, use simply 1 rather than 1P unless there is good reason to add the P for person or possessive.  For a first person object marker, 1O might serve as well as 1OBJ.

 3. Use only uppercase alphabetic characters and numbers for contrast with root morphnames, which are generally made up of lowercase alphabetic characters.  Be cautious in using hyphens, periods, underscores, slashes, backslashes, or other nonalphanumeric characters.  The reason to avoid these is that other programs which apply to the resulting analysis may make use of nonalphanumerics in different ways.

 4. Design a syntax of names and stick to it for inflectional morphemes which combine more than one semantic notion.  For example, for Latin nominal inflections, which indicate gender, number, and case, the syntax might be

          MORPHNAME = GENDER CASE NUMBER

     where GENDER is M for masculine, F for feminine and N for neuter; CASE is N for nominative, A for accusative, G for genitive, and so on; and NUMBER is S for singular and P for plural.  The name for masculine nominative singular would then be MNS.

Root morphnames
---------------

Root morphnames are generally either glosses or etymologies. Etymologies are frequently marked with a leading asterisk (*). (This is used by STAMP to indicate regular sound changes.)

If the morphname field contains only an asterisk, the morphname becomes an asterisk followed by whatever allomorph is matched.  If the morphname field is omitted, or if it contains only a comment, UNKNOWN_PROGRAM puts whatever allomorph was matched in the text into the analysis.  If the morpheme contains any alternate forms, it is wise to include an explicit morphname field.

Order class (internal code O)
=============================

The order class of an affix is a number indicating its position relative to other morphemes.  Prefixes should be assigned negative numbers and suffixes should be assigned positive numbers.  Infixes should be assigned order class values appropriate to where they can appear in the word relative to the prefixes and suffixes.

If the order class field is omitted, then a default value of zero (0) is assigned to the affix.  Order class values must be between -32767 and 32767.

Order classes are used only by tests in the analysis data file.  They are needed only if appropriate tests are written to take advantage of them.

The order class field is used only in affix type dictionary entries: prefix, infix, and suffix.  Roots always have an implicit order class of zero.

Beginning with AMPLE version 3.6.0, one may have up to two order class numbers in an order class field (separated by white space).  These represent the minimum and the maximum values of the positions this affix can span.  The first number is the minimum and the second is the maximum. Therefore the first number should be less than or equal to the second.  If only one number appears, both the minimum and maximum values are set to that number.  If no number appears, then both the minimum and maximum are set to zero.

Note that for STAMP, only the first order class number has any use (it is used for transfer insertion rules whose environments do not indicate a location where the morpheme is to be inserted).

Morpheme property (internal code P)
===================================

This field contains one or more morpheme properties.  These properties must have been defined by a \mp field in the analysis data file. A morpheme property is inherited by all allomorphs of the morpheme.

The morpheme property field is optional, and may be repeated.  If multiple properties apply to a morpheme, they may be given all in a single field or each in a separate field.

Morpheme properties typically indicate a characteristic of the morpheme which conditions the occurrence of allomorphs of an adjacent morpheme. Morpheme properties are used in tests defined in the analysis data file and in morpheme environment constraints.

The morpheme property field is used in all types of dictionary entries: prefix, infix, suffix, and root.

Morpheme type (internal code T)
===============================

In a unified dictionary, the type of an entry is determined by the first letter following the morpheme type field code: p or P for prefixes, i or I for infixes, s or S for suffixes, and r or R for roots.  The morpheme type field is not needed for root entries because the entry type defaults to root.

The morpheme type field is used only in unified dictionary files, since the morpheme type is otherwise implicit.

Do not load (internal code !)
=============================

When a ``do not load'' field is included in a record, UNKNOWN_PROGRAM ignores the record altogether.  This makes it possible to include records in the dictionary for linguistic purposes, while not needlessly taking up memory space if the dictionary is used for some other purpose.

The ``do not load'' field is used in all types of dictionary entries: prefix, infix, suffix, and root.