This chapter describes the expected characteristics of an input text file, and the options offered for describing these characteristics by a text input control file. (This chapter is adapted from chapters 7, 8, and 9 of Weber (1988).)

Input text files
================

Text input control files define a simple model of input text files. They are plain text files with two types of embedded format markers.

 1. A primary format marker consists of one or more contiguous characters beginning with a special flag character.  The default character initiating format markers is the backslash (\).  Thus, each of the following would be recognized as a format marker and would not be processed by the program:

          \
          \p
          \sp
          \begin{enumerate}
          \very-long.and;muddled/format*marker,to#be$sure

     Note that format markers cannot have a space or tab embedded in them; the first space or tab encountered terminates the format marker.

     One final note: the format character under discussion here applies only to the input text files which are to be processed.  It has absolutely nothing to do with the use of backslash (\) to flag field codes in control files such as the text input control file.

 2. A secondary type of marker consists of a flag character followed by a single character from a list of known values.  This secondary flag character must be different than the primary flag character.  Its default value is the vertical bar (|), causing this type of format marker to be frequently called a bar code.  The following could be valid (secondary) format markers and would not be processed by the program:

          |b
          |i
          |r

Consider the following two lines of input text:

     \bgoodbye\r
     |bgoodbye|r

Using the default definitions of format markers, the first line is considered to be a single format marker, and provides nothing which the program should try to parse.  The second line, however contains two format markers, |b and |r, and the word goodbye which would be processed by the program.

The primary format markers serve to divide the text into fields.  See \excl and \incl for details on how these fields are used.  There is no requirement that the format markers be at the beginning of a line as with the field codes used in UNKNOWN-PROGRAM control files.

Ambiguity Marker Character: \ambig
==================================

The \ambig field defines the character used to mark ambiguities and failures in the analysis output file.  For example, to use the hash mark (#), the text input control file would include:

     \ambig  #

This would cause an ambiguous analysis to be output as follows:

     \a #3#< N0 kay >#< V1 ka > IMP#< V1 ka > INF#

It makes sense to use the \ambig field only once in the text input control file.  If multiple \ambig fields do occur in the file, the value given in the first one is used.  If the text input control file does not have an \ambig field, the percent sign (%) is used.

The first printing character following the \ambig field code is used as the ambiguity marker.  The character currently being used to mark comments cannot be assigned to also mark ambiguities in the output file. Thus, the semicolon (;) cannot normally be used as the ambiguity marker.  Logically, this field should be in the file rather than the text input control file since it affects output instead of input.  Nevertheless, compatibility demands that it stays this way.

Bar code format marker character: \barchar
==========================================

The \barchar defines the character that begins a two-character secondary format marker.  For example, if this type of format marker begins with the dollar sign ($), the following would be placed in the text input control file:

     \barchar  $

An empty \barchar field in the text input control file prevents any bar code format markers from being recognized.  Thus, the following field effectively turns off special treatment of this style of format marking (assuming the ; is marking comments):

     \barchar       ; no bar character

It makes sense to use the \barchar field only once in the text input control file.  If multiple \barchar fields do occur in the file, the value given in the first one is used.

The first printing character following the \barchar field code is used as the bar code format marker.  The character currently being used to mark comments cannot be assigned to also flag format markers in input text files.

Bar Code Format Code Characters: \barcodes
==========================================

In conjunction with the special format marking character discussed in the previous section, the \barcodes field defines the individual characters used with in bar codes.  These characters may be separated by spaces or lumped together.  Thus, the following two fields are equivalent:

     \barcodes    abcdefg         ; lumped together
     \barcodes    a b c d e f g   ; separated

If provided more than one \barcodes field in the text input control file, the combination of all characters defined in all such fields is used.  No check is made for repeated characters: the previous example would be accepted without complaint despite the redundancy of the second line.

Text Orthography Change: \ch
============================

     [file changes.txi not included]

Decomposition Separation Character: \dsc
========================================

The \dsc field defines the character used to separate the morphemes in the decomposition field of the output analysis file.  For example, to use the equal sign (=), the text input control file would include:

     \dsc  =

This would cause a decomposition field to be output as follows:

     \d %3%kay%ka=y%ka=y%

It makes sense to use the \dsc field only once in the text input control file.  If multiple \dsc fields do occur in the file, the value given in the first one is used.  If the text input control file does not have an \dsc field, a dash (-) is used.

The first printing character following the \dsc field code is used as the morpheme decomposition separator character.  The same character cannot be used both for separating decomposed morphemes in the analysis output file and for marking comments in the input control files.  Thus, one normally cannot use the semicolon (;) as the decomposition separation character.

Logically, this field should be in the file rather than the text input control file since it affects output instead of input.  Nevertheless, compatibility demands that it stays this way.

Fields to Exclude: \excl
========================

The \excl field excludes one or more fields from processing. For example, to have the program ignore everything in \co and \id fields, the following line is included in the text input control file:

     \excl  \co  \id      ; ignore these fields

If more than one \excl field is found in the text input control file, the contents of each field is added to the overall list of text fields to exclude.  This list is initially empty, and stays empty unless the text input control file contains an \excl field. Thus, no text fields are excluded from processing by default.

If the text input control file contains \excl fields, then only those text fields are not processed.  Every word in every text field not mentioned explicitly in an \excl field will be processed.

Note that every text field in the input text files is processed unless the text input control file contains either an \excl or an \incl field.  One or the other is used to limit processing, but never both.

Primary format marker character: \format
========================================

The \format field designates a single character to flag the beginning of a primary format marker.  For example, if the format markers in the text files begin with the at sign (@), the following would be placed in the text input control file:

     \format  @

This would be used, for example, if the text contained format markers like the following:

     @
     @p
     @sp
     @make(Article)
     @very-long.and;muddled/format*marker,to#be$sure

If a \format field occurs in the text input control file without a following character to serve for flagging format markers, then the program will not recognize any format markers and will try to parse everything other than punctuation characters.

It makes sense to use the \format field only once in the text input control file.  If multiple \format fields do occur in the file, the value given in the first one is used.

The first printing character following the \format field code is used to flag format markers.  The character currently used to mark comments cannot be assigned to also flag format markers.  Thus, the semicolon (;) cannot normally be used to flag format markers.

Fields to Include: \incl
========================

The \incl field explicitly includes one or more text fields for processing, excluding all other fields.  For instance, to process everything in \txt and \qt fields, but ignore everything else, the following line is placed in the text input control file:

     \incl  \txt  \qt      ; process these fields

If more than one \incl field is found in the text input control file, the contents of each field is added to the overall list of text fields to process.  This list is initially empty, and stays empty unless the text input control file contains an \incl field.

If the text input control file contains \incl fields, then only those text fields are processed.  Every word in every text field not mentioned explicitly in an \incl field will not be processed.

Note that every text field in the input text files is processed unless the text input control file contains either an \excl or an \incl field.  One or the other is used to limit processing, but never both.

Lowercase/uppercase character pairs: \luwfc
===========================================

To break a text into words, the program needs to know which characters are used to form words.  It always assumes that the letters A through Z and a through z are used as word formation characters.  If the orthography of the language the user is working in uses any other characters that have lowercase and uppercase forms, these must given in a \luwfc field in the text input control file.

The \luwfc field defines pairs of characters; the first member of each pair is a lowercase character and the second is the corresponding uppercase character.  Several such pairs may be placed in the field or they may be placed on separate fields.  Whitespace may be interspersed freely.  For example, the following three examples are equivalent:

     \luwfc  éÉ ñÑ

or

     \luwfc  éÉ      ; e with acute accent
     \luwfc  ñÑ      ; enyee

or

     \luwfc  é É  ñ Ñ

Note that comments can be used as well (just as they can in any UNKNOWN-PROGRAM control file).  This means that the comment character cannot be designated as a word formation character.  If the orthography includes the semicolon (;), then a different comment character must be defined with the -c command line option when UNKNOWN-PROGRAM is initiated; see

The \luwfc field can be entered anywhere in the text input control file, although a natural place would be before the \wfc (word formation character) field.

Any standard alphabetic character (that is a through z or A through Z) in the \luwfc field will override the standard lower- upper case pairing.  For example, the following will treat X as the upper case equivalent of z:

     \luwfc z X

Note that Z will still have z as its lower-case equivalent in this case.

The \luwfc field is allowed to map multiple lower case characters to the same upper case character, and vice versa.  This is needed for languages that do not mark tone on upper case letters.

Multibyte lowercase/uppercase character pairs: \luwfcs
======================================================

The \luwfcs field extends the character pair definitions of the \luwfc field to multibyte character sequences.  Like the \luwfc field, the \luwfcs field defines pairs of characters; the first member of each pair is a multibyte lowercase character and the second is the corresponding multibyte uppercase character.  Several such pairs may be placed in the field or they may be placed on separate fields.  Whitespace separates the members of each pair, and the pairs from each other.  For example, the following three examples are equivalent:

     \luwfcs  e' E` n~ N^ ç C&

or

     \luwfcs  e' E`      ; e with acute accent
     \luwfcs  n~ N^      ; enyee
     \luwfcs  ç  C&      ; c cedilla

or

     \luwfcs  e' E`
              n~ N^
              ç  C&

Note that comments can be used as well (just as they can in any UNKNOWN-PROGRAM control file).  This means that the comment character cannot be designated as a word formation character.  If the orthography includes the semicolon (;), then a different comment character must be defined with the -c command line option when UNKNOWN-PROGRAM is initiated; see

Also note that there is no requirement that the lowercase form be the same length (number of bytes) as the uppercase form.  The examples shown above are only one or two bytes (character codes) in length, but there is no limit placed on the length of a multibyte character.

The \luwfcs field can be entered anywhere in the text input control file.  \luwfcs fields may be mixed with \luwfc fields in the same file.

Any standard alphabetic character (that is a through z or A through Z) in the \luwfcs field will override the standard lower- upper case pairing.  For example, the following will treat X as the upper case equivalent of z:

     \luwfcs z X

Note that Z will still have z as its lowercase equivalent in this case.

The \luwfcs field is allowed to map multiple multibyte lowercase characters to the same multibyte uppercase character, and vice versa. This may be useful in some situations, but it introduces an element of ambiguity into the decapitalization and recapitalization processes.  If ambiguous capitalization is supported, then for the previous example, z will have both X and Z as uppercase equivalents, and X will have both x and Z as lowercase equivalents.

Maximum number of decapitalizations: \maxdecap
==============================================

The \maxdecap field sets the maximum number of different decapitalizations allowed.  Since the \luwfc field can map several lowercase characters onto a single uppercase character, a word with uppercase characters can (logically) generate a number of alternatives when decapitalized.  This is especially true of words that are entirely capitalized to begin with.  The default limit is 100.

Prevent Any Decapitalization: \nocap
====================================

The usual behavior is to normalize input words to lowercase.  The program remembers the case of the word as one of four possibilities:

 1. all uppercase

 2. all lowercase

 3. only the first letter uppercase

 4. mixed uppercase and lowercase

However, not all orthographies use the concept of capitalization.  To help deal with these, the field code \nocap disables all case normalization if it appears anywhere in the text input control file.

Prevent Decapitalization of Individual Characters: \noincap
===========================================================

The handling of mixed uppercase and lowercase is limited in utility, and sometimes causes more problems than it solves.  For this reason, the \noincap field code turns off mixed case decapitalization. The program would still decapitalize words that are entirely capitalized and words that begin with a capital letter.

String class: \scl
==================

A string class is defined by the \scl field code followed by the class name, which is followed in turn by one or more contiguous character strings or (previously defined) string class names.  A string class name used as part of the class definition must be enclosed in square brackets.

The class name must be a single, contiguous sequence of printing characters.  Characters and words which have special meanings in tests should not be used.  The actual character strings have no such restrictions.  The individual members of the class are separated by spaces, tabs, or newlines.

Each \scl field defines a single string class.  Any number of \scl fields may appear in the file.  The only restriction is that a string class must be defined before it is used.

String classes must be defined before being used.  For example, the first two lines of the simpler Caquinte example above could be given as follows:

     \scl  -bilabial  c t qu
     \ch  "m"  >  "N"  / _ p
     \ch  "n"  >  "N"  / _ [-bilabial]

The string class definition could be in another control file: string classes defined elsewhere can be used in the text input control file as well.

If no \scl fields appear in the text input control file, then UNKNOWN-PROGRAM does not allow any string classes in text input orthography change environment constraints unless they are defined in the file.

Caseless word formation characters: \wfc
========================================

To break a text into words, the program needs to know which characters are used to form words.  It always assumes that the letters A through Z and a through z are used as word formation characters.  If the orthography of the language the user is working in uses any characters that do not have different lowercase and uppercase forms, these must given in a \wfc field in the text input control file.

For example, English uses an apostrophe character (') that could be considered a word formation character.  This information is provided by the following example:

     \wfc  '    ; needed for words like don't

Notice that the characters in the \wfc field may be separated by spaces, although it is not required to do so.  If more than one \wfc field occurs in the text input control file, the program uses the combination of all characters defined in all such fields as word formation characters.

The comment character cannot be designated as a word formation character. If the orthography includes the semicolon (;), then a different comment character must be defined with the -c command line option when UNKNOWN-PROGRAM is initiated; see

Multibyte caseless word formation characters: \wfcs
===================================================

The \wfcs field allows multibyte characters to be defined as ``caseless'' word formation characters.  It has the same relationship to \wfc that \luwfcs has to \luwfc.  The multibyte word formation characters are separated from each other by whitespace.

A sample text input control file
================================

The following is the complete text input control file for Huallaga Quechua (a language of Peru):

     \id HGTEXT.CTL - for Huallaga Quechua, 25-May-88

     \co         WORD FORMATION CHARACTERS

     \wfc  ' ~

     \co         FIELDS TO EXCLUDE

     \excl  \id            ; identification fields

     \co         ORTHOGRAPHY CHANGES

     \ch  "aa" > "a:"      ; for long vowels
     \ch  "ee" > "i:"
     \ch  "ii" > "i:"
     \ch  "oo" > "u:"
     \ch  "uu" > "u:"
     \ch  "qeki" > "qiki"  ; for cases like wawqeki
     \ch  "~n" > "n~"      ; for typos
     ; for Spanish loans like hwista
     \scl sib s c          ; sibilants
     \ch  "hw" > "f"  / ~[sib]_