1. Prepare to partition your brain to learn a whole new formalism.
The lexc Language
Prepare to partition your brain to learn a whole new formalism.

2. The lexc language “lexc” stands for “LEXicon Compiler”
lexc is a high-level, declarative programming language
lexc is different from regular expressions and from xfst
the syntax is different
the assumptions are different
the special characters are different
the interfaces are different
BUT, the lexc compiler produces STANDARD Xerox networks
these networks are fully compatible with networks from xfst
you can sometimes choose to use lexc or xfst for building a network
This is all fertile ground for confusion!

3. Why a Separate lexc Language?
Lexc is intended for use by lexicographers.
Regular expressions in xfst are often hard to read, especially big ones
Typing spaces between all the letters , e.g. e l e p h a n t , to be concatenated in xfst is a nuisance, especially if you need to type 40,000 words
You can also write {elephant} in xfst regular expressions, but that’s a nuisance too
Lexc is more efficient for compiling large natural-language lexicons (it optimizes the union operation)
Lexc has better error messages
But remember:
lexc is just another formalism for defining finite-state languages and relations
you can (and will) use lexc and xfst together in building significant applications

4. The lexc Source File: Multichar_Symbols
The lexc compiler and the xfst regular-expression compiler have completely opposite assumptions about multicharacter symbols:
In xfst Regular Expressions, the default is to treat a string of symbols written together, e.g. %+Noun or cat, as a single symbol. Concatenation of separate symbols is indicated by manually separating symbols with white space, e.g. [ c a t ], or by using the curly-brace notation, e.g. {cat}.
In lexc, in contrast, the default is to treat strings, e.g. cat, as a concatenation of three symbols. Any multicharacter symbols must be explicitly declared at the top of the source file.

5. Multichar_Symbols declaration
Multichar_Symbols +Noun +Verb +Adj +Adv +Sg +Pl +1P +2P +3P ^FEAT1 ^FEAT2
The Multichar_Symbols statement is formally optional and is placed at the top of your lexc source file. You can declare as many multicharacter symbols as you find necessary or useful. The compiler uses this declaration to separate the strings of your lexc program into symbols. You are strongly encouraged to include a non-alphabetic character like the plus sign or the circumflex to help the multicharacter symbol stand out visually.

6. The Body of your lexc Program
The body of a lexc program is composed of LEXICONs.
There should be one LEXICON named Root. It corresponds to the Start State in the resulting Network.
If you don’t define a LEXICON Root, lexc will try to use the first LEXICON in the file as the Start State.
LEXICON Root
dog N ;
cat N ;
bird N ;

7. Entries in a LEXICON
Each defined LEXICON must have at least one entry.
An entry consists of two parts and is terminated with a semicolon
data continuation-class ;
The data part has to fit one of four formats:
string e.g dog
upper:lower e.g swim:swam
< regular-expression > e.g < a b* c >
empty e.g.

8. upper:lower Entries
The upper:lower entries are the simplest way to specify portions of the network where the upper-side and lower-side differ. They are especially useful for irregularies/suppletions.
Multichar_Symbols +Verb +Past +Noun +Sg +Pl
LEXICON Root
swim+Verb+Past:swam # ;
go+Verb+Past:went # ;
child+Noun+Pl:children # ;
ox+Noun+Pl:oxen # ;

9. upper:lower Entries
In upper:lower entries, you can overtly indicate where the epsilons should go.
Multichar_Symbols +Verb +Past +Noun +Sg +Pl +Nom
LEXICON Root
poder+Verb:pod0r FutCond ;
Danger: the lexc upper:lower notation is not quite the same as the regular-expression colon notation.

10. Regular Expressions in lexc
Any data written as a regular expression must be surrounded with angle brackets, e.g.
< e l e p h a n t > CC ;
< a b* c+> CC ;
Inside angle brackets, you revert to all the assumptions suitable for xfst regular expressions, including the treatment of multicharacter symbols vs. concatenation of symbols.
This is fertile ground for confusion and errors.

11. Continuation Classes
The Continuation Class is just the name of a defined LEXICON or #, indicating end-of-word (a final state).
Multichar_Symbols +Noun +Sg +Pl
LEXICON Root
dog N ;
cat N ;
LEXICON N
+Noun+Sg:0 # ;
+Noun+Pl:s # ;

12. Thinking About lexc LEXICONS
A LEXICON should hold a coherent class of morphemes
The entries in a lexc LEXICON are unioned together by the compiler; the order of the entries in a LEXICON is not significant.
Think of LEXICONs as potential “targets”
Entries “point at” a LEXICON via the ContinuationClass
But each entry in a LEXICON could itself point to a different ContinuationClass
During development, you may have to subdivide lexicons
Avoid having copies of the same material (if possible)
You may change an entry in one place and forget to change the copy

13. Formally Speaking
Lexc syntax is a kind of right-recursive phrase-structure grammar.
Phrase-structure grammars can in general describe languages beyond finite-state power, including languages with balanced parentheses.
But with the right-recursive limitation, a phrase-structure grammar can define only finite-state languages.
Lexc can describe only finite-state languages.
Lexc descriptions compile into finite-state networks.

14. Lexc Idiom: Optional Morphemes via By-Pass
LEXICON Vroot
kant V ;
dir V ;
don V ;
pens V ;
LEXICON V
AdLex ;
Vend ;
LEXICON AdLex
ad Vend ;
LEXICON Vend
as # ;
is # ;
os # ;
us # ;
u # ;
i # ;

15. Lexc Idiom: Optional Morphemes via “Escape” Entries
LEXICON Vroot
kant AdLex ;
dir AdLex ;
don AdLex ;
pens AdLex ;
LEXICON AdLex
ad Vend ;
Vend ; ! escape
LEXICON Vend
as # ;
is # ;
os # ;
us # ;
u # ;
i # ;

16. Lexc Idiom: Loops LEXICON Nroot LEXICON Plur
kat N ; j Case ; ! Opt. plural ending
hund N ; Case ;
elefant N ;
LEXICON N LEXICON Case
eg N ; ! loop n # ; ! Opt. case ending
et N ; ! loop # ;
in N ; ! loop
Nend ;
LEXICON Nend
o Plur ;

17. Stem compounding (loops) in lexc
LEXICON Nroot LEXICON Plur
kat N ; j Case ; ! Opt. plural ending
hund N ; Case ;
elefant N ;
LEXICON N LEXICON Case
Nroot ; n # ;
Nend ; # ;
LEXICON Nend
o Plur ;

18. Special Characters in lexc
Overall, there are far fewer special characters in lexc than in regular expressions. In lexc, the following are special:
Special Literalized
: used in upper:lower notation %:
; terminates an entry %;
< begins a regular expression %<
> ends a regular expression %>
0 denotes the empty string (epsilon) %0
! introduces a comment line %!
# continuation-class for end-of-word %#
% literalizing prefix %%

19. Lexc source files
Lexc sources files are ascii, typically edited with xemacs
Lexc programs for natural language can get very large
Typically 8000 to entries for verbs
Tens of thousands of entries for nouns and proper nouns

20. The lexc interface
Invoke the lexc interface by simply entering ‘lexc’ at the UNIX prompt.
unixprompt% lexc
You communicate with the interface using lexc commands. Type ‘?’ to see all the possible commands. Invoke ‘help commandname’ to see some terse online documentation.
Enter ‘quit’ to leave lexc and return to the operating system.
lexc: quit

21. The Three lexc Registers
To understand lexc commands, you must understand that they refer to and operate on networks held in three registers, visualized as
SOURCE
Typically used to store a lexicon FST.
RULES
Typically used to store a rule FST or FSTs.
Typically used to store the result of composing the rule FST(s) under the source FST.
RESULT

22. Basic lexc commands SOURCE RULES RESULT compile-source filename
compiles the lexc source code in filename and stores the resulting network in SOURCE
SOURCE
read-rules filename
reads the binary file filename and stores the network(s) in RULES. The binary file may be from xfst or twolc.
RULES
compose-result
composes the RULE FST(s) under the SOURCE FST and stores the resulting FST in RESULT
RESULT

23. Some Other lexc Commands
read-source load a pre-compiled binary network into the SOURCE register
save-source filename store network in SOURCE to binary file
save-result filename store network in RESULT to binary file
lookup word (equivalent to the xfst ‘apply up’)
lookdown word (equivalent to the xfst ‘apply down’)
result-to-source move the network from the RESULT
register to SOURCE
Enter ‘?’ to see all the lexc interface commands.

24. Using lexc and xfst together
Write a lexc source file (e.g. mysrc.lex) using xemacs or a similar editor
Write suitable alternation rules (in xfst or even twolc). Compile them and save the network(s) to file, e.g. to myrul.fst
Then from the lexc interface:
lexc: compile-source mysrc.lex
lexc: read-rules myrul.fst
lexc: compose-result
lexc: save-result mylang.fst

25. Using lexc and xfst together
Lexc lexicons build “words” (strings) using union and concatenation
Entries within a LEXICON are unioned (the order of entries is not significant)
The LEXICON Root corresponds to the start state
The special # continuation class corresponds to final states
Other continuation classes translate into concatenation
By Xerox convention, upper-side strings consist of a baseform and “tags”
By convention, a surface (or more surfacy) form appears on the lower-side
The surfacy forms generated by lexc may still be rather abstract, hyper-regular, or “morphophonemic”. They may sometimes contain multicharacter symbols.
Replace Rules (perhaps a whole cascade of them) map from the surfacy strings produced by lexc to real surface strings; rules are applied using composition.
Composition can also be used to “filter” out various kinds of overgeneration.

26. A Typical Finite-State System
Filters (xfst)
.o.
Core Lexicon (lexc)
.o.
Orthographical or Phonological Alternation Rules (xfst)

27. A System may be a Union of Subsystems
Nouns (lexc)
Verbs (lexc)
Adjs (lexc)
Numbers (lexc)
.o.
.o.
Verb Rules
Noun Rules
Then, in xfst:
define final NounFST | VerbFST | AdjFST | NumberFST ;

28. Review: Outputs and Inputs
Unix Pipe:
cat wordlist.in | sort | uniq -c | sort -rnb > myfile.out
The output of one routine is the input to the next
NOT reversible
Cascade of Replace Rules:
read regex [ N -> m || _ p ] .o. [ p -> m || m _ ] ;
Reversible/bidirectional relation
apply down: the output of the first rule is the input to the second
the lower side of the top rule is the upper side of the bottom rule

29. Review: Up and Down In xfst (regular expressions) In lexc
a:b swim+Verb+Past:swam
%+Pl:s upper:lower
[ a .x. b ]
a -> b
a <- b

30. Review: Xerox Conventions
The lexical side language, and all intermediate languages, have to be defined by the linguist writing the grammar.
Upper (lexical) language: baseform+Tag+Tag+Tag
Lower (surface) language: orthographical-string
(mapping via rules)
The surface language is usually determined for you by the standard orthography.

31. Review: Up and Down with Composition
A rule that refers to tags on its lower side
.o.
baseform+Tag+Tag+Tag
surfacy-form
An FST defined using lexc
.o.
A rule that refers to a surfacy form on its upper side

32. Review: Think in terms of Languages and Relations
Lexical Language
Core Lexicon FST
Surfacy Language
Rule1
Intermediate Language
Rule2
Rule n
Final Surface Language

33. Other Important Topics in The Book
Composition is Our Friend
Modify a common “core” network to handle
Multiple orthographies
Multiple dialects
Multiple registers
Testing with the Finite-State Calculus
Bulk testing against corpora
Regression testing/comparison
Testing against wordlists
Testing the well-formedness of the upper-side strings

34. Advanced Features “Flag Diacritic” features and feature unification
Simplify lexc descriptions
Help keep transducers small
The compile-replace Algorithm
Useful for non-concatenative morphology
Reduplication
Semitic Interdigitation