Outline of Class 5: Syntax

Held: Wednesday, January 28, 1998

Administrivia

• Tomorrow at 4:15 in Science 2424 the department will be showing "The Proof", a film about the proof of Fermat's last theorem. This had been a major open problem in mathematics, and was only solved in the past few years.
• The program I used to generate the syllabus got some counts wrong. It's also taking a little bit longer to cover some topics. Hence, I've removed the presentations from the syllabus.
• While I'm trying to get the class outlines done in advance, it's looking more and more doubtful that I'll always have them done the day in advance. Sorry; it's an incredibly busy term for me.
• Don't forget that assignment one is due today (at the beginning of class).
• I was sorry to see so few of you at the exotic programming languages discussion group. I hope to see more of you next week.

Language Syntax

• The syntax of a language is (describes?) the way a language looks: the symbols and combinations of symbols that form valid utterances.
  • At times, the syntax describes a superset of the valid utterances; if we said that every "noun-phrase transitive-verb noun-phrase" sequence was an English sentence, we'd allow some meaningless sentences like "the dog flew an ice-cream cone".
• The syntax is also used to assign "types" to utterances (e.g., this is a noun-phrase, this is a sentence, this is a program, this is a function call).
• From the syntactic perspective, a language is a potentially infinite set of strings built from symbols in a base alphabet.
  • A string is a fine linear sequence of symbols from the alphabet.
• Often, we build a language from multiple grammars. One simple grammar is used to build a simpler alphabet which the next grammar can then process in more interesting ways.
• For example, the process of finding identifiers, numbers, and such is much easier than that of finding the various types of statements in a language.
  • If it's possible to build a fast program to find identifiers and such, this can simplify and speed the other program.
• The simple processing is often called lexical analysis.
• The more complicated processing is then called syntax analysis.

Grammars

• How do you describe the syntax of the language?
  • It depends on the language.
  • If the language is finite, you could list all the valid utterances.
  • If the language is infinite, you'd like a finite method of describing the language.
• A grammar is a formal set of rules that describe the valid sentences in a language. It defines the syntax of the language.
• From one perspective, grammars provide predicates. Given a string, a grammar can answer the question "is this string in the language".
• But grammars might be used for other purposes. They might be used to generate strings in the language.
• As computer scientists, we look for grammars that provide computationally efficient ways of determining whether or not a string is in a language.
• Often, we work with generative grammars, grammars whose rules describe how to generate the strings of the language.
• The most typical form of grammar is the BNF ("Backus Naur Form") grammar, but there are others.

Regular Expressions

• One particularly convenient form of grammar is the regular expression.
• Regular expressions can only describe relatively simple languages, but they permit very efficient membership tests and matching.
• [Note that we need a language to describe regular expressions, but we're only now learning how to describe languages, which leads to an interesting problem. We'll use an informal description of regular expressions.]
• Like all languages, the language of regular expressions is based on an underlying alphabet, sigma.
• We'll denote the set of utterances a regular expression denotes by L(exp).
• There is a special symbol, epsilon, not in sigma.
  • epsilon is written as epsilon
  • L(epsilon) is the set of the empty string, { "" }.
• Any single symbol in sigma is a regular expression.
  • The regular expression for that symbol is that symbol
  • For each symbol s in sigma, L(s) = { s }.
• The concatenation of any two regular expressions is a regular expression denoting the combinations of strings from those two regular expressions.
  • If R and S are regular expressions, the concatenation of R and S is written (R)(S)
  • L((R)(S)) = { concat(x,y) | x in L(R), y in L(S) }
  • concat(epsilon,s) = concat(s,epsilon) = s
  • (R)(R) may also be written as (R)^2; (R)(R)(R) as (R)^3; and so on and so forth.
• The alternation of any two regular expressions is a regular expression denoting strings in either language.
  • If R and S are regular expressions, the alternation of R and S is written (R)+(S).
  • L((R)+(S)) = L(R) union L(S)
• If R is a regular expression, then the Kleene star of R is a regular expression.
  • If R is a regular expression, the Kleene star of R is written (R)*.
  • L((R)*) = L(epsilon) union L((R)^1) union L((R)^2) ...
• Because the parentheses are cumbersome, you may remove them if the meaning is obvious without them.
  • Kleene star has highest precedence; concatenation next; alternation least.
  • ab* is (a)((b)*) and not ((a)(b))*
  • a+bc is (a)+((b)(c)) and not ((a)+(b))(c)
• There are a number of shorthands for regular expression. None add any expressive power and all can be "compiled" into normal regular expressions.
• What are some sample regular expressions?
  • All strings of a's and b's: (a+b)*
  • Strings of a's and b's with exactly one b: a*ba*
  • Strings of one or more a's: aa* (also a*a, also a*aa*)
• Often, we name our regular expressions so that we can more easily identify different things. Generally, it is legal to use any previously defined name (this restriction prevents recursive or mutually recursive definitions).
• For example

  lowercase:  a+b+c+d+e+f+g+h+i+j+k+l+m+n+o+p+q+r+s+t+u+v+w+x+y+z
  uppercase:  A+B+C+D+E+F+G+H+I+J+K+L+M+N+O+P+Q+R+S+T+U+V+W+X+Y+Z
  letter:     (uppercase)+(lowercase)
  digit:      0+1+2+3+4+5+6+7+8+9
  number:     (digit)(digit)*
  word:       (letter)(lowercase)*
  identifier: ((letter)+_)((digit)+(letter)+_)*
  

Backus-Naur Form (BNF) Grammars

• History:
  • In addition to leading the Fortran team, Backus worked as part of the Algol team. One of his key roles in that team was formally defining the syntax of Algol.
  • Backus worked on a formal system for defining syntaxes. Many of his ideas were governed by previous work of Chomsky.
  • Naur worked on improving Backus's notation.
• BNF grammars are four-tuples that contain
  • Sigma, the alphabet from which strings are built (note that this alphabet may have been generated from the original program via a lexer). These are also called the terminal symbols of the grammar.
  • N, the nonterminals in the language. You can think of these as names for the structures in the language or as names for groups of strings.
  • S in N, the start symbol. In BNF grammars, all the valid utterances are of a single type.
  • P, the productions that present rules for generating valid utterances.
• The set of productions is the core part of a grammar. Often, language definitions do not formally specify the other parts because they can be derived from the grammar.
• A production is, in effect, a statement that one sequence of symbols (a string of terminals and nonterminals) can be replaced by another sequence of terminals and nonterminals. You can also view a production as an equivalence: you can represent the left-hand-side by the right-hand-side.
• What do productions look like? It depends on the notation one uses for nonterminals and terminals, which may depend on the available character set.
• We'll use words that begin with capital letters to indicate nonterminals, words that begin with lowercase letters to indicate terminals, and stuff in quotation marks to indicate particular phrases. We'll use ::= to separate the two parts of a rule.
• We might indicate the standard form of a Pascal program with

  Program ::= 'program'
              identifier
              '('
              Identifier-List
              ')'
              Declaration-List
              Compound-Statement
              '.'
  

• Similarly, we might indicate a typical compound statement in Pascal with

  Compound-Statement ::= 'begin'
                        Statement-List
                        'end'
  

• Lists are defined recursively, using multiple definitions. Here's one for non-empty statement lists. It says, "a statement list can be a statement; it can also be a statement followed by a semicolon and another statement list".

  Statement-List ::= Statement
  Statement-List ::= Statement ';' Statement-List
  

• The statements may then be defined with

  Statement ::= Assignment-Statement
  Statement ::= Compound-Statement
  ...
  Assignment-Statement ::= identifier '=' Expression
  ...
  

• How do we use BNF grammars? We can use them to derive legal strings in the language by starting with the start symbol and repeatedly replacing strings that match left-hand-sides with the corresponding right-hand-sides. This is called a derivation.
• When do we stop the derivation? When we run out of nonterminals in the string.
• While typical language grammars have only one nonterminal on the left, it is possible to have multiple things on the left. For example,

  Rock Anything Hardplace ::= Rock Hardplace
  

• It turns out to be much harder to parse using grammars that have multiple nonterminals on the left, so we'll work mostly with grammars that have only one nonterminal on the left. These are called context-free grammars, since the replacement of a nonterminal requires no context. That is, we can replace a nonterminal without considering the surrounding symbols.