Properties

Here’s a funny little class I’ve been playing with in some development work.

class Property[T](var value:T) {
  def get:T = value
  def set(newval:T) = {
    value = newval
    this
  }
  def :=(newval:T) = set(newval)
  override def toString = "Property(" + value + ")"
}
implicit def Property2value[T](prop:Property[T]) = prop.get

I think that’s just too neat. The property can be set with the (admittedly PL/1-ish) := operator, and the value can be extracted with a cast or assignment.

If I could override the assignment then I could make this very, very C++-ish-ish. Note that I could have written something like the following.

def `=`(newval:T) = set(newval)

But then I’d have to use this as follows.

scala> val p = new Property(12)
scala> p `=` 13

That’s just not the same thing. Scala does not allow overriding assignment; no copy constructor for you.

If you are willing to declare p to be a var, then you can do the following. But this is really not what you want since it creates a new property, and does not modify the value of the existing property.

scala> def ValueToProperty[T](value:T) = new Property(value)
 
scala> var p = new Property(12)
p: Property[Int] = Property(12)
 
scala> p = 171
p: Property[Int] = Property(171)

Watched Property

So what’s the point of having this mutable property class? Well, now that I have the above I can add code to watch the property.

trait Watcher[T,U] {
  def notify(value:T, data:U)
}
trait Watchable[T,U] {
  var watchers:Map[Watcher[T,U],U] = Map()
  def register(watch:Watcher[T,U], data:U) = {
    watchers += (watch -> data)
  }
  def notify(value:T) = {
    watchers foreach {
      case (watcher, data) => watcher.notify(value, data)
    }
  }
}
class WatchedProperty[T,U](value:T) extends Property(value) with Watchable[T,U] {
  override def set(newval:T) = {
    super.set(newval)
    notify(newval)
    this
  }
}

Now I have a watcher and a watchable. I mix in the watchable trait with the property, and presto! I have a watched property. It’s somewhat of a big deal for some work I’m doing now; this would have made my life much easier.

class See[T,U]() extends Watcher[T,U] {
  def notify(value:T, data:U) = {
    println("Value is now " + value + " and data is " + data + ".")
  }
}

Finally, we do some “work.”

scala> val see = new See[Int,String]
see: See[Int,String] = See@fab91
 
scala> val watch = new WatchedProperty[Int,String](1)
watch: WatchedProperty[Int,String] = Property(1)
 
scala> watch register (see,"watch")
 
scala> watch := 12
Value is now 12 and data is watch.
res22: Property[Int] = Property(12)
 
scala> watch := 31
Value is now 31 and data is watch.
res23: Property[Int] = Property(31)

Of course, I can also avoid new by declaring the companion object for WatchedProperty and adding a factory method. Life in Scala is pretty good (so far).

object WatchedProperty {
  def apply[T,U](value:T) = new WatchedProperty[T,U](value)
}

Cleaner Watched Property

There is a problem with the implementation of watched property, and it is the type annotation U. Why should the property have to know the type for the data? The data only exists for the benefit of the watcher. Worse, the watched property isn’t a drop-in replacement for property; it has two parameters instead of just one. Finally, everyone who wants to register must use the same data type U (or a subclass). Here are some alternatives.

1. The watcher can maintain a map from watched property to the associated data. The watched property can then send itself with the property change notification. This is bad. The watcher has to maintain the map. If it is watching a lot of properties then it may have to maintain a big map, and there is lookup overhead. We avoid that by having the property return the data with the notification.
2. We could treat the data as an Any. Then it is the responsibility of the watcher to correctly cast the value during notification. This is a nuisance because the compiler cannot catch type problems for us, but Java programmers should be used to that by now.

Let’s try the second approach.

trait Watcher[T] {
  def notify(value:T, data:Any)
}
trait Watchable[T] {
  var watchers:Map[Watcher[T],Any] = Map()
  def register(watch:Watcher[T], data:Any) = {
    watchers += (watch -> data)
  }
  def notify(value:T) = {
    watchers foreach {
      case (watcher, data) => watcher.notify(value, data)
    }
  }
}
class WatchedProperty[T](value:T) extends Property(value) with Watchable[T] {
  override def set(newval:T) = {
    super.set(newval:T)
    notify(newval)
    this
  }
}

That’s better, actually. Watched property is again a referentially-transparent replacement for property, and watchers can use any data type they want. What could be better than this?

Better Watched Property

Actually that whole “Any” value is the problem now. This implementation cleans up the design a bit with better information hiding and referential transparency, but it also musses up type safety a bit. The watcher knows the type of the data; why can’t it get back properly-typed data during notification?

Maybe we’ve been asking the wrong question. The issue is, I think, the Java-esque nature of the solution. Why does notify really have to be passed the data? Why not do something very different? In fact, why not get rid of the watcher trait altogether? Why do we need it? We really just want to invoke a function (or method) when the property is changed. Let’s just directly register the callback, and forget the middleman.

trait Watchable[T] {
  var watchers:List[(T) => Unit] = List()
  def register(notify:(T) => Unit) {
    watchers += notify
  }
  def notify(value:T) = {
    watchers foreach (notify => notify(value))
  }
}
class WatchedProperty[T](value:T) extends Property(value) with Watchable[T] {
  override def set(newval:T) = {
    super.set(newval)
    notify(newval)
    this
  }
}

Watcher is gone. The data item is gone. The funny type problem is gone. The code is simpler, and even more flexible, since anything can be a watcher, including lambdas.

scala> val prop = new WatchedProperty(21)
prop: WatchedProperty[Int] = Property(21)
 
scala> def see(x:Int) = println("Value changed to: " + x + ".")
see: (Int)Unit
 
scala> prop.register(see)
 
scala> prop.register((x:Int) => println("Lambda sees value: " + x + "."))
 
scala> prop := 413
Value changed to: 413.
Lambda sees value: 413.
res10: Property[Int] = Property(413)

Data Again

Well, what about data? The value for data can be captured in a closure. Here’s an example, where the data is a name for the property being monitored.

class ChangeWatcher {
  def watch[T](prop:WatchedProperty[T], name:String) {
    prop register {
      (value:T) => println(name + ": " + value)
    }
  }
}

Did you get all that? The above is a fully-generic watcher. You pass it a property and a name for the property. Whenever the property is changed, it notifies you. This works for any kind of property.

scala> val prop = new WatchedProperty(21)
prop: WatchedProperty[Int] = Property(21)
 
scala> val status = new WatchedProperty("okay")
status: WatchedProperty[java.lang.String] = Property(okay)
 
scala> val temp = new WatchedProperty(21.8)
temp: WatchedProperty[Double] = Property(21.8)
 
scala> val cw = new ChangeWatcher
cw: ChangeWatcher = ChangeWatcher@3da8bc
 
scala> cw.watch(prop, "prop")
 
scala> cw.watch(status, "status")
 
scala> cw.watch(temp, "temperature")
 
scala> status := "Danger!"
status: Danger!
res18: Property[java.lang.String] = Property(Danger!)
 
scala> temp := 73.2
temperature: 73.2
res19: Property[Double] = Property(73.2)

The important thing to note is that you don’t have to have data. Or you can have lots of data. Or whatever you want. The responsibility for data is all back to the watcher, and off the shoulders of the watchable. This seems (to me) to be the correct division of responsibility.

To sum up: Functional Programming + Objects = Good.

def show[T](implicit thetype:scala.reflect.Manifest[T]) {
  thetype.erasure.getMethods() foreach println
}
def help(x:AnyRef) {
  x.getClass().getMethods() foreach println
}

These silly little functions are useful – at least, to me. Now when I can’t remember the name of a method I can type something like the following at the Scala prompt.

scala> help(Map())
public scala.Option scala.collection.immutable.EmptyMap.get(java.lang.Object)
public int scala.collection.immutable.EmptyMap.hashCode()
public boolean scala.collection.immutable.EmptyMap.equals(java.lang.Object)
...

The output isn’t very Scala-ish, but I could format it if I weren’t so lazy.

scala> help(scala.runtime.RichString)
public int scala.runtime.RichString$.$tag() throws java.rmi.RemoteException
public final boolean scala.runtime.RichString$.scala$runtime$RichString$$parseBoolean(java.lang.String)
public final char scala.runtime.RichString$.scala$runtime$RichString$$LF()
public final char scala.runtime.RichString$.scala$runtime$RichString$$CR()
public final char scala.runtime.RichString$.scala$runtime$RichString$$FF()
public final void java.lang.Object.wait(long,int) throws java.lang.InterruptedException
public final void java.lang.Object.wait() throws java.lang.InterruptedException
public final native void java.lang.Object.wait(long) throws java.lang.InterruptedException
public native int java.lang.Object.hashCode()
public final native java.lang.Class java.lang.Object.getClass()
public boolean java.lang.Object.equals(java.lang.Object)
public java.lang.String java.lang.Object.toString()
public final native void java.lang.Object.notify()
public final native void java.lang.Object.notifyAll()

These functions aren’t in any way brilliant, or even clever, but they are short and ridiculously trivial to write.

• A const reference.  I don’t want to copy it, but I promise not to modify it, either.
• A reference.  I might modify it.
• A copy.  I might modify my local copy, but not the original.  I’m getting a copy, after all.

Easy peasy in C++.  In Java?  Uh, I pass all objects by reference. The interface says nothing about modification.  Pass me a Map and maybe I’ll modify it, maybe not.  You don’t know, and you can’t enforce it.  Unless you pass me a Collections.unmodifiableMap().  Then my code might break!  There might be some odd little case left over where I do modify it, in spite of documentation comments to the contrary.  C++?  I declare it to be a const reference, and now the compiler will prevent me from modifying it.  I like that quite a bit.

I also like const methods, and you really can’t have const without const methods.  The Java way it to throw exceptions.  I’d rather have the compiler tell me something at compile-time than have the runtime system fail at runtime.  But I’m weird, I suppose.

Programming

I spend my time (recently) writing a mishmash of Python, C++, and Java. It’s interesting to switch back and forth.

What’s a good idea and what’s a bad (or dangerous) idea in computer language design? We’ve got a lot of candidates, and a lot of opinions.

I’ll list a few here, along with a few places where they show up. I’m not saying C macros are the same as, or even similar to, Lisp macros. Just trying to clarify the spectrum.

I’ve heard someone, at some point, say each one of the items below is a good idea / bad idea. Dangerous in the hands of a novice? Yes, but then so is any computer with a network connection.

• Preprocessors (C, C++)
• Macros (C, Lisp)
• Multiple Inheritance (C++, Smalltalk)
• Operator Overloading (C++, Python)
• Closures (Lisp, Haskell)
• Generics (Java, Ada)
• Template Metaprogramming (C++)
• Casts (C, C++)
• Strong or Weak Typing (Scala, Lisp, Python, ML)
• Fixed-Point Math (C# and COBOL)
• Fixed Integer Types vs. “Numbers” (C, Lisp, Python)
• Garbage Collection (Java, Lisp, ML)
• Manual Allocation / Deallocation (C, C++)
• Everything is an Object / Some Things are Primitives (Smalltalk, Java)
• Auto-(Un)Boxing (Java, C#)
• Indentation is Significant (Python)
• First-Order Functions (ML, Lisp)
• Lambdas (ML, Lisp, Python)
• Type Erasure (Java)

Did I leave out your favorite / most reviled language feature? Complain below. I included type erasure at the end of the list. Does anyone think this was a good idea for Java???

Programming

I recently heard about a Car Talk puzzler.  I don’t listen to Car Talk as much as I used to.  Anyway, you can read about the puzzler and the fellow who solved it here.  This is an excerpt.

During Christmas week on the popular National Public Radio show Car Talk, the weekly puzzler required listeners to find the longest English word that remains a valid English word as you remove its letters one at a time, but without rearranging any of the letters. For example: sprite, spit, pit, it, I. There are many such words, but, as Barr discovered, only one with 11 letters.

So, only one word with eleven letters: complecting.  Maybe.  I’m not convinced, but finding out is easy.

Recursion

Okay, so how would you know?  Well, suppose you had a list of valid words.  The statement of the problem suggests a simple recursive algorithm to find a solution.

• Put the words in a heap, ordered longest to shortest.
• Iterate until the heap is empty.
  • Take the next word $w$ from the heap, and perform $\mbox{search}(w)$

The procedure $\mbox{search}(w)$ works as follows.  It returns true if it finds a solution.

• If $w$ is the empty string, then return true.
• For each letter $l$ of $w$, do the following.
  • Drop $l$ from $w$, generating the candidate word $v$.
  • If $v$ is in the dictionary, then execute $r=\mbox{search}(v)$.
    • If $r$ is true, then write out $w$ and return true.
• The word $w$ is a dead end; drop it from the dictionary and return false.

Dropping the words from the dictionary, or otherwise marking them as dead ends in some other way is necessary to avoid re-searching them later.  The time complexity is roughly $O(n)$, where $n$ is the size of the dictionary to search.  If the average length of a word is $k$, then we can be more specific, and say the complexity is $O(kn)$, but $k$ is very small with respect to $n$, and does not vary much with the data size, so it can be safely ignored.

Iteration

The problem was explained to me in the reverse fashion.  Given a letter, generate words by adding a single letter at a time.  Every word must be in the dictionary.  Find the longest word you can generate from a single letter.  This suggests a different algorithm.

• Put the words in a heap, ordered shortest to longest.
• Mark the empty string as reachable and set $long$ to the empty string.
• Iterate through the heap, considering each word $w$.
  • For each position $i$ of $w$, do the following.
    • Drop the $i$th letter from $w$, generating the candidate word $v$.
      • If $v$ is marked as reachable, then mark $w$ as reachable by omitting $i$, and set $long=w$.
• While $long$ is not the empty string, do the following.
  • Write $long$.
  • Omit the character $i$ by which $long$ was reached, and set $long$ equal to the new word.

We can improve this by checking against the length of the longest word reached.  If we ever have $|w|>|longest|+1$, then we can stop the loop immediately, since we cannot reach any further words.  We can’t do that with the prior algorithm.

So… What’s the Longest Word?

The solution given was complecting, at 11 letters.  When I download the aspell dictionaries, and run the algorithm (the iterative one), I find the following derivation.

-> a
-> ah
-> ach
-> bach
-> banch
-> banchi
-> branchi
-> branchia
-> branchiae
-> branchiate
-> abranchiate
-> abranchiates

Ooh.  12 letters.  But, it’s a plural.  Does that count?  Well, complecting is a participle, so it seems it should.  Time to send a note to Tom and Ray?

So… Which Algorithm?

Well, I coded up the iterative one in Java, since originally I was asked how I would code it in Java.  Again, the question suggesting the solution.  I suspect there are a lot of long words that are not reachable, so it seemed better to use the trick to terminate the search when the current word is longer than the longest found (so far) plus one.  Plus, it isn’t recursive, and I don’t have to modify the data structure by deleting an item from the heap.

The Java

So… here’s the Java source.  Enjoy!

/**
 * Given some iterable object that provides strings, find the longest
 * string in the collection that can be reached from the empty string
 * by adding one character at a time such that every intermediate step
 * is also in the collection.
 *
 
 * For example, if the collection is {@code {@literal {}a,
 * at, cat, chat, chart, cart, car{@literal }}},
 * then the longst word is {@code chart}, and it's derivation is:
 * {@code a => at => cat => cart => chart}.  If there are multiple
 * longest words, then only one is detected.
 *
 
 * Note that if single letters are not in the collection, then no words
 * can be derived, and only the empty derivation is returned.
 *
 * @param strings    An iterable providing the strings.
 * @return    The derivation of the longest, from the shortest word to the
 *             longest.  If there is no longest word, then the empty
 *             derivation is returned.
 */
public List search(Iterable strings) {
    if (strings == null) {
        throw new NullPointerException("The string iterator is null.");
    }
 
    // Read each string from the iterable.  The strings assumed to be
    // single words, and the words are not assumed to occur in any
    // particular order.  We put the words we read into a heap, ordered
    // by length.  Our "heap" is a sorted map.
    //
    // Every entry in the map has an associated integer value, which
    // tells us the position of the last inserted character.  This can
    // be used to "unwind" the derivation of the word.  If there is not
    // (yet) any way to reach the word, then the word is mapped to -1.
    TreeMap heap =
        new TreeMap(new Comparator() {
            public int compare(String first, String second) {
                // Okay, we sort first by length, and then alphabetically.
                // This is necessary because of the way TreeMap tests
                // equality; it uses the comparator.  Thus we must only
                // return zero if the two are actually equal.  So, we
                // first order by length, and then for items of the same
                // length, we order using the natural comparator for
                // strings.
                int order = first.length() - second.length();
                if (order == 0) {
                    order = first.compareToIgnoreCase(second);
                }
                return order;
            }
        });
    for (String str : strings) {
        heap.put(str,-1);
    } // Add all strings to the heap.
 
    // Prime the pump by adding the empty string as "reachable."
    String longest = "";
    int length = 0;
    heap.put("", 0);
 
    // At this point the heap is constructed.  We now iterate over the
    // heap looking for connected words.  If we find one, we mark it.
    for (String str : heap.keySet()) {
        // Watch for a case where we skip a length.  If we do, we're done.
        if (str.length() > length+1) {
            break;
        }
        length = str.length();
 
        // Now we have a string, we need to find out if the string is
        // reachable from a previously-reached string.  We drop each
        // character, and then check if the resulting string is
        // present in the heap and is marked.
        for (int index = 0; index < str.length(); index++) {
            // Omit the character at position index.
            String test = str.substring(0,index) + str.substring(index+1);
            // See if the resulting test string is in the heap.
            Integer pos = heap.get(test);
 
            // Now there are three cases:
            // (1) The test string is not in the heap.
            // (2) The test string is in the heap, but not reachable.
            if (pos == null || pos < 0) {
                continue;
            }
            // (3) The test string is in the heap, and is reachable.
            // In this last case we know we can reach the current string
            // from the test string by adding the character at position
            // index, so put this in the heap.
            heap.put(str, index);
            // Note that whenever this happens, we might have just found
            // (one of) the longest words, so save it.
            longest = str;
            // We don't need to test any other positions.
            break;
        } // Construct new words by omitting each character.
    } // Search the heap for words.
 
    // Now we have (hopefully) found a long word.  We need to generate
    // the derivation and return it.
    List derivation = new LinkedList();
    while (longest.length() > 0) {
        // Get the insertion position from the heap.
        int pos = heap.get(longest);
        // Add the word to the derivation.
        derivation.add(0, longest);
        // Generate the prior string in the derivation.
        longest = longest.substring(0, pos) + longest.substring(pos+1);
    } // Build the derivation.
 
    // Done!  Return the derivation.
    return derivation;
}

The code isn’t optimal, but it is commented. I assume you are all capable of putting this in a class and feeding it a dictionary. Have fun!

Programming

I need to generate multipart/form-data (see here) messages from Python. Never mind why. I dug around in the documentation for httplib, urllib, and urllib2, but it seems this is not currently supported (it’s Issue 3244). I didn’t like the code I found on the web to do it, because I needed to set additional headers on each piece. So… I wrote something. Here it is. If it’s useful to you, great! If you find bugs in it, please let me know. I think this is pretty easy to use.

Make an instance of Multipart, and then add parts using the field and file methods.

>>> from multipart import Multipart
>>> m = Multipart()
>>> m.field('search','searchish term')
>>> m.file('greet','greet.txt','Hello multipart world!',{'Content-Type':'text/text'})
>>> ct,body = m.get()
>>> print ct
multipart/form-data; boundary=----------AaB03x
>>> print body
------------AaB03x
Content-Type: application/octet-stream
Content-Disposition: form-data; name="search"
 
searchish term
------------AaB03x
Content-Type: text/text
Content-Disposition: form-data; name="greet"; filename="greet.txt"
 
Hello multipart world!
------------AaB03x--

If no content type is specified for a value, then the default of application/octet-stream is chosen. If none is specified for a file, then the mime libraries are consulted to guess one based on the filename, and again the default is application/octet-stream if none can be guessed.

If you want to specify the content type, make sure to use the string ‘Content-Type’ (note the caps) or it will be ignored. I’ve seen the capitalization all over the map, but needed to choose one to use in the case-sensitive dict. If you don’t like my choice… change the code. See the constants at the start of the Client class.

Here’s how you might continue the above example to send this out in an HTTP request.

request = urllib2.Request(url='http://my.fake.server',
    headers={'Content-Type':ct},
    data=body)
reply = urllib2.urlopen(request)
print reply.read()

Finally, here’s the code. Enjoy! This comes, of course, with no warranty, use at your own risk, etc.

#!/usr/bin/python
'''
Classes for using multipart form data from Python, which does not (at the
time of writing) support this directly.
 
To use this, make an instance of Multipart and add parts to it via the factory
methods field and file.  When you are done, get the content via the get method.
 
@author: Stacy Prowell (http://stacyprowell.com)
'''
 
import mimetypes
 
class Part(object):
    '''
    Class holding a single part of the form.  You should never need to use
    this class directly; instead, use the factory methods in Multipart:
    field and file.
    '''
 
    # The boundary to use.  This is shamelessly taken from the standard.
    BOUNDARY = '----------AaB03x'
    CRLF = '\r\n'
    # Common headers.
    CONTENT_TYPE = 'Content-Type'
    CONTENT_DISPOSITION = 'Content-Disposition'
    # The default content type for parts.
    DEFAULT_CONTENT_TYPE = 'application/octet-stream'
 
    def __init__(self, name, filename, body, headers):
        '''
        Make a new part.  The part will have the given headers added initially.
 
        @param name: The part name.
        @type name: str
        @param filename: If this is a file, the name of the file.  Otherwise
                        None.
        @type filename: str
        @param body: The body of the part.
        @type body: str
        @param headers: Additional headers, or overrides, for this part.
                        You can override Content-Type here.
        @type headers: dict
        '''
        self._headers = headers.copy()
        self._name = name
        self._filename = filename
        self._body = body
        # We respect any content type passed in, but otherwise set it here.
        # We set the content disposition now, overwriting any prior value.
        if self._filename == None:
            self._headers[Part.CONTENT_DISPOSITION] = \
                ('form-data; name="%s"' % self._name)
            self._headers.setdefault(Part.CONTENT_TYPE,
                                     Part.DEFAULT_CONTENT_TYPE)
        else:
            self._headers[Part.CONTENT_DISPOSITION] = \
                ('form-data; name="%s"; filename="%s"' %
                 (self._name, self._filename))
            self._headers.setdefault(Part.CONTENT_TYPE,
                                     mimetypes.guess_type(filename)[0]
                                     or Part.DEFAULT_CONTENT_TYPE)
        return
 
    def get(self):
        '''
        Convert the part into a list of lines for output.  This includes
        the boundary lines, part header lines, and the part itself.  A
        blank line is included between the header and the body.
 
        @return: Lines of this part.
        @rtype: list
        '''
        lines = []
        lines.append('--' + Part.BOUNDARY)
        for (key, val) in self._headers.items():
            lines.append('%s: %s' % (key, val))
        lines.append('')
        lines.append(self._body)
        return lines
 
class Multipart(object):
    '''
    Encapsulate multipart form data.  To use this, make an instance and then
    add parts to it via the two methods (field and file).  When done, you can
    get the result via the get method.
 
    See http://www.w3.org/TR/html401/interact/forms.html#h-17.13.4.2 for
    details on multipart/form-data.
 
    Watch http://bugs.python.org/issue3244 to see if this is fixed in the
    Python libraries.
 
    @return: content type, body
    @rtype: tuple
    '''
 
    def __init__(self):
        self.parts = []
        return
 
    def field(self, name, value, headers={}):
        '''
        Create and append a field part.  This kind of part has a field name
        and value.
 
        @param name: The field name.
        @type name: str
        @param value: The field value.
        @type value: str
        @param headers: Headers to set in addition to disposition.
        @type headers: dict
        '''
        self.parts.append(Part(name, None, value, headers))
        return
 
    def file(self, name, filename, value, headers={}):
        '''
        Create and append a file part.  THis kind of part has a field name,
        a filename, and a value.
 
        @param name: The field name.
        @type name: str
        @param value: The field value.
        @type value: str
        @param headers: Headers to set in addition to disposition.
        @type headers: dict
        '''
        self.parts.append(Part(name, filename, value, headers))
        return
 
    def get(self):
        '''
        Get the multipart form data.  This returns the content type, which
        specifies the boundary marker, and also returns the body containing
        all parts and bondary markers.
 
        @return: content type, body
        @rtype: tuple
        '''
        all = []
        for part in self.parts:
            all += part.get()
        all.append('--' + Part.BOUNDARY + '--')
        all.append('')
        # We have to return the content type, since it specifies the boundary.
        content_type = 'multipart/form-data; boundary=%s' % Part.BOUNDARY
        return content_type, Part.CRLF.join(all)

Some methodologies, like Cleanroom, prohibit programmers testing or in some cases even compiling their own programs.

As evidence of this he cites the text Cleanroom Software Engineering: Technology and Process, by, among others, yours truly.  I for one have never said that people should not run unit tests, nor does the above text say that.  I emailed Mr. Beck about it, and he confirmed that he did not get this from the cited text.  Whew!

Just to be clear, here is my position.  I would never tell anyone not to run a test they felt was necessary. I may personally believe there are efficient and inefficient ways to test, but I’m not an expert in every domain.  The people who create a product are responsible for the consequences of release of that product, and should act accordingly.

The Purpose of Testing

The following is my philosophical position on software testing.

The purpose of software testing is to gain evidence that software release will not be harmful.

In particular, I believe it is impractical and inefficient to take the position that the purpose of testing is to find bugs.  It leads to poor practice, and information that isn’t really “actionable.”

Some may insist that the purpose of testing is specifically to find bugs, but they’re misguided.  If you want to find bugs, testing is a poor way to do it.  Use inspections or verification.  But more on that later.

Cleanroom

I’m a late-comer to Cleanroom.  Since Harlan Mills introduced the notion to the world in the 1980’s, it’s been an ever-evolving set of practices.  I’d say the following are the essential elements of Cleanroom software engineering.

• Emphasize defect prevention rather than defect correction.  I’d say this is the fundmental characteristic of Cleanroom.
• Statistical certification of software quality.
• Iterative and incremental development under statistical process control.
• Formal or rigorous specification.
• Stepwise refinement with verification.

Okay, I cheated.  Here’s what Harlan Mills, Mike Dyer, and Rick Linger have to say on the subject in “Cleanroom Software Engineering,” IEEE Software, 14(2), 1987.

With the Cleanroom process, you can engineer software under statistical quality control.  As with cleanroom hardware development, the process’s first priority is defect prevention rather than defect removal (of course, any defects not prevented should be removed).  This first priority is achieved by using human mathematical verification in place of program debugging to prepare software for system test.

Its next priority is to provide valid, statistical certification of the software’s quality through representative-user testing at the system level.  The measure of quality is the mean time to failure in appropriate units of time (real or processor time) of the desired product.  The certification takes into account the growth of reliability achieved during system testing before delivery.

To gain the benefits of quality control during development, Cleanroom software engineering requires a development cycle of concurrent fabrication and certification of product increments that accumulate into the system to be delivered.  This lets the fabrication process be altered on the basis of early certification results to achieve the quality desired.

Unit testing is not mentioned in the article, because the emphasis is on breaking the code-compile-debug cycle.

Prohibiting Unit Testing

So, unit testing.  Phil Hausler, Rick Linger, and Carmen Trammell have this to say in “Adopting Cleanroom Software Engineering With a Phased Approach,” IBM Systems Journal, March 1994.  (Read it here.)

In traditional, craft-based software development errors were accepted as inevitable, and programmers were encouraged to get software into testing quickly in oder to begin debugging. Programs were subjected to unit testing and debugging by their authors, then integrated into components, subsystems, and systems for more debugging. Product use by customers resulted in still more debugging to correct errors discovered in operational use. The most virulent errors were often the result of fixes to other error, and it was not unusua for software products to reach a steady-state error population, with new errors introduced as fast as old ones were fixed. Today, however, craft-based processes that dependent on testing and debugging to improve reliability are understood to be inefficient and ineffective. Experience has shown that craft-based processes often fail to achieve the level of reliability essential to a society dependent on software for the conduct of human affairs.

They cite the Adams data in support of the above.  (E. N. Adams, “Optimizing Preventive Service of Software Products,” IBM Journal of Research and Development, 28(1):2-14, 1984.)

I believe it is clear from the above what sort of practices they are targeting.  They go on to write the following.

In the Cleanroom process, correctness is built into the software by development teams through a rigorous engineering process of specification, design, and verification. The more powerful process of team correctness verification replaces unit testing and debugging, and software enters system testing directly, with no execution by development teams. All errors are accounted for from first execution on, with no private unit testing necessary or permitted.

So there you have it; the prohibition against unit testing.  This is done to break the cycle of code-compile-test by taking the test and possibly compile phases out of the hands of the developers.  And I agree, up to a point.  I would even go so far as to claim that Kent Beck might agree, but break the cycle in a different way.  He requires you to write the tests first.  Then you write the code that passes these tests.

Inspection is Insufficient

Software inspection is the most effective means to find and remove defects.  I can say this because the studies all support this.  Even informal code reviews are far more effective than testing, when both are done properly.

The fact is that inspections are not enough.  Nobody understands the semantics of modern high-level languages.  I’m serious; you do not.  You may think you do, in which case you are dangerously deluded.

There are three primary reasons for this.

1. Language standards are seldom complete or consistent.  It is possible, for instance, to have legally constructed C programs whose meaning is not specified by the C standard.  Compiler writers are free to make their own decisions.
2. Compiler writers make mistakes.  Library writers make mistakes.
3. Computers are weird.  Can you explain the IEEE-754 standard to me, along with the details of the particular implementation you are using?  How about the fact that there is one more negative number than positive numbers in the two’s-complement world?

The short of this is that you have to execute the code.  You have to run tests.  I see nothing wrong with test early, test often.  Is the library documentation insufficient?  Then you’re going to code up and run some tests, aren’t you?

Even if you work at the assembly level, and don’t worry with compilers and high-level languages, computers are still a bit dodgy.  Are “add eax,dword 1″ and “inc eax” the same?  Many emulators assume they are, but they differ in one particular way.  Do you know what it is?  The people who write malicious code do.

Verification

Testing is insufficient, since you cannot prove the absence of defects, bad behavior, or malicious code by testing.  Inspection, as I just argued, is insufficient for a number of reasons.

The answer is verification.  Verification is challenging, mathematical, and requires a precise model of the system.  Fortunately, automated verification is on its way.  By creating a formal model of the processor, warts and all, and then by computing the functional behavior of code, one can achieve results that cannot be achieved by either testing or inspection.

Automated verification is essential.  Manual verification is simply not practical, thanks to the complexity of modern processors.  I am capable of proving the correctness of an algorithm, but then it is implemented on a finite precision machine with lots of non-linear and discontinuous behavior, and all bets are off.  I would argue that humans are simply not good enough at the bookkeeping to do the job for large software systems.  Computers are.

Projects like CERT’s Function Extraction (FX) effort may finally deliver on the promise of software verification.  Until then, you’ll just have to run your tests.

Do It Correctly

Here’s my advice on unit testing.  It’s short.  Feel free to extend and revise it as necessary.

• Have someone other than the programmer write the unit test.
• Test the interface.  That is, write the test as a user of the module, not the developer.  The same could be said for documentation.
• Test the GUI.  It isn’t as hard as you think it is.
• Write your tests to generate random values and also test boundary values.
• For stateful modules, write tests that generate sequences of inputs.

• Reified generics or
• Closures

In the interim, I offer the following Java code to the world.  Does it help?  Use at your own risk.

This is a class that wraps the collections interface to provide for recovery of the type parameter.  To use it, make a new instance and pass the class object for the parameter to the constructor, along with the original collection.

package gypsum.util;
 
import java.util.Collection;
import java.util.Iterator;
 
/**
 * This class packages some implementation of {@link Collection} along with
 * the class object for the type parameter of the collection.  This allows
 * getting the type of objects in the collection using the {@link #getType()}
 * method, and then using the resulting class to perform casts.  This basically
 * creates a "reified" generic collection (of sorts).
 *
 * @param    Class of data held in this collection.
 * @author sprowell
 * @version $Revision$ $Date$
 */
public class TypedCollection implements Collection {
 
    private Class type;
    private Collection implementation;
 
    /**
     * Make a new instance.
     * @param type              The class object for the type of elements.
     * @param implementation    The implementation.
     */
    public TypedCollection(Class type, Collection implementation) {
        if (type == null) {
            throw new NullPointerException("The type is null.");
        }
        if (implementation == null) {
            throw new NullPointerException("The implementation is null.");
        }
        this.type = type;
        this.implementation = implementation;
    }
 
    /**
     * Get the type held in this collection.
     * @return  The type held in this collection.
     */
    public Class getType() {
        return type;
    }
 
    /**
     * Recover the original typed collection.  In order for this to work,
     * the correct class must be supplied; otherwise you will get a
     * {@link ClassCastException}.  Use this as follows.
     *
<pre>     * TypedCollection&lt;Foo&gt; coll = oldcoll.recover(oldcoll.getType());
     *
     *
 
     * This converts from {@code TypedCollection&lt;?&gt;} to
     * {@code TypedCollection&amp;ltT&gt;}, so you recover the generic type
     * parameter.
     *
 
     * This may seem backward, or even a little contrived.  Blame it on
     * the Java generics system.  The reason you must pass the class
     * object <em>in</em>, even though it is maintained here, is that
     * the compiler must know the type of the object <em>at the point
     * you use this method</em>.  Thus you can do the following.
     *
<pre>     * public void print(TypedCollection&lt;?&gt; coll) {
     *   if (coll.getType() == String.class) {
     *     TypedCollection&lt;String&gt; coll = coll.recover(String.class);
     *     for (String str : coll) {
     *       System.out.println(str);
     *     }
     *   }
     * }
     * &lt;/pre&gt;
     *
     * @param    The type of the desired collection.
     * @param kind  The class object for type T.
     * @return  The correctly cast generic collection.
     * @throws  ClassCastException
     *          This instance is <em>not</em> the type desired.
     */
    @SuppressWarnings("unchecked")
    public  TypedCollection recover(Class kind) {
        if (kind == type) {
            return (TypedCollection) this;
        }
        throw new ClassCastException("Incorrect cast.");
    }
 
    /**
     * {@inheritDoc}
     * @see java.util.Collection#add(java.lang.Object)
     */
    @Override
    public boolean add(E arg) {
        return implementation.add(arg);
    }
 
    /**
     * {@inheritDoc}
     * @see java.util.Collection#addAll(java.util.Collection)
     */
    @Override
    public boolean addAll(Collection arg) {
        return implementation.addAll(arg);
    }
 
    /**
     * {@inheritDoc}
     * @see java.util.Collection#clear()
     */
    @Override
    public void clear() {
        implementation.clear();
    }
 
    /**
     * {@inheritDoc}
     * @see java.util.Collection#contains(java.lang.Object)
     */
    @Override
    public boolean contains(Object arg) {
        return implementation.contains(arg);
    }
 
    /**
     * {@inheritDoc}
     * @see java.util.Collection#containsAll(java.util.Collection)
     */
    @Override
    public boolean containsAll(Collection c) {
        return implementation.containsAll(c);
    }
 
    /**
     * {@inheritDoc}
     * @see java.util.Collection#isEmpty()
     */
    @Override
    public boolean isEmpty() {
        return implementation.isEmpty();
    }
 
    /**
     * {@inheritDoc}
     * @see java.util.Collection#iterator()
     */
    @Override
    public Iterator iterator() {
        return implementation.iterator();
    }
 
    /**
     * {@inheritDoc}
     * @see java.util.Collection#remove(java.lang.Object)
     */
    @Override
    public boolean remove(Object o) {
        return implementation.remove(o);
    }
 
    /**
     * {@inheritDoc}
     * @see java.util.Collection#removeAll(java.util.Collection)
     */
    @Override
    public boolean removeAll(Collection c) {
        return implementation.removeAll(c);
    }
 
    /**
     * {@inheritDoc}
     * @see java.util.Collection#retainAll(java.util.Collection)
     */
    @Override
    public boolean retainAll(Collection c) {
        return implementation.retainAll(c);
    }
 
    /**
     * {@inheritDoc}
     * @see java.util.Collection#size()
     */
    @Override
    public int size() {
        return implementation.size();
    }
 
    /**
     * {@inheritDoc}
     * @see java.util.Collection#toArray()
     */
    @Override
    public Object[] toArray() {
        return implementation.toArray();
    }
 
    /**
     * {@inheritDoc}
     * @see java.util.Collection#toArray(T[])
     */
    @Override
    public  T[] toArray(T[] a) {
        return implementation.toArray(a);
    }
}

So… I wanted a way to trap CTRL+C and clean up.  That is, I wanted to have transactions without actually having to use transactions.  I’m funny that way.  (In other news, I’m probably moving to PostgreSQL at some point.)

Anyway, here’s what I came up with.  You might find it useful, too.

#!/usr/bin/python -i
 
'''
Trap keyboard interrupts.  No rights reserved; use at your own risk.
 
@author: Stacy Prowell (http://stacyprowell.com)
'''
import signal
 
class BreakHandler:
    '''
    Trap CTRL-C, set a flag, and keep going.  This is very useful for
    gracefully exiting database loops while simulating transactions.
 
    To use this, make an instance and then enable it.  You can check
    whether a break was trapped using the trapped property.
 
    # Create and enable a break handler.
    ih = BreakHandler()
    ih.enable()
    for x in big_set:
        complex_operation_1()
        complex_operation_2()
        complex_operation_3()
        # Check whether there was a break.
        if ih.trapped:
            # Stop the loop.
            break
    ih.disable()
    # Back to usual operation...
    '''
 
    def __init__(self, emphatic=9):
        '''
        Create a new break handler.
 
        @param emphatic: This is the number of times that the user must
                    press break to *disable* the handler.  If you press
                    break this number of times, the handler is automagically
                    disabled, and one more break will trigger an old
                    style keyboard interrupt.  The default is nine.  This
                    is a Good Idea, since if you happen to lose your
                    connection to the handler you can *still* disable it.
        '''
        self._count = 0
        self._enabled = False
        self._emphatic = emphatic
        self._oldhandler = None
        return
 
    def _reset(self):
        '''
        Reset the trapped status and count.  You should not need to use this
        directly; instead you can disable the handler and then re-enable it.
        This is better, in case someone presses CTRL-C during this operation.
        '''
        self._count = 0
        return
 
    def enable(self):
        '''
        Enable trapping of the break.  This action also resets the
        handler count and trapped properties.
        '''
        if not self._enabled:
            self._reset()
            self._enabled = True
            self._oldhandler = signal.signal(signal.SIGINT, self)
        return
 
    def disable(self):
        '''
        Disable trapping the break.  You can check whether a break
        was trapped using the count and trapped properties.
        '''
        if self._enabled:
            self._enabled = False
            signal.signal(signal.SIGINT, self._oldhandler)
            self._oldhandler = None
        return
 
    def __call__(self, signame, sf):
        '''
        An break just occurred.  Save information about it and keep
        going.
        '''
        self._count += 1
        # If we've exceeded the "emphatic" count disable this handler.
        if self._count >= self._emphatic:
            self.disable()
        return
 
    def __del__(self):
        '''
        Python is reclaiming this object, so make sure we are disabled.
        '''
        self.disable()
        return
 
    @property
    def count(self):
        '''
        The number of breaks trapped.
        '''
        return self._count
 
    @property
    def trapped(self):
        '''
        Whether a break was trapped.
        '''
        return self._count > 0

To use this, make an instance and then enable the instance.  If you repeatedly hit CTRL+C you will eventually disable it (see emphatic in the __init__).

# Trap break.
bh = BreakHandler()
bh.enable()
for item in big_set:
    if bh.trapped:
        print 'Stopping at user request (keyboard interrupt)...'
        break
    do_thing_1()
    do_thing_2()
    do_thing_3()
do_cleanup()
bh.disable()

If you press CTRL+C, it is detected at the start of the loop, and break is called.  Then do_cleanup() runs, and the break handler is disabled, returning things to normal.  The idea is that do_thing_1(), do_thing_2(), and do_thing_3() all run to completion, even when the user presses CTRL+C, and do_cleanup() always runs when the loop is exited.

I consider this public domain; if you like it, use it. If you don’t like it, don’t use it. If you have suggestions, please leave me a comment.

computer science: n. a study akin to numerology and astrology, but lacking the precision of the former and the success of the latter.

– Stan Kelly-Bootle, The Devil’s DP Dictionary

I’d say the above is no longer true: computer science is successful.  But it really isn’t very precise.

The field of computer science is probably the area of knowledge that is closest to pure gibberish. On my shelf are books with the titles Python in a Nutshell, Enterprise Java Beans, and MySQL & mSQL.  A random passage reads:

In this example, gmpy manages to compensate for the float’s “representation error” and emit the exact fraction that you presumably meant rather than the mathematically exact one, which tends to be less useful.

– Python in a Nutshell, p. 374

You get the point.

This is because where other fields invent new names for new concepts, computer science tends to borrow old names for new concepts. Thus “bean” and “object” take on new meanings. Even mathematics tends to coin terms carefully, and use names for specific concepts, such as “Jacobi identity.”  You won’t confuse that with a desert topping, and other mathematicians will typically know what you mean.

Sometimes things get really confusing as perfectly good words get used up.  Like the word word, which means 16 bits (bits?) on most computers.  Wouldn’t want to confuse that with a dword, or double word.  You get the confusion of the specific versus the general, when words like object, entity, and attribute get used up.  Then you get text like the following (I plead the Fifth on where this originates):

If the entity is a class object then the collection of known object classes is updated with the provided class object.  Otherwise the object’s class is declared to the container manager prior to being serialized.

Fred Brooks pointed out that most fields study what’s out there; natural phenomena, if you will. Computer science studies its own invented concepts, and they could have been implemented in different ways and often are; there are many competing ways of doing things.

All this is relatively important. Computer scientists, programmers, and others who work in the field develop an affinity for this sort of silliness and help perpetuate it, often just for fun.  Need a name for your just-in-time compiler?  How about ElectricalFire?  (Apparently the other name proposed was, famously, “sexual chocolate.”)

The goal isn’t to be perspicuous (or I wouldn’t have used that word, after all), but to have fun.  And in the end, isn’t that what computer science really should be all about?