Developing on Windows - Replacing the basic tools.

As anyone who has to work with Windows as their development environment knows, the tools Windows comes with are crap. Notepad, Windows Explorer and the Command prompt to name a few. Tools such as these are vital to programming productivity, and are simply substandard in all current versions of Windows. As far as it goes, Explorer is fine for the average Windows user, but is awkward if you need to keep track of different folders, and transfer files between them. In addition, there is no easy way of customising Explorer without hacking the registry. Notepad isn't fit even for the average user - particularly since it is the default text editor. It can't handle anything other than ASCII, and won't even automatically convert between different line endings. The Command prompt is pretty inexcusable. A tool only used by programmers and sys admins, you would have thought that the console program in Windows might have been a little more developed by now, but seems stuck in the dark ages, and years behind the unix console.

Luckily there are some free, or at least very cheap alternatives for all of the above tools.

Xplorer2 - Explorer on steroids.

An Explorer replacement that I've been using for years, and remains one of the few development tools I've paid money for (there being no free or open source equivalent with the same sort of power) is [Xplorer2] [http://zabkat.com/]. This tool is superb - featuring tabs, dual pane, scrap folders, user defined commands, user defined toolbars, a very powerful search facility etc etc.

The tabs are great - you can keep a tab open for each of your most visited folders. And they are nested within dual panes if you wish, so you can keep different sets of tabs open for each pane.

Dual panes not only make moving files about much quicker and easier than the traditional Explorer tree view, but they have a Mirror browsing mode for comparing trees of folders with different roots. For example, you have trunk and a branch checked out from your source repository. If you have trunk open in the left view, and the branch in the right, with Mirror browsing switched on, moving through the source tree in one pane will be mirrored in the other. With the built in folder sync tool, you can check for differences between the folders based on a flexible selection of parameters (date, content etc)

Scrap panes can be used to collect together files from various folders into a virtual folder. For example the 'Browse Flat' menu item puts all files in a folder subtree into one virtual folder. These files can be operated on as if this were an actual folder, compared via a dual pane view. Useful for many purposes, including keeping sets of disparate but related files together, as extended bookmarks etc.

User defined functions can be used to great effect. One of my favorites is integrating WinMerge into Xplorer2, by defining a pair of commands, the first of which calls WinMerge on the currently open directories in left and right panes; the second calls WinMerge on the currently selected left and right files. So, using mirror browse on my trunk and branch, I select the file in trunk I want to compare (the branch file if automatically selected correspondingly), hit CTRL-1 and WinMerge opens comparing those two files.

Not free, but well worth the $30 fee for the professional version (especially in the UK with the pound as strong as it is at the moment!). There's also a lite version for home and academic use.

GVim, Cream

Actually, just about any open source editor is better than Notepad. For programming though, certain things are indispensable: syntax highlighting; correct handling of line endings; smart indentation handling etc. I prefer [GVim] [http://www.vim.org/] these days - several years editing stuff on unix servers has taught me the power of vim (through a painful and long induction...). Not an editor for novices, it requires a lot of practice and a different way of thinking than your usual text editor, but is very powerful, especially since it can be programmed with my language de jour, Python.

For the more faint of heart, [Cream] [http://cream.sourceforge.net/] is GVim, but configured for ease of use. By default it works like a normal text editor, but has all of the power of vim working in the background and available for use in advanced mode.

Apart from the features already described, I use the following Vim features (in no particular order) on a regular basis:

1) Auto-complete 2) Marks - mark lines with a letter, and jump between these. 3) Macros - record macros to perform repetitive tasks. 4) Block select - one of my favorites, this lets you for example insert the same text in the middle of a block of 7 lines. I use this frequently for tasks such as editing repetitive code and property file editing. 5) Python scripting - for example sort the lines in the file into alphabetical order: :py lines = vim.current.buffer[:]; lines.sort(); vim.current.buffer[:] = lines 6) Spelling checker 7) Global command - match lines using a combination of a line range and pattern, and then do stuff to those lines. :2,6g/^#/dd deletes all lines between 2 and 6 that start with a # 8) Folding - hide blocks of code from view. Useful to get a higher level view of source code.

There are other editors than Vim with features like these, Emacs and JEdit come to mind, but for me Vim is the winner since it is small, powerful and ubiquitous on unix/linux based machines where I spend quite a bit of time.

Console and Cygwin - a better Command shell.

I've just discovered the delights of [Console] [http://sourceforge.net/projects/console/]. It's a tabbed console for windows which allows you to pre-define tab definitions to open up specific configurations. For example, I have tabs set up for bash, Python, Jython, Beanshell etc. Very nice, and in addition circumvents the problems the standard Windows command interface has with selection and the fact that starting a selection pauses the process currently running (I find this incredibly irritating - though I know others find it a feature. You can get the same behaviour in Console, or in Windows command console using the pause key on the keyboard.) In addition, for those into such frivolities, there are lots of special effect to make your tabs transparent, have wallpaper etc.

It brings to windows some of the convenience of programs such as Konsole on Linux, and in combination with [Cygwin] [http://cygwin.com/][1], you can even use the bash shell and gnu tools with all of their power. Actually things may have improved with the introduction of [PowerShell] [http://www.microsoft.com/windowsserver2003/technologies/management/powershell/default.mspx], but I haven't had the time to experiment with it yet.

Swing Testing with Jython and Jemmy Part 1 - Setting up the system

Introduction

Testing is an important part of the software development process, and various types of testing need to be combined in order to make sure a product is sufficiently tested. A full set of unit tests goes a long way toward making a stable product, and brings with it a safety net to catch errors made when refactoring the code base. Unit tests a far from the whole story however; they are by nature highly localised, and so can't test the overall behaviour of a system. Functional tests go a step further in this regard than unit tests, wiring together various different components and checking that they work together correctly. At a higher level still are system tests, which should test the system as a whole, and that you can use the software to achieve what it is designed to do. Orthogonal to system tests, but at the same sort of level are acceptance tests, tests designed to see whether the product is suitable for the customers to use. Acceptance tests need to check the more superficial (but nevertheless important) aspects of the product. Does the product meet usability targets? Can the user cause crash the system by doing things wrong?

This post is intended to address the latter two types of testing: automating system and acceptance tests on the Java GUI. The basics of the technique requires two open source libraries: Jemmy, a GUI automation tool from the Netbeans team, and Jython, an implementation of Python written in Java.

Setting up the System

1. Create a system-test folder, with tests and lib subdirectories. The lib subdirectory should further have java and python child directories. The java directory is where the jemmy and jython jars will reside, lib/python is where the python standard library will live.
2. Download Jemmy and Jython. If Jython hasn't reached 2.2 final yet, 2.2a1 is very stable.
3. Install Jython, and then copy jython.jar to lib/java and the contents of the Lib directory from within the installation directory to lib/python.
4. Copy jemmy.jar into lib/java

You will now need to create a script to run your tests. Typically there will already be a script that is used to run the application that you want to test, and a copy of this script can be adapted to the job. In the case that the application is run by running the jar directly, you will need to examine the manifest in order to extract the classpath and main class for the script. The script will look something like the following:

#!/bin/bash
# file: run.sh
APP_DIR=/my/app
TEST_DIR=$APP_DIR/system-test

APP_LIB=$APP_DIR/lib
JAVA_LIB=$TEST_DIR/lib/java
PYTHON_LIB=$TEST_DIR/lib/python

CLASSPATH=$CLASSPATH\;$APP_LIB/anImportantLib.jar
CLASSPATH=$CLASSPATH\;$APP_LIB/anotherLib.jar

CLASSPATH=$CLASSPATH\;$JAVA_LIB/jemmy.jar
CLASSPATH=$CLASSPATH\;$JAVA_LIB/jython.jar

PYTHON_PATH=-Dpython.path="$PYTHON_LIB;./scripts;$TEST_DIR/tests;"

$JAVA_HOME/bin/java -cp $CLASSPATH $PYTHON_PATH org.python.util.jython $@

Note that this script will do nothing unless provided with a script name as an argument, and any arguments the script requires. As we will see later, the application itself is started within the test script. On windows, this can be converted to a batch script, or else run using Cygwin.

Writing the first script

The first script will simply start the application. Nothing more. It will go in the system-test/tests directory. Note that we start the app in a separate thread so that the running app doesn't block our tests from running!

# file: test.py
from co.uk.mycompany.myapp import MainApp
import sys

def run():
    MainApp.main(sys.argv[1:])

def main():
    startup = Thread(target=run)
    startup.start()

if __name__ == "__main__":
    main()

So, after that bit of work, we are back where we started, except that now we have to do something like the following to start the app:

./run.sh tests/test.py more args here

The next article will go into more detail about using jython and jemmy to do some actual work.

Java Readers and Streams rant.

Another bit of Java design genius is the Reader/Stream architecture. Streams are low level things dealing with streams of bytes. Readers are higher level, and make working with character arrays easier. There is a useful class called InputStreamReader which takes a Stream, and converts it into a Reader so that you can work with the data as a stream of characters. All very nice, but also all very unidirectional...

Consider this: you have an object which loads data in via a load(InputStream is) method. The data you want to load is in a String you have obtained from elsewhere. OK, so what are your options? Is there a StringInputStream? no - but there is a StringBufferInputStream which looks spot on. Wait a minute - StringBufferInputStream is deprecated - apparently it doesn't properly convert characters to bytes. So rather than fix the implementation they say 'well actually, this class is rubbish - use StringReader instead'.

But I don't want a Reader, I want a Stream! So I have to write a wrapper class, which exposes the Readers read() method to the read() method of the new ReaderInputStream class. So, what would have been so hard about providing a ReaderInputStream to allow the backward conversion in the standard library?

Python Coroutines

The latest release of Python (version 2.5) has a new feature called coroutines. This post looks at what coroutines are and how to use them. Firstly a brief recap on generator functions. The following function is a generator, since it's definition contains the keyword 'yield':

def rota(people):
    _people = list(people)
    current = 0
    while len(_people):
        yield _people[current]
        current = (current + 1) % len(_people)

if __name__ == "__main__":
    people = ["Ant", "Bernard", "Carly", "Deb", "Englebert"]
    r = rota(people)
    for i in range(10):
        print "It's %s's turn." % r.next()

The generator function returns an iterable object, which returns the value given by the yield statement whenever next() is called on it. In the above example, the iterator is never exhausted - it acts like a circular linked list. Note that we create a new list of people - this allows us to pass in an arbitrary sequence or iterator as the initial argument and isolates the generator from other parts of the program changing the list. So what are coroutines? Coroutines are essentially generators which allow you to pass data back into the generator function. For example, lets say we a almost happy with our rota generator, but we would like a way of updating the internal list of people on the rota. This is where coroutines and the 'send()' function come in:

def rota(people):
    _people = list(people)
    current = 0
    while len(_people):
        command = yield _people[current]
        current = (current + 1) % len(_people)
        if command:
            comm, name = command
            if comm == "add":
                _people.append(name)
            elif comm == "remove" and name in _people:
                _people.remove(name)


def printname(name):
    print "It's %s's turn." % name

if __name__ == "__main__":
    people = ["Ant", "Bernard", "Carly", "Deb", "Englebert"]
    r = rota(people)
    
    for i in range(6):
        printname(r.next())
    
    printname(r.send(("add", "Fred")))
    
    for i in range(7):
        printname(r.next())
    
    printname(r.send(("remove","Deb")))
    
    for i in range(6):
        printname(r.next())

You can see from this example that we can use send() instead of next() to get the next value, but first the argument provided to send is given to the 'command' variable in the coroutine. In this case it expects a pair of things, a command string and a name (note that in a real world example you'd make the coroutine more robust by adding some tests on the return value from the yield). If the command is "add" the name is added to the internal list, if the command is "remove", then the name is removed from the list if it is there.

Refactoring Function Names in Python

I read an interesting article yesterday about how automating name refactoring in dynamically written languages can be done: http://dogbiscuit.org/mdub/weblog/Tech/Programming/Ruby/RubyMethodRenamed The idea is that you rename your function, and then create a method with the original name that will redirect to the new function and in addition track down the file it was called from, and rename the function call in that file. Thought I'd give it a go in Python, and so created a decorator for the job. Here it is in a basic form sans exception handling and logging.

import inspect, os, re, os.path

class renameto(object): 
 def __init__(self, new_function):
     self.new_function = new_function

 def __call__(self, function):
     self.function = function
     return self.decorator
  
 def decorator(self, *args, **kw):
     f = inspect.currentframe().f_back
     fn = f.f_code.co_filename
     lineno = f.f_lineno
  
     in_f = file(fn)
     out_l = []
  
     for i, line in enumerate(in_f):
         if i == (lineno - 1):
             line = re.sub(r"\b%s\b" % self.function.__name__,
                                         self.new_function.__name__, line)
         out_l.append(line)
  
     in_f.close()
  
     temp = fn + "~"
  
     if os.path.exists(temp):
         os.remove(temp)
      
     os.rename(fn, temp)
  
     out_f = file(fn, "w")
  
     for line in out_l:
         out_f.write(line)
     out_f.close()
  
     return self.new_function(*args, **kw)


def newf(arg):
 print "New trace Function: " + arg

@renameto(newf)
def bob(arg):
 pass

Note that what needs to be done to refactor the name is to copy the old method to the new method name, and then decorate the old method with the @renameto(newname) decorator. Optionally the body of the old method can be removed (or an exception thrown - it should never actually get called). Now run all of your tests - the refactoring should be automatically done (save for perhaps a handful of cases). This of course won't work in it's current form with functions that are assigned to a different variable. For example:

disguised_name = bob
disguised_name("Bet this one can't be changed!")

So some more work is required here to handle odd cases like this. Interesting idea though...

Shadowing and Overriding in Java and Python Inheritance.

This morning I was trying to do something in Java which would tidy up the code in some custom JSP tags I was writing. The idea was to have an abstract superclass for the tags which declared a method "getLog()". This method was to return the LOG object defined in the concrete subclass (I wanted to do some logging to the subclasses 'personal' log from a method within the superclass).

Something was nagging at me that this wasn't going to work, as I recalled that there was something different about overriding methods and variables in subclasses. I did a bit of research, and in fact you can't override variables, only methods. Variables are merely "shadowed".

What this means in practice is that subclasses may call superclass variables as if they were their own, unless they have shadowed them with their own copy of the variable (in which case you have to do "super.variable" to get the superclass version).

In Python on the other hand, you can override superclass variables in subclasses. In my opinion this behaviour is more intuitive and consistent than the Java behaviour. Here are some examples illustrating the difference between the languages:

Java example

// In file A.java
public class A {
  String x = "In class A";
    String getX(){
      return x;
  }
    public static void main(String[] args){
      B b = new B();
      System.out.println(b.getX());
  }
}
class B extends A {
  String x = "In class B";
}

Compiling and running this example will print "In class A" to the console.

Python example

# In A.py
class A(object):
  def __init__(self):
      self.x = "In class A"
  def getX(self):
      return self.x
class B(A):
  def __init__(self):
      self.x = "In class B"
b = B()
print b.getX()

Running this example will print "In class B".

It's not a huge deal, more of an irritation. If I want to do the sort of dependency inversion that I wanted, then I will just have to make all subclasses override a getLog() method in the superclass rather than using the LOG variable.

Partial Functions in Python

Recently I have found many uses for the "function that returns a function" python idiom. The basic idea is something like this:

def get_is_small(limit):

    def is_small(i):
        return i < limit
    
    return is_small

The net result is that you have a function reusable in many different situations, without having to keep track of the limit value:

>> is_quite_small = get_is_small(15)
>> is_very_small = get_is_small(3)
>> is_quite_small(6)
True
>> is_very_small(6)
False

This should all become much easier to do once python 2.5 is released, as there is a general purpose function to provide the outer function for you. I believe it will look something like:

import functional
def is_small(limit, i):
    return i < limit

is_quite_small = partial(is_small, 15)
is_very_small  = partial(is_small,3)

Update: Now that 2.5 has been released, I can confirm that it does have a partial function in the standard library - but in the module functools.

Moving to Beta

With the features promised by the new Blogger beta, and too impatient to wait for the "Switch" button, I've decided to move my blog. The original posts should appear shortly, copied over from the old blogger...

Part of the reason is that I have discovered a Vim plugin for the new Blogger which, for someone who's quite often inside Vim, will make things much easier to publish.