I have started a new blog here: Velocity. I have done this as I prefer to write my posts in Restructured Text using Vim as my editor. It is a statically generated site powered by Python and Pelican and hosted on github. All nice and Open Source!

Android first tips.

I've had the new android phone for a few days now, and thought I'd share some tips.

Transferring Contacts

One of the first things I found read that there was no easy way to copy my contacts from my old phone to the new one. I managed to send all of my contacts to the phone as vcards, via bluetooth. However, there seems to be no easy way of importing them into the contacts lusty.

The method I settled on was to combine all of the cards into one via a simple python script (using sl4a) and then uploading them into my gmail account. The contacts list syncs with gmail, and so job done.

Sl4a is the Scripting Languages for Android by Google, and gives you a programming interface tothe phone, without the need to develop on a PC - you can actually write and run scripts on the phone itself. The python script itself is:

import os, sys 
  
 base = "/sdcard/bluetooth" 
  
 with open (os.path.join (base,"contacts.vcf"), "w") as fh: 
  
   for f in os.listdir(base): 
     if not f.endswith ("vcf") or f == "contacts.vcf": 
       continue 
  
     print >> fh, "" 
     with open (os.path.join (base, f)) as fhx: 
       print >> fh, fhx.read () 
 

Transferring files from Kubuntu

The next puzzle was how to transfer music and other files to the phone from my Kubuntu laptop. Now the phone I have is the Motorola Defy, and it seems to restrict access to through it's propriety software. The only drive that shows up when I plug in the USB cable is a small volume containing some Windows drivers. The simplest solution to this that I could find, which satisfied the criteria that the drive should show up in the File Manger as if it were a mounted drive, was to install SwiFTP, an ftp server, on the phone, expose the sdcard directory as the root dir, and then I can browse the phone, upload and download files through Dolphin as I'd wanted, without going through some clunky third party application...

Angry Birds

Don't do it - it is a massive, massive time waster. Time to finish the post now though - I have level 2 to complete.

Published with Blogger-droid v1.6.8

Cycling GPS Tools

We got a GPS unit (Garmin eTrex Venture) a year or so ago to go Geo-caching with the kids, and I also bought a cycle mounting kit for it. One of the features I've been playing with recently is the capability to upload routes - the GPS will show you which direction the next point in the route lies.
My netbook runs Fedora Linux and there are several good tools available for manipulating gps data for Linux, if only you know how to use them! The primary one I use is gpsbabel, which I believe will be packing a GUI in the next version.
In addition, I have found a handful of very useful websites for creating routes. Unfortunately, because of the multitude of different formats for gps data, it is not as straightforward as I would have liked to download and use the data from these files.

Linux Tools

Firstly a pre-requisite: on Fedora at least, things need tweaking a little to get the operating system to access the Garmin GPS device.

  echo "blacklist garmin_gps" >> /etc/modprobe.d/blacklist.conf
  echo "blacklist usbserial" >> /etc/modprobe.d/blacklist.conf

This disables the module kernel Garmin drivers, which apparently don't work very well.
Then install gpsbabel. Fedora has this in it's repositories, so can be easily installed through the Package Manager, or with yum (yum install gpsbabel)
The main commands I use with GpsBabel are to download and upload gpx files from/to the Garmin:

gpsbabel -r -t -w -i gpx -f "my_route.gpx" -o garmin -F usb:
gpsbabel -r -t -w -i garmin -f usb: -o gpx -F "my_route.gpx"

Note the flags. -r, -t, -w cause gpsbabel to process Routes, Tracks and Waypoints respectively. -i specifies the input format, -f the input file or device (usb: is the usb port), -o and -F are the same for the output format/file.
In addition, I have a collection of Python scripts that I've written to help process GPS data, including one to compare elevation profiles of multiple routes in graphical format. I may stick these up online sometime.

Websites

Bike Route Toaster
The most useful site I have come across is the Bike Route Toaster. This site allows you to create routes, add course points to give directions/notes at particular places and save them to your profile using Google Maps. If you want a route entirely on the road, you can opt for the site to auto route you between the points you have clicked. This can be switched off, and the satellite view taken advantage of for creating off-road routes.
The site fetches elevation data for the route, and lets you view data on the route, such as total distance, total height gain and a graph showing the elevation profile of the route. You can also download the route in gpx (track points only), mbcrs (track and course points) and Google Earth files.
Ordance Survey Mapping and GPXLoader
GPXLoader, an ingenious bookmarklet turns any website using the Ordance Survey Map API into something similar to the Bike Route Toaster. You can create, upload and download routes in gpx format.
I find this most useful for verifying a route from the Bike Route Toaster against a proper OS map.

El Chorro

Had a long weekend out with the lads climbing in El Chorro. It rained on the two middle days, so we didn't do all that we'd hoped for, but still got some great climbing in at Vallee d' Abdjelas and on the Arabian Steps. Amtrax, a long multi-pitch 6a was on the radar, but will have to wait for another trip, as it rained the day we'd hoped to do it.
On one of the rainy days we did the Caminito del Rey, a walkway built for the King in the 1920's, and now in rather a shaky state. Video clips here:
Part 1 Part 2 Part 3

Developing on Windows - Replacing the basic tools.

As anyone who has to work with Windows as their development environment knows, the tools Windows comes with are crap. Notepad, Windows Explorer and the Command prompt to name a few. Tools such as these are vital to programming productivity, and are simply substandard in all current versions of Windows. As far as it goes, Explorer is fine for the average Windows user, but is awkward if you need to keep track of different folders, and transfer files between them. In addition, there is no easy way of customising Explorer without hacking the registry. Notepad isn't fit even for the average user - particularly since it is the default text editor. It can't handle anything other than ASCII, and won't even automatically convert between different line endings. The Command prompt is pretty inexcusable. A tool only used by programmers and sys admins, you would have thought that the console program in Windows might have been a little more developed by now, but seems stuck in the dark ages, and years behind the unix console.

Luckily there are some free, or at least very cheap alternatives for all of the above tools.

Xplorer2 - Explorer on steroids.

An Explorer replacement that I've been using for years, and remains one of the few development tools I've paid money for (there being no free or open source equivalent with the same sort of power) is [Xplorer2] [http://zabkat.com/]. This tool is superb - featuring tabs, dual pane, scrap folders, user defined commands, user defined toolbars, a very powerful search facility etc etc.

The tabs are great - you can keep a tab open for each of your most visited folders. And they are nested within dual panes if you wish, so you can keep different sets of tabs open for each pane.

Dual panes not only make moving files about much quicker and easier than the traditional Explorer tree view, but they have a Mirror browsing mode for comparing trees of folders with different roots. For example, you have trunk and a branch checked out from your source repository. If you have trunk open in the left view, and the branch in the right, with Mirror browsing switched on, moving through the source tree in one pane will be mirrored in the other. With the built in folder sync tool, you can check for differences between the folders based on a flexible selection of parameters (date, content etc)

Scrap panes can be used to collect together files from various folders into a virtual folder. For example the 'Browse Flat' menu item puts all files in a folder subtree into one virtual folder. These files can be operated on as if this were an actual folder, compared via a dual pane view. Useful for many purposes, including keeping sets of disparate but related files together, as extended bookmarks etc.

User defined functions can be used to great effect. One of my favorites is integrating WinMerge into Xplorer2, by defining a pair of commands, the first of which calls WinMerge on the currently open directories in left and right panes; the second calls WinMerge on the currently selected left and right files. So, using mirror browse on my trunk and branch, I select the file in trunk I want to compare (the branch file if automatically selected correspondingly), hit CTRL-1 and WinMerge opens comparing those two files.

Not free, but well worth the $30 fee for the professional version (especially in the UK with the pound as strong as it is at the moment!). There's also a lite version for home and academic use.

GVim, Cream

Actually, just about any open source editor is better than Notepad. For programming though, certain things are indispensable: syntax highlighting; correct handling of line endings; smart indentation handling etc. I prefer [GVim] [http://www.vim.org/] these days - several years editing stuff on unix servers has taught me the power of vim (through a painful and long induction...). Not an editor for novices, it requires a lot of practice and a different way of thinking than your usual text editor, but is very powerful, especially since it can be programmed with my language de jour, Python.

For the more faint of heart, [Cream] [http://cream.sourceforge.net/] is GVim, but configured for ease of use. By default it works like a normal text editor, but has all of the power of vim working in the background and available for use in advanced mode.

Apart from the features already described, I use the following Vim features (in no particular order) on a regular basis:

1) Auto-complete 2) Marks - mark lines with a letter, and jump between these. 3) Macros - record macros to perform repetitive tasks. 4) Block select - one of my favorites, this lets you for example insert the same text in the middle of a block of 7 lines. I use this frequently for tasks such as editing repetitive code and property file editing. 5) Python scripting - for example sort the lines in the file into alphabetical order: :py lines = vim.current.buffer[:]; lines.sort(); vim.current.buffer[:] = lines 6) Spelling checker 7) Global command - match lines using a combination of a line range and pattern, and then do stuff to those lines. :2,6g/^#/dd deletes all lines between 2 and 6 that start with a # 8) Folding - hide blocks of code from view. Useful to get a higher level view of source code.

There are other editors than Vim with features like these, Emacs and JEdit come to mind, but for me Vim is the winner since it is small, powerful and ubiquitous on unix/linux based machines where I spend quite a bit of time.

Console and Cygwin - a better Command shell.

I've just discovered the delights of [Console] [http://sourceforge.net/projects/console/]. It's a tabbed console for windows which allows you to pre-define tab definitions to open up specific configurations. For example, I have tabs set up for bash, Python, Jython, Beanshell etc. Very nice, and in addition circumvents the problems the standard Windows command interface has with selection and the fact that starting a selection pauses the process currently running (I find this incredibly irritating - though I know others find it a feature. You can get the same behaviour in Console, or in Windows command console using the pause key on the keyboard.) In addition, for those into such frivolities, there are lots of special effect to make your tabs transparent, have wallpaper etc.

It brings to windows some of the convenience of programs such as Konsole on Linux, and in combination with [Cygwin] [http://cygwin.com/][1], you can even use the bash shell and gnu tools with all of their power. Actually things may have improved with the introduction of [PowerShell] [http://www.microsoft.com/windowsserver2003/technologies/management/powershell/default.mspx], but I haven't had the time to experiment with it yet.

Swing Testing with Jython and Jemmy Part 1 - Setting up the system

Introduction

Testing is an important part of the software development process, and various types of testing need to be combined in order to make sure a product is sufficiently tested. A full set of unit tests goes a long way toward making a stable product, and brings with it a safety net to catch errors made when refactoring the code base. Unit tests a far from the whole story however; they are by nature highly localised, and so can't test the overall behaviour of a system. Functional tests go a step further in this regard than unit tests, wiring together various different components and checking that they work together correctly. At a higher level still are system tests, which should test the system as a whole, and that you can use the software to achieve what it is designed to do. Orthogonal to system tests, but at the same sort of level are acceptance tests, tests designed to see whether the product is suitable for the customers to use. Acceptance tests need to check the more superficial (but nevertheless important) aspects of the product. Does the product meet usability targets? Can the user cause crash the system by doing things wrong?

This post is intended to address the latter two types of testing: automating system and acceptance tests on the Java GUI. The basics of the technique requires two open source libraries: Jemmy, a GUI automation tool from the Netbeans team, and Jython, an implementation of Python written in Java.

Setting up the System

1. Create a system-test folder, with tests and lib subdirectories. The lib subdirectory should further have java and python child directories. The java directory is where the jemmy and jython jars will reside, lib/python is where the python standard library will live.
2. Download Jemmy and Jython. If Jython hasn't reached 2.2 final yet, 2.2a1 is very stable.
3. Install Jython, and then copy jython.jar to lib/java and the contents of the Lib directory from within the installation directory to lib/python.
4. Copy jemmy.jar into lib/java

You will now need to create a script to run your tests. Typically there will already be a script that is used to run the application that you want to test, and a copy of this script can be adapted to the job. In the case that the application is run by running the jar directly, you will need to examine the manifest in order to extract the classpath and main class for the script. The script will look something like the following:

#!/bin/bash
# file: run.sh
APP_DIR=/my/app
TEST_DIR=$APP_DIR/system-test

APP_LIB=$APP_DIR/lib
JAVA_LIB=$TEST_DIR/lib/java
PYTHON_LIB=$TEST_DIR/lib/python

CLASSPATH=$CLASSPATH\;$APP_LIB/anImportantLib.jar
CLASSPATH=$CLASSPATH\;$APP_LIB/anotherLib.jar

CLASSPATH=$CLASSPATH\;$JAVA_LIB/jemmy.jar
CLASSPATH=$CLASSPATH\;$JAVA_LIB/jython.jar

PYTHON_PATH=-Dpython.path="$PYTHON_LIB;./scripts;$TEST_DIR/tests;"

$JAVA_HOME/bin/java -cp $CLASSPATH $PYTHON_PATH org.python.util.jython $@

Note that this script will do nothing unless provided with a script name as an argument, and any arguments the script requires. As we will see later, the application itself is started within the test script. On windows, this can be converted to a batch script, or else run using Cygwin.

Writing the first script

The first script will simply start the application. Nothing more. It will go in the system-test/tests directory. Note that we start the app in a separate thread so that the running app doesn't block our tests from running!

# file: test.py
from co.uk.mycompany.myapp import MainApp
import sys

def run():
    MainApp.main(sys.argv[1:])

def main():
    startup = Thread(target=run)
    startup.start()

if __name__ == "__main__":
    main()

So, after that bit of work, we are back where we started, except that now we have to do something like the following to start the app:

./run.sh tests/test.py more args here

The next article will go into more detail about using jython and jemmy to do some actual work.

Java Readers and Streams rant.

Another bit of Java design genius is the Reader/Stream architecture. Streams are low level things dealing with streams of bytes. Readers are higher level, and make working with character arrays easier. There is a useful class called InputStreamReader which takes a Stream, and converts it into a Reader so that you can work with the data as a stream of characters. All very nice, but also all very unidirectional...

Consider this: you have an object which loads data in via a load(InputStream is) method. The data you want to load is in a String you have obtained from elsewhere. OK, so what are your options? Is there a StringInputStream? no - but there is a StringBufferInputStream which looks spot on. Wait a minute - StringBufferInputStream is deprecated - apparently it doesn't properly convert characters to bytes. So rather than fix the implementation they say 'well actually, this class is rubbish - use StringReader instead'.

But I don't want a Reader, I want a Stream! So I have to write a wrapper class, which exposes the Readers read() method to the read() method of the new ReaderInputStream class. So, what would have been so hard about providing a ReaderInputStream to allow the backward conversion in the standard library?