python

Periodic Status Update (exclaimation, exclaimation, one, exclaimation)

brad — Fri, 02 May 2008 19:59:39 +0200

Because Blogging has become a secondary industry to Having a Family, and Working, and Living In Cape Town, I figured it was time for a generic status update, for those of you who dont use Twitter, and/or live near me. It is a hodge-podge, so apologies to any random internet search victims who think this page contained what they were looking for.

We moved to Cape Town at the beginning of the year, for me to take up a position as a Web Architect at openVOICE, and for the whijos to start living the Cape Town dream. About a month after I joined OV the decision was taken to split the company into different parts, to maintain focus. I was offered a position in the still to be formed company (which is now called nvent, and for a short time was called Richtrau no. 201). So, the company split, and I started to work for the new company (April 1st). In the run up to the split I became a bit unhappy with the uncertainty involved, and I ran into the problem (which I have had for a while) of being "the Web Guy", and, while talented and handsome, I am very much keen to be part of a web team, in a web company, to see what it feels like, to have peers who worry about the same things I do, etc. So I put my CV out there, hoping to find a web firm that does python (my one true programming love after pascal). I didn´t find that, but I did find a very nice PHP web firm called Persuasion Lab. So, I interviewed with them, and decided to quit my position at OV/nvent, and take up a position in PL.
Now, I was not convinced of PL at first, on paper, since they specialised in...Online Dating. I chuckled to myself, and decided to give them an interview, since it was either going to be new material at parties, or something genuinely intriguing. They won points out the box because they are located within walking distance of our home. To be honest I was expecting red velvet, and a handful of programmers wearing black silk shirts with long, slicked back pony tails. The former was true, the latter was devastatingly not. I interviewed with the directors of the company, David Burstein (there is only one n in burstein), and Duncan Forrest. They were nice, and I enjoyed the interview (I was unusually nervous, FNAR). I did a small amount of research before I went to the interview, and was satisfied by the info that turned up (even if it is not 100%, 4 errors is better than this web page). So I was pleasantly surprised. They have the online dating game wrapped up in SA, powering most of the newspaper dating sites, including ¨Date Your Destiny¨ IOL dating (which always makes me smile when I read an IOL story). They seem to have grown, or split, out of Ogilvy Interactive (information courtesy of some retrospective facebook research), they have been going since 2000 (which is dog years in the web game), and Duncan seems to have done a lot of the development, and currently still dives into the code (which is something that makes me happy). They have a good number of employees, split over varying roles (which also makes me happy), and they also have a London office, for their British clients. I think I stand to learn a lot there, and hopefully pick up the skills and understanding I will need when I build my own empire.
We visited JHB for two weeks to go to Pippa and Vaughn´s not-a-wedding-wedding, which was frankly amazing, and, to date, is one of the most heartfelt and enjoyable weddings I have ever been to, but maybe it is just because they invited all our friends. The time we did spend with them we used productively to encourage them to get pregnant, so Fin has friends. We stayed with Pippa and Vaughn, Hannah and Mau, and saw a lot of Marc Bradley Lewis and Bokkie/Sweetie, Alison Green, caught up with old school friends Marc Patrick Lewis, and Breton VDB, among others. Unfortunately we did not see everyone I would have liked to, but JHB is a big place hey, and, with traffic, two weeks goes by quick.
In May we are going to visit Grahamstown, to attend the wedding of Mamello Wheee Thinyane, and Dr. Hannah Barbie Sparkles Slay, and I did not want to fork out for the plane trips, so Mandy said we could drive if we had a new car (I am sorry Telburt, I loved you like a man loves a car, and It Was Her Idea). I really wanted a Toyota Prius, but they are PRICY, and still the first generation of hybrids, so I am a bit cautious. Instead, after narrowing down the choice between a low mileage VW Polo Classic, and a low mileage Golf 5, we went with the Golf. The Polo Classic came with mostly the same features, but the Golf FEELS incredible. I find the fittings and interior of the Polos to be quite plasticy, and sensible. So, parked outside our house is possibly the most expensive toy we have ever bought, but shew, it´s nice. I have taken every opportunity to drive it over the long weekend (we recieved it on Wednesday, when I traded Telburt in), including an airport run for Russell and Les. It feels like heaven to drive, and features like being able to wind down the electic windows using the remote blows my mind.
We bought a new VW Golf 5
Over the past few months I have steadily been attending more Geek things, like CLUG (actually, the first CLUG talk I attended was the one given by Jonathan and Me). Attended two[1][2] Geek Dinner(s), during the second of which I did slideshow karaoke, and designed the random grape graphic (SVG here, needs gradients and blur filters) using inkscape. There are some blog posts about GeekDinner on Planet Geekdinner.
I upgraded to Ubuntu 8.04 ´Hardy Heron´, and I am in love with Linux once again (every six months Ubuntu re-ignites the spark)
I have accounts on, and I am going to write for, ILML (the brainchild of my brother Kyle, and his mates), and the still-to-be-launched TechLeader

That should cover it for now, thanks for your patience and I hope you enjoyed your stay.

Intrepid Ibex

brad — Thu, 21 Feb 2008 10:09:42 +0200

In celebration of the newly announced "Intrepid Ibex" (or Ubuntu 8.10), I once again bring your attention to my python script which generates alternative Ubuntu names. I brute forced some names starting with I and came up with these alternatives (which suggests maybe Intrepid Impala would be a nicer name for 8.10). So,

radbrad@renton:~/python/ubuntu/choose_ubuntu$ while ((1));do python namer.py |grep ^I; done;
Inspirational Insect
Insolent Imp
Inexpensive Imp
Illustrious Ichthyosaur
Ingenious Indri
Incredible Impala
Immense Inkanyamba
Insurgent Ibis
Indefatigable Insect
Inescapable Ichthyosaur
Impressive Ichthyosaur
Inquisitive Imp
Ignited Impala
Ineffable Imp
Inevitable Insect
Inquisitive Ibis
Immodest Imp
Incomparable Ibex
Incredible Ichthyosaur
Ill Insect
Impressive Ibex
Indefatigable Ichthyosaur
Icky Iceweasel

Like the kubuntu, xubuntu, gubuntu names confuse newbies and make Ubuntu difficult to market. I dig the naming scheme, just not the names that come out of it.

Also, the register points out that:

We're informed that a male ibex will try and tempt a female into sex by licking and sniffing the lass and by issuing a low scream. It's suspected that Bono responds to a similar ritual. Sometimes the female will demonstrate her interest by urinating or by ramming the male with her horns. The male then returns the favor through the flehmen response, which is a curling of the upper lip. This goes on for about 30 minutes until Bono shows up or the ibexes get it on.

The Zen of Python, by Tim Peters

brad — Wed, 05 Dec 2007 11:18:02 +0200


radbrad@Cylon:~$ python
Python 2.5.1 (r251:54863, May  2 2007, 16:56:35) 
[GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

Hardy Heron

brad — Sun, 02 Sep 2007 15:37:04 +0200

In celebration of the newly announced "Hardy Heron" version of Ubuntu (8.04) I built a script to randomly generate a name for Ubuntu releases. ~~I did not have a huge corpus of animal names, and I tried to find as many adjectives as I could find, but~~ I used the adjectives and animals from the ubuntu wiki, and here is the script in all it's glory:

import re
from random import randrange

def get_adjectives():
        f = open('adjectives','r')
        adjectives = f.readlines()
        f.close()
        return adjectives

def get_animals():
        f = open('animals','r')
        animals = f.readlines()
        f.close()
        return animals

def choose_adjective(adjectives):
        return adjectives[randrange(len(adjectives))].strip()

def choose_ubuntu(adjectives,animals):
        adj = choose_adjective(adjectives)
        letter = '^%s' % adj[0]
        letter = re.compile(letter)
        animal = [i for i in animals if letter.match(i)]
        if animal == []:
                return choose_ubuntu(adjectives,animals)
        if len(animal)>1:
                animal = animal[randrange(len(animal))].strip()
        else:
                animal = animal[0].strip()
        return [adj,animal]

adjectives = get_adjectives()
animals = get_animals()
ubuntu = choose_ubuntu(adjectives,animals)

print('%s %s' % (ubuntu[0],ubuntu[1]))

The script and the animal and adjective corpusesses (not sure of the plural of corpus) are attached in a zipfile.

P.S. I would have preferred a name like Happy Hummingbird, or something that doesn't sound so much like Hadeda (the reference to a bird does not help it's case). Surely in light of Hoary Hedgehog they should skip H entirely?

Statistics logging for Django - part 2

brad — Sun, 29 Jul 2007 21:52:25 +0200

In part 1 I explained how to build middleware and an associated model to capture page accesses, and tie them to a user session. Now that we have all this useful info logged we need to do something with it, like, display it. Unfortunately Django doesn't have a facility for using GROUP BY with mysql, so you have two major choices (there are more but we can ignore them): implement a custom request in a custom Manager (see snippet and snippet, or tagged snippets), or exploit a mysql view and model it in Django. Now for me I prefer the latter because it means my custom sql becomes a mysql customisation and as far as Django is concerned it is dealing with a normal table (but don't tell Django that it is read only), and thus the model code works, so subsequent queries and manipulations can exploit the ORM easily. My subjective and non-scientific experience is that using views is a lot more efficient/quick than using custom queries in the manager (it probably has to do with whatever optimisations exist with views, and the fact that you only fetch items when Django decides you need to fetch a row). So, how the hell do we do it?

First I created a model that describes what information I want to deal with (something which maps neatly on to our other model):

class UserActivity(models.Model):
        session = models.OneToOneField(Session,
                                        db_index=True, 
                                        null=True,blank=True,
                                        primary_key=True)
        user = models.ForeignKey(User,null=True,blank=True)
        date = models.DateTimeField(
                       help_text="Date Request started processing",
                       auto_now_add=True,
                       db_index=True)
        processing_time = models.IntegerField(
                       help_text="Total time spent on this user")
        requests = models.IntegerField(
                       help_text="Total Requests in this session")
        stats = UserActivityManager()
        def __str__(self):
                return '%s: %s %s - %s - %s' % (self.user,self.session,self.date,self.processing_time,self.requests)
        class Admin:
                list_display= ('user','session','date','processing_time','requests')

The nice thing about this set up is when we aggregate our activity logs we can pull out random stuff like total processing time for requests for a user/session, along with number of requests/user/session (and thus average request time)

But that is just our model, we still need the magic. To implement the magic nicely I put some custom initial SQL into the sql directory of my application (in my case the housing application for this is called accounts, so I make a file called accounts/sql/useractivity.sql), you can read more about initial data here, Django fixtures).My SQL looks like this:

DROP TABLE accounts_useractivity;
CREATE OR REPLACE VIEW accounts_useractivity AS 
SELECT i.session_id,
       i.user_id,
       MAX(i.date) as date,
       sum(i.request_time) AS processing_time, 
       count(*) AS requests 
FROM accounts_activitylog i 
GROUP BY 1 
ORDER BY NULL;

So first I tell mysql to drop the table that django just created (accounts_useractivity), and create a view in it's place. The view is very simple, in that it just GROUP BY the session_id. The real hair puller for me was figuring out that I needed to use the MAX(i.date) (see more about aggregate functions) to get the most recent access to float to the top when it normalises the data (otherwise the GROUP BY normally ORDER BY the session_id, which helps no one), the ORDER BY NULL is an optimisation to tell GROUP BY not to ORDER BY. I am hoping that because date is an INDEX (from our logging model) it shouldn't cost too much to do a MAX. (I would like someone with Much MYSQL-fu to point out any further optimisations to this, or even alternative approaches to the whole thing).

So now we have an aggregating VIEW which Django maps using it's ORM, so that to figure out sessions which have been active in the last x minutes (where x is a datetime.timedelta object) we simply do a:

UserActivity.objects.get_query_set().filter(date__gte=datetime.now()-x)

I wrote a custom manager for getting recent sessions etc., but that is an exercise for the reader. What I did include in my model is something which returns a stepped "request_weight" i.e. session requests / largest session request x steps, which in my case defaults to 6. This means I can style my users like one would a "tag cloud", so very active sessions will grow bigger than less active sessions. I needed to implement a helper function in the custom manager to return the session with the most requests.

The final tip is to use a context processor to make the information available to all your templates, although you could do it with middleware (maybe middleware is the proper way to do it?).

Hey Rhythmbox, go pause yourself!

brad — Thu, 26 Jul 2007 16:41:49 +0200

A while back NBM wrote a script to pause Amarokk when his screensaver turned on. Since I get competitive I felt I could do it better, and do it for the media player that I love, Rhythmbox. If you are wondering what purpose this really serves, imagine you are away from your desk for a long time, you want to make sure the music is the same when you come back, and you don't want to bug any co-workers. It also serves the purpose of deepening my understanding of python, and scratches the itch of customising desktops (and feeling better than windows people). So...

I started out writing a daemonisd script like the one nbm built, which listens on the d-bus (the Linux/Freedesktop inter-process messaging/control bus), and connects to Rhythmbox on the same bus to get in control. Taking his script and changing the message passing to point to RB instead of Amarokk was trivial (but a mission due to lack of documentation) and I threw in some improvements.

Then I got itchy and figured I could write a plugin for RB and push control of the functionality to RB (where it belongs), and drop the need for a daemon script. So I opened up the artdisplay plugin for Rhythmbox (since it is a python based plugin) to get an idea of what is involved. After browsing through the plugin, and reading the Rhythmbox plugins writing guide I rolled my own.

I call my plugin dbusevents because while it pauses when the screen saver is on it could be extended to support multiple events and actions (maybe you want the volume turned up louder when you first plug in your mouse so you can party). So, a Rhythmbox plugin inhabits a directory and has atleast two parts: A descriptor file, and the plugin.

You can download the whole thing: dbusevents.tar.gz. To use it drop the extracted directory in /usr/lib/rhythmbox/plugins/

My descriptor file looks like:

[RB Plugin]
Loader=python
Module=dbusevents
IAge=1
Name=DBUS Events
Name[en_GB]=DBUS Events
Description=Connects Rhythmbox actions to DBUS events (for e.g. pausing when the screensaver is on)
Description[en_GB]=Connects Rhythmbox actions to DBUS events (for e.g. pausing when the screensaver is on)
Authors=Bradley Whittington 
Copyright=Copyright © 2007 Bradley Whittington
Website=http://www.rhythmbox.org

Pretty straightforward stuff eh? The net result is:

The meat of the plugin is contained in dbusevents/__init__.py which implements a Rhythmbox plugin, which means it inherits from rb.plugin (import rb), implements at least __init__, activate, and deactivate.

So:

class DbusEventsPlugin (rb.Plugin):
#Initialisation 
        def __init__ (self):
                rb.Plugin.__init__ (self)

        def activate (self, shell):
                self.shell = shell
                self.rb = shell.get_player ()
                self.session_bus = dbus.SessionBus()
                self.load_events()
                self.connect_events()

        def deactivate (self, shell):
                self.disconnect_events()
                self.save_events()
                self.shell = None
                self.rb = None
                self.session_bus = None

When my plugin is told to activate it assigns self.rb to an instance of the Rhythmbox shell, makes a connection to the D-BUS session bus, and assigns the instance to self.session_bus, then it calls load_events (which could load events and actions from a config file), and finally connect_events which tells the session_bus which functions to call when certain events happen. The deactivate function does everything, in reverse ;). Ultimately I would like to make a configuration dialogue which allows you to have your custom event-action set up, but I havent found anything more beyond pausing when the screensaver is on.

The magic in connect_events and disconnect_events is pretty basic, at the moment it walks a dictionary with relevant information:

        events = {
                        'Screensaver Play/Pause':{
                                '__doc__':'Pauses Rhythmbox when the Screensaver is running',
                                'handler':'playPause',
                                'signal':('SessionIdleChanged','org.gnome.ScreenSaver'),
                                'enabled':True,
                        }
                }

The functions use self.session_bus.add_signal_receiver and self.session_bus.remove_signal_receiver to attach events to actions. I have only defined a PlayPause function at the moment:

        def playPause(self,state):
                if not state:
                        if self.wasplaying:
                                self.rb.play()
                else:
                        self.wasplaying = self.rb.get_playing()
                        self.rb.pause()

Which checks if it is playing when the screensaver is turned on, and pauses if it is, and unpauses when the screensaver turns off, if it was paused when the screensaver went on.

I would appreciate feedback, so drop me a comment if you have any!

Statistics logging for Django

brad — Thu, 19 Jul 2007 12:05:13 +0200

Last night I built some middleware/models for a django application to log visitor/user activity on the site. The intention is to be able to do better user tracking, and build more comprehensive statistics stored in the mysql db (obviously I am also logging everything with apache). The current set up still needs some periodical scripts to conflate data into statistics. I was thinking of doing a daily-weekly-monthly routine (i.e. once a day stats are conflated for yesterday's stats, and once a week they are turned into weekly stats, and once a month they are minimised into a monthly overview. It was actually really simple to implement, but I butted my head against some django issues (more at the end).

So, first we build a model to represent a request:


class UserActivity(models.Model):
        user = models.ForeignKey(
                      User,
                      null=True, blank=True,
                      db_index=True
               )
        session = models.ForeignKey(
                      Session,
                      db_index=True,
                      null=True, blank=True
                  )
        date = models.DateTimeField(
                      help_text="Date Request started processing",
                      auto_now_add=True,
                      db_index=True)
        request_time = models.IntegerField(
                              help_text="Processing time (in ms)",
                              null=True, blank=True)
        request_url = models.CharField(maxlength=800,db_index=True)
        referer_url = models.URLField(
                              verify_exists=False,
                              db_index=True,
                              blank=True, null=True)
        client_address = models.IPAddressField(
                              blank=True,null=True)
        client_host = models.CharField(
                              maxlength=256,
                              blank=True,null=True)
        browser_info = models.TextField(null=True,blank=True)
        error = models.TextField(null=True,blank=True)
        def set_request_time(self):
                from datetime import datetime
                self.request_time = (
                                      datetime.now() - 
                                      self.date 
                                    ).microseconds
                self.save()

(download models.py_.txt)

I think the model captures all the relevant info (we tie a request to a session and user, we have the time they made the request (and using middleware we can calculate how long the request took), the referer, and some info about the client).

Most of the fields can be blank/null because we are not always going to have a session (see below), etc.

The function set_request_time is called by the outgoing middleware function (process_response) and just notes how long the request took, and saves the object.

Next we need some middleware to handle the object creation:


from datetime import datetime
from django.conf import settings
from my_app.models import UserActivity

class Activity(object):
        def process_request(self,request):
                if request.META.has_key('HTTP_REFERER'):
                        referer = request.META['HTTP_REFERER']
                else:
                        referer = ''

                self.activity = UserActivity(
                        user = request.user,
                        session = request.session,
                        date = datetime.now(),
                        request_url = request.META['PATH_INFO'],
                        referer_url = referer,
                        client_address = request.META['REMOTE_ADDR'],
                        client_host = request.META['REMOTE_HOST'],
                        browser_info = request.META['HTTP_USER_AGENT']
                )

        def process_exception(self,request,exception):
                self.activity.error = exception
                self.activity.save()

        def process_response(self,request,response):
                self.activity.set_request_time()
                return response

(download middleware.py_.txt)

You may (or may not) have noticed that we only actually save our model on the outgoing response, so we only have one db write per request. The middleware system is very easy to build for, and is documented here. The nice thing is the process_exception will keep a record of the exception (but I am not sure if this could be done so it stores more information than just the exception.__str__()?)

To install this you would need to have your model within in an app that is "installed" and "syncdb". The middleware needs to be placed after the session middleware, for e.g. in settings.py (in MIDDLEWARE_CLASSES):

    
    "django.middleware.common.CommonMiddleware",
    "django.contrib.sessions.middleware.SessionMiddleware",
    "django.contrib.auth.middleware.AuthenticationMiddleware",
    "league.middleware.Activity",

The next step is to build a context_processor that will include some useful stats like who is logged in etc. but that will need a more models, or mysql view or UserActivityManager that does a custom sql request with some "group by" magic. I have not built those parts yet, so I won't speak about them yet.

My gripes about this implementation doing regular user activity stats is a relatively costly request (you need to do a SELECT COUNT(*) WHERE date>now()-(20 minutes) GROUP BY user). This could be cheapened by having a OneToOne join table with the user table which just has an indexed recent_activity field against a User which is touched every request from that user. To get anonymous user activity we can only really rely on ip addresses, since sessions are not set until a user logs in/logs out, so we would need to do a similar system to the user OneToOne table, and use the REPLACE syntax of mysql (not sure if this is possible using django).

My gripes about the session middleware is that users do not get sessions until they log in/log out. This is good because once of visitor etc. do not get sent a cookie, and you don't allocate them a session in the DB, but it means unique sessions are more difficult to track because anonymous, first time visitors are only unique by their IP address, and nothing else. I can obviously change this, by setting any session variable for visitors without a session in the process_request of the activity middleware. This is neat because it is an opt in db hit, but after wrestling for ages with session middleware appreciating opt in is something to be done in the sober light of day.

My gripes about Django's ORM are that there is no neat way to do custom sql requests (the nicest group by sql snippet I have seen is this one because it uses django's _meta to get the table names). Newer changes in Django introduced the extra parameter, which means less completely custom sql (i.e. you can just append your customisations to the existing sql statement), but it still doesn't allow you to use very specific stuff like GROUP BY (which not all DBs support). The way to remedy this is to figure out some way you can still send sanitised sql to a db server in an extra statement, while allowing more appended customisations for developers. The alternative is to build group_by functions which either translate to DB specific requests, or do it virtually (much like the transactions infrastructure). I prefer the latter solution because I think GROUP BY is very relevant and very useful, but the latter solution does mean that if your DB doesnt support it, then it could be a very costly operation in python-space.

Python and cybersmart Caps

brad — Sun, 24 Jun 2007 10:07:17 +0200

I recently got DSL at home (after 22 working days), and signed up with a local ISP (cybersmart - you can tell them bwhittington@cybersmart.co.za referred you) because it was the cheapest 3 Gig cap available. What I also like about them is they provide a very simple web page that helps you keep track of your usage. I knew a bit of python, and I wanted to further my skills, so I decided to tap out a python script to bring my cap usage to the desktop. The result:

What this re-iterated for me is that python is awesome, linux is awesome, gnome is awesome, and python is awesome. You can get python wrappers for just about everything, and more and more programs include python as an avenue to build extensions (my first non-django, python project was a rhythmbox plugin that pauses the music when your screensaver turns on).

So I started out on the problem. It was a pretty straightforward issue: download the html from cybersmart, walk the DOM, grab the elements with the relevant data, typecast the data to a floating point number, and, for cream, calculate daily averages (per day usage, per day remaining). Finally, display the data. For this little project I used urllib, pyxml, pynotify (a thin wrapper around libnotify), datetime and time (you can download the full file as an attachment to this blog post).

We start with the data, the cybersmart page looks like this:

The relevant bits of information are the "Used for this month" and "Total left for month". Luckily these are stored in elements with class names that are relevant (although they do reuse the same class name for multiple things, instead of doing a class="used usedTotal" and a class="remaining remainingTotal", and styling the relevant classes, while getting the semantic meaning, and visual identity across).

So, we download the file (sending relevant log in details as a POST), and send it to the HtmlReader class of pyxml:

print("Downloading HTML")
data = urllib.urlencode(
  {   "accountName":"account@cybersmart.co.za",
      "password":"password",
      "login":"login"})
f = urllib.urlopen(
      "http://www.cybersmart.co.za/getAccountDetails.cgi",
      data
    )
doc = f.read()
f.close()
reader= HtmlLib.Reader()

Then we parse the document, and walk the DOM tree, pulling out relevant info (sidebar: I couldn't do this more efficiently than pulling out all the td's and then checking each class entry, the getElementByClass doesn't seem to work out properly?? I also cheat because there are multiple savedTotal elements, but I am only worried about the last one):

print("Parsing Document")
dom = reader.fromString(doc)

print("Walking the tree")
td_elements = dom.getElementsByTagName("td")
remaining=''
used=''
for td in td_elements:
        temp = td.attributes.getNamedItem('class')
        if temp:
                if temp.value == "savedTotal":
                        remaining=collect_text(td)
                if temp.value == "usedTotal":
                        used=collect_text(td)

Now we have strings in our "remaining" and "used" variables which we can easily typecast to floats and process them further. I used some helper functions that I found on the net for dealing with number of days past and remaining in the month, I didn't note where they came from, so apologies to the author.

print("Caluclating averages")
daysleft = 
    mkLastOfMonth(datetime.datetime.now())-datetime.datetime.now()
daysleft = daysleft.days
remaining_float = float(remaining)
used_float = -float(used)
days = int(datetime.datetime.now().strftime("%d"))

Then, the magic...displaying the data as a notification bubble is flipping simple, you just import pynotify, initialise it with your namespace, and tell it what to display. You can get wrappers that allow you more flexibility for your notification (like images etc.), but I didn't need it.

print("Displaying data")
usage=
   "%.2f Gb used in %s days\nUsage average of %.2f Mb" % 
   (used_float,days,(used_float*1000)/days)

projected=
   "\n\n%s days left to use %s Gb\ndaily allowance of %.2f Mb" % 
   (daysleft,remaining,(remaining_float*1000)/daysleft)

content = ''.join([usage,projected])
pynotify.init("cybersmart")
n = pynotify.Notification("Cybersmart Remaining Cap",content)
n.show()

It is not the most wonderful, or extensible code ever, but it works for me, and hopefully this can help someone else do something else. I would like to find a platform agnostic way to do notifications, and I have found some code to build a platform agnostic systray icon (which could keep track of you usage daily as a graph, and pop up a detailed notification when clicked). Splitting the code into classes would allow it to become more extensible in that multiple ISPs could be catered for using different approaches, with the same display code.

I am stoked with how easy it was to roll it together (POSTing, downloading, parsing, processing and display in just over 30 lines), how easy it was to hook into libnotify (the freedesktop notification lib) for display. I have upgraded from edgy to feisty (which bumped python from 2.4 to 2.5) and it wasn't bothered either. You will need to install python-xml, and python-notify. The nice thing is that this is also an appropriate area for python, since it is just message passing.

Update 24/06/2007 - Jonathan pointed out BeautifulSoup as a more appropriate alternative to htmllib

As a side note, I fell in love with python when I learnt it so I could program in Django. I have slowly been trying to spread my python wings beyond simply writing web apps, because it is a flipping rad language, with awesome readability, and crazy power. I suppose you could say I script in python, because python is interpreted at run time, then all the java programmers are scriptors too, so ne ner ne ner ne ner.