Python and cybersmart Caps

brad — Sun, 24 Jun 2007 10:07:17 +0200

I recently got DSL at home (after 22 working days), and signed up with a local ISP (cybersmart - you can tell them bwhittington@cybersmart.co.za referred you) because it was the cheapest 3 Gig cap available. What I also like about them is they provide a very simple web page that helps you keep track of your usage. I knew a bit of python, and I wanted to further my skills, so I decided to tap out a python script to bring my cap usage to the desktop. The result: What this re-iterated for me is that python is awesome, linux is awesome, gnome is awesome, and python is awesome. You can get python wrappers for just about everything, and more and more programs include python as an avenue to build extensions (my first non-django, python project was a rhythmbox plugin that pauses the music when your screensaver turns on). So I started out on the problem. It was a pretty straightforward issue: download the html from cybersmart, walk the DOM, grab the elements with the relevant data, typecast the data to a floating point number, and, for cream, calculate daily averages (per day usage, per day remaining). Finally, display the data. For this little project I used urllib, pyxml, pynotify (a thin wrapper around libnotify), datetime and time (you can download the full file as an attachment to this blog post). We start with the data, the cybersmart page looks like this: The relevant bits of information are the "Used for this month" and "Total left for month". Luckily these are stored in elements with class names that are relevant (although they do reuse the same class name for multiple things, instead of doing a class="used usedTotal" and a class="remaining remainingTotal", and styling the relevant classes, while getting the semantic meaning, and visual identity across). So, we download the file (sending relevant log in details as a POST), and send it to the HtmlReader class of pyxml:

print("Downloading HTML")
data = urllib.urlencode(
  {   "accountName":"account@cybersmart.co.za",
      "password":"password",
      "login":"login"})
f = urllib.urlopen(
      "http://www.cybersmart.co.za/getAccountDetails.cgi",
      data
    )
doc = f.read()
f.close()
reader= HtmlLib.Reader()

Then we parse the document, and walk the DOM tree, pulling out relevant info (sidebar: I couldn't do this more efficiently than pulling out all the td's and then checking each class entry, the getElementByClass doesn't seem to work out properly?? I also cheat because there are multiple savedTotal elements, but I am only worried about the last one):

print("Parsing Document")
dom = reader.fromString(doc)

print("Walking the tree")
td_elements = dom.getElementsByTagName("td")
remaining=''
used=''
for td in td_elements:
        temp = td.attributes.getNamedItem('class')
        if temp:
                if temp.value == "savedTotal":
                        remaining=collect_text(td)
                if temp.value == "usedTotal":
                        used=collect_text(td)

Now we have strings in our "remaining" and "used" variables which we can easily typecast to floats and process them further. I used some helper functions that I found on the net for dealing with number of days past and remaining in the month, I didn't note where they came from, so apologies to the author.

print("Caluclating averages")
daysleft = 
    mkLastOfMonth(datetime.datetime.now())-datetime.datetime.now()
daysleft = daysleft.days
remaining_float = float(remaining)
used_float = -float(used)
days = int(datetime.datetime.now().strftime("%d"))

Then, the magic...displaying the data as a notification bubble is flipping simple, you just import pynotify, initialise it with your namespace, and tell it what to display. You can get wrappers that allow you more flexibility for your notification (like images etc.), but I didn't need it.

print("Displaying data")
usage=
   "%.2f Gb used in %s days\nUsage average of %.2f Mb" % 
   (used_float,days,(used_float*1000)/days)

projected=
   "\n\n%s days left to use %s Gb\ndaily allowance of %.2f Mb" % 
   (daysleft,remaining,(remaining_float*1000)/daysleft)

content = ''.join([usage,projected])
pynotify.init("cybersmart")
n = pynotify.Notification("Cybersmart Remaining Cap",content)
n.show()

It is not the most wonderful, or extensible code ever, but it works for me, and hopefully this can help someone else do something else. I would like to find a platform agnostic way to do notifications, and I have found some code to build a platform agnostic systray icon (which could keep track of you usage daily as a graph, and pop up a detailed notification when clicked). Splitting the code into classes would allow it to become more extensible in that multiple ISPs could be catered for using different approaches, with the same display code. I am stoked with how easy it was to roll it together (POSTing, downloading, parsing, processing and display in just over 30 lines), how easy it was to hook into libnotify (the freedesktop notification lib) for display. I have upgraded from edgy to feisty (which bumped python from 2.4 to 2.5) and it wasn't bothered either. You will need to install python-xml, and python-notify. The nice thing is that this is also an appropriate area for python, since it is just message passing. Update 24/06/2007 - Jonathan pointed out BeautifulSoup as a more appropriate alternative to htmllib As a side note, I fell in love with python when I learnt it so I could program in Django. I have slowly been trying to spread my python wings beyond simply writing web apps, because it is a flipping rad language, with awesome readability, and crazy power. I suppose you could say I script in python, because python is interpreted at run time, then all the java programmers are scriptors too, so ne ner ne ner ne ner.

Lazyweb, Lazyweb: Commandline hinting

brad — Tue, 19 Jun 2007 11:53:13 +0200

The geeks out there who use commandline on a popular set of operating systems (linux) may or may not know about bash_completion. bash_completion is awesome, it adds more tabbing to your typing. You type ssh <tab> <tab> and it starts suggesting what you might want to type, like hosts, or switches. It autocompletes when it can, and it makes life very easy. To the non-geek this translates into very efficient commandlines, and that is why geeks dig the commandline. I always thought bash_completion was awesome, but lacking a critical feature. bash_completion doesn't work for programs it doesn't know about. This is logical, because, how can it. So I propose, oh lazyweb, that we agree upon a switch like --hint for all commandline apps which spits out it's help in a form that an add on like bash completion can easily use, so even if bash_completion doesn't know about it, it can find out about it as you tab. It would allow richer things like this:

yourhost$ ls -l <tab> <tab> 
       -l     use a long listing format
       Other Suggestions: ...

It would mean that everyone@opensource would need to figure out and agree upon a standard, but it would mean implementing hinting under different shells would be trivial to implement, and the problem gets farmed out to everyone who maintains their own packages, so hopefully we would see wider support. It is just a rad geek dream, really. A lot like my tab completion IRC/IM/Blogging dream (the one where you hit a <tab> <tab> <tab> <tab> and it tells your friend what you want to say, without having to type it all out).