django

Statistics logging for Django - part 2

brad — Sun, 29 Jul 2007 21:52:25 +0200

In part 1 I explained how to build middleware and an associated model to capture page accesses, and tie them to a user session. Now that we have all this useful info logged we need to do something with it, like, display it. Unfortunately Django doesn't have a facility for using GROUP BY with mysql, so you have two major choices (there are more but we can ignore them): implement a custom request in a custom Manager (see snippet and snippet, or tagged snippets), or exploit a mysql view and model it in Django. Now for me I prefer the latter because it means my custom sql becomes a mysql customisation and as far as Django is concerned it is dealing with a normal table (but don't tell Django that it is read only), and thus the model code works, so subsequent queries and manipulations can exploit the ORM easily. My subjective and non-scientific experience is that using views is a lot more efficient/quick than using custom queries in the manager (it probably has to do with whatever optimisations exist with views, and the fact that you only fetch items when Django decides you need to fetch a row). So, how the hell do we do it?

First I created a model that describes what information I want to deal with (something which maps neatly on to our other model):

class UserActivity(models.Model):
        session = models.OneToOneField(Session,
                                        db_index=True, 
                                        null=True,blank=True,
                                        primary_key=True)
        user = models.ForeignKey(User,null=True,blank=True)
        date = models.DateTimeField(
                       help_text="Date Request started processing",
                       auto_now_add=True,
                       db_index=True)
        processing_time = models.IntegerField(
                       help_text="Total time spent on this user")
        requests = models.IntegerField(
                       help_text="Total Requests in this session")
        stats = UserActivityManager()
        def __str__(self):
                return '%s: %s %s - %s - %s' % (self.user,self.session,self.date,self.processing_time,self.requests)
        class Admin:
                list_display= ('user','session','date','processing_time','requests')

The nice thing about this set up is when we aggregate our activity logs we can pull out random stuff like total processing time for requests for a user/session, along with number of requests/user/session (and thus average request time)

But that is just our model, we still need the magic. To implement the magic nicely I put some custom initial SQL into the sql directory of my application (in my case the housing application for this is called accounts, so I make a file called accounts/sql/useractivity.sql), you can read more about initial data here, Django fixtures).My SQL looks like this:

DROP TABLE accounts_useractivity;
CREATE OR REPLACE VIEW accounts_useractivity AS 
SELECT i.session_id,
       i.user_id,
       MAX(i.date) as date,
       sum(i.request_time) AS processing_time, 
       count(*) AS requests 
FROM accounts_activitylog i 
GROUP BY 1 
ORDER BY NULL;

So first I tell mysql to drop the table that django just created (accounts_useractivity), and create a view in it's place. The view is very simple, in that it just GROUP BY the session_id. The real hair puller for me was figuring out that I needed to use the MAX(i.date) (see more about aggregate functions) to get the most recent access to float to the top when it normalises the data (otherwise the GROUP BY normally ORDER BY the session_id, which helps no one), the ORDER BY NULL is an optimisation to tell GROUP BY not to ORDER BY. I am hoping that because date is an INDEX (from our logging model) it shouldn't cost too much to do a MAX. (I would like someone with Much MYSQL-fu to point out any further optimisations to this, or even alternative approaches to the whole thing).

So now we have an aggregating VIEW which Django maps using it's ORM, so that to figure out sessions which have been active in the last x minutes (where x is a datetime.timedelta object) we simply do a:

UserActivity.objects.get_query_set().filter(date__gte=datetime.now()-x)

I wrote a custom manager for getting recent sessions etc., but that is an exercise for the reader. What I did include in my model is something which returns a stepped "request_weight" i.e. session requests / largest session request x steps, which in my case defaults to 6. This means I can style my users like one would a "tag cloud", so very active sessions will grow bigger than less active sessions. I needed to implement a helper function in the custom manager to return the session with the most requests.

The final tip is to use a context processor to make the information available to all your templates, although you could do it with middleware (maybe middleware is the proper way to do it?).

Statistics logging for Django

brad — Thu, 19 Jul 2007 12:05:13 +0200

Last night I built some middleware/models for a django application to log visitor/user activity on the site. The intention is to be able to do better user tracking, and build more comprehensive statistics stored in the mysql db (obviously I am also logging everything with apache). The current set up still needs some periodical scripts to conflate data into statistics. I was thinking of doing a daily-weekly-monthly routine (i.e. once a day stats are conflated for yesterday's stats, and once a week they are turned into weekly stats, and once a month they are minimised into a monthly overview. It was actually really simple to implement, but I butted my head against some django issues (more at the end).

So, first we build a model to represent a request:


class UserActivity(models.Model):
        user = models.ForeignKey(
                      User,
                      null=True, blank=True,
                      db_index=True
               )
        session = models.ForeignKey(
                      Session,
                      db_index=True,
                      null=True, blank=True
                  )
        date = models.DateTimeField(
                      help_text="Date Request started processing",
                      auto_now_add=True,
                      db_index=True)
        request_time = models.IntegerField(
                              help_text="Processing time (in ms)",
                              null=True, blank=True)
        request_url = models.CharField(maxlength=800,db_index=True)
        referer_url = models.URLField(
                              verify_exists=False,
                              db_index=True,
                              blank=True, null=True)
        client_address = models.IPAddressField(
                              blank=True,null=True)
        client_host = models.CharField(
                              maxlength=256,
                              blank=True,null=True)
        browser_info = models.TextField(null=True,blank=True)
        error = models.TextField(null=True,blank=True)
        def set_request_time(self):
                from datetime import datetime
                self.request_time = (
                                      datetime.now() - 
                                      self.date 
                                    ).microseconds
                self.save()

(download models.py_.txt)

I think the model captures all the relevant info (we tie a request to a session and user, we have the time they made the request (and using middleware we can calculate how long the request took), the referer, and some info about the client).

Most of the fields can be blank/null because we are not always going to have a session (see below), etc.

The function set_request_time is called by the outgoing middleware function (process_response) and just notes how long the request took, and saves the object.

Next we need some middleware to handle the object creation:


from datetime import datetime
from django.conf import settings
from my_app.models import UserActivity

class Activity(object):
        def process_request(self,request):
                if request.META.has_key('HTTP_REFERER'):
                        referer = request.META['HTTP_REFERER']
                else:
                        referer = ''

                self.activity = UserActivity(
                        user = request.user,
                        session = request.session,
                        date = datetime.now(),
                        request_url = request.META['PATH_INFO'],
                        referer_url = referer,
                        client_address = request.META['REMOTE_ADDR'],
                        client_host = request.META['REMOTE_HOST'],
                        browser_info = request.META['HTTP_USER_AGENT']
                )

        def process_exception(self,request,exception):
                self.activity.error = exception
                self.activity.save()

        def process_response(self,request,response):
                self.activity.set_request_time()
                return response

(download middleware.py_.txt)

You may (or may not) have noticed that we only actually save our model on the outgoing response, so we only have one db write per request. The middleware system is very easy to build for, and is documented here. The nice thing is the process_exception will keep a record of the exception (but I am not sure if this could be done so it stores more information than just the exception.__str__()?)

To install this you would need to have your model within in an app that is "installed" and "syncdb". The middleware needs to be placed after the session middleware, for e.g. in settings.py (in MIDDLEWARE_CLASSES):

    
    "django.middleware.common.CommonMiddleware",
    "django.contrib.sessions.middleware.SessionMiddleware",
    "django.contrib.auth.middleware.AuthenticationMiddleware",
    "league.middleware.Activity",

The next step is to build a context_processor that will include some useful stats like who is logged in etc. but that will need a more models, or mysql view or UserActivityManager that does a custom sql request with some "group by" magic. I have not built those parts yet, so I won't speak about them yet.

My gripes about this implementation doing regular user activity stats is a relatively costly request (you need to do a SELECT COUNT(*) WHERE date>now()-(20 minutes) GROUP BY user). This could be cheapened by having a OneToOne join table with the user table which just has an indexed recent_activity field against a User which is touched every request from that user. To get anonymous user activity we can only really rely on ip addresses, since sessions are not set until a user logs in/logs out, so we would need to do a similar system to the user OneToOne table, and use the REPLACE syntax of mysql (not sure if this is possible using django).

My gripes about the session middleware is that users do not get sessions until they log in/log out. This is good because once of visitor etc. do not get sent a cookie, and you don't allocate them a session in the DB, but it means unique sessions are more difficult to track because anonymous, first time visitors are only unique by their IP address, and nothing else. I can obviously change this, by setting any session variable for visitors without a session in the process_request of the activity middleware. This is neat because it is an opt in db hit, but after wrestling for ages with session middleware appreciating opt in is something to be done in the sober light of day.

My gripes about Django's ORM are that there is no neat way to do custom sql requests (the nicest group by sql snippet I have seen is this one because it uses django's _meta to get the table names). Newer changes in Django introduced the extra parameter, which means less completely custom sql (i.e. you can just append your customisations to the existing sql statement), but it still doesn't allow you to use very specific stuff like GROUP BY (which not all DBs support). The way to remedy this is to figure out some way you can still send sanitised sql to a db server in an extra statement, while allowing more appended customisations for developers. The alternative is to build group_by functions which either translate to DB specific requests, or do it virtually (much like the transactions infrastructure). I prefer the latter solution because I think GROUP BY is very relevant and very useful, but the latter solution does mean that if your DB doesnt support it, then it could be a very costly operation in python-space.

Even when you are winning (Drupal vs. Mambo, Germalism, Django, Schools, more bean)

brad — Sat, 05 Aug 2006 15:45:25 +0200

You haven't heard from me in a while because I have been pretty darn busy. So, what have I been busy with? lots...and to be a dork, I will lump the unrelated together in this monumental post:

Drupal and MamboJoomla?
I have been doing some free-lance work here and there. Lots of CMS setting up, notably, lots of Drupal. After using Mambo for a long time, and dealing with hacks here and there to get it to behave in a sane manner wears one down. I cannot vouch for if Joomla has improved the sif codebase that Mambo runs on, but I know Drupal is the closest you can come to a good multi-purpose CMS that Doesnt make your head burn when problem solving. I don't want to say that the many hours that went into making Mambo were wasted. It is very clear that Mambo started as something, grew into something else, got released as open source, and became something else, split into two competing products (Mambo/Joomla), etc. Drupal, on the other hand, is community plumbing. It started out as community plumbing, grew into better community plumbing, and has a very interactive community, built using...Drupal. If you want a clear illustration of where Mambo/Joomla fails download yourself a mambo module, and a drupal module. Open the two codebases (in parallel, or series), and notice how the joomla/module has No style, no API it is adhering to, behaving generally like a wild west php script. Notice how drupal module has an API it adheres to, notice how it can augment many different areas in the engine. Drupal starts as an engine, you plug modules into it, and it becomes a CMS, or a Blog, or a community advocacy site. Mambo/Joomla is a piece of bad ex-commercial monolithic code which heard about plugins during one of it's augmentations.

What I do know is that Drupal is incredibly easy to work with, well documented, stable, and current. It cuts my web development time in half. In fact the work I normally have to do lands in building a theme (using any one of the templating engines available), installing and configuring a few specific modules, and smiling a lot. I think there is probably enough business in just building themes for drupal. For web development...Inkscape++, Drupal++.
Germalisms
I have been lecturing some Germalisms (or Journalists) in XHTML and CSS (plus some mentions of Web 2.0.1b rc3 stuffs). I have been using our local Moodle installation to build up the course and so on. Obviously I did my homework, and being an advocate of standards where possible, I looked into using s5: A Simple, Standards-Based, Slide Show System within moodle. Good news is it is possible. What does all that jibber jabber mean? well, using an s5 plugin for moodle means I can create lecture slides on moodle, and they will run in any web browser. I don't need a proprietry product, I don't even need a fancy web browser, I can actually build the slides using my cellphone (if the urge gripped me) from anywhere in the world. Sure it doesnt have fancy effects (it can be easily augmented to have some fancy effects), but it allows me to build good looking slides quickly and easily, without ever leaving my browser. that is k-rad. Up yours proprietry-vendor-of-your-choice.

The kids are cool, but we have only had two lectures so far, what with me being in Dwesa last week, and them being in jhb this week. It is going well though. We have pet names and everything. Well, not so much, but we could. Some may take offense at me calling them Germalisms, but it is a pet name some of us have for Journalists, I am a victim of circumstances.
Djangalising
Another highlight in my life has been wrapping my keys around some Django. Django is a web application framework written in the (imho) best-language-around python. I am currently writing a custom app for the Freedom League. Mambo has been seriously deficient at this task, and I felt that building a custom app was the way to go. Django makes a lot of life easy. It does take ten minutes to do the basic core code, but it obviously takes longer to develop an application, with bells and whistles. I have been bad in not posting any code snippets etc. but I haven't had the inclination to spend the time on such things. The project is going to be open sourced as soon as possible, but i want to get it up and running, have a people use the site, improve it, get some peeps to review it, and then release it. Either way, python, and Django, have improved my outlook on webdevelopment even more. The exciting things about Django include easy scaling up, high response times (i.e. higher than any web framework in php, j2ee, etc.), easy interactions with multiple database backends, abstracted modelling code, plenty of freebies (like the admin module, comments, multiple markdown syntaxes, powerful templating). There are arguments for different web frameworks, and they are nice, but Django is a comfort zone that I like. I tried Ruby on Rails, knowing no ruby, and got pretty far, but very frustrated. I tried Django, knowing no python, and got most places I wanted to get. Django++
Schools
A significant portion of my time has been dedicated to the e-Yethu schools project. Besides the technical side of things (helping schools get things done), the project is having increasing interactions with the Department of Education in the Eastern Cape. We are starting to produce documents which are of use to the department. They have to cater for lots of schools getting computer labs by 2013, like all of them. We can, and are, presenting them with research and knowledge which has grown from our actual experience working with the schools in grahamstown. We can discuss multiple lab setups with expertise (Thin clients, Fat clients, Dual boot thick and thin clients, Windows networks managed by Open source software, school relevant open source software on linux and windows), as well as connectivity issues (Modem, DSL, Wireless (wifi and wimax), GPRS/EDGE/HDPSISASAAS). It is nice that we are producing something worthwhile, building our own experience and knowledge. Without the work that came before us we would be nowhere, and now we are trying to ensure that the work we and others are doing can be used to the benefit of lots of school kids.
Bean and Amanda are the underlying current of my life. We have a reasonably strong indication that Bean is a girl, although I think people are seeing chickens before the eggs crack. Becoming a parent brings a whole wealth of experiences and worries, but it seems like together we are getting somewhere. I cannot wait till bean is born and I take the time off to spend with Mandy and bean. The prospect of a long holiday at the end of the year, spent getting to know my spawn and all of it's crying-spitting-shitting-ness is going to be rad. It seems like everyone else is more worried about what-is-going-to-happen than me, or worrying for me. The nice thing is that there is nothing to worry about, because it will work out fine in the end. I think people will only believe me when I am old enough to embarrass bean. I am already old enough to. I will know my work is paid off when my child tells me "you are such a dork". Because, I am ;)