Statistics logging for Django - part 2

In part 1 I explained how to build middleware and an associated model to capture page accesses, and tie them to a user session. Now that we have all this useful info logged we need to do something with it, like, display it. Unfortunately Django doesn't have a facility for using GROUP BY with mysql, so you have two major choices (there are more but we can ignore them): implement a custom request in a custom Manager (see snippet and snippet, or tagged snippets), or exploit a mysql view and model it in Django. Now for me I prefer the latter because it means my custom sql becomes a mysql customisation and as far as Django is concerned it is dealing with a normal table (but don't tell Django that it is read only), and thus the model code works, so subsequent queries and manipulations can exploit the ORM easily. My subjective and non-scientific experience is that using views is a lot more efficient/quick than using custom queries in the manager (it probably has to do with whatever optimisations exist with views, and the fact that you only fetch items when Django decides you need to fetch a row). So, how the hell do we do it?

First I created a model that describes what information I want to deal with (something which maps neatly on to our other model):

class UserActivity(models.Model):
        session = models.OneToOneField(Session,
                                        db_index=True, 
                                        null=True,blank=True,
                                        primary_key=True)
        user = models.ForeignKey(User,null=True,blank=True)
        date = models.DateTimeField(
                       help_text="Date Request started processing",
                       auto_now_add=True,
                       db_index=True)
        processing_time = models.IntegerField(
                       help_text="Total time spent on this user")
        requests = models.IntegerField(
                       help_text="Total Requests in this session")
        stats = UserActivityManager()
        def __str__(self):
                return '%s: %s %s - %s - %s' % (self.user,self.session,self.date,self.processing_time,self.requests)
        class Admin:
                list_display= ('user','session','date','processing_time','requests')

The nice thing about this set up is when we aggregate our activity logs we can pull out random stuff like total processing time for requests for a user/session, along with number of requests/user/session (and thus average request time)

But that is just our model, we still need the magic. To implement the magic nicely I put some custom initial SQL into the sql directory of my application (in my case the housing application for this is called accounts, so I make a file called accounts/sql/useractivity.sql), you can read more about initial data here, Django fixtures).My SQL looks like this:

DROP TABLE accounts_useractivity;
CREATE OR REPLACE VIEW accounts_useractivity AS 
SELECT i.session_id,
       i.user_id,
       MAX(i.date) as date,
       sum(i.request_time) AS processing_time, 
       count(*) AS requests 
FROM accounts_activitylog i 
GROUP BY 1 
ORDER BY NULL;

So first I tell mysql to drop the table that django just created (accounts_useractivity), and create a view in it's place. The view is very simple, in that it just GROUP BY the session_id. The real hair puller for me was figuring out that I needed to use the MAX(i.date) (see more about aggregate functions) to get the most recent access to float to the top when it normalises the data (otherwise the GROUP BY normally ORDER BY the session_id, which helps no one), the ORDER BY NULL is an optimisation to tell GROUP BY not to ORDER BY. I am hoping that because date is an INDEX (from our logging model) it shouldn't cost too much to do a MAX. (I would like someone with Much MYSQL-fu to point out any further optimisations to this, or even alternative approaches to the whole thing).

So now we have an aggregating VIEW which Django maps using it's ORM, so that to figure out sessions which have been active in the last x minutes (where x is a datetime.timedelta object) we simply do a:

UserActivity.objects.get_query_set().filter(date__gte=datetime.now()-x)

I wrote a custom manager for getting recent sessions etc., but that is an exercise for the reader. What I did include in my model is something which returns a stepped "request_weight" i.e. session requests / largest session request x steps, which in my case defaults to 6. This means I can style my users like one would a "tag cloud", so very active sessions will grow bigger than less active sessions. I needed to implement a helper function in the custom manager to return the session with the most requests.

The final tip is to use a context processor to make the information available to all your templates, although you could do it with middleware (maybe middleware is the proper way to do it?).

Bookmark/Search this post with: