statistics

On Internet statistics

brad — Wed, 23 Jul 2008 10:05:35 +0200

Because I am on the internet, and because the apps I use offer statistics, I figure I would share some of them, feel free to do the same, if you feel the urge.

Google Reader

From your 157 subscriptions, over the last 30 days you read 1,698 items, starred 17 items, shared 48 items, and emailed 0 items.

Statistics logging for Django - part 2

brad — Sun, 29 Jul 2007 21:52:25 +0200

In part 1 I explained how to build middleware and an associated model to capture page accesses, and tie them to a user session. Now that we have all this useful info logged we need to do something with it, like, display it. Unfortunately Django doesn't have a facility for using GROUP BY with mysql, so you have two major choices (there are more but we can ignore them): implement a custom request in a custom Manager (see snippet and snippet, or tagged snippets), or exploit a mysql view and model it in Django. Now for me I prefer the latter because it means my custom sql becomes a mysql customisation and as far as Django is concerned it is dealing with a normal table (but don't tell Django that it is read only), and thus the model code works, so subsequent queries and manipulations can exploit the ORM easily. My subjective and non-scientific experience is that using views is a lot more efficient/quick than using custom queries in the manager (it probably has to do with whatever optimisations exist with views, and the fact that you only fetch items when Django decides you need to fetch a row). So, how the hell do we do it? First I created a model that describes what information I want to deal with (something which maps neatly on to our other model):

class UserActivity(models.Model):
        session = models.OneToOneField(Session,
                                        db_index=True, 
                                        null=True,blank=True,
                                        primary_key=True)
        user = models.ForeignKey(User,null=True,blank=True)
        date = models.DateTimeField(
                       help_text="Date Request started processing",
                       auto_now_add=True,
                       db_index=True)
        processing_time = models.IntegerField(
                       help_text="Total time spent on this user")
        requests = models.IntegerField(
                       help_text="Total Requests in this session")
        stats = UserActivityManager()
        def __str__(self):
                return '%s: %s %s - %s - %s' % (self.user,self.session,self.date,self.processing_time,self.requests)
        class Admin:
                list_display= ('user','session','date','processing_time','requests')

The nice thing about this set up is when we aggregate our activity logs we can pull out random stuff like total processing time for requests for a user/session, along with number of requests/user/session (and thus average request time) But that is just our model, we still need the magic. To implement the magic nicely I put some custom initial SQL into the sql directory of my application (in my case the housing application for this is called accounts, so I make a file called accounts/sql/useractivity.sql), you can read more about initial data here, Django fixtures).My SQL looks like this:

DROP TABLE accounts_useractivity;
CREATE OR REPLACE VIEW accounts_useractivity AS 
SELECT i.session_id,
       i.user_id,
       MAX(i.date) as date,
       sum(i.request_time) AS processing_time, 
       count(*) AS requests 
FROM accounts_activitylog i 
GROUP BY 1 
ORDER BY NULL;

So first I tell mysql to drop the table that django just created (accounts_useractivity), and create a view in it's place. The view is very simple, in that it just GROUP BY the session_id. The real hair puller for me was figuring out that I needed to use the MAX(i.date) (see more about aggregate functions) to get the most recent access to float to the top when it normalises the data (otherwise the GROUP BY normally ORDER BY the session_id, which helps no one), the ORDER BY NULL is an optimisation to tell GROUP BY not to ORDER BY. I am hoping that because date is an INDEX (from our logging model) it shouldn't cost too much to do a MAX. (I would like someone with Much MYSQL-fu to point out any further optimisations to this, or even alternative approaches to the whole thing). So now we have an aggregating VIEW which Django maps using it's ORM, so that to figure out sessions which have been active in the last x minutes (where x is a datetime.timedelta object) we simply do a:

UserActivity.objects.get_query_set().filter(date__gte=datetime.now()-x)

I wrote a custom manager for getting recent sessions etc., but that is an exercise for the reader. What I did include in my model is something which returns a stepped "request_weight" i.e. session requests / largest session request x steps, which in my case defaults to 6. This means I can style my users like one would a "tag cloud", so very active sessions will grow bigger than less active sessions. I needed to implement a helper function in the custom manager to return the session with the most requests. The final tip is to use a context processor to make the information available to all your templates, although you could do it with middleware (maybe middleware is the proper way to do it?).

Statistics logging for Django

brad — Thu, 19 Jul 2007 12:05:13 +0200

Last night I built some middleware/models for a django application to log visitor/user activity on the site. The intention is to be able to do better user tracking, and build more comprehensive statistics stored in the mysql db (obviously I am also logging everything with apache). The current set up still needs some periodical scripts to conflate data into statistics. I was thinking of doing a daily-weekly-monthly routine (i.e. once a day stats are conflated for yesterday's stats, and once a week they are turned into weekly stats, and once a month they are minimised into a monthly overview. It was actually really simple to implement, but I butted my head against some django issues (more at the end). So, first we build a model to represent a request:


class UserActivity(models.Model):
        user = models.ForeignKey(
                      User,
                      null=True, blank=True,
                      db_index=True
               )
        session = models.ForeignKey(
                      Session,
                      db_index=True,
                      null=True, blank=True
                  )
        date = models.DateTimeField(
                      help_text="Date Request started processing",
                      auto_now_add=True,
                      db_index=True)
        request_time = models.IntegerField(
                              help_text="Processing time (in ms)",
                              null=True, blank=True)
        request_url = models.CharField(maxlength=800,db_index=True)
        referer_url = models.URLField(
                              verify_exists=False,
                              db_index=True,
                              blank=True, null=True)
        client_address = models.IPAddressField(
                              blank=True,null=True)
        client_host = models.CharField(
                              maxlength=256,
                              blank=True,null=True)
        browser_info = models.TextField(null=True,blank=True)
        error = models.TextField(null=True,blank=True)
        def set_request_time(self):
                from datetime import datetime
                self.request_time = (
                                      datetime.now() - 
                                      self.date 
                                    ).microseconds
                self.save()

(download models.py_.txt) I think the model captures all the relevant info (we tie a request to a session and user, we have the time they made the request (and using middleware we can calculate how long the request took), the referer, and some info about the client). Most of the fields can be blank/null because we are not always going to have a session (see below), etc. The function set_request_time is called by the outgoing middleware function (process_response) and just notes how long the request took, and saves the object. Next we need some middleware to handle the object creation:


from datetime import datetime
from django.conf import settings
from my_app.models import UserActivity

class Activity(object):
        def process_request(self,request):
                if request.META.has_key('HTTP_REFERER'):
                        referer = request.META['HTTP_REFERER']
                else:
                        referer = ''

                self.activity = UserActivity(
                        user = request.user,
                        session = request.session,
                        date = datetime.now(),
                        request_url = request.META['PATH_INFO'],
                        referer_url = referer,
                        client_address = request.META['REMOTE_ADDR'],
                        client_host = request.META['REMOTE_HOST'],
                        browser_info = request.META['HTTP_USER_AGENT']
                )

        def process_exception(self,request,exception):
                self.activity.error = exception
                self.activity.save()

        def process_response(self,request,response):
                self.activity.set_request_time()
                return response

(download middleware.py_.txt) You may (or may not) have noticed that we only actually save our model on the outgoing response, so we only have one db write per request. The middleware system is very easy to build for, and is documented here. The nice thing is the process_exception will keep a record of the exception (but I am not sure if this could be done so it stores more information than just the exception.__str__()?) To install this you would need to have your model within in an app that is "installed" and "syncdb". The middleware needs to be placed after the session middleware, for e.g. in settings.py (in MIDDLEWARE_CLASSES):

    
    "django.middleware.common.CommonMiddleware",
    "django.contrib.sessions.middleware.SessionMiddleware",
    "django.contrib.auth.middleware.AuthenticationMiddleware",
    "league.middleware.Activity",

The next step is to build a context_processor that will include some useful stats like who is logged in etc. but that will need a more models, or mysql view or UserActivityManager that does a custom sql request with some "group by" magic. I have not built those parts yet, so I won't speak about them yet. My gripes about this implementation doing regular user activity stats is a relatively costly request (you need to do a SELECT COUNT(*) WHERE date>now()-(20 minutes) GROUP BY user). This could be cheapened by having a OneToOne join table with the user table which just has an indexed recent_activity field against a User which is touched every request from that user. To get anonymous user activity we can only really rely on ip addresses, since sessions are not set until a user logs in/logs out, so we would need to do a similar system to the user OneToOne table, and use the REPLACE syntax of mysql (not sure if this is possible using django). My gripes about the session middleware is that users do not get sessions until they log in/log out. This is good because once of visitor etc. do not get sent a cookie, and you don't allocate them a session in the DB, but it means unique sessions are more difficult to track because anonymous, first time visitors are only unique by their IP address, and nothing else. I can obviously change this, by setting any session variable for visitors without a session in the process_request of the activity middleware. This is neat because it is an opt in db hit, but after wrestling for ages with session middleware appreciating opt in is something to be done in the sober light of day. My gripes about Django's ORM are that there is no neat way to do custom sql requests (the nicest group by sql snippet I have seen is this one because it uses django's _meta to get the table names). Newer changes in Django introduced the extra parameter, which means less completely custom sql (i.e. you can just append your customisations to the existing sql statement), but it still doesn't allow you to use very specific stuff like GROUP BY (which not all DBs support). The way to remedy this is to figure out some way you can still send sanitised sql to a db server in an extra statement, while allowing more appended customisations for developers. The alternative is to build group_by functions which either translate to DB specific requests, or do it virtually (much like the transactions infrastructure). I prefer the latter solution because I think GROUP BY is very relevant and very useful, but the latter solution does mean that if your DB doesnt support it, then it could be a very costly operation in python-space.

statistics

On Internet statistics

Google Reader

Last.fm

Twitter

Statistics logging for Django - part 2

Statistics logging for Django