Workload Allocation Modelling Update - Scalability

I have been doing some more work on my software to handle Academic Workload Modelling, developing a roadmap for two future versions, one being modifications needed to run real allocations for next year without scrapping existing data, and another being code to handle the moderation of exams and coursework (which isn't really anything to do with workload modelling, there's some more mission creep going on).

Improvements to Task Handling

Speaking of mission creep I noted in the last article I'd added some code to capture tasks that staff members would be reminded off and could self-certify as complete. I improved this a lot with more rich detail about when tasks were overdue and UI improvements.

I wanted to automate some batch code to send emails from the system periodically. I discovered that using a Django management command provided an elegant way to the batch mode code into the project that could be called with cron through the usual Django manage.py script that it creates to handle its own internal related tasks for the project from the command line.

#
# Regular cron jobs for the wam package
#
#  m h  dom mon dow user command
# Every week
#
# Each Monday at 7 am, send all reminders
0 7 * * 1 root /usr/bin/python3 /usr/local/share/wam/manage.py email_reminders
# Each Wednesday and Friday at 7 am, send reminders for overdue and less than 7 days to deadline tasks
0 7 * * 3,5 root /usr/bin/python3 /usr/local/share/wam/manage.py email_reminders --urgent-only

It was easy to use this framework to add command switches and configuration of verbosity (you might note I haven't disabled all output at the moment so I can monitor execution at this stage). I have set this up to email folks on a Monday morning with all the tasks, but also on Wednesday and Friday if there are urgent tasks still outstanding (less than a week to deadline).

I've been using this functionality live and it has worked very well. I used Django templates to help provide the email bodies, both in HTML and plain text.

Sample Task Reminder Email

Sample Task Reminder Email

Issues of Scale

My early prototype handled data for one academic year, albeit with fields in the schema to try and solve this at a later stage. It also suffered from a problem in that if other Schools wanted to use the system, how would I disaggregate the data both for security and convenience?

In the end I hit upon a solution for both issues, a WorkPackage model that allows a range of dates (usually one academic year) and a collection of Django User Groups to be specified. This allows all manually allocated activities, and module data to be specified with a package and therefore both invisible to other packages (users in other Schools, or in other Academic Years). I was also able to put the constants I'm using to model workload into the Django model, making it easier to tweak year on year.

class WorkPackage(models.Model):
    '''Groups workload by user groups and time
    
    A WorkPackage can represent all the users and the time period
    for which activities are relevant. A most usual application
    would be to group activities by School and Academic Year.
    
    name        the name of the package, probably the academic unit
    details     any further details of the package
    startdate   the first date of activities related to the package
    enddate     the end date of the activities related to the package
    draft       indicates the package is still being constructed
    archive     indicates the package is maintained for record only
    groups      a collection of all django groups affected
    created     when the package was created
    modified    when the package was last modified
    nominal_hours
                the considered normal number of load hours in a year
    credit_contact_scaling
                multiplier from credit points to contact hours
    contact_admin_scaling
                multiplier from contact hours to admin hours
    contact_assessment_scaling
                multiplier from contact hours to assessment hours
    
    '''
    
    name = models.CharField(max_length = 100)
    details = models.TextField()
    startdate = models.DateField()
    enddate = models.DateField()
    draft = models.BooleanField(default=True)
    archive = models.BooleanField(default=False)
    groups = models.ManyToManyField(Group, blank=True)
    nominal_hours = models.PositiveIntegerField(default=1600)
    credit_contact_scaling = models.FloatField(default=8/20)
    contact_admin_scaling = models.FloatField(default=1)
    contact_assessment_scaling = models.FloatField(default=1)
    created = models.DateTimeField(auto_now_add = True)
    modified = models.DateTimeField(auto_now = True)
    
    def __str__(self):
        return self.name + ' (' + str(self.startdate) + ' - ' + str(self.enddate) + ')'
    
    class Meta:
        ordering = ['name', '-startdate']

I'm pretty much ready to use the system for a real allocation now without having to purge the test data I used this this year. I can simply create a new WorkPackage.

I need to write some functionality to allow one package's allocations to be automatically rolled over to the next as a starting point, but I reckon that's maybe two or three more hours.

Future Plans for the Application

The next part of planned functionality is an ability to handle coursework and examination and the moderation process. It will be quite a big chunk of new functionality and moving the system again to something quite a bit bigger than just a workload allocation system.

This of course means I need a better Application name, (WAM isn't so awesome anyway). Suggestions on a post card.

Django Issues

I think I'm getting more to grips with Django all the time - although I often have the nagging feeling I'm writing several lines of code that would be simpler if I had a better feel for its syntax for dealing with QuerySets.

The big problem I hit, again, was issues in migrations. I created and executed migrations on my (SQLite) development system, but when I moved these over to production (MySQL) it barfed spectacularly.

Once again the lack of idempotent execution means you have to work out what part of the migration worked and then tag the migration as "faked" in order to move onto the next. This was sufficient this time, and I didn't have to write custom migrations like last time, but it's really not very reassuring.

Further Details

As before, the code is on GitHub, and the development website on foss.ulster.ac.uk, if you want more details.

Follow me!

Leave a Reply

Your email address will not be published. Required fields are marked *