Assessment handling and Assessment Workflow in WAM

Sometime ago I began writing a Workload Allocation Modeller aimed at Higher Education, and I've written some previous blog articles about this.

As is often the way, the scope of the project broadened and I found myself writing in support for handling assessments and the QA processes around them. At some point this necessitates a new name for WAM to something more general (answers on a post card please) but for now, development continues.

Last year I added features to allow Exams, Coursework, and their Moderation and QA documents to be uploaded to WAM. This was generally reasonably successful, but a bit clunky. We gave several External Examiners access to the system and they were able to look in at the modules for which they were an examiner and the feedback was pretty good.

What Worked

One of the things that worked best about last year's experiment was that we put in information about the Programmes (Courses) each Module was on. It's not at all unusual for many Programmes to have the same Module within them.

This can cause a headache for External Examination since an External Examiner is normally assigned to a Programme. In short, the same Module can end up being looked at by several Examiners. While this is OK, it can be wasteful of work, and creates potential problems when two Examiners have a different perspective on the Module.

So within WAM, I put in code an assumption of what we should be doing in paper based systems - that every Module should have a "Lead Programme". The examiner for that Programme should be the one that has primacy, and furthermore, where they are presented other Modules on the Programme for which they aren't the "lead" Examiner, they should know that this is for information, and they may not be required to delve into it in so much detail - unless they choose to.

This aspect worked well, and the External Examiners have a landing screen that shows which Modules they are examining, and which they are the lead Examiner.

What Didn't Work

I had written code that was intended to look at what assessment artefacts had been uploaded since a last user's login, and email them the relevant stuff.

This turned out to be problematic, partly because one had to unpick who should get what, but mostly because I'm using remote authentication with Django (the Python framework in which WAM is written), and it seems that the last login time isn't always updated properly when you aren't using Django's built in authentication.

But the biggest problem was a lack of any workflow. This was a bit deliberate since I didn't want to hardcode my School or Faculty's workflow.

You should never design your software product for HE around your own University too tightly. Because your own University will be a different University in two years' time.

So, I wanted to ponder this a bit. It made visibility of what was going on a little difficult. It looked a bit like this (not exactly, as this is a screenshot from a newer version of an older module):

with items shown from oldest at the bottom to newest at the top. You can kind of infer the workflow state by the top item, and indeed, I used that in the module list.

But staff uploaded files they wanted to delete (and that was previously disallowed for audit reasons) and the workflow wasn't too clear and that made notifications more difficult.

What's New

So, in a beta version of 2.0 of the software I have implemented a workflow model. I did this by:

• defining a model that represented the potential states a Module could be in, each state defines who can trigger it, and what can happen next, and who should be notified;
• defining a model that shows a "sign off" event.

Once it became possible to issue a "sign off" of where we were in the workflow, a lot of things became easier. This screenshot shows how it looks now.

Ok, it's a bit of a dumb example, since I'm the only user triggering states here (and I can only do that in some cases since I'm a Superuser, otherwise some states can only be triggered by the correct stakeholder - the moderator of examiner).

However, you can see that now we can still have all the assessment resources, but with sign offs at various stages. The sign off could (and likely would) have much more detailed notes in a real implementation.

This in turn has made notification emails much easier to create. Here is the email triggered by the final sign off above.

The detailed notes aren't shown in the email, in case other eyes are on it and there are sensitive comments.

All of this code is available at GitHub. It's working now, but I'm probably do a few more bits before an official 2.0 release.

I will be demoing the system at the Royal Academy of Engineering in London next Monday, although that will focus entirely on WAM's workload features.