Skip to content.
Personal tools
You are here: Home » Features » PyAssess Project: a toolkit for QTI2 migration

PyAssess Project: a toolkit for QTI2 migration

Steve Lay
Last modified 27 Apr, 2007
Published 14 Mar, 2007
Steve Lay talks about the PyAssess project in the e-assessment domain which aimed to create an IMS QTI v2 compliant assessment system using Python and a means of migrating content between QTI versions 1 and 2. In the article Steve details the language incompatibilities that arose during the project and suggests that web services may provide the answer allowing: "better defined web services may help applications to break out of their language community silos".


The aim of the PyAssess project was to further develop existing work in the areas of Assessment and Marking by building on the then newly released IMS QTI version 2 specification. We wanted to develop a toolkit that would enable developers to quickly add basic assessment functionality to their own java or python based systems. In addition we hoped the PyAssess toolkit could be used to build demonstrators and inform the development of emerging service protocols, “particularly in the area of external response processing systems such as those providing automated marking of free-text responses or human marking services.” [1]

Question and Test Interoperability specification

The origins of the PyAssess project can be traced to the work of the IMS QTI project team. QTI remains the only "version 2" specification published by IMS. The version number change represented a major step forward for the specification but, critically, one that was incompatible with version 1 at the XML binding level.

In 2003, Version 1 of QTI was already widely implemented but interoperability remained elusive because there were often several different ways to represent the same information. This led to different conventions in different tools. An important goal for version 2 of the specification project was to reconcile these differences. One way to test that this was successful was to write a script that mapped existing version 1 content into the new version 2 specification. I tackled this problem by developing a python script that could read in QTI version 1 XML files and output an IMS content package containing the converted items marked up with QTI version 2.[2]

Once the specification was complete the code for the migration script had served its purpose but I was reluctant to let it gather dust. The PyAssess project was originally conceived as a way of ensuring that the code would be published and made available to the community and could serve as a platform for developing further python applications that support QTI. PyAssess was a small project by JISC toolkit standards. This was largely due to the limited resource available to work on it within the ICT Group at Cambridge Assessment where the work was carried out.

Language level integration

The use of python for the original migration script was guided by there being a good match between the capabilities of the language and python libraries and the requirements of this application. The Python interpreter is also open source and binary distributions are available for most major platforms so people should be able to use the migration script without having to be in an expert in building tools from source code. In fact, Python itself is accessible to other developers as demonstrated by Pierre Gorissen (now co-chair of the QTI project team) who modified the script to provide a Windows compatible binary distribution with a simple graphical interface without any prior knowledge of python at all.

Unfortunately, the use of python was a bad match for the JISC toolkit project requirements which emphasized the need for implementations in widely used languages such as Java or .Net. To address this issue the project undertook to investigate whether Jython could be used to make the python classes that implemented QTI available to java programmers. Jython is an implementation of the Python interpreter in java, done in such a way that java classes can call the methods of python-defined classes.

Our experiments with Jython ran into problems when we hit differences between the low-level representation of python string objects in Jython and the standard python interpreter. Quickly finding these problems was critical to the problem and was a by product of our strategy of writing comprehensive unit tests to constantly check for consistent operation between the two environments.

Although Jython is actively under development it is still having to play catch up with the main python interpreter code. Options for integrating python source into other language environments are even more esoteric. Perhaps the quest for convergence is best summarized by the Parrot Code project which is attempting to create a virtual machine that runs code written in a range of languages including python and perl. The "Parrot" project started as an April fools joke claiming that the python and perl communities had settled their differences and were prepared to work together on a joint project. The conclusion I draw from this is that useful integration of code written by different programming language communities is unlikely to be achieved in an application domain like e-learning any time soon.

Web service solutions

With no foreseeable way to integrate code at the language level integration is most likely to happen when we use web services. Crossing the language divide doesn't seem to be a key driver for the uptake of web services. Almost by definition they are designed to allow applications on different servers to communicate. However, better defined web services may help applications to break out of their language community silos and allow "best in class" systems to work together in the way described by Patrick Masson [3]

In the assessment domain, the QTI specification offers a starting point for the breakdown of assessment into a set of co-operating components. The diagram below is derived from the component diagram in the specification itself but has been coloured to indicate the areas (in orange) that are currently being funded through capital programme projects started in March 2007.

New JISC capital projects in the QT1 2 space

New JISC capital projects (orange)in the QT1 2 space (adapted by Gary Wills)

More recently, the delivery engine has been further broken down into services by the R2Q2 project [4] building on earlier work done by the APIS family of projects. One particular part of the delivery engine of interest in the PyAssess project was the marking. In QTI this is called response processing because it is the process that transforms the candidate's responses into a set of outcomes.

QTI defines an expression language that allows responses to be transformed in fairly complex ways. The job of the delivery engine is to evaluate these expressions to obtain the outcomes. Some questions require marking processes that can't be represented in QTI. Essay type questions typically require human marking, often involving fairly sophisticated workflows. More recently these workflows have even included automatic essay marking software. In both cases there is a clear benefit from taking a web service based approach to response processing.

PyAssess was developed at Cambridge Assessment at the same time as a parallel project evaluating a new approach to the automated marking of short free-text answers. The marking engine was written in Prolog, a language well suited to describing the complex pattern matching rules required to analyse free text. The engine had a simple XML-based input/output mode of operation which made it an obvious candidate for recasting as a web service.

In order to demonstrate the PyAssess source code a simple command-line delivery engine was constructed with a web service client to enable automated processing of the student responses in real time. When doing this an important decision had to be made about the scope of such a web service. Should the expression language in QTI be extended to allow custom operators to invoke web services or should the entire response processing be carried out remotely? Demonstrating a solution to this problem at the CETIS Conference at the end of the project (November 2005) was a significant outcome. It seems clear to me now that the latter approach is actually the most appropriate.

You can keep up to do date with the latest developments in QTI, and with news from the Minibix projects at the QTI Tools website [5].


[1] Project proposal on JISC web site

[2] QTI Migration Tool

[3] Developing an SOA at SUNY; Lessons learned

[4] R2Q2: Rendering and Response processing services for QTIv2 questions

[5] QTI Tools


Supported by JISC Supported by CETIS
Powered by Plone