Using a comprehensive framework for guiding formative and summative evaluations of OER course offerings at the University of Southern Queensland

My name is Tim McCallum [1], I work closely with Professor Jim Taylor [2] designing and building OER course offerings at the University of Southern Queensland.

CIPP model for evaluation

In this blog post I would like to discuss the CIPP model for evaluation Stufflebeam & Shinkfield (2007), but more importantly how we are using it to build our OER courses here at the University of Southern Queensland. The CIPP model provides 4 evaluation types which are as follows:


Firstly, Context, for example assessing the user’s needs and identifying opportunities or problems relating to particular users or groups (a little more on this later).


The second evaluation type is Input, this relates to the design aspect of how knowledge is acquired by the user. Some courses lend themselves to a more social or open design where micro-blogging or social media play a big part in the learning experience, personal mentoring is another consideration here. Perhaps the most important factor to consider when looking at this evaluation type is the software platform or delivery mechanism for the course in question. If we are designing the course for engagement, we need to be sure that the technology is able to support this.


Process is the third evaluation type. This relates to the actual implementation of the course and its related activities. We, will be focussing on what learners actually do during the course.


Finally Product; Product in a nutshell relates to return on investment, comparing actual outcomes against efforts made.

Putting CIPP into practice

Just to elaborate further on the first context evaluation type and pull this all together, let’s say we perform a survey at the start of a given course where a given number of users identify their intention to gain full credit for the course. If we have a KPI relating to this metric, we can make this part of the CIPP model for evaluation. We can combine survey results with empirical data or automated analytics (data mined from a course) then use this data to derive information and make decisions which improve the KPIs and ultimately the success of future course offerings. Essentially we are creating an opportunity to find out what is working, or more importantly what is not working for particular users or groups. Once this information is derived, we can improve the course with a degree of certainty.

We are focusing on improving user grades and creating independent self-directed learners. We are working to  improve user retention and we consider future enrolment and further engagement a sign of successful implementation.

Software platforms

I recently spoke with Professor Ken Udas [3], Professor Jim Taylor [2] and Dr Wayne Mackintosh [4] about delivery mechanisms for open education resources. This conversation took place shortly after I had completed the ‘open content licensing for educators’ course (an OER course delivered on the WikiEducator platform).

We discussed the Wiki environment as a platform and commented on its usefulness around version control, co-authoring and collaboration.

The WordPress platform was introduced as a potential delivery mechanism. WordPress offers many advantages in this arena such as responsive design templates and WordPress themes as well as plug-ins, all of which allow rapid development and reuse.

Jim Tittsler and I have unofficially discussed a WordPress plug-in called wiki-embed which lets you embed MediaWiki pages into your WordPress site. I am in the process of installing and testing this plug-in.

A colleague and PhD candidate, Neil Martin [5] and I are also working together. Neil is investigating how User Experience design techniques could potentially improve the learning experience within the OERu. 

Be sure to keep a keen eye on the Google summer of code as I understand The OER Foundation has been accepted as a mentor organisation for 2014. 

Certainly a lot happening in this space, please leave any questions or comments below and enjoy open education week.


[1] Senior Analyst and Demonstrator at the University of Southern Queensland

[2] Board of Directors of the OER Foundation and Emeritus Professor for the Australian Digital Futures Institute (ADFI) at the University of Southern Queensland

[3] Deputy Vice-Chancellor, Academic Services and Chief Information Officer (CIO) at the University of Southern Queensland

[4] Founder of WikiEducator,  founding Director of the OER Foundation and UNESCO-COL Chair in OER at Otago Polytechnic

[5] Learning Technologist for the Australian Digital Futures Institute (ADFI) at the University of Southern Queensland


Stufflebeam, D. L., & Shinkfield, A. J. (2007). Evaluation theory, models, and applications (pp. 240-241). San Francisco, CA: Jossey-Bass.


One thought on “Using a comprehensive framework for guiding formative and summative evaluations of OER course offerings at the University of Southern Queensland

  1. Great blog post and as you know, we’re keen to host a copy of your post on the OERu site as well to profile your OpenEd Week contribution. Don’t be shy to link back to your own blog. In OER I believe we should replicate good content on as many sites as possible!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s