Using a Retrospective Pretest Design in Elearning Evaluation
The Retrospective Pre-Test method is an interesting if controversial data collection design for evaluating e-learning programs. In this post, I will begin examining its advantages and negative aspects, and later argue for including this strategy in assessing organizational learning initiatives. The retrospective pre-test method (see Table 1) is an extension of the one-shot case study design, where data are gathered from participants following a learning intervention (for example, a series of non-formal Information Sessions).
Table 1 Retrospective Pre-test Design
|
Retrospective Pre-test Design |
|||||
|
Groups |
|
Non-equivalent |
Intervention |
Observation |
|
|
Locally-based |
|
N |
X |
O1 |
O2 |
|
Remotely-based |
|
N |
X |
O1 |
O2 |
|
<- Time -> |
|||||
However, participants (N) report on their knowledge, understanding or skills (Observation O1) after the intervention (X), and then also reflect and answer (O2) what they believe their understanding or skill was before the intervention. Rockwell & Kohn (1989) applied this method to testing the effectiveness of achieving program outcomes when interventions such as training programs are implemented and concluded that “using a post-then-pre design [retrospective pre-test] to identify self-reported behavioral changes can provide substantial evidence for program impact”. The retrospective pre-test enables researchers to reduce the response shift bias – defined by Klatt & Taylor-Powell in their 2005 paper Synthesis of Literature relative to Retrospective Pretest Design as
…the change in the participant’s metric for answering questions from the pre-test to the post-test due to a new understanding of a concept being taught…
because the participants are able to give pre-test responses which are based on a post-intervention frame of reference.
Using the retrospective pre-test, response shift bias can be reduced increasing the likelihood that the observable results are due to intended intervention effects (Pratt, McGuigan & Katzev, 2001).
This proposition is complicated and has its critics: Theodore Lamb (2005), an advocate of the retrospective pre-test method, describes it as “an imperfect but useful tool.” Robson (2002, p.139-141) expresses concerns that used purely as a quasi-experimental method, this strategy could lead to an internal validity threat through regression to the mean.
More Next Time…
________
References:
Klatt, J. and Taylor-Powell, E. (2005). Synthesis of Literature relative to Retrospective Pretest Design. Presentation to the 2005 Joint CES/AEA Conference, Toronto [Internet] Available from: http://www.citra.org/Assets/documents/evaluation%20design.pdf [Accessed 5th August 2009]
Lamb, T. (2005). The Retrospective Pretest: An Imperfect but Useful Tool. Evaluation Exchange. [Internet] 11(2). Available from: http://www.gse.harvard.edu/hfrp/eval/issue30/spotlight.html [Accessed 21st March 2010]
Pratt, C.C., McGuigan, W.M., & Katzev, A.R. (2000). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21(3). 341-349.
Robson, C. (2002) Real World Research. 2nd ed. Oxford: Blackwell Publishing.
Rockwell, S. K. & Kohn, H. (1989). Post-Then-Pre Evaluation. Journal of Extension. [Internet] 27(2). Available from: http://www.joe.org/joe/1989summer/a5.html [Accessed 21st March 2010]
–
July 14 2010 03:00 pm | e-learning
2 Responses to “Using a Retrospective Pretest Design in Elearning Evaluation”
virginia Yonkers on 14 Jul 2010 at 11:17 pm #
So if I am reading this correctly, what you are talking about is an assessment after an intervention of the trainee’s skills BEFORE the intervention (pre-test)? One problem is self-reporting. However, there has been a lot of studies in which a person’s self reporting was in fact accurate, especially in online studies (comparing student, teacher, and researchers perceptions).
I think many think of “pre-tests” as objective tests which are the same as “post-tests”. This would be quantitative methods in this case. But if I am reading you post correctly, you are talking about something different than these traditional measures.
Kirkpatrick's Guidelines for Evaluating Training Programs | E-Learning Curve Blog on 19 Jul 2010 at 3:41 pm #
[...] Returning to the topic of learning – particularly non-formal learning – evaluation using the retrospective pre-test design model. [...]