A Research Design to Evaluate E-Learning Projects
In a previous blog entry, I mentioned the importance of data triangulation, particularly if you use a case study approach when evaluating an e-learning initiative. Another strategy I recommend is to collect data over a number of phases, using a number of techniques.
Case studies involve in-depth, descriptive data collection and analysis of a case, or a number of related instances of the same type within the case. In particular, the case study design is useful when answering “how” and “why” questions, and in understanding the particulars, and diversity of the case (Yin, 1994). As Yin asserted in his 1993 paper Applications of Case Study Research:
the case study is the method of choice when the phenomenon under study is not readily distinguishable from its context.
(p.3)
The case study design (see Table 1) suited the conditions of this research project very well as it allowed naturalistic methods of enquiry such as individual interviews to be carried out, side-by-side with quantitative data gathering from surveys and archival records.
1 Features of a case study (Russ-Eft & Preskill, 2001, pp.173-174)
|
Advantages |
Disadvantages |
|
Provides descriptive data |
Results may not lead to scientific generalizability |
|
Does not require control of participants or setting |
Researcher bias may interfere with validity of the findings |
|
Reports include verbatim quotes |
May take too long to conduct |
|
Leads to a greater understanding of the context of the evaluand |
May produce more data than can be analyzed in an effective manner |
|
Gather data using multiple methods |
|
|
Provides data that are rich in examples |
|
|
Captures what is important to the participants |
|
|
Portrays the multiplicity of causes that are associated with various outcomes |
|
|
Embraces diversity of perspectives and experiences of participants |
|
|
Allows the researcher to collect information on outcomes not known prior to the learning and performance initiative |
|
As Robson points out, the flexibility of the case study approach allows the design to “emerge” (2002, p.89) during data collection and analysis. Similarly, the case study facilitates the application of other research designs within the case study design framework, enabling the learning professional to capture data during different phases of a project.
More…
__________
References:
Robson, C. (2002) Real World Research. 2nd ed. Oxford: Blackwell Publishing.
Russ-Eft, D. & Preskill H. (2001) Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance and Change. New York, NY. Perseus Books.
Yin, R. K. (1993). Applications of Case Study Research. Newbury Park, CA. Sage.
Yin, R. K. (1994). Discovering the future of the case study method in evaluation research. Evaluation Practice [Internet] 15. Available from: http://aje.sagepub.com/cgi/reprint/15/3/283 [Accessed 15th January 2010, subscription required]
–
July 08 2010 03:00 pm | e-learning
4 Responses to “A Research Design to Evaluate E-Learning Projects”
virginia Yonkers on 08 Jul 2010 at 3:30 pm #
Many qualitative researchers would say that any results that lead to scientifically generalizable findings was not qualitative research, in fact. However, “May produce more data than can be analyzed in an effective manner” is probably the biggest problem. Related to this is that qualitative analysis requires a great deal of training and conferencing for analysts/researchers. The more analysts you have, the greater reliability of the analysis, but also the greater chance of different interpretation of the study. As a result, the best qualitative studies include conferencing between researchers on a regular basis to ensure consistency in study goals and processes. This is both expensive and time consuming.
A. Williams-Nash on 08 Jul 2010 at 11:53 pm #
As e-learning becomes more prominent in society and shifts downward towards the high school and elementary school levels, standards must be developed to evaluate and assess whether the content met the objectives. Since anyone can develop an e-learning module, the question becames how does an instructional designer or person charged with acquiring a turn key solution compare different vendors and ensure validity of the proposed course?
The advantages and disadvantages provided in the case study serve as a foundation to streamline the process. Perhaps a checklist or list of does and don’ts can be provided. Additional do you have any thoughts on how much data should be collected at the various points to make the findings statistically valid?
Case Study Techniques for Evaluating E-Learning Initiatives | E-Learning Curve Blog on 13 Jul 2010 at 3:01 pm #
[...] I asserted in a previous post here on the E-Learning Curve Blog, case studies allow naturalistic methods of enquiry such as [...]
Posts and Articles To Read: July 23, 2010 « 4R x T on 23 Jul 2010 at 2:34 pm #
[...] “A Research Design to Evaluate E-Learning Projects” [...]