Evaluating Non-formal Learning: Validity in Research
When considering appropriate research designs to evaluate the effectiveness of learning initiatives, it’s important to consider the types of data that you have access to, particularly in the context of the issues evaluating a non-formal learning initiative at Kirkpatrick’s Levels 3 and 4 (Shelton. & Alliger, 1993, pp.43-46, Horton, 2006, p.109), I feel it is essential to ensure validity of the naturalistic information by capturing a range of supporting quantitative data.
Table 1. Relationship between parallel criteria
| Truth Value | Internal Validity | Credibility |
| Applicability | External Validity | Fittingness/transferability |
| Consistency | Reliability | Dependability/auditability |
| Neutrality | Objectivity | Confirmability |
To assist evaluators understand and provide a context for the analysis of the information collected in the final, qualitative phase of the study, Russ-Eft and Preskill (2001, pp.155-156) describe a number of useful techniques for establishing the validity of quantitative and qualitative data, among them accuracy checking in data recording and encoding, persistent participant observation and member checking.
As the resultant evaluation will use a mix of qualitative and quantitative methods as well as data collected from multiple sources, use I recommend that you use data triangulation used as a strategy to enhance the rigor and validity of you evaluation.
The term triangulation originated in cartography where two or more reference points are used to locate an exact position.
Knowing a single landmark only locates you somewhere along a line in a direction from the landmark, whereas with two landmarks you can take bearings in two directions and locate yourself at their intersection.
(Patton, 1990, p.187).
Denzin (1988) has identified four types of triangulation (see Table 2) – data triangulation, methodological triangulation, investigator triangulation, and theory triangulation; the I use these three triangulation techniques were used to evaluate the effectiveness of non-formal learning initiatives:
Table 2 Denzin’s (1988) Definition of triangulation methods
|
Triangulation method |
Definition |
|
Data triangulation |
Collecting data from a variety of sources. For example, in evaluating the transfer of learning from a three-hour workshop, the evaluator collects information from the learners, their managers, and their peers. In this case, three different sources have been queried. |
|
Methodological triangulation |
Using more than one method to collect data. For example, we may interview 20 per cent of a department’s employees and survey the remaining 80 per cent. By using two methods, the weaknesses of one may be compensated fro by the other. |
|
Theory triangulation |
Using different theoretical perspectives to interpret the same data. By applying different theories to make sense of the data, it is possible to see how different assumptions and beliefs influence one’s interpretations. By making these explicit, stakeholders can see how their assumptions might influence various actions taken because of the findings. |
More…
_________
References:
Denzin, N. K. (1988) The Research Act: A Theoretical Introduction to Sociological Methods. 3rd Ed. Englewood Cliffs, NJ: Prentice-Hall.
Guba, E.G. & Lincoln, Y.S. (1981) Effective Evaluation. San Francisco: Jossey-Bass.
Horton, W. (2006) So how is elearning different? IN: Kirkpatrick, P. & Kirkpatrick, J. Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.
Patton, M. Q. (1990) Qualitative Evaluation and Research Methods. 2nd Ed. Thousand Oaks, CA. Sage
Russ-Eft, D. & Preskill H. (2001) Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance and Change. New York, NY. Perseus Books.
Shelton, S. & Alliger, G. (1993) Who’s Afraid of Level 4 Evaluation? A Practical Approach. Training & Development. [Internet] 41 (3). Available from: http://www.afc-ispi.org/Repository/ Approaches%20for%20Assessing%20Outcomes%20ROI.doc [Accessed 24th May 2010]
–
July 01 2010 03:30 pm | e-learning
5 Responses to “Evaluating Non-formal Learning: Validity in Research”
virginia Yonkers on 02 Jul 2010 at 5:32 pm #
Being neck deep in a qualitative study, I feel qualified to comment on your article. In fact, quantitative and qualitative research addresses two different types of questions. Quantitative addresses the “what” whereas qualitative addresses the “how”.
As you point out, triangulation helps to make findings valid. However, often methodical qualitative analysis (whose process could be replicated) is also required. Grounded theory, ethnomethodology, case analysis, and discourse analysis all have standardized procedures which helps validate qualitative research, resulting in a deep level of analysis. To do it correctly, qualitative analysis is very time consuming (I currently am in my second year of analysis using grounded theory).
I also see qualitative research as a construction of the participant’s reality. Drawing from the qualitative data, the researcher systematically draws out relevant data, and reconstructs it during the analysis to create a story/narrative from which theory will be developed. Qualitative analysis is subjective, both from the researcher and participant’s perspective. Then again, so in quantitative, but good qualitative research identifies bias, perspective, and subjectivity. Quantitative does not always identify the biases.
Michael Hanley on 06 Jul 2010 at 12:31 pm #
Hi Virginia,
Thanks for your insights – as perceptive as ever. I agree with your views regarding pretty much everything you say. I don’t know the domain you’re researching, but have you any views on using a qualitative approach when investigating student behavior change following a learning intervention?
Best,
Michael
–
A Research Design to Evaluate E-Learning Projects | E-Learning Curve Blog on 08 Jul 2010 at 3:01 pm #
[...] a previous blog entry, I mentioned the importance of data triangulation, particularly if you use a case study approach [...]
virginia Yonkers on 08 Jul 2010 at 6:35 pm #
My research is on collaborative writing in the workplace. As my degree will be in Education and I hope to find a job in Communication that’s the focus of my research.
However, as a graduate assistant I worked with a professor on various ways to research online learning (Song, Yonkers, The Development and Evaluation of Multi-Level Assessment Framework for Analyzing Online Interaction). This article looked at a number of different qualitative methods.
The importance would be to look at how an intervention was effective or not effective. For example, I’ve used case studies and conversation analysis. However, a researcher could look at the change of processes used in a project or problem, which would be ethnomethodology. After identifying differences in the processes, it would be important to analyze what those differences were, compare it to the training or intervention, then have follow up questions about why one processes was preferred over another.
Evaluating Non-Formal Learning Programs - Table of Links | E-Learning Curve Blog on 05 Aug 2010 at 3:32 pm #
[...] Evaluating Non-formal Learning: Validity in Research [...]