Evaluating Non-Formal Learning using Kirkpatrick’s Four-level model: More…
I’m going to begin this post by mentioning Donald Clark’s view on Donald Kirkpatrick’s model – the post has the tongue-in-cheek title Donald Talks Bollocks, and it gives an alternative view on the four levels; have a read, it’s good stuff. However, as Mr. Clark points out, rejecting Kirkpatrick doesn’t provide a answer to how else we should evaluate the effectiveness of learning (particularly non-formal learning) so in lieu of another approach I will carry on discussing how I think Kirkpatrick can be used to assess the effectiveness of NFL initiatives.
Pay attention, here comes the science bit… I’ve already discussed why I think Kirkpatrick’s guidelines for evaluating learning is incompatible with the assessment of non-formal learning (particularly in the context of NFL), and today I’ll discuss how you can begin to counter these deficiencies.
Like Steven Kerr, CLO at Goldman Sachs, I think that Kirkpatrick’s Level 3 is the most interesting; in the context of NFL, Level 2 cannot be used as an instrument for evaluation: with no summative assessment at the end of an NFL session, you cannot evaluate knowledge transfer by traditional testing methods. To understand why NFL cannot be assessed at Level 2, look at what Michael Eraut says in Non-formal learning, implicit learning and tacit knowledge (2000). Eraut makes a clear distinction between his interpretation of the term ‘non-formal learning’ and what others including Scribner (1988), Conner (2002) and Cross (2003) would call ‘informal learning’ – what Eraut terms ‘incidental learning’ or ‘the acquisition of knowledge independently of conscious attempts to learn and the absence of explicit knowledge about what was learned’ (Reber, 1993, quoted by Eraut 2000, p.12)
This is, he argues, because most workplace learning takes place outside formal learning contexts, and informal learning carries with it connotations of
so many other features of a situation, such as dress, discourse, behaviour, diminution of social differences – that its colloquial application as a descriptor of learning contexts may have little to do with learning per se.
Not only does the term carry unwanted and confusing implications, but it is too vague to be of any real utility. For Eraut, an analysis of learning must focus on activities and the outcomes that that contribute to significant changes in capability or understanding.
In a sense, Eraut does not define non-formal learning; rather, he defines the characteristics of formal learning (p.12) as:
- A prescribed learning framework
- An organized learning event or package
- The presence of a designated teacher or trainer
- The award of a qualification or credit
- The external specification of outcomes.
The implication of this categorization is that any learning that does not exhibit all of these characteristics should be classed as either informal or non-formal learning. Some reviewers (Colley, Hodkinson & Malcolm, 2002) make the point that Eraut does not make clear what the status is of learning in situations that meet some, but not all, of his ‘formal’ criteria. My interpretation of his characterization is that the very nature of a formal activity – “following or according with established form, custom, or rule” (Merriam-Webster Online, 2007) validates Eraut’s description.
I am particularly interested in exploring the relationship between Level 1 (Reaction) and Level 3 (Behavior); as Kirkpatrick & Kirkpatrick (2006, p.22) assert
learning has taken place …when one or more of the following occurs: Attitudes (sic) are changed. Knowledge is increased. Skill is improved. One or more of these changes must take place if a change in behavior is to occur.
In this context, I believe a mixed method of quantitative data-gathering and a more naturalistic or qualitative information collection approaches can be employed to capture data that will assist in evaluating the success of a non-formal learning initiative. And it is this approach that I will be discussing tomorrow.
__________
References:
Colley, H. Hodkinson, P. Malcolm, J. (2002). non-formal learning: mapping the conceptual terrain. a consultation report. [Internet] Available from: http://www.infed.org/archives/e-texts/colley_informal_learning.htm [Accessed 28th May 2010]
Conner, M. (2002). Informal Learning: more than “What we do between classes”. [Internet] Available from: http://www.productivity-solutions.net/documents/ informallearning-mconner-071202.pdf [Accessed 19th October, 2006]
Cross, J. (2004) An informal history of eLearning. On the Horizon [Internet] 12(3). pp.103-110. Available from: http://www.emeraldinsight.com/Insight/viewPDF.jsp? Filename=html/Output/Published/EmeraldFullTextArticle/ Pdf/2740120301.pdf [Accessed 20th May, 2010]
Kirkpatrick, D. & Kirkpatrick, P. (2006) Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.
Scribner, S. (1988) Mental and manual work: An activity theory orientation. In E. Tobah, R. J. Falmagne, M. B. Parlee, L. M. Martin & A. S. Kapelman (Eds.), Mind and Social Practice: Selected Writings of Sylvia Scribner. (pp 367-374).Cambridge: CUP
Whitney, K. (2004) Steve Kerr: Managing the Business of Learning [Internet] Available from: http://www.clomedia.com/content/templates/clo_cloprofile.asp?articleid=582&zoneid=4 [Accessed 28th May 2010]
–
June 08 2010 03:30 pm | e-learning
I see value in many of...
virginia Yonkers on 09 Jun 2010 at 7:01 pm #
This has been a very interesting series. I am wondering if there is any research on the use of portfolios as a means of assessing employees. It seems to me that a portfolio of work is one way that services make their product tangible for potential customers. Why not use it in the workplace (as it is increasingly being done in schools at all levels). This could be submitted individually, at the department level, and at the organizational level.