More on evaluating NFL using Kirkpatrick’s Four-level model

I’m going to begin this post by mentioning Donald Clark’s view on Donald Kirkpatrick’s model – the post has the tongue-in-cheek title Donald Talks Bollocks, and it gives an alternative view on the four levels; have a read, it’s good stuff.

However, as Mr. Clark points out, rejecting Kirkpatrick doesn’t provide a answer to how else we should evaluate the effectiveness of learning (particularly non-formal learning) so in lieu of another approach I will carry on discussing how I think Kirkpatrick can be used to assess the effectiveness of NFL initiatives.

Pay attention, here comes the science bit…

I’ve already discussed why I think Kirkpatrick’s guidelines for evaluating learning is incompatible with the assessment of non-formal learning (particularly in the context of NFL), and today I’ll discuss how you can begin to counter these deficiencies. Like Steven Kerr, CLO at Goldman Sachs, Kirkpatrick Level 3 is the most interesting to me; in the context of NFL, Level 2 cannot be used as an instrument for evaluation: with no summative assessment at the end of an NFL session, you cannot evaluate knowledge transfer by traditional testing methods.

To understand why NFL cannot be assessed at Level 2, look at what Michael Eraut says in Non-formal learning, implicit learning and tacit knowledge (2000). Eraut makes a clear distinction between his interpretation of the term ‘non-formal learning’ and what others including Scribner (1988), Conner (2002) and Cross (2003) would call ‘informal learning’ – what Eraut terms ‘incidental learning’ or ‘the acquisition of knowledge independently of conscious attempts to learn and the absence of explicit knowledge about what was learned’ (Reber, 1993, quoted by Eraut 2000, p.12)

This is, he argues, because most workplace learning takes place outside formal learning contexts, and informal learning carries with it connotations of


so many other features of a situation, such as dress, discourse, behaviour, diminution of social differences – that its colloquial application as a descriptor of learning contexts may have little to do with learning per se.


(2000, p.12).


Not only does the term carry unwanted and confusing implications, but it is too vague to be of any real utility.For Eraut, an analysis of learning must focus on activities and the outcomes that that contribute to significant changes in capability or understanding. In a sense, Eraut does not define non-formal learning; rather, he defines the characteristics of formal learning (p.12) as:

  1. A prescribed learning framework
  2. An organised learning event or package
  3. The presence of a designated teacher or trainer
  4. The award of a qualification or credit
  5. The external specification of outcomes.

The implication of this categorisation is that any learning that does not exhibit all of these characteristics should be classed as either informal or non-formal learning. Some reviewers (Colley, Hodkinson & Malcolm, 2002) make the point that Eraut does not make clear what the status is of learning in situations that meet some, but not all, of his ‘formal’ criteria. My interpretation of his characterisation is that the very nature of a formal activity – “following or according with established form, custom, or rule” (Merriam-Webster Online, 2007) validates Eraut’s description.

I am particularly interested in exploring the relationship between Level 1 (Reaction) and Level 3 (Behaviour); as Kirkpatrick & Kirkpatrick (2006, p.22) assert “learning has taken place …when one or more of the following occurs: Attitudes (sic) are changed. Knowledge is increased. Skill is improved. One or more of these changes must take place if a change in behaviour is to occur.” In this context, I believe a mixed method of quantitative data-gathering and a more naturalistic or qualitative information collection approaches can be employed to capture data that will assist in evaluating the success of a non-formal learning initiative.

And it is this approach that I will be discussing next time.


Colley, H. Hodkinson, P. Malcolm, J. (2002) non-formal learning: mapping the conceptual terrain. a consultation report [Internet] Available from: [Accessed 28th January 2014]

Conner, M. (2002) Informal Learning: more than “What we do between classes”. [Internet] Available from:
[Accessed 19th October, 2013]

Cross, J. (2004) An informal history of eLearning. On the Horizon [Internet] 12(3). pp.103-110. Available from:
[Accessed 20th February, 2013]

Kirkpatrick, D. & Kirkpatrick, P. (2006) Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

Scribner, S. (1988) Mental and manual work: An activity theory orientation. In E. Tobah, R. J. Falmagne, M. B. Parlee, L. M. Martin & A. S. Kapelman (Eds.), Mind and Social Practice: Selected Writings of Sylvia Scribner. (pp 367-374).Cambridge: CUP

Whitney, K. (2004) Steve Kerr: Managing the Business of Learning [Internet] Available from:
[Accessed 28th February 2013]

Related Posts with Thumbnails

February 14 2014 09:00 am | e-learning

2 Responses to “More on evaluating NFL using Kirkpatrick’s Four-level model”

  1. mason moore on 15 Feb 2014 at 6:34 pm #

    Am I allowed to post this on my twitter?

  2. Michael Hanley on 16 Feb 2014 at 11:09 pm #

    Hi Mason,
    Thanks for your note; please fee free to post on your Twitter.
    Best regards,
    Michael Hanley (the E-Learning Curve Blog)