Using Quantitative Data when Evaluating Non-Formal Learning
To support the qualitative data captured using the retrospective pre-test method, I strongly recommend that you gather as much related, quantitative data as you can.
Now read on…
Information associated with a learning initiative or program you’re evaluating can derive from a range of source, and I advise using as many as is practicable. Collecting quantitative data can seem daunting and difficult to achieve, but happily there are a range of accessible sources of quantitative data that every learning professional can use.
For example…
- Always collect Kirkpatrick Level 1 attendee feedback forms from classroom-based and synchronous online participants at the end of each individual event: over time you accumulate substantial amounts of valuable feedback about learners’ reaction to the initiative.
- Use Web-hosted feedback forms to collect data from asynchronous participants (i.e. for online and DVD-ROM based content).
- Data can also be collected automatically from systems including Web servers, learning management systems and learning content management systems; Horton (2006) describes these types of archive data as “meaningful statistics” (p.102): they record detailed information about what participants did while taking the learning event, particularly in the e-learning channels.
- Other “meaningful statistics” include non-automated but standardized data recording processes such as recording learner attendance captured in the synchronous online and classroom-based context, and off-line learner activity such as DVD-ROM requests.
By examining logs and reports from these systems and processes, useful data can be collected on:
- Frequency and pattern of course access
- Number of slides/pages or learning objects accessed
- Duration of access to learning objects
- Number of supporting collateral downloads
- Feedback submitted
- Participation in discussion and question & answer sessions
- Rate of individual learner attendance or online access
- Rate of attendance or online access by role (i.e. developer, support, business analyst etc)
- Rate of group level (i.e. Sales & Marketing, Support, Manufacturing, Engineering etc) attendance or online access
- Rate of attendance or online access by length of service in an organization (i.e. new hire, with a company 0-12 months, 1-2 years, 2-3 years, 3-5 years, +5 years etc)
More Next Time…
_________
References:
Horton, W. (2006) So how is elearning different? IN: Kirkpatrick, P. & Kirkpatrick, J. Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.
–
July 22 2010 03:30 pm | e-learning
I see value in many of...
Ellen on 22 Jul 2010 at 7:09 pm #
Michael — Great to see this post on quantitative data measurement of elearning
I see value in many of the items in your final list, but wonder what usefulness you see in “Number of slides/pages or learning objects accessed”? And I’m intrigued by your reference to “rates” — “Rate if individual learner attendance or online access” is especially unclear, but even if this is a typo for “Rate of” I’m not sure I’d understand what measure you’re describing. Could you clarify?
Thanks so much!
Michael Hanley on 23 Jul 2010 at 9:39 am #
Thanks Ellen,
I don’t know why there isn’t more information about e-learning quantitative data capture in the public domain: it’s a bit of a “no-brainer” in my view, considering the richness of the data that’s available. Anyhoo, in regrad to your queries:
1 – “Number of slides/pages or learning objects accessed” – I’ll will discuss this in more detail in a future blog post
2 – The term “rate” or “rates” – hmmm… interesting. Not that you highlight it, this term can be interpreted in a number of ways including
and
as well as
In fact a quick skim through my dictionary provides 26 separate meanings for the word. So, I need to tighten up my terminology here, so leave it with me…
Best regards,
Michael
–
virginia Yonkers on 23 Jul 2010 at 4:53 pm #
One other place to capture quantitative information is in the registration process. We used to get a lot of valuable information in face to face training, and I get a lot of information from my students by having them fill out a general information form.
I also think a pre-test gives some good quantitative information in terms of a starting point for participants. This does not have to be “graded” but rather can be useful in analyzing as an aggregate questions participants had trouble with, which answers where given, and other trends in their knowledge base (i.e. people from the same department got the same question wrong with the same answer…maybe there is differences in terminology, understanding of what is “correct”, or a difference in philosophy). In fact, I think pre-tests are much more useful than post tests.