Designing It Right: Making an Impact with Diagnostic Assessments : Articles (2024)

It has been a while sincediagnostic assessments have been at the forefront of learning and development discussions.However, they have the potential to address emerging issues in your workplace. Let’srevisit the purpose of diagnostics and the impact they can make in onlinelearning.

First, how would you define adiagnostic assessment? It is often described as a set of questions thatevaluates a learner’s prior knowledge in relation to specified learningobjectives. While this definition appears to be straightforward, if you ask twoonline learning professionals why theyuse diagnostics, their answers will likely be different. In reality, theresults of a diagnostic can be used in a wide variety of ways to achieve manydifferent objectives. Diagnostics can extend beyond typical knowledge-basedquestions to include items that assess a learner’s experience, confidence level,motivations, interests, or personal needs.

In our experience, manylearning professionals have not been using diagnostics to their full potential.Before reading on, ask yourself these questions:

• Do I currentlyuse diagnostics in my online learning?
• What value are mydiagnostics bringing to learners? To the organization?
• Could I usediagnostics more effectively?

In this article, we willexplore the opportunities that diagnostics present to learning professionals.We will discuss the benefits of diagnostics in asynchronous online learning andoffer practical steps and ideas to designing a diagnostic that meets your businessgoals.

Benefits of a diagnostic

Diagnostics offer a broad rangeof benefits when incorporated into asynchronous online courses. Consider thefollowing:

• Personalizedlearning paths: Each learner is unique and may benefit from personalizedlearning paths based on diagnostic results. For example, you may provide advancedlearners with a recommendation report that outlines their identified areas ofweakness. The report will help them to focus on priority content and movethrough the course more efficiently.
• The rightlearners in the right courses: A diagnostic that aligns with a final test canhelp you determine which learners should enroll in a course, an effort thatallows your organization to manage employee time more effectively.
• Measurableevidence of learning: Diagnostics can provide learning managers with numericalevidence to demonstrate the value of a course. You can measure the progress oflearners by comparing their first-attempt results from the diagnostic to thoseof the final test.
• Global learningneeds: You can aggregate diagnostic results for multiple learners or evenmultiple courses to identify where to allocate training resources based on high-prioritylearning gaps. In this way, diagnostics can be part of a more comprehensive approachto performance management.

While there are many benefits,diagnostics may not be appropriate for all situations. When would they beideal? If you can answer yes to any of the following questions, a diagnosticmight be right for you:

• Do my learnershave a degree of prior knowledge in the subject area?
• Is my audiencemade up of both novice and advanced learners?
• Is my course areasonable enough length to provide custom recommendations?
• Can some learnersbenefit from approaching the course in a non-linear fashion?
• Do I want to givelearners more control over their learning experience?
• Is my departmentrequired to show tangible evidence of learning?

This is by no means acomprehensive list, but it can help point you in the right direction.

Steps for designing a diagnostic

Now that we have discussed whydiagnostics are useful and when they are suitable, let’s get started on how todesign an effective diagnostic assessment. We have come up with five simplesteps:

1. Define your goal
2. Identify impact oncourse design
3. Assess learningobjectives
4. Determine questionformat
5. Develop a messageto learners

Step 1: Define your goal

Your first step is to clearlydefine your goal for using a diagnostic. If you have multiple stakeholders, werecommend validating the goal to ensure the solution aligns with eachstakeholder’s expectations. You may be tempted to create a diagnostic to servemultiple and/or dissimilar needs, but keep in mind that conflicting prioritiesmay ultimately dilute the effectiveness of your diagnostic.

Example goal:

The diagnostic in Accounting101 will generate study recommendations for learners based on their personalstrengths and weaknesses, allowing them to plan their study scheduleaccordingly. This solution responds to the different learning gaps amonglearners who come from varied educational backgrounds and experiences.

Step 2: Identify impact on course design

Next, you need to determinewhether or not the results of the diagnostic will change the course design andin what way. If you choose to change the course experience for the learner, Table1 provides some suggestions. The options are not mutually exclusive, so you maywant to choose a blended solution.

Table 1: Suggestionsfor course adaptation based on context

Step 3: Assess learning objectives

Once you have defined your goal and knowyour intended impact on course design, you can begin to build a diagnostic thatwill meet your needs.

First, establish which topics and howmany questions should be in the diagnostic. We recommend analyzing yourlearning objectives so that the diagnostic focuses on core learning areasrather than “nice-to-know” ideas. A common practice for instructional designersis to prioritize learning objectives by ranking them according to criticality,frequency, task difficulty, or domain size. (Note that assessing learningobjectives may not be as critical if the purpose of the diagnostic is to identifywhich topics are of interest to the learner or how they prefer to learn—theirmotivation.)

With your prioritized learningobjectives, you can determine the domain weightings for the diagnostic. If youare building a diagnostic that must be comparable to a final test, ensure that youbreak down the results of both assessments in the same way. Here are a couple ofchallenges we have encountered when aligning the diagnostic to the final test.

Challenge 1: The diagnostic needed to be shorter in length thanthe final test.

Our solution: Reduce the numberof questions for the diagnostic but ensure that the number of questions areproportionate to the weightings of the final test. Table 2 shows a samplearrangement.

Table2: Making the diagnostic proportionate to the final test

Challenge 2: We needed to deliver diagnostics at the module levelbut deliver the final test at the course level, making it difficult to measurethe progress of learners from start to finish.

Our solution: Ensure youcategorize the final test results in the same way as the diagnostic results.Display the results to learners using these categories so they can clearly see theirimprovements. Here is a sample report:

Step 4: Determine question format

Now it is time to design the questionsthemselves. Questions may evaluate any of the following including knowledge,skills, experience, confidence levels, interests, and motivations. The goal ofyour diagnostic should drive which type of questions are appropriate. If youmean to compare the results of your diagnostic to be compared with the finaltest, your question format will likely be exam-quality items mapped to taxonomylevels. On the other hand, if you mean for the diagnostic results to provide amore efficient study path, your question format may include rating scales toaddress more subjective measures such as confidence levels or study interests.

Challenge 3: We were concerned that learners who did not know theanswer would guess correctly, which would produce misleading diagnosticresults.

Our solution: Include an “I don’tknow” option in the question. In the diagnostic instructions, encouragelearners to select the option when they are unsure of the correct answer sothat they can receive a more accurate recommendation report. From a trackingperspective, treat the“I don’t know” option as an incorrect answer.

Challenge 4: We thought that learners with high levels of experience maypossibly answer knowledge-level questions incorrectly, which would producemisleading “beginner” recommendations on how to approach the course.

Our solution: Integratequestions that ask learners to rate their levels of experience. Map the scoresfrom both the experience and knowledge questions to recommendation levels.Learners who have high experience ratings but low knowledge scores will nolonger receive a “beginner” recommendation but rather an “intermediate” one.Table 3 below shows a sample mapping.

Table 3: Samplerecommendation mapping

Step 5: Develop a message to learners

Before launching a diagnostic,it is important to position it properly with learners so they understand whyand how they are being assessed. If a learner fails a test prior to starting acourse, it could negatively impact the learner’s motivation. You can mediate hisoutcome through proper positioning of the diagnostic. In your message tolearners, we recommend addressing the following questions:

• Why am I taking adiagnostic before starting the course?
• Do I have to takethe diagnostic?
• How long will thediagnostic take to complete?
• Can I re-do the diagnostic?
• How do I use the recommendationreport (if provided)?

The answers to the abovequestions will vary based on the goal of your diagnostic. Consider thequestion, “Do I have to take the assessment?” If your diagnostic is intended toprovide a streamlined learning path for advanced learners, it will not beuseful for novice learners and should not be mandatory. On the other hand, ifyour goal is to track evidence of learning, the diagnostic should be taken byall learners. In both situations, you should clearly communicate the purpose ofthe diagnostic to learners so they can approach it accordingly.

Summary remarks

You have explored the benefits of diagnostic assessments and when theyare suitable. If you are ready to create a diagnostic, try integrating the five-stepprocess into your work. We believe there is a great opportunity for diagnosticsto resolve critical challenges in our field today. Learning professionals arelooking for new solutions as the demand grows for personalized learningexperiences, as organizations increasingly require accurate reporting, and asthe field of knowledge management emerges. Diagnostics are powerful measurementtools that have the potential to both meet organizational needs and transformlearning experiences.

How have diagnostics helped you and yourorganization? What challenges have you faced? We are interested in yourexperiences on how you have used diagnostics in your online courses.

February 9, 2015