Pillars of sound educational assessment - THE EDUCATION HUB (2024)

Alleducational assessments should have a clear purpose, provide information that is trustworthy, dependable and appropriate, and have beneficial outcomes foralllearners.

Validity, reliability and fairness are the three pillars of sound educational assessment and are interconnected in ways that demand careful attention be given to each. They are universally important qualities that apply to:

• informal and formal assessments
• assessmentsforlearning and assessmentsoflearning
• assessment information gathered in early childhood education (ECE) settings, kura and schools
• classroom/ECE-based assessments and large-scale tests and examinations

Teachers can ensure assessment is valid, reliable and fair by asking these questions:

Assessment in ECE,kuraand school settings

The national curricula(Te Whāriki, Te Marautanga o Aotearoa, andThe New Zealand Curriculum)identifyvalued learning outcomesand achievement objectivesconsistent with theirrespectivemission statements andprinciples.Information aboutlearners’progressand achievement can begatheredin a variety ofinformal and formal ways.Examples include:

• teacherobservations
• annotated photographs,written narratives orlearning stories
• teacher-developed assessment activities or tests andexams
• published tests, such as Progressive Achievement Tests
• external examinations for qualifications, such as NCEA
• online resources, such as e-asTTle

Teachers in ECE settings donot use test-based assessments.Here,assessmentismore holistic andbased on noticing, recognising and responding to achild’s interests.A child’s learning is documented in learning stories in relation to learning outcomes, and based on their interests, strengths and dispositions. 

Evaluating the quality of assessment–anargument-based approach

Statistical evidence of validityandreliabilityis usually reported for published tests and large-scale tests and examinations. However, for assessmentsundertaken byteachers in ECE,kuraand schoolsan argument-based approachis more appropriateforevaluatingthe validity, reliability and fairness of an assessment or programme of assessment.Teachers can ensure their assessments are asvalid, reliable,and fairas possible by considering both the evidence that supports their claims and the impact of any factors that may threaten or weaken them.

Validity

Validity isconcerned withgathering assessment informationand ensuring the assessment is fit for purposeand does no harm.Validityhas traditionally referred toanassessmentinstrumentit*elf.More recentlythough, and of more relevance for teachersin ECE settings, kura and schools, validity is better understood asthequality of theinterpretationsanddecisionsthat are madeon the basis of assessmentresultsor observations.Teacherscan do this byusingavalidity argument.In evaluating the validityofan assessment, it is important to consider both the evidence that supports claims for validity and the impact of any factors that may weaken claims for validity.There are different types of validity.

• Construct validity:Does the assessment cover the learning intended or capture the learning noticed?

Construct validityis at the heart of a validity argument.Constructs arethespecific attributes or traitsthatyou want to assess, such as reasoning skillsorreading comprehension.For example,aconstructin a maths assessmentmay be problemsolving.Constructs are alsospecificlearning,interests and dispositionsyou are seeking to capture.For example, an ECE assessment about making connections between people and places could be captured in a learning story about a visit to a local marae.Teachers need to have a good understanding of the construct and traitsof the learning activity, and how they can be demonstratedor capturedthrough assessment.

Construct validity can bestrengthened by includinga reasonable coverage oftheimportanttraits of the main constructin an assessmentandexcludingother‘confounding’factorsthatareunrelated to the construct.Forexample,when the reading demands ofthemaths problemsin a maths assessmentbecome an obstacletostudentsunderstanding the problem, this isa confounding factor.Theinformationfrom the maths assessmentwill be less validfor less able readers,andtherefore less ofa true reflection of their maths ability.

• Content validity:Does the assessmentincludea reasonable coverage of thelearningdomain to be assessed?

A domain isanarea of learning,suchas learning outcomes ofTe Whārikiormathsachievement objectivesinThe New Zealand Curriculum(NZC).A singleassessment cannot cover the whole domain you are interested in assessing,soit is important that the assessment includesafairsample of the domainor captures sufficient evidenceso that you can be confidentaboutany inferencesyou make aboutlearners’ abilitiesin that domain.This means that teachers need to be able to matchthe scope and nature of thelearningdomainto assessment tasksand learning outcomes.Sometimes, this might require teachers to write anassessment planthat detailsthe skills,knowledge, dispositionsand contextsto be assessed/capturedwithin an assessment or across a programme of assessment.Content validitycan be improved by matchinglearning opportunities to learning outcomes,bymatchingassessment tasksto learning intentions, andbyhaving discussions with colleagues.

• Consequential validity:Is the assessment information usedfor the purposefor whichit is gathered andin ways that are beneficial forlearners?

This refers to the consequences of the assessment,andthe uses to which it is put.Consequential validity will be high when theassessment information is used to support and promote individuals’ learning.However, consequential validity will be low when the assessment information is used for a purposefor whichit was not intended.For example, iftheassessmentinformation collectedto inform individuals’ learning in the ECE, kura or school settingsis also usedto judgeateacher’sperformance, then the assessmentinformationbecomes‘highstakes’for the teacher,andassessmentmay becomedistorted.

In evaluating the consequential validity of an assessment, teachers should consider the potentialimpact the assessment will have onlearners,onthemselves,andonothers whowill usetheinformation.Teachersparticularlyneed to question the validity of assessment when there is evidence thattheinformationintended to inform teaching and learningis used in ways that may be detrimental or harmful tolearners, such as when informal data gathered in class is used for high stakes purposes.

Reliability

Reliability is concerned withtheaccuracy and consistency oftheinformation collectedfrom an assessmentand the dependability ofthe judgments that teachers make about learners’ progress and achievement.When assessment information isaccurate andconsistent,we can be confident that it is areliablerepresentation of what is being assessed.When there is sufficientmoderatedassessment information, we can be confident that teachers make dependable judgments and decisions.Teachers should aim to develop assessment procedures and tasks that are as reliable as possible. The level of reliability that is acceptable for an assessmentor judgmentalso depends on the purpose of the assessment and how the information is to be used.

Reliabilitymay beconsidered as different forms of consistency:

• Teacherconsistency:Does a teacher apply the assessment criteria in the same way for alllearners?

Teacher consistencywillbestrengthened if the teachercanapplyassessmentcriteriaconsistently for differentlearnersin an unbiased way.

• Consistency acrossteachers:Would the assessment information collected be similar if someone else applied the same criteria to observations of individuallearnersor marked the assessment?

Consistency across teachersgives an indication of how much differentteachers agree when theyjudgethe same assessment using the same criteria. Consistency across teacherswill bestrengthenedif teachershave a shared or common understanding of thecriteriaand learning outcomes,andtheyapply themin the same way.

• Consistency acrossassessment information:Would teachers get the same information if they used differentassessmentsto assess the same learning or if they observedlearnersin different contexts?

When theinformationfrom two differentsets ofassessmentthatassess the same constructissimilar, we have reasonable evidence to claim that the assessment hasconsistency acrosstheinformation collected.

• Consistency acrosstime:Would teachers get the same assessment information iflearnerswere assessed at a different time with the same or equivalent assessment?

When the information fromtheassessment thatis used at different points in time is similar,we have reasonable evidence to claim that the assessmentinformation collectedhas consistency acrosstime.

Dependable judgments

Teachers are far more likely to make dependable and meaningfuljudgmentsabout learners’ progress and achievement when they collect and consider assessment information from multiple sources, engage in moderation activities, and/or usebenchmarksto judge learners’ performances or learning. This is particularly important when teachers use the information to make decisions about teaching and learning, and when reporting back to learners and their parents or whānau.

Teachers can ensure theirjudgments are asdependableas possible byconsidering the following:

• Sufficient information:Doesthe teacher haveenough information to make adependablejudgment aboutlearners’ progress and achievement?

Whenteachers gather multiple sources of evidence from assessments withstrongreliability,it ismore likelythatdependablejudgments anddecisions will be made.Usingthree different sources of evidence(triangulation)may provide sufficient information formakingthisjudgment.Thesemightinclude observations of learning,products thatlearnerscreate(includingartefacts andassessment results) and learning conversations with thelearners.

• Moderation: Do teachers make similar judgments based onthe sameassessment information?

It isalsoimportant tocompare howother teachers apply criteria to the same assessment task,todiscussyour interpretations of the criteriaandtoadjustthemso that they converge. This willhelpteachers develop a shared understanding of the criteria as well as promotingcollegial support and professional development.Moderation will also contribute to improved consistency across teachers.

• Benchmarks: Are teachers’judgments about learners’ progress and achievementin line withcurriculum expectations?

Nationalcurricula set out expectations for whatlearnersshould know and do at different points in their learning. These may beaccompanied byprogressionsof learning, exemplarsand performance indicators.Theseelaborate on criteria and includelearners’actualperformancesalong with explanationsof the features of thoseperformancesthat contribute to the judgmentmadeabout thelearning. As nationally developedbenchmarks forthe curricula,such toolsguideteachers’ understandingsofexpected interpretations oflearners’ performances.

Fairness

Fairness is concerned with assessinglearnersin ways that are appropriate for them,andthatrecognisethe diversity oflearners.It ensures thatassessment allowsalllearnersto show what they know and can do.It is also important to consider whether somelearnersmay needadditional assistance, and what other opportunitieslearnersmaybenefit from.Fairness isnecessary for ethical assessment practice andto give alllearnersequitable opportunities to succeed in their learning.

Achieving fairerassessment requires flexibleconditions and strategies as differentlearnersmay have different needs. Flexibility is more acceptable when assessment is used to support individuals’ learning but less acceptable when making decisions about awarding qualifications. Assessment fairness iscomplex and cannot be ensured through one set of practices. Fairer assessments can be achieved using differentconditionsand strategiesdepending on the purpose of the assessmentand thelearnersassessed,including:

• Opportunitiesto learn: Do all learners have the same opportunity to show what they know and can do?

Alllearnersshouldhavesimilarand appropriateopportunities to learn.This may relate to whatlearnersexperience in their ECE or school settings, and to other social and educational factors that enable learning, including the availability and quality of resources (teachers, learning materials, technology,and so on) andlearners’ability to use them.

• Aconstructive environment: Do school students feel comfortable when they are assessed?

Schoolstudentsneed tosee that assessment is worthwhile, or at least necessary, andberespectfullyencouragedto participate in assessmentin ordertodemonstrate what they know and can do.Thisrequireshigh levels of trust and respect between teachers andstudents,andamongpeers.Studentsneed to feel safeabout demonstratingtheir learning in an assessmentand know their learning will be supported accordingly.

InECE,childrenneed toexperiencelearningopportunities that allow them to freely explore and show their learning.To enable this, teachersshould aim tominimiseanyfactors that might inhibitchildrenfromfully showingtheirlearning. This is often achievedthroughthe useof photographs, videos and artifacts ofchildren‘in action’, and documented later.

• Teacherreflection:Do teachers avoid bias and recognise diversity?

Assessment may be influenced by several factorsof whichteachers need to be mindful. Theseincludeteachers’assumptions andbeliefsthat may lead to bias,theiropennessto the knowledge and learning of diverselearners, and their commitment to fairness. Other factors includethoughtful planning, administration, and interpretation.

• Transparency:Do school students clearly understand what the teacher requires in an assessment?

Are the instructions clear? Are theassessmentcriteriaexplicit,clearandsharedwithstudents?Transparency makes assessment fairer by reducing irrelevant factors (although this is less relevant forchildreninECE).

• Opportunities to demonstrate learning:Do learners have multiple, varied, equitable, and meaningful opportunities to demonstrate their learning?

Multiple opportunitiesallow teachers to gather sufficient information to make fairer decisions;varied opportunitiesenabledifferent types oflearnersto succeed;equitable opportunities,usingappropriate accommodations,allowlearnersto demonstrate what they knowand can do; andmeaningful opportunitiesare engaging and challengingwithout being superficial and impossible.

The relationships between validity, reliability and fairness

Validity, reliability and fairness are interconnected in important ways.Validity is considered to be the most important quality ofsoundeducational assessmentand makingvalid interpretations of learningneeds reliableassessmentinformation. But reliable assessmenton itsowndoes not necessarily mean that it is assessing what you intend to assess(i.e.,itis valid).Reliability then is a necessary but not sufficient condition for validity.Achieving fairness in assessment may impact on both reliability and validity. Thus, optimal assessments balance validity, reliability and fairness to best suit the purposes of the assessment.

References

Darr, C. (2005). A hitchhiker’s guide to validity,Set, 2, pp. 55-56.

Darr, C. (2005). A hitchhiker’s guide toreliability,Set, 3, pp. 59-60.

Tierney, R. D. (2016). Fairness in educational assessment. In M. A. Peters (Ed.),Encyclopedia of Educational Philosophy and Theory. Singapore: Springer Science+Business Media. DOI 10.1007/978-981-287-532-7_400-1

By Alison Gilmore