Table 2 Summary of evidence about validity of the Personal Statement as a test

 Face validity

A close examination of the test criteria reveals some concern about evidence for validity in terms of the three classic concepts of validity. On content validity there is no evidence of judgement by ‘experts’ on the content or relevance of the criteria used. No case is made for their use as against other important criteria. The criteria used appear appropriate on their face but are not well defined.

On construct validity, the four criteria do seem necessary, appropriate, but no case has been made that they are sufficient, that taken together, they capture the (unspecified) quality of ‘suitability for a nursing career’? They are also broadly conceived and poorly defined making consistent data analysis (marking the criteria) difficult. On the face of it the criteria have been useful. They address some important attributes and there is statistical evidence to suggest, via correlation and factor analysis, that the overall concept assessed is coherent (unidimensional). However, the validity of the concept of using an essay about self in ‘assessment at a distance’ is open to questions about authorship, collaboration, integrity, support, powers of expression in writing etcetera and of subjectivity and unfairness in marking it making it unsatisfactory. The evidence is that the PS can be an unfair test and may simply assess the capacity to write a decent essay in praise of self and not capture anything real about potential as a nurse.

 Content validity

 Construct validity

 Credibility

The PS given its uncertain derivation is not a reliable source of information and this is compounded by loosely defined criteria and subjective interpretation by the marker.

 Fitness

Intuitively the criteria appear relevant, but they are broad and general concepts without clear definition, developmental history, or other justification.

 Robustness

Procedure and calculations are systematic, but it is difficult to be sure of a valid estimate of a candidate’s ability since no attempt can be made when marking the PS to deal with breaches of the assumption of genuine self-report made evident here.

 Reliability

Item reliability: the coefficient of reliability (internal consistency) is acceptable, alpha = 0.77 (HEI 1) 0.78 (HEI 2). Invigilation is not a requirement and there is otherwise no attempt to ensure external consistency of PS completion under similar conditions so undermining dependability. There are several variables confounding use: the UCAS PS may be written for several Providers, sources of guidance and support are many and varied and access to it uneven, the concepts being tested are not well defined and subjective interpretation of the concepts and evaluation of the personal statement make for less reliable standards of marking.

 Integrity

The PS is a free text open response so potentially genuine honest and moral: but advice and assistance are readily available with expert advice, if it can be afforded, so uncertainty exists about the true voice of the applicant while marking appears to assume an applicant to be sole author.

 Representativeness

Standardisation of process and representativeness of the PS is undermined by modest validity and reliability; scores are ordinal not interval or ratio scale, so distributions of scores are layered creating multiple ties when rank ordering candidates.

 Coherence

The broad and general concepts used as criteria for marking the PS are interpreted subjectively and scored differently by tutors: coherence is undermined.

 Transparency

Information about the candidate comes to an unknown extent from co-authorship and guidance of variable quality from other people on how to make a good impression.