Date on Master's Thesis/Doctoral Dissertation

12-2022

Document Type

Doctoral Dissertation

Degree Name

Ph. D.

Department

Counseling and Human Development

Degree Program

Counseling and Personnel Services, PhD

Committee Chair

Valentine, Jeffrey C.

Committee Co-Chair (if applicable)

Immekus, Jason C.

Committee Member

Natesan Batley, Prathiba

Committee Member

Dossett, Dena

Author's Keywords

School climate; practitioner-developed instruments; alignment method; measurement invariance; alidity testing

Abstract

School climate is increasingly recognized by scholars and policymakers as a crucial factor associated with students’ educational experiences. Hence, practitioners endeavor to equitably measure and improve school climate to promote favorable student academic and behavior outcomes. Unfortunately, school climate research is fragmented, and a research-practice gap exists in best scale development and validity testing practices. The result is a proliferation of practitioner-developed school climate measures lacking solid theory-grounding and evidence to support intended score interpretations and uses. In response, Whitehouse et al. (2021) proposed a validity testing framework for practitioner-developed instruments aimed at supporting culturally responsive school climate measurement. Their framework, however, suffers from key limitations regarding the transparency of content validity assessment, breadth of validity evidence reported, and methods used to examine measurement invariance. Therefore, this study sought to replicate and extend their validity testing framework by using a standardized rubric to assess content validity, examining measurement invariance via the alignment method, and analyzing the predictive validity of group mean scores. By applying the extended validity testing framework to a practitioner-developed school climate student survey, the study also aimed to provide useful evidence to a large urban district with respect to the validity of comparing survey scores across Black and White middle school student groups and using scores to inform continuous improvement of student learning. Results suggest the extended framework is superior to the original for obtaining general content, factorial, and predictive validity evidence, and assessing measurement invariance across racial subgroups, provided the number and size of groups are adequate. Findings suggest the district’s middle school student survey is culturally responsive, although it may not sufficiently address all critical school climate dimensions. To improve the survey, the district must settle on a clear definition and taxonomy of school climate to facilitate a program of validity testing, and publicly document all available validity evidence. Future studies should clarify alignment sample size and simulation study requirements and extend the framework to assess additional validity concerns and for use with person-centered approaches.

Download

Share

COinS