Date of Graduation

8-2024

Document Type

Dissertation

Degree Name

Doctor of Philosophy in Educational Statistics and Research Methods (PhD)

Degree Level

Graduate

Department

Counseling, Leadership, and Research Methods

Advisor/Mentor

Boykin, Allison

Committee Member

Liang, Xinya

Second Committee Member

Robinson, Samantha

Third Committee Member

Turner, Ronna C.

Keywords

Bayesian simulation; Reliability; Standard error of measurement (SEM); Standardized tests; Teachers; Value-added models (VAMs)

Abstract

Abstract Background: Value-added models (VAMs) are statistical tools used to gauge a teacher’s impact on student performance by analyzing standardized test scores. These models project students’ future performance based on past scores and compare the projection to actual outcomes, accounting for differences in student backgrounds. However, the standard error of measurement (SEM) inherent in all measurement tools is often overlooked in VAMs. Aims and Objectives: This study aims to investigate the impact of test reliability on teacher and school score estimates within a Bayesian framework. We will precisely manipulate the reliability of standardized tests by adjusting the standard error of measurement (SEM) and compare the resulting estimates with those generated by the Tennessee Value-Added Assessment System (TVAAS) Composite score, a widely used VAM. Methods: Employing a Monte Carlo simulation design, we explore how varying levels of test SEM, sample size at the classroom level, number of years or grades, and number of teachers affect estimates of teacher effectiveness produced by TVAAS. By systematically altering these parameters, we assess their individual and collective influence on the accuracy and reliability of teacher effectiveness measures. Results: Our analysis indicates that higher levels of test reliability correspond to more precise measures of teacher effectiveness and rankings. Additionally, we highlight SEM’s significant role in shaping value-added modeling outcomes. Furthermore, we identify the interplay between factors such as sample size, number of grades, and intraclass correlation (ICC) in determining the reliability and accuracy of teacher effectiveness measures and rankings. Conclusion: Our findings underscore the importance of considering not only the reliability parameter but also the number of teachers involved in the TVAAS model. The study illuminates the complex dynamics in estimating teacher effectiveness and emphasizes the need for a comprehensive understanding of factors influencing VAM outcomes. Keywords: VAMs, standardized tests, reliability, SEM, Bayesian

Available for download on Wednesday, September 10, 2025

Share

COinS