初中学业水平考试中固定比例法标准设定的信度分析
|
摘要
旨在综合应用现代测量理论分析我国现有初中学业水平考试中固定分数法标准设定的信度指标。采用分层随机取样设计,分别从我国东中西部地区各选择一个区县,并分别从中随机抽取初三学生3000名,对被试的数学学业水平考试的数据进行分析。综合应用现代测量理论分析学业水平考试标准设定的信度指标,包括经典测量理论的决策一致性系数(kappa)、概化理论的等级线决策信度Φ_λ和项目反应理论的信息量I_θ。研究结果显示,固定比例法标准设定下,决策信度处于0.7左右;等级线决策信度大于0.7,大部分在0.8左右;分界点的信息量大部分低于16。这些结果说明,我国现有的学业水平考试标准设定质量一般,对于毕业和升学的高利害性考试来说需要进一步提高。
The main purpose of this study is to examine the reliability of current standard setting method of Fixed Ratio in academic level examination for Secondary School. Using stratified random sampling design to select three counties from the East,Middle,and West of China respectively,3000 students of each county are chosen. The data from the mathematic Academic Level Examination for Secondary School is used. A comprehensive application of modern measurement theory to analyze the reliability indicators of standard setting,including Decision Consistency Index( Kappa) in Classical Testing Theory(CTT),Cut – score Dependability Φ(λ) in Generalizability Theory(GT),and the amount of information index I(θ) from Item Response Theory. The results show that:(i) the Decision Consistency Index of Academic Level Examination for Secondary School are around 0. 7;( ii) Φ( λ) of the cut-scores is greater than 0. 7,mostly beyond 0. 8;( iii) I( θ) are less than 16 regardless of methods to Fix Score or Fix Ratio. All these results suggest that the quality of existing tests' standards setting is barely satisfactory,and it should be improved for high-stakes examinations.
The main purpose of this study is to examine the reliability of current standard setting method of Fixed Ratio in academic level examination for Secondary School. Using stratified random sampling design to select three counties from the East,Middle,and West of China respectively,3000 students of each county are chosen. The data from the mathematic Academic Level Examination for Secondary School is used. A comprehensive application of modern measurement theory to analyze the reliability indicators of standard setting,including Decision Consistency Index( Kappa) in Classical Testing Theory(CTT),Cut – score Dependability Φ(λ) in Generalizability Theory(GT),and the amount of information index I(θ) from Item Response Theory. The results show that:(i) the Decision Consistency Index of Academic Level Examination for Secondary School are around 0. 7;( ii) Φ( λ) of the cut-scores is greater than 0. 7,mostly beyond 0. 8;( iii) I( θ) are less than 16 regardless of methods to Fix Score or Fix Ratio. All these results suggest that the quality of existing tests' standards setting is barely satisfactory,and it should be improved for high-stakes examinations.
引文
[1]罗照盛.项目反应理论[M].北京:北京师范大学出版社,2012:4-43.
[2]漆书青,戴海崎.项目反应理论及其应用研究[M].南昌:江西高校出版社,1992.
[3]陈平,李珍,辛涛等.标准参照测验决策一致性指标研究的总结与展望[J].心理发展与教育,2011,(02):210-215.
[4]戴海琦.基于项目反应理论的测验编制方法研究[J].考试研究,2006,(04):31-44.
[5]黎光明,张敏强,张文怡.人事测评中的概化理论应用[J].心理科学进展,2013,(01):166-174.
[6]韩宁.评价考试质量的新指标:决策一致性和决策准确性[J].中国考试(研究版),2008,(06):3-6.
[7]教育部关于基础教育课程改革实验区初中毕业考试与普通高中招生制度改革的指导意见[J].中华人民共和国教育部公报,2005,(04):38-41.
[8]教育部办公厅关于印发《国家基础教育课程改革实验区2004年初中毕业考试与普通高中招生制度改革的指导意见》的通知[J].中华人民共和国教育部公报,2004,Z1:70-73.
[9]漆书青,周骏,张青华等.用信息函数法对标准参照测验作质量分析[J].心理与行为研究,2003,(01):34-39.
[10]沈玉顺.中招考试制度改革若干政策问题分析[J].华东师范大学学报(教育科学版),2014,(3):26-30.
[11]涂冬波,蔡艳.信息函数在标准参照测验中的应用研究[J].江西师范大学学报(自然科学版),2005,(02):167-172.
[12]肖永琴.目前中考理化学科评价体系的调查与分析[J].福建基础教育研究,2011,(05):106-109.
[13]徐远征.对普通高中学业水平考试命题技术的初步探讨[J].课程·教材·教法,2013,(02):104-108.
[14]徐敏,黄光扬.从考试信度角度解析中考等级制[J].中小学管理,2006,(06):25-27.
[15]杨志明,张雷.改进普通话测试的概化理论分析[J].湖南师范大学教育科学学报,2003,(01):76-82.
[16]杨志明,标准参照测验及其等级线信度的概化理论分析[J].心理学探新,2003,(3):52-57.
[17]周彩莺,沈启正,季芳.普通高中学业水平考试命题研究(二)——难度控制技术探究[J].教育测量与评价(理论版),2013,(10):35-38.
[18]张雨强,魏梦其.初中毕业生学业考试的市域比较研究[J].教育参考,2015,(05):28-34+53.
[19]杜佳萱,陈平,辛涛.基于IRT的决策一致性系数在大规模教育测量中的应用[J].北京师范大学学报(自然科学版),2015,(06):643-648.
[20]陆一萍.HSK高等考试信度的多元概化理论研究[J].中国考试,2011,(05):20-23.
[21]李建平.解析初中毕业学业考试改革新思路[N].中国教育报,2005,4(4).
[22]AERA,APA,&NCME.Standards for Educational and Psychological Testing[M].Washington,DC:Author,1999:35-36.
[23]Brennan R.L.Generalizability Theory[M].New York:Springer-Verlag,2001.
[24]Brennan,R.L.Manual for BB-CLASS:A Computer Program that Uses the Beta-Binomial Model for Classification Consistency and Accuracy Version 1.1[M].Iowa City,IA:Iowa Testing Programs,University of Iowa,2004.
[25]Crick,J.E.&Brennan,R.L.Manual for GENOVA:A Generalized Analysis of Variance System(American College Testing Technical Bulletin No.43)[M].Iowa City,IA:ACT,Inc,1983.
[26]Fischer,G.H.,&Molenaar,I.W.(Eds.).Rasch Models:Foundations,Recent Developments and Applications[M].New York:Springer-Verlag,1995.
[27]Hambleton,R.K.,&Pitoniak,M.J.Setting Performance Standards[A].In:R.L.Brennan(Ed.).Educational Measurement(4th Ed.)[M].Washington,DC:American Council on Education,2006:433–470.
[28]Hanson,B.A.,&Brennan,R.L.An Investigation of Classification Consistency Indexes Estimated under Alternative Strong True Score Models[J].Journal of Educational Measurement,1990,27:345-359.
[29]Impara,J.C.,&Plake,B.S.A Comparison of Cut Scores Using Multiple Standard Setting Methods[R].Paper presented at the Meeting of the American Educational Research Association,New Orleans,LA,2000.
[30]Lee,W.C.Classification Consistency and Accuracy for Complex Assessments Using Item Response Theory(CASMA Research Report No.27)[R].Iowa City,IA:Center for Advanced Studies in Measurement and Assessment,The University of Iowa,2008.
[31]Linn,R.L.Performance standards:Utility for different uses of Assessments[R].Education Policy Analysis Archives,11(31).Retrieved August 13,2010.
[32]Livingston,S.A.,&Lewis,C.Estimating the Consistency and Accuracy of Classifications Based on Test Scores[J].Journal of Educational Measurement,1995,32:179-197.
[33]Sireci,S.G.Standard Setting Using Cluster Analysis[A].In:C.J.Cizek(Ed.)Standard Setting:Concepts,Methods,and Perspectives[M].Mahwah,NJ.:Lawrence Erlbaum Associates,Inc.,2001.
[34]Subkoviak,M.J.Decision-consistency Approaches[A].In:R.A.Berk(Ed.),Criterion Referenced Measurement[M].Baltimore:Johns Hopkins University Press,1980.
[35]Wu,M.L,Adams,R.L,Wilson,M.R.,&Haldane,S.A.Manual for ACER Con Quest version 2.0,Australia[M].ACER PRESS,2007.
[36]Wyse,A.E.The Issue of Range Restriction in Bookmark Standard Setting[J]Educational Measurement:Issues and Practice,Summer,2015,Vol.34,No.2:47-54.
[37]Nedelsky,L.Absolute Grading Standards for Objective Tests[J].Educational&Psychological Measurement,2010,14:3-19.
[38]Jaeger,R M.An Iterative Structured Judgment Process for Establishing Standards on Competency Tests:Theory and Application[J].Educational Evaluation&Policy Analysis,1982,4(4):461-475.
[39]Baker,F.B.,&Kim,S.H.Item Response Theory:Parameter Estimation Techniques,(2nd eds.)[M].New York:Marcel Dekker,2004.
[40]Yen,W.M.,&Fitzpatrick,A.R.Item Response Theory[A].In:R.L.Brennan(Ed.),Educational Measurement(4th ed.,pp.111–153)[M].Westport,CT:Praeger,2006.
[2]漆书青,戴海崎.项目反应理论及其应用研究[M].南昌:江西高校出版社,1992.
[3]陈平,李珍,辛涛等.标准参照测验决策一致性指标研究的总结与展望[J].心理发展与教育,2011,(02):210-215.
[4]戴海琦.基于项目反应理论的测验编制方法研究[J].考试研究,2006,(04):31-44.
[5]黎光明,张敏强,张文怡.人事测评中的概化理论应用[J].心理科学进展,2013,(01):166-174.
[6]韩宁.评价考试质量的新指标:决策一致性和决策准确性[J].中国考试(研究版),2008,(06):3-6.
[7]教育部关于基础教育课程改革实验区初中毕业考试与普通高中招生制度改革的指导意见[J].中华人民共和国教育部公报,2005,(04):38-41.
[8]教育部办公厅关于印发《国家基础教育课程改革实验区2004年初中毕业考试与普通高中招生制度改革的指导意见》的通知[J].中华人民共和国教育部公报,2004,Z1:70-73.
[9]漆书青,周骏,张青华等.用信息函数法对标准参照测验作质量分析[J].心理与行为研究,2003,(01):34-39.
[10]沈玉顺.中招考试制度改革若干政策问题分析[J].华东师范大学学报(教育科学版),2014,(3):26-30.
[11]涂冬波,蔡艳.信息函数在标准参照测验中的应用研究[J].江西师范大学学报(自然科学版),2005,(02):167-172.
[12]肖永琴.目前中考理化学科评价体系的调查与分析[J].福建基础教育研究,2011,(05):106-109.
[13]徐远征.对普通高中学业水平考试命题技术的初步探讨[J].课程·教材·教法,2013,(02):104-108.
[14]徐敏,黄光扬.从考试信度角度解析中考等级制[J].中小学管理,2006,(06):25-27.
[15]杨志明,张雷.改进普通话测试的概化理论分析[J].湖南师范大学教育科学学报,2003,(01):76-82.
[16]杨志明,标准参照测验及其等级线信度的概化理论分析[J].心理学探新,2003,(3):52-57.
[17]周彩莺,沈启正,季芳.普通高中学业水平考试命题研究(二)——难度控制技术探究[J].教育测量与评价(理论版),2013,(10):35-38.
[18]张雨强,魏梦其.初中毕业生学业考试的市域比较研究[J].教育参考,2015,(05):28-34+53.
[19]杜佳萱,陈平,辛涛.基于IRT的决策一致性系数在大规模教育测量中的应用[J].北京师范大学学报(自然科学版),2015,(06):643-648.
[20]陆一萍.HSK高等考试信度的多元概化理论研究[J].中国考试,2011,(05):20-23.
[21]李建平.解析初中毕业学业考试改革新思路[N].中国教育报,2005,4(4).
[22]AERA,APA,&NCME.Standards for Educational and Psychological Testing[M].Washington,DC:Author,1999:35-36.
[23]Brennan R.L.Generalizability Theory[M].New York:Springer-Verlag,2001.
[24]Brennan,R.L.Manual for BB-CLASS:A Computer Program that Uses the Beta-Binomial Model for Classification Consistency and Accuracy Version 1.1[M].Iowa City,IA:Iowa Testing Programs,University of Iowa,2004.
[25]Crick,J.E.&Brennan,R.L.Manual for GENOVA:A Generalized Analysis of Variance System(American College Testing Technical Bulletin No.43)[M].Iowa City,IA:ACT,Inc,1983.
[26]Fischer,G.H.,&Molenaar,I.W.(Eds.).Rasch Models:Foundations,Recent Developments and Applications[M].New York:Springer-Verlag,1995.
[27]Hambleton,R.K.,&Pitoniak,M.J.Setting Performance Standards[A].In:R.L.Brennan(Ed.).Educational Measurement(4th Ed.)[M].Washington,DC:American Council on Education,2006:433–470.
[28]Hanson,B.A.,&Brennan,R.L.An Investigation of Classification Consistency Indexes Estimated under Alternative Strong True Score Models[J].Journal of Educational Measurement,1990,27:345-359.
[29]Impara,J.C.,&Plake,B.S.A Comparison of Cut Scores Using Multiple Standard Setting Methods[R].Paper presented at the Meeting of the American Educational Research Association,New Orleans,LA,2000.
[30]Lee,W.C.Classification Consistency and Accuracy for Complex Assessments Using Item Response Theory(CASMA Research Report No.27)[R].Iowa City,IA:Center for Advanced Studies in Measurement and Assessment,The University of Iowa,2008.
[31]Linn,R.L.Performance standards:Utility for different uses of Assessments[R].Education Policy Analysis Archives,11(31).Retrieved August 13,2010.
[32]Livingston,S.A.,&Lewis,C.Estimating the Consistency and Accuracy of Classifications Based on Test Scores[J].Journal of Educational Measurement,1995,32:179-197.
[33]Sireci,S.G.Standard Setting Using Cluster Analysis[A].In:C.J.Cizek(Ed.)Standard Setting:Concepts,Methods,and Perspectives[M].Mahwah,NJ.:Lawrence Erlbaum Associates,Inc.,2001.
[34]Subkoviak,M.J.Decision-consistency Approaches[A].In:R.A.Berk(Ed.),Criterion Referenced Measurement[M].Baltimore:Johns Hopkins University Press,1980.
[35]Wu,M.L,Adams,R.L,Wilson,M.R.,&Haldane,S.A.Manual for ACER Con Quest version 2.0,Australia[M].ACER PRESS,2007.
[36]Wyse,A.E.The Issue of Range Restriction in Bookmark Standard Setting[J]Educational Measurement:Issues and Practice,Summer,2015,Vol.34,No.2:47-54.
[37]Nedelsky,L.Absolute Grading Standards for Objective Tests[J].Educational&Psychological Measurement,2010,14:3-19.
[38]Jaeger,R M.An Iterative Structured Judgment Process for Establishing Standards on Competency Tests:Theory and Application[J].Educational Evaluation&Policy Analysis,1982,4(4):461-475.
[39]Baker,F.B.,&Kim,S.H.Item Response Theory:Parameter Estimation Techniques,(2nd eds.)[M].New York:Marcel Dekker,2004.
[40]Yen,W.M.,&Fitzpatrick,A.R.Item Response Theory[A].In:R.L.Brennan(Ed.),Educational Measurement(4th ed.,pp.111–153)[M].Westport,CT:Praeger,2006.