Main Article Content


The present paper aims at presenting the Critical Thinking (CT) Skills assessment results in teachers participating in the Erasmus+ KA203 CRITHINKEDU summit (Critical Thinking Across the European Higher Education Curricula), organised in Leuven in June 2019. Within the summit, a workshop was organized to promote in participants’ CT skills knowledge, especially in terms of CT assessment methods through open-ended questions. Based on our theoretical assumptions, description and interpretation activities of written text promote skills such as Analysis, Argumentation, Inference and Critical evaluation, which can also be defined in terms of improvement of language skills. Teachers participating in the workshop were assessed through a test composed by literary text paraphrase and commentary exercises; a prototype for the automatic assessment of CT in open-ended answers was used to evaluate the open-answers. Also three human raters evaluated the answers’ texts. The goal of the present research was to verify the assessment method reliability and to collect some data useful for the implementation of the automatic prototype.


Critical Thinking Evaluation Automatic Assessment Open-Ended Questions Writing Activities

Article Details

How to Cite
Re, M. R., Amenduni, F., De Medio, C., & Valente, M. (2019). How to use assessment data collected through writing activities to identify participants’ Critical Thinking levels. Journal of E-Learning and Knowledge Society, 15(3), 117-133.


  1. Benedetti, F., Beneventano, D., & Bergamaschi, S. (2016, October). Context semantic analysis: a knowledge-based technique for computing inter-document similarity. In International Conference on Similarity Search and Applications (pp. 164-178). Springer, Cham.
  2. Bloom H. (2000). How to Read and Why. New York: Touchstone.
  3. Bovi, C. D., Telesca, L., & Navigli, R. (2015). Large-scale information extraction from textual definitions through deep syntactic and semantic analysis. Transactions of the Association for Computational Linguistics, 3, 529-543.
  4. Byrnes J. P., & Dunbar K. N. (2014). The nature and development of critical-analytic thinking. Educational Psychology Review, 26(4), pp. 477-493.
  5. Dominguez C. (Ed.) (2018a). A European collection of the Critical Thinking skills and dispositions needed in different professional fields for the 21st century. Vila Real: UTAD.
  6. Dominguez C. (Ed.) (2018b). A European review on Critical Thinking educational practices in Higher Education Institutions. Vila Real: UTAD.
  7. Dominguez C. (Ed.) (2018c). The CRITHINKEDU European course on critical thinking education for university teachers: from conception to delivery. Vila Real: UTAD.
  8. Elen, J., Jiang L., Huyghe S., Evers M., Verburgh A., & Palaigeorgiou, G. (2019). Promoting Critical Thinking in European Higher Education Institutions: towards an educational protocol. Vila Real: UTAD.
  9. Ennis R.H. (1993). Critical Thinking Assessment. Theory into Practice, 33(3), pp. 179-186.
  10. Ennis R. H., & Weir E. (1985). The Ennis–Weir critical thinking essay test. Pacific Grove (CA): Midwest Publications.
  11. Esplugas C., & Landwehr M. (1996). The Use of Critical Thinking Skills in Literary Analysis. Foreign Language Annals, 29(3), pp. 449-461.
  12. Facione P. A. (1990a). Executive summary of ‘The Delphi Report’. Millbrae (CA): The California Academic Press.
  13. Facione P. A. (1990b). The California Critical Thinking Skills Test. Millbrae (CA): California Academic Press.
  14. Facione N. C., Facione P. A., & Sanchez C. A. (1994). Critical thinking disposition as a measure of competent clinical judgment. The development of the California Critical Thinking Disposition Inventory. Journal of Nursing Education, 33(8), pp. 345-350.
  15. Fleiss, J. L., & Cohen, J. (1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 33, 613–619
  16. Gordon, T. F., Prakken, H., & Walton, D. (2007). The Carneades model of argument and burden of proof. Artificial Intelligence, 171(10-15), 875-896.
  17. Halpern D. F. (2007). Halpern critical thinking assessment using everyday situations. Background and scoring standards. Claremont (CA): Claremont McKenna College.
  18. Johnson R. H., & Hamby B. (2015). A meta-level approach to the problem of defining ‘Critical Thinking’. Argumentation, 29(4), pp. 417-430.
  19. Ku K. Y. (2009). Assessing students’ critical thinking performance. Urging for measurements using multi-response format. Thinking skills and creativity, 4(1), pp. 70-76.
  20. Liu O. L., Frankel L., & Roohr K. C. (2014). Assessing critical thinking in higher education. Current state and directions for next‐generation assessment. ETS Research Report Series, 2014(1), pp. 1-23.
  21. Liu, O. L., Brew, C., Blackmore, J., Gerard, L., Madhok, J., & Linn, M. C. (2014). Automated scoring of constructed response science items: Prospects and obstacles. Educational Measurement: Issues and Practice, 33, 19–28. doi:10.1111/emip.12028
  22. Mao, L., Liu, O. L., Roohr, K., Belur, V., Mulholland, M., Lee, H. S., & Pallant, A. (2018). Validation of automated scoring for a formative assessment that employs scientific argumentation. Educational Assessment, 23(2), 121-138.
  23. Moore T. (2013). Critical thinking: Seven definitions in search of a concept. Studies in Higher Education, 38(4), pp. 506-522.
  24. Newman D. R., Webb B., & Cochrane C. (1995). A content analysis method to measure critical thinking in face-to-face and computer supported group learning. Interpersonal Computing and Technology, 3(2), pp. 56-77.
  25. Paul R.W., & Elder L. (2006). Critical Thinking Reading & Writing Test. Tomales (CA): The Foundation for Critical Thinking.
  26. Poce A., Vertecchi B., Agrusti F., Re M. R. (2016). Pen or keyboard. An empirical study on the effects of technology on writing skills. CADMO, 24(2), 33-44.
  27. Poce A. (2017). Verba Sequentur. Pensiero e scrittura per uno sviluppo critico delle competenze nella scuola secondaria. Milano: Franco Angeli.
  28. Powers B. (1989). Illinois Critical Chinking Annual. Champaign (IL): University of Illinois College of Education.
  29. Scott C. L. (2015). The futures of learning 3: What kind of pedagogies for the 21st century. Education Research and Foresight, 15, pp. 1-21.
  30. Serianni L., Della Valle V., & Patota G. (2003). L’italiano parlato e scritto. Agenda salvascrittura. Milano: Bruno Mondadori Editori.
  31. Watson G., & Glaser E. M. (1980). Watson–Glaser critical thinking appraisal. Cleveland (OH): Psychological Corporation.
  32. Wegerif, R., McLaren, B. M., Chamrada, M., Scheuer, O., Mansour, N., Mikšátko, J., & Williams, M. (2010). Exploring creative thinking in graphically mediated synchronous dialogues. Computers & Education, 54(3), 613-621.
  33. World Economic Forum (2016). The 10 skills you need to thrive in the Fourth Industrial Revolution. Retrieved from: