Skip to main content

Advertisement

SpringerLink
Log in

Uncovering Patterns in Process Data to Analyze Interactions and Learning Outcomes Within a Computer-Based Learning Environment

  • Published:
Research in Science Education Aims and scope Submit manuscript

Abstract

Computer-based learning environments (CBLEs) are powerful tools to support student learning. Increasingly of interest is the data that is recorded as learners interact with a CBLE. This process data yields opportunities for researchers to examine learners’ engagement with a CBLE and explore whether specific interactions are associated with learner variables, with direct implications for improving learning outcomes and CBLE design. As CBLEs increase in number and complexity, researchers continue to seek more effective strategies to analyze process data. While a variety of strategies are in use, from visualizations to predictive modeling, none yet offer the capabilities to both uncover hidden, meaningful interactions and descriptively analyze those interactions rapidly across the complete data set. This paper details a method that addresses current challenges, and then applies the method to existing data from a prior study which investigated the effects of adding a visual scaffold to a chemistry-based CBLE. Using a biochemical coding approach through a cultural-historical activity theory (CHAT) framework, the method successfully identified 257 unique, meaningful patterns of interaction that were strategically grouped into nine categories of mediated actions. Though no differences in mediated actions were observed between learners in the experimental (visual scaffolds) and control conditions, three mediated actions were significantly and positively associated with higher learning outcomes in the visual scaffold condition. The results not only provide insight into why the addition of visual scaffolds led to higher learning outcomes among participants but have broader implications for filling a gap in the field of process data analytics for CBLEs in science education.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data Availability

All the materials on the Molecules & Minds website, including the simulations and graphics, are available for use free of charge under the Creative Commons Attribution-NonCommercial 3.0 Unported License.

References

  • Almasri, F. (2022). Simulations to teach science subjects: Connections among students’ engagement, self-confidence, satisfaction, and learning styles. Education and Information Technologies, 1–21. https://doi.org/10.1007/s10639-022-10940-w

  • Augustyniak, P., & Tadeusiewicz, R. (2006). Assessment of electrocardiogram visual interpretation strategy based on scanpath analysis. Physiological Measurement, 27(7), 597. https://doi.org/10.1088/0967-3334/27/7/004

    Article  Google Scholar 

  • Belland, B. R. (2017). Instructional scaffolding in STEM education. Springer International. https://doi.org/10.1007/978-3-319-02565-0

  • Brady, A. G., Schwartz, R. N., Milne, C. E., Plass, J. L., Homer, B. D., Jordan, T. & Letourneau, S. (2014). Exploring Patterns in Multimedia Interactions: Implications for Student Thinking. Poster presented at American Educational Research Association Annual Meeting, Philadelphia, PA.

  • Chang, Y. K. (2010). Examining metacognitive processes in exploratory computer-based learning environments using activity log analysis. (Doctoral Dissertation). Retrieved from ERIC. (ED519966)

  • CREATE. (2013). Science Simulations, Molecules & Minds project at New York University. create.nyu.edu/mm. Accessed 4 Apr 2016

  • da Silva, R. A., & de Vasconcelos, F. C. G. C. (2022). Learning through chemistry simulations: An analysis of cognitive skill levels. Education and Information Technologies, 27(5), 6967–6987. https://doi.org/10.1007/s10639-022-10911-1

    Article  Google Scholar 

  • Engeström, Y. (2001). Expansive learning at work: Toward an activity theoretical reconceptualization. Journal of Education and Work, 14(1), 133–156. https://doi.org/10.1080/13639080020028747

    Article  Google Scholar 

  • Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191. https://doi.org/10.3758/BF03193146

    Article  Google Scholar 

  • Gasevic, D., Siemens, G., & Rosé, C. P. (2017). Guest editorial: Special section on learning analytics. IEEE Transactions on Learning Technologies, 1, 3–5. https://doi.org/10.1109/TLT.2017.2670999

    Article  Google Scholar 

  • Geden, M., Emerson, A., Carpenter, D., et al. (2021). Predictive student modeling in game-based learning environments with word embedding representations of reflection. International Journal of Artificial Intelligence in Education, 31, 1–23. https://doi.org/10.1007/s40593-020-00220-4

    Article  Google Scholar 

  • Greiff, S., Wüstenberg, S., & Avvisati, F. (2015). Computer-generated log-file analyses as a window into students’ minds? A showcase study based on the PISA 2012 assessment of problem solving. Computers & Education, 91, 92–105. https://doi.org/10.1016/j.compedu.2015.10.018

    Article  Google Scholar 

  • Greiff, S., Niepel, C., Scherer, R., & Martin, R. (2016). Understanding students’ performance in a computer-based assessment of complex problem solving: An analysis of behavioral data from computer-generated log files. Computers in Human Behavior, 61, 36–46. https://doi.org/10.1016/j.chb.2016.02.095

    Article  Google Scholar 

  • Havanki, K. L., & VandenPlas, J. R. (2014). Eye tracking methodology for chemistry education research. In Tools of chemistry education research (pp. 191–218). American Chemical Society. https://doi.org/10.1021/bk-2014-1166.ch011

  • He, Q., Borgonovi, F., & Paccagnella, M. (2021). Leveraging process data to assess adults’ problem solving skills: Using sequence mining to identify behavioral patterns across digital tasks. Computers & Education, 166, 104–170. https://doi.org/10.1016/j.compedu.2021.104170

    Article  Google Scholar 

  • Herrington, D. G., Sweeder, R. D., & VandenPlas, J. R. (2017). Students’ independent use of screencasts and simulations to construct understanding of solubility concepts. Journal of Science Education and Technology, 26, 359–371. https://doi.org/10.1007/s10956-017-9684-2

    Article  Google Scholar 

  • Huey, E.B. (1908). The psychology and pedagogy of reading. The Macmillan Company.

  • Jiang, S., Tatar, C., Huang, X., Sung, S. H., & Xie, C. (2022b). Augmented reality in science laboratories: Investigating high school students’ navigation patterns and their effects on learning performance. Journal of Educational Computing Research, 60(3), 777–803. https://doi.org/10.1177/07356331211038764

    Article  Google Scholar 

  • Jiang, S., Huang, X., Sung, S. H., & Xie, C. (2022a). Learning analytics for assessing hands-on laboratory skills in science classrooms using Bayesian network analysis. Research in Science Education, 1–20. https://doi.org/10.1007/s11165-022-10061-x

  • Johnstone, A. H. (1982). Macro- and microchemistry. School Science Review, 64, 377–379.

    Google Scholar 

  • Leontyev, A. N. (1978). Activity, consciousness, and personality. Prentice Hall. https://www.marxists.org/archive/leontev/works/activity-consciousness.pdf. Accessed 26 Mar 2023

  • Letourneau, S. M., Brady, A. G., Milne, C., Homer, B., Jordan, T., Schwartz, R., & Plass, J. (2013). Effects of controlled experimentation and a problematized narrative on learning outcomes in a chemistry simulation. Poster presented at National Association for Research in Science Teaching Annual Meeting, Rio Grande, PR.

  • Maseleno, A., Sabani, N., Huda, M., Ahmad, R., Jasmi, K. A., & Basiron, B. (2018). Demystifying learning analytics in personalised learning. International Journal of Engineering & Technology, 7(3), 1124–1129. https://doi.org/10.14419/ijet.v7i3.9789

  • Mayer, R.E. (2005). Principles for reducing extraneous processing in multimedia learning: Coherence, signaling, redundancy, spatial contiguity, and temporal contiguity principles. In R. E. Mayer (Ed.), Cambridge handbook of multimedia learning (pp. 183–200). Cambridge. https://doi.org/10.1017/CBO9780511816819.013

  • Milne, C., Homer, B., Schwartz, R., Jordan, T., Khan, M., Chang, Y. K., Ching, D. & Plass, J. (2011, April). Developing narrative scaffolds for use within multimedia chemistry simulations: Challenges and possibilities. Paper presented at American Educational Research Association Annual Meeting, New Orleans, LA.

  • O’Keefe, P. A., Letourneau, S. M., Homer, B. D., Schwartz, R. N., & Plass, J. L. (2014). Learning from multiple representations: An examination of fixation patterns in a science simulation. Computers in Human Behavior, 35, 234–242.

    Article  Google Scholar 

  • Oliver, J. S., Hodges, G. W., Moore, J. N., Cohen, A., Jang, Y., Brown, S. A., Kwon, K. A., Jeong, S., Raven, S. P., Jurkiewicz, M., & Robertson, T. P. (2019). Supporting high school student accomplishment of biology content using interactive computer-based curricular case studies. Research in Science Education, 49(6), 1783–1808. https://doi.org/10.1007/s11165-017-9675-6

    Article  Google Scholar 

  • Park, Y., & Jo, I. H. (2017). Using log variables in a learning management system to evaluate learning activity using the lens of activity theory. Assessment & Evaluation in Higher Education, 42(4), 531–547.

    Article  Google Scholar 

  • Plass, J. L., Milne, C., Homer, B. D., Schwartz, R. N., Hayward, E. O., Jordan, T., Jay Verkuilen, Florrie Ng, Yan Wang, & Barrientos, J. (2012). Investigating the effectiveness of computer simulations for chemistry learning. Journal of Research in Science Teaching, 49(3), 394–419. https://doi.org/10.1002/tea.21008

  • Plass, J.L., Letourneau, S., Milne, C., Homer, B.D., & Schwartz, R.N. (2013a). Effects of visual scaffolds on attention patterns and pupil size in a computer-based simulation. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA.

  • Plass, J.L., Homer, B.D., Schwartz, R.N., Milne, C., Jordan, T. & Yavner, S. (2013b). What you see is what you get: Using visual scaffolds in multimedia simulations. Paper presented at the Annual Meeting for the American Educational Research Association, San Francisco, CA.

  • Scherer, R., Greiff, S., & Hautamäki, J. (2015). Exploring the relation between time on task and ability in complex problem solving. Intelligence, 48, 37–50. https://doi.org/10.1016/j.intell.2014.10.003

    Article  Google Scholar 

  • Slater, S., Joksimović, S., Kovanovic, V., Baker, R. S., & Gasevic, D. (2017). Tools for educational data mining: A review. Journal of Educational and Behavioral Statistics, 42(1), 85–106. https://doi.org/10.3102/1076998616666808

    Article  Google Scholar 

  • Stetsenko, A. (2005). Activity as object-related: Resolving the dichotomy of individual and collective planes of activity. Mind, Culture, and Activity, 12(1), 70–88. https://doi.org/10.1207/s15327884mca1201_6

    Article  Google Scholar 

  • Stieff, M. (2019). Improving learning outcomes in secondary chemistry with visualization-supported inquiry activities. Journal of Chemical Education, 96(7), 1300–1307. https://doi.org/10.1021/acs.jchemed.9b00205

    Article  Google Scholar 

  • Strømme, T. A., & Mork, S. M. (2021). Students’ conceptual sense-making of animations and static visualizations of protein synthesis: A sociocultural hypothesis explaining why animations may be beneficial for student learning. Research in Science Education, 51(4), 1013–1038. https://doi.org/10.1007/s11165-020-09920-2

    Article  Google Scholar 

  • Sullivan, G. M., & Feinn, R. (2012). Using effect size-or why the P value is not enough. Journal of Graduate Medical Education, 4(3), 279–282. https://doi.org/10.4300/JGME-D-12-00156.1

    Article  Google Scholar 

  • Supianto, A. A., Hayashi, Y., & Hirashima, T. (2016). Visualizations of problem-posing activity sequences toward modeling the thinking process. Research and Practice in Technology Enhanced Learning, 11(1), 14. https://doi.org/10.1186/s41039-016-0042-4

    Article  Google Scholar 

  • Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Harvard University Press. https://doi.org/10.2307/j.ctvjf9vz4

    Book  Google Scholar 

  • Wertsch, J. V. (1991). Voices of the mind: A sociological approach to mediated action. Harvard UP.

    Google Scholar 

  • West, J. M., Haake, A. R., Rozanski, E. P., & Karn, K. S. (2006). eyePatterns: Software for identifying patterns and similarities across fixation sequences. In Proceedings of the 2006 symposium on Eye tracking research & applications (pp. 149–154). https://doi.org/10.1145/1117309.1117360

  • Wise, A. F., & Shaffer, D. W. (2015). Why theory matters more than ever in the age of big data. Journal of Learning Analytics, 2(2), 5–13. https://doi.org/10.18608/jla.2015.22.2

  • Wüstenberg, S., Stadler, M., Hautamäki, J., & Greiff, S. (2014). The role of strategy knowledge for the application of strategies in complex problem-solving tasks. Technology, Knowledge, and Learning, 19(1–2), 127–146. https://doi.org/10.1007/s10758-014-9222-8

    Article  Google Scholar 

  • Yamagata-Lynch, L. C. (2010). Activity systems analysis methods: Understanding complex learning environments. Springer Science & Business Media.

  • Zendler, A., & Greiner, H. (2020). The effect of two instructional methods on learning outcome in chemistry education: The experiment method and computer simulation. Education for Chemical Engineers, 30, 9–19. https://doi.org/10.1016/j.ece.2019.09.001

    Article  Google Scholar 

Download references

Acknowledgements

The author gratefully acknowledges access to, and use of, technology, support, facilities, and Gas Laws Simulation from the Molecules and Minds Group at the Consortium for Research and Evaluation of Advanced Technologies in Education (CREATE) Lab.

Funding

This project was funded in part by the United States Department of Education’s (USDOE) Institute of Education Sciences (IES) through grants R305K050140 (Jan Plass [PI], Bruce D. Homer, Trace Jordan, and Catherine Milne [Co-PI’s]) and R305A090203 (Catherine Milne [PI], Jan Plass, Bruce D. Homer, and Trace Jordan [Co-PI’s]).

Author information

Authors and Affiliations

  1. School of Education, Quinnipiac University, Hamden, CT, USA

    Anna G. Brady

Authors
  1. Anna G. Brady
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anna G. Brady.

Ethics declarations

Conflict of Interest

The author declares no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Molecules & Minds was launched in 2005 in New York University’s Steinhardt School of Culture, Education, and Human Development.

This manuscript is an adaptation of Chapter 2 of the following Doctoral Dissertation: Brady, A. G. (2019). Examining How an Online Chemistry Simulation Supports Native (L1) Spanish Speakers at the Secondary-Level (Doctoral dissertation, New York University).

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Brady, A.G. Uncovering Patterns in Process Data to Analyze Interactions and Learning Outcomes Within a Computer-Based Learning Environment. Res Sci Educ 54, 83–100 (2024). https://doi.org/10.1007/s11165-023-10109-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11165-023-10109-6

Keywords

Search

Navigation