Oulun ammattikorkeakoulu
ePooki 5/2018

The Rhetoric Of Online Testing In Higher Education: A Mixed Methods Longitudinal Case Study

29.1.2018 ::

Metatiedot

Nimeke: The Rhetoric Of Online Testing In Higher Education: A Mixed Methods Longitudinal Case Study

Tekijä: Brown Ken; Hurme Jarkko

Aihe, asiasanat: kokeet (arviointi), korkeakouluopetus, matematiikka, opetusteknologia, oppilasarviointi, verkko-opetus

Tiivistelmä: Online assessment is not a true reflection of their work effort according to some students. This paper reports on a continuous collaborative international project within two higher education institutions to research issues of student perception of effort in online assessment. The very beginning of this longitudinal study offered a brief insight into the thinking of students at first year of study in engineering. It may be suggested that alternative approaches to curriculum and pedagogical design are necessary to alleviate student concerns. The research outputs are forming a baseline for the design of improved programmes utilizing online assessment methods.

Julkaisija: Oulun ammattikorkeakoulu, Oamk

Aikamääre: Julkaistu 2018-01-29

Pysyvä osoite: http://urn.fi/urn:nbn:fi-fe2017120855533

Kieli: englanti

Suhde: http://urn.fi/URN:ISSN:1798-2022, ePooki - Oulun ammattikorkeakoulun tutkimus- ja kehitystyön julkaisut

Oikeudet: Julkaisu on tekijänoikeussäännösten alainen. Teosta voi lukea ja tulostaa henkilökohtaista käyttöä varten. Käyttö kaupallisiin tarkoituksiin on kielletty.

Näin viittaat tähän julkaisuun

Brown, K. & Hurme, J. 2018. The Rhetoric Of Online Testing In Higher Education: A Mixed Methods Longitudinal Case Study. ePooki. Oulun ammattikorkeakoulun tutkimus- ja kehitystyön julkaisut 5. Hakupäivä 19.11.2019. http://urn.fi/urn:nbn:fi-fe2017120855533.

The research on Online Testing In Higher Education is an international research project. It emerged three years ago as a result of Erasmus exchange.  The research gathered information from students and lecturers on their respective expectations and perceptions about online assessment of first year engineering mathematics. These results presented here were first recognized on ECER 2017 conference in Copenhagen. It provides the latest findings from a two year longitudinal study conducted by Letterkenny Institute of Technology (LYIT) and Oulu University of Applied Sciences (OUAS) and is a continuation to the report Hurme, J. & Brown, K. 2016. Reflections of online assessment between BEng programmes at the dawn of digitalization - A case study of online assessment of mathematics programmes in two partner institutions. ePooki. Oulun ammattikorkeakoulun tutkimus- ja kehitystyön julkaisut 12. Hakupäivä 8.12.2017. http://urn.fi/urn:nbn:fi-fe201604049008..

Kuva: Shutterstock

Photo: Ditty_about_summer / Shutterstock.com

Research Question and Research Framework

The application of computer aided assessment (CAA) is widely employed across many domains in Higher Education. It is suggested that computer aided methods of assessment offer several benefits in Higher Education such as reduction in lecturer workload for large class groups and the possibility of a higher degree of consistent objectivity in the assessment process. The increasing sophistication offered by CAA systems makes the use of such technologies attractive for assessment purposes. The assessment develops a communication with the student where feedback and estimation of performance are the main components. Research focusing on the assessment technologies Jordan, S. 2013. E-assessment: Past, present and future. New Directions 9 (1), 87–106. Sangwin, C. 2012. Computer Aided Assessment Of mathematics Using STACK. 12th International Congress on Mathematical Education, South Korea. Narciss, S., Sosnovsky, S., Schnaubert, L., Andrès, E., Eichelmann, A., Goguadze, G. & Melis, E. 2014. Exploring feedback and student characteristics relevant for personalizing feedback strategies. Computers & Education 71, 56–76. suggests growing maturity in the domain of mathematics and the promise of greater availability of such technologies to lecturing staff. The benefits and positive aspects of online assessment have been outlined in the literature for engineering applications Rasila, A. & Zenger, K. 2007. Automatic assessment of mathematics exercises – Experiences and future prospects. ResearchGate. Accessed online 19.9.2017. http://www.researchgate.net/publications/228382934 Cox, M.J. 2012. Formal to informal learning with IT: research challenges and issues for e-learning. Journal of Computer Assisted Learning 29 (1), 85–105., immediate feedback Lamberti, F., Sanna, A., Paravati, G. & Carlevaris, G. 2014. Automatic Grading of 3D Computer Animation Laboratory Assignments. IEEE Transactions on Learning Technologies 7 (3), 280–90. Hamilton, I.R. 2009. Automating formative and summative feedback for individualised assignments. Campus-Wide Info Systems 26 (5), 355–64., online quizzing Jordan, S. 2013. E-assessment: Past, present and future. New Directions 9 (1), 87–106. and formative assessment Tempelaar, D.T., Niculescu, A., Rienties, B., Gijselaers, W.H. & Giesbers, B. 2012. How achievement emotions impact students' decisions for online learning, and what precedes those emotions. The Internet and Higher Education 15 (3), 161–9.. The growth of Computer Algebra Systems (CAS) Sangwin, C. 2012. Computer Aided Assessment Of mathematics Using STACK. 12th International Congress on Mathematical Education, South Korea. extends the mechanics of computer aided assessment systems by introducing greater capability to consider concepts that are too abstract for lower order testing methods; for example, Multiple Choice Questioning. The importance of the role of assessment technologies cannot be understated and their encapsulation within Higher Education curricula is becoming an accepted element of programme design.

In Ireland, the mathematics curriculum was redesigned Prendergast, M., Breen, C., Carr, C. & Faulkner, F. 2016. Investigating Third Level Lecturers' Awareness of Second level Curriculum Reform. 13th International Congress on Mathematical Education. in an attempt to address reported deficiencies in the mathematical skillsets of students at second level. A review of the mathematics provision McCraith, B. 2015. Average is no longer good enough – it’s time for a step change in STEM education in Ireland. In Education Matters, Yearbook 2016–2016, 13–18. raised further issues about the skillset with particular reference to the transition from second level to third level, the use of Information and Communications Technologies, and international comparison. Discussion with lecturing colleagues in a similar Higher Education institution in Finland revealed similar concerns Kinnari, H. 2010. A study of the mathematics proficiency. 1st Intl Workshop on Maths and ICT: Education, Research and Applications. Bucharest, 35–39. Rinneheimo, K. 2010. Methods for teaching mathematics case Tampere University of Applied Sciences. 1st Intl Workshop on Maths and ICT: Education, Research and Applications. Bucharest, 48–55. Cole, J.S., McCartan, C.D., Tuohi, R. & Steinby, P. 2014. Mathematics background of engineering students in Northern Ireland and Finland. Proceedings of 10th International CDIO Conference. June 16 th–19th, Universitat Politecnica de Catalunya, Barcelona, Spain..

This paper presents the findings from a two year longitudinal study conducted by the partner Higher Education institutes in Finland and Ireland using a mixed methods approach beginning in academic year 2015/2016. The research gathered information from students and lecturers on their respective expectations and perceptions about online assessment of first year engineering mathematics. The basis for the research was anecdotal observation of negative behavioural attributes, prior to, and shortly after, online assessment. Evidence within the literature suggests these attributes may be deeply embedded in students as they make the transition to third level Gallimore, M. & Stewart, J. 2014. Increasing the impact of mathematics support on aiding student transition in higher education. Teaching Mathematics and its Applications 33 (2), 98–109., leading to greater support requirements in third level Gill, O., Mac An Bhaird, C. & Ni Fhloinn, E. 2010. The Origins. Development and Evaluation of Mathematics Support Services. Irish Maths Society 66, 51–63.. The research outputs are forming a baseline for the design of improved programmes utilizing online assessment methods. The interaction of students and staff with the technologies is vital to the success of these programmes, as is an understanding of perceptions held by students Tempel, T. & Neumann, R. 2014. Stereotype threat, test anxiety, and mathematics performance. Social Psychology of Education 17 (3), 491–501. in relation to barriers and how students engage online. 

The research was designed within the boundaries of a social-cognitive approach, to self-efficacy theory, within engineering mathematics Bandura, A., 1977. Self-efficacy: Toward a Unifying Theory of Behavioral Change. Psychological Review 84 (2), 191–215. Marchand, G.C. & Gutierrez, A.P. 2012. The role of emotion in the learning process: Comparisons between online and face-to-face learning settings. The Internet and Higher Education 15 (3), 150–60. Artino, A.R. & Jones, K.D. 2012. Exploring the complex relations between achievement emotions and self-regulated learning behaviors in online learning. The Internet and Higher education 15 (3), 170–175., as the core concept, to help the researchers gain a better understanding of the online assessment experiences, prior to and post assessment. The premise is that the actions and reactions of the students are influenced by their own observations and experiences; these pre-existing attributes are determining factors of self-efficacy and awareness of learning. The study of lecturers as a parallel activity was conducted to determine if any mismatch exists between the expectations and perceptions of lecturers and those of students. 

Method

The longitudinal study commenced in April 2015 with a short pilot questionnaire and the main study commenced in October 2015. The research has been conducted within the domain of engineering mathematics for students in their first year of study of a Bachelor of Engineering programme. A mixed methods approach was designed to validate the findings from the study. Data was collected from the students in both countries simultaneously at the end of October in each year by means of a questionnaire. The questionnaire contained dichotomous questions, six-point Likert scale questions and two open questions. The cohort for this questionnaire was n=186 in academic year 2015/2016, and n=158 in 2016/2017, respectively. An Interpretative Phenomenological Analysis of the response to the open questions within the questionnaires formed the basis of qualitative group interviews with students (n=50). The primary focus of the student interviews was students’ self-efficacy, confidence, and barriers, whilst engaging with online assessment. All student interviewees were asked the same questions to ensure comparability between responses. Utterances by the students were analysed and a schema developed for coding. The code was altered once a thematic change was detected. The topics were revisited for the second year of the study and found to be representative and relevant. A further group interview of students in Finland is planned for early 2017 to ensure balance within the data.

In parallel with the student group interviews, a series of individual interviews were conducted with lecturers from Ireland (n=6) and Finland (n=2). The questions for this qualitative process were developed from the thematic outputs, derived from the student responses to questions, in the group interviews, and open questions from the questionnaire. The interview questions were formed around themes of confidence, barriers, and optimal assessment. 

A rigorous process of ethical scrutiny was engaged in by all Higher Education institutions involved in the research prior to the commencement of the research.

Conclusions

Analysis of the responses, to the quantitative questionnaire, and qualitative interviews of students, reinforces the initial findings reported at ECER 2016; they were still pertinent in the second year within the longitudinal study. At the commencement of study, many first year engineering mathematics students reported feelings of low confidence and high levels of anxiety in relation to online assessment. The experience of online assessment for mathematics was new for the vast majority of students in both countries and this is reflected in the comments:

hipsut_oranssi.png"I was a lot better than I thought I would be."

Evidence of anxiety came through in the interviews revealing that anxiousness was carried through from second level affecting self-confidence in the initial stages of study. Confidence levels improved as the students progressed to the second year of study however, the negative anxiety attributes transferred to other forms of assessment outside of the scope of this study. 

hipsut_oranssi.png"I am frustrated and panicked because I feel that I am all over the place."

Comments relating to negative experiences, linked to online assessment, reduced considerably, from 28% in year 1 of study to 8% in year 2, as students became more familiar with techniques. Issues raised, relating to the current online assessment system, continue to demonstrate that students consider the system to be deficient in providing a fully objective analysis of students’ performance, and ability, because partial credits are not provided for Ashton, H.S., Beevers, C.E., Korabinski, A.A. & Youngson, M.A. 2006. Incorporating partial credit in computer-aided assessment of mathematics in secondary education. Br J Educ Technol, 37 (1), 93–119. Sangwin, C., Hunt, T. & Butcher, P. 2013. Embedding and enhancing eAssessment in the leading open source VLE, The Higher Education Academy.

The challenge of abstract questioning is an area to be addressed in the second stage of this research project in both institutions.

References

    Kommentit

    blog comments powered by Disqus