Hae sivustolta

Reflections of online assessment between BEng programmes at the dawn of digitalization - A case study of online assessment of mathematics programmes in two partner institutions

Some students think that online assessment is not a true reflection of their work effort. Students have preconceived ideas about assessment, may be uncomfortable with new forms of assessment, and are unsure about how they should interact with online assessment systems. Perceptions of effort and reward as seen by students is at variance with those held by teachers. A brief insight is offered into the thinking of students at first year of study in engineering.

Introduction

Mathematics and its associated subjects are fundamental elements of engineering education and proficiency in the area is expected. Engineers are required to be analytical and be able to utilize their mathematical toolkit to solve problems that may be ill or well defined depending on the contextual situation of the engineer. Until quite recently the determination of learning was undertaken using face to face techniques such as hand written assessments, private and public communication and observation to name but a few. Assessment and programme delivery underwent a seed change with the new millennium when Educational Authorities and Professional bodies adapted their validation methods to include Learning Outcomes within programmes of study. The assessment techniques within the programmes altered accordingly to address these requirements. In addition the forces applied to programme designers to engage with online learning methods have meant that many programmes of study now have an online presence. Anecdotal evidence gathered from longitudinal observation, of issues affecting students, has led to a PhD study within Letterkenny Institute of Technology to examine the learning processes of online testing of engineering students under the supervision of University of Glasgow. The initial focus of the study commenced with first year engineering students of the Letterkenny Institute of Technology. 

Partnership and teacher exchanges engaged within the Erasmus programme and the resulting communication of teaching ideas between lecturers has resulted in an extension of the study to include the University of Applied Sciences of Oulu. As partner organisations, there is an established agreement for collaboration and knowledge of similarities and differences in their respective areas of study. The study paths in Oulu University of Applied Sciences and Letterkenny Institute of Technology are described to provide information regarding the contextual situation of the respective programmes leading to the degree of bachelor of engineering. The theoretical framework in curriculum issues is restricted to the teaching of mathematics within the engineering degree programmes in Oulu and Letterkenny. A description of the education systems of both countries is provided to establish an understanding of the frameworks of the research. 

To enable educational comparisons to be made between Letterkenny and Oulu, the International Standard Classification of Education (ISCED) has been used. ISCED maps the national educational systems of member states within the United Nations according to the standard. This standard, ensures cross-comparability, provides tools for organizing educational information, utilizes common indicators and analysis and standard interpretation of data. The mappings for Oulu University of Applied Sciences and Letterkenny Institute of Technology are provided in figures 1 and 3 in the ISCED United Nations Educational, Scientific and Cultural Organization. Institute for Statistics, ISCED. 2015. ISCED mappings. Last accessed online 2.2.2016. http://www.uis.unesco.org/Education/ISCEDMappings/Pages/default.aspx framework.

Background to Bachelor of Engineering at Oulu University of Applied Sciences (OUAS)

The Finnish education system provides a couple of different paths to higher education as described in figure 1. The majority of students making applications to OUAS from the upper secondary schools have mainly mastered their matriculation exams. If an applicant opted for a vocational route, and obtains a profession after finishing their vocational studies, he/she is also qualified to apply  for higher degree education. A variety of criteria is applied to every degree programme to select the new students without discrimination, as there is a restricted amount of entries to every field of study Ministry of Education and Culture. 2016. Education. Last accessed online 2.2.2016. http://minedu.fi/OPM/Koulutus/?lang=en. It is worth mentioning that success in the matriculation exam does not provide for any particular occupation or profession – it is intended that upper secondary schools give the permission to apply in higher degree education.

FIGURE 1. Finnish Education System Oulu University of Applied Sciences. 2016. Studying and applying. Bachelor Programmes. Last accessed online 2.2.2016. http://www.oamk.fi/en/studies-and-applying/studying-ouas/curricula/bachelor/

The foundation of the Finnish education system is a comprehensive school for the whole age groups. This provides a nine-year basic education. After completing basic education, students mainly choose upper secondary education. This means a path to a general education or to a vocational education. It is worth mentioning, that children usually go to a preceding pre-primary education, which is voluntary.

Student selection to upper secondary schools is done based on the leaving certificate of the basic education. On the other hand, the student selection in vocational institutions can be determined differently. It can include for example work experience and some other comparable factors, such as entrance or tailored aptitude tests.

In upper secondary schools the syllabus is designed to last three years, but depending on the student’s studying pace, it takes from two to four years to cover the curriculum. The upper secondary school ends with a matriculation exam. After successfully mastering the matriculation exams, students are awarded with a detailed certificate indicating the grades and levels studied. Vocational institutions aim to offer vocational qualifications and further and specialist qualifications. The latter two of these are intended for mature students. Vocational studies constitute of three years of study. In each qualification at least a half-year practical trainee period is included. This on the job learning is done in work and therefore it grants to the student a proper, realistic professional qualification. In vocational education for adults it is possible to obtain a qualification by a competence-based approach. This provides an excellent way to maintain, enhance and obtain new skills needed. A specific benefit of this system is that it offers a nice way to recognise and accept an individual’s vocational competences regardless of whether they have been acquired through work experience, studies or other activities Finnish National Board of Education. 2016. Last accessed online 29.3.2016. http://www.oph.fi Ministry of Education and Culture. 2016. Education. Last accessed online 2.2.2016. http://minedu.fi/OPM/Koulutus/?lang=en.

Focusing on mathematics education, students in upper secondary schools may complete a long course or short course in mathematics. When the amount of mathematics is considered in comparison to that given during the vocational studies, the difference in the applicant’s skills in mathematics can be distinctive. In this perspective this will provide very interesting information when comparisons with the Irish system of education are made in future research. This project has not attempted to differentiate between students who have completed the short or long course in mathematics.

Mathematics for engineering students in Oulu 

The amount of mathematics taught is not constant in every branch of engineering in Oulu. The subjects of the first year of study have a high degree of commonality, however, some field-specific related issues exist. The overview of the courses in mathematics of a student in the degree programme in Electrical and Automation Engineering is shown in Table 1.

TABLE 1. The structure of studies in mathematics in OUAS Oulu University of Applied Sciences. 2016. Studying and applying. Bachelor Programmes. Last accessed online 2.2.2016. http://www.oamk.fi/en/studies-and-applying/studying-ouas/curricula/bachelor/

Course	Content	Credits
Mathematics 1	Algebra. Equations and inequalities. Linear algebraic equations. Trigonometry and trigonometric functions. Vectors. Matrices. Complex numbers.	4
Mathematics 2	Functions of one variable. Partial derivatives. Differential and integral calculus. Applications.	4
Mathematics for automation engineering 1 / Mathematics 3	Integral transforms. Series. Difference equations. Applications.	4
Mathematics and Physics Based Project for Engineers	The knowledge gathered in the introductory courses will be pursued further. The contents of the course vary. The main purpose is to focus on the different phenomena of mathematics and physics which are natural to electrical and automation engineering.	3+3

The curriculum described in table 1 is for those students who started their studies in fall 2015. The last course in the curriculum is a mixture of two different courses and it will be held for the first time in spring 2017. The mathematics curriculum is evenly spread to the first two years of study.

The Bachelor degree programme in Electrical and Automation Engineering within OUAS is mainly instructed in Finnish.  A total amount of 30 ECTS credits is reserved for studies abroad or studies in English taken in OUAS. The 4-year degree of Bachelor of Engineering in Automation and Electrical Engineering is 240 ECTS points.  Studies are roughly divided into basic and professional studies and to practice periods. The amount of mathematics studied is 15 ECTS points out of total 240 Oulu University of Applied Sciences. 2016. Studying and applying. Bachelor Programmes. Last accessed online 2.2.2016. http://www.oamk.fi/en/studies-and-applying/studying-ouas/curricula/bachelor/.

Background to Letterkenny Institute of Technology

The  education system in Ireland provides several routes to higher education as outlined in figure 2. All students enter compulsory primary education around age 4 or 5 years and proceed to secondary education at level 3 around age 11 or 12 years. Secondary level education is compulsory to Junior Certificate level around age 15. The student will study approximately thirteen subjects of which Irish, English, and Mathematics are compulsory at Junior Certificate. The Junior Certificate is not sufficient for a student to enter Higher education such as University or Institute of Technology. Students may opt to leave school after the Junior Certificate and enter employment, vocational programmes or remain in school until completion of the Foundation Certificate / Ordinary Leaving Certificate / Higher Leaving Certificate around age 18 years. The Higher Leaving Certificate curriculum enables students to study to a greater depth than the Leaving Certificate. 

Matriculation to Higher Education is administered by the Central Applications Office (CAO) Central Applications Office. 2016. Last accessed online 14.4.2016. http://www.cao.ie and points are awarded for grades achieved in the examinations. English must be presented within the subjects studied when applying to Higher Education – the ultimate choice may be dictated by the chosen career path e.g. medicine or veterinary science requires certain subjects to have been studied. Engineering usually requires mathematics to have been studied to at least Ordinary certificate Level and in many cases to Higher certificate Level.

Entry to Level 7 Ordinary Bachelor of Engineering programmes at LYIT requires students to have obtained minimum grade OD3 (a pass grade at Ordinary Certificate Level of between 40 % and 44 %) or better in mathematics in the Leaving Certificate. In comparison, entry to University Level 8 Honours Bachelor of Engineering  may require the student to have obtained a minimum grade HC3 (Higher Leaving Certificate Level of between 70 % and 80 %).

A student presenting alternative qualifications such as vocational studies or who is returning to education as a mature student will also be able to participate in Higher Education programmes. Equivalent qualifications deemed to be comparable are available through vocational routes to enable students from vocational programmes to continue to Higher Education. It is recognized that within the Irish Higher Education sector that Institutes of Technology take a higher proportion of mature students than the Universities O’Sullivan, C., Mac an Bhaird, C., Fitzmaurice, O.F. & Ni Fhloinn, E. 2014. An Irish Mathematics Learning support Network (IMLSN) Report on Student Evaluation of Mathematics Learning Support: Insights from a large scale multi-institutional survey. National Centre for Excellence in Mathematics and Science Teaching and Learning. Last accessed 22.4.2016. http://supportcentre.maths.nuim.ie/sites/supportcentre.maths.nuim.ie.mathsnetwork/files/uploads/docs/IMLSNFinalReportPrePrinters.pdf.

FIGURE 2. ISCED equivalence for the Irish Education System United Nations Educational, Scientific and Cultural Organization. Institute for Statistics, ISCED. 2015. ISCED mappings. Last accessed online 2.2.2016. http://www.uis.unesco.org/Education/ISCEDMappings/Pages/default.aspx

The standardised framework of qualifications in Ireland is provided in figure 3 to allow comparability with the Bologna Framework within the European Union.

FIGURE 3. National Framework of Qualifications in Ireland (NFQ) FIGURE 3. National Framework of Qualifications in Ireland (NFQ). In National Framework for Qualifications. 2016. Ireland, Last Accessed online 2.2.2016. http://www.nfq-qqi.com/index.html

Mathematics for Engineering Students in LYIT 

in LYIT, mathematics is taught in every semester of the BEng (ordinary) degree programmes in engineering for all disciplines as shown in table 2. All disciplines receive the same mathematics curriculum and are taught together within a modular system. Mathematics 1, 2, 3, 4, 5 and 6 are core components of the level 7 Ordinary Bachelor of Engineering programmes in Building Services Engineering, Civil Engineering, Computer Engineering, Electronic Engineering, Fire Safety Engineering and Mechanical Engineering. Mathematics 7 and 8 are core components of the level 8 Honours Bachelor of Engineering add-on programmes in Embedded Systems and Mechanical Engineering.

TABLE 2. The structure of the engineering mathematics curriculum in LYIT

Course	Content	Credits
Mathematics 1	Numbers, Trigonometry, Vectors, Algebra, Statistics. Vectors.	5
Mathematics 2	Trigonometry, Vectors, Complex Numbers, Differentiation, Integration.	5
Mathematics 3	Functions & Curves, Sequences & Series, Trigonometry, Differentiation, 2nd order Differential Equations.	5
Mathematics 4	Vectors, Matrices, Integration, Probability.	5
Mathematics 5	Trigonometric & Hyperbolic Functions, Power Series & Root Solving, 1st order Differential Equations, Linear Algebraic Equations.	5
Mathematics 6	Laplace Transforms, Fourier Series, Vector Calculus, Sampling Distributions, Regression & Correlation.	5
Mathematics 7	Logic, Graph Theory, Finite State Machines, Interpolation & Splines, Numerical Differentiation & Integration, Numerical Methods for Differential Equations.	5
Mathematics 8	Events & Discrete Random Variables, Continuous Random Variables, Reliability Theory, Markov Chains, Poisson Process, Queuing Theory.	5

The amount of mathematics studied is 30 ECTS out of a total of 180 points for the Ordinary Bachelor Degree programmes.The amount of mathematics studied is 40 ECTS points out of total 240 ECTS points for the Honours Bachelor Degree programmes. 

Similar to the situation regarding entry to the bachelor degree programmes in OUAS, a student may present an application to LYIT with an alternative educational route that differs, significantly, from the standard route through secondary school. It is possible to leave secondary level education aged 15 years to enter the workplace or engage in a vocational programme. Such non-standard entrants are assessed according to status where mature students enter with minimum age 23 years or if the student has studied a vocational course. Special educational support is offered to these students in the areas of literacy and numeracy. Similar support is also made available to all registered students within LYIT. Such students are described as non-CAO in this document.

Comparison of LYIT and OUAS

The case study shown in table 3 consisted of 127 engineering students in LYIT and OUAS. The groups were almost of equal size however, issues of gender cannot be drawn from this sample because female students account for only 4 from the total of 127. The intention of this study was to obtain an insight into the students’ possible exposure to computer based tests prior to commencing their studies in the higher degree institutions. The study – based on a questionnaire – was designed to reveal any problems, obstacles or support needs, faced by the students when engaging in online testing.

TABLE 3. LYIT and OUAS Group Size profile 2015/16

Group		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	LYIT	67	52.8	52.8	52.8
	OULU	60	47.2	47.2	100.0
	Total	127	100.0	100.0

The groups participating to the study had different kind of underlying distributions. This is due to the education systems of the countries. To perform the analysis, it was necessary to use ISCED as the theoretical framework.

FIGURE 4. CAO points (LYIT) and Matriculation (OUAS) grades on entry to programme 2015/16

The combined data indicates a slightly negative skewness (-0.075) and medium flattening (Kurtosis -1.463). The LYIT data indicates a slightly positive skewness (1.011) and medium peak (Kurtosis 0.222). The OUAS data indicates a negative skewness (-1.371) and medium peak (Kurtosis 0.524). (Figure 4.)

The Bar Chart for LYIT is similar to typical entry to engineering programmes in LYIT since 2012/13. The number of non-CAO entries is on an increasing trend. Of interest is the high level of non-matriculation entry to the programme in OUAS.

One of the striving ideas of the study was to discover student perceptions prior entering the programme. Also it was of interest how they have experienced computer based testing and assessment during the first semester of their studies. 

FIGURE 5. Exposure to Computer Based Testing prior to entry onto programme

The combined data indicates the majority of students (73.2 %) have experienced some form of Computer based Testing prior to entry onto the programme. The LYIT data indicates that 67.2 % of students have experienced some form of Computer based Testing prior to entry onto the programme. The OUAS data indicates that 73.2 % of students have experienced some form of Computer based Testing prior to entry onto the programme. (Figure 5.)

FIGURE 6. Student experiences of Computer Based Testing

The majority of students (64.5 %, n=93) having prior experience of Computer Based Testing indicated a positive experience within the combined data. The question was open and the responses produced four main themes when analysed. LYIT (n=45) 55.5 % of students noted a positive experience. OUAS (n=48) 73.0 % of students noted a positive experience. (Figure 6.)

On the recent years, computer based testing has become more popular. Still, in many cases, there exists several barriers, why students have not had online assesments at all. 

FIGURE 7. Students not having experiences of Computer Based Testing

The majority of students (88.2 %, n=34) stated that the reason they did not have any experience of Computer Based Testing is because all tests were paper based. The question was open and the responses produced three themes when analysed. LYIT (n=22) 95.5 % of students stated that all tests were paper based. OUAS (n=12) 75.0 % of students stated that all tests were paper based. 1 student in LYIT and 1 student in OUAS stated that they did not have access to a computer at home to engage in Computer based Tests. Of interest is that 2 students in OUAS expressed total dislike for Computer based Tests. (Figure 7.)

Questions relating online testing and the support to the students given in the beginning of the studies is a main issue when considering online learning supporting systems. Not so many research is done in this branch of education. Neither on the topic, of students’ thoughts, perceptions and reflections relating to online studies. 

FIGURE 8. What students think about the support and training they receive with the general computer system in their university?

The combined data indicates slightly negative skewness (-0.343) and slightly flattened Kurtosis (-0.294). The majority of students in both groups indicate that the level of support and training received in general regarding use of online systems is acceptable. 80.6 % of LYIT students consider the level of support and training received in general use of computer systems to be moderate or better. 81.7 % of OUAS students consider the level of support and training received in general use of computer systems to be moderate or better. (Figure 8.)

FIGURE 9. What students think about the support they receive when engaging with an online Learning Support System?

The combined data indicates negative skewness (-0.382) and considerable flattening of the profile (Kurtosis -0.688). The profile suggests that the majority of students received considerable support when engaging with an online system. The individual groups reveal some differences in their experiences.

75 % of the LYIT group indicate that they received moderate support or greater. There is a slightly negative skewness (-0.312) and considerable flattening of the normal distribution (Kurtosis -0.955) and large variance (1.342). 98 % of the OUAS group indicate that they received moderate support or greater. There is a negligible skewness (-0.034) and considerable flattening of the normal distribution (Kurtosis -1.297) and reduced variance (0.741). No students in OUAS indicated zero support for use of online Learning Support Systems. This result raises questions about the type and level of support received by LYIT students in the use of the online VLE. (Figure 9.)

FIGURE 10. What students think about the support they receive when engaging with quizzes online?

The combined data indicates negligible skewness (0.053) and flattened Kurtosis (-0.620). 38.8% of the LYIT group suggest they receive little or no support in the general use of online quizzes. In comparison 26.7 % of the OUAS group suggest they receive little or no support in the general use of online quizzes. (Figure 10.)

As the focus of the study was in the learning process of mathematics the following is particularly of importance. The online testing of students in OUAS is in the design and construction stage, whereas in LYIT some testing has been already performed. 

FIGURE 11. What students think about the support they receive when engaging with Mathematics quizzes online?

The combined data is a normal distribution with negligible skewness (-0.33) and flattened tails (Kurtosis=-0.810). 60% of the combined group felt they received moderate levels of support or greater when engaging with Mathematics online quizzes.

The LYIT group responses reveal a normal distribution with negligible skewness (0.048) and slightly flattened tails (Kurtosis=-0.376). 73.1 % of the LYIT group felt they received moderate levels of support or greater when engaging with Mathematics online quizzes. (Figure 11.)

As mentioned earlier, questions relating to online testing and the support given to the students formed one of the groundstones of the study. Student perceptions of confidence and  preparedness to online testing formed the other branch of this study. The main question to be answered was about possible barriers in online testing.

FIGURE 12. Student perceptions of how confident they feel when using Computer Based Tests

The combined data is a normal distribution with negligible skewness (-0.178) and flattened tails (Kurtosis=-0.879). 88.2 % of the combined group perceived their confidence levels as moderate or greater when engaging in computer based tests. 80.6 % of the LYIT group perceived their confidence levels as moderate or greater. 96.7 % of the OUAS group perceived their confidence levels as moderate or greater. (Figure 12.)

FIGURE 13. Student perceptions of how prepared they feel when using Computer Based Tests

Within the combined data 73.4 % of students considered themselves to be moderately prepared or better prepared for engaging with computer based tests. The distribution is normal with skewness=0.215 and a sharp profile (Kurtosis=-0.080). 68.7 % within the LYIT group consider themselves to be moderately or better prepared for engaging with computer based tests. The distribution is normal with skewness=0.067 and Kurtosis=-0.027. 76.7 % within the OUAS group consider themselves to be moderately or better prepared for engaging with computer based tests. The distribution is normal with skewness=-0.102 and Kurtosis=-0.071. The distributions are quite similar, however it suggests that OUAS students feel they are better prepared than LYIT students. (Figure 13.)

FIGURE 14. Student perceptions of the quantity of Computer Based Tests they have engaged with

Within the combined data 52 % of respondents stated they had conducted very few computer based tests. Skewness=0.846 and Kurtosis=0.166. 67 % of LYIT respondents stated they had conducted very few computer based tests. Skewness=1.481 and Kurtosis=2.727. 35 % of LYIT respondents stated they had conducted very few computer based tests. Skewness=0.354 and Kurtosis=-0.539. Disparity in responses from student groups reveal that LYIT students consider the amount of computer based tests they were exposed to prior to the programme was low in comparison with students in OUAS. (Figure 14.)

FIGURE 15. Student perceptions of the barriers experienced while engaging with Computer Based Tests

92.1 % of the combined group felt that at least one barrier existed while engaging with computer based tests. Of note is that 19 % considered there to be a lot of barriers or more. 

The skewness=0.227 and Kurtosis=-0.252. 89.6 % of the LYIT group felt at least one barrier existed while engaging with computer based tests. 20.9 % considered there to be a lot of barriers or more. 95.0 % of the OUAS group felt at least one barrier existed while engaging with computer based tests. 15.0 % considered there to be a lot of barriers or more. (Figure 15.)

Discussion

The students participating in this study do not comprise the complete populations in LYIT or OUAS; however, they are viewed as being representative of each Institution. The study sets the scene for further investigation as both institutions increase the quantity of online assessment within their respective programmes.

A number of questions or issues arising from the study include:

• The training or support received by LYIT students using the VLE is in need of improvement.
• Students in LYIT appear to be less confident when working online compared to OUAS students.
• Students in LYIT consider that more support for Maths quizzes online is needed.
• Students within OUAS appear to feel they are better prepared than students in LYIT for online testing.
• Students in OUAS give the impression that they have experienced more Computer Based Testing than LYIT students prior to their programme.
• Students in LYIT appear to see more barriers to using online testing than OUAS students.

The online testing of students in OUAS is in the design and construction stage. This research will feed into the OUAS decision making process promoting further investigation. In parallel, the research will feed into the design and construction of a potential alternative testing system in LYIT.

References

Picture References

Uusimmat artikkelit