Back to AAIR Journal Volume 16, No. 2

Marked for Success: Secondary School Performance and University Achievement in Biology

Published in Volume 16, No. 2, 30 December 2011

Printer friendly version

Keith Comer, Erik Brogt & Kaylene Sampson
Academic Development Group, University of Canterbury, New Zealand

Submitted to the Journal of Institutional Research July 2011, accepted for publication August 2011.

Abstract

Building on Shulruf, Hattie and Tumen (2008), this work examines the capacity of various National Certificate in Educational Achievement (NCEA)-derived models to predict first-year performance in Biological Sciences at a New Zealand university. We compared three models: (1) the ’best-80‘ indicator as used by several New Zealand universities as a predictor of grade point average (GPA); (2) the ’best-80‘ as a predictor of outcome grade in biology courses and (3) ’domain status‘ in biology and chemistry as a predictor of outcome grade. These models span quantity, quality and competency measures in examining student performance and success at both university-wide and specific disciplinary levels. Results show that the models explain between 25 and 45% of the variance, which may present challenges if one of these models were to become the sole determinant for enrolment and limitation of entry policies, but can be useful in an advisory capacity.

Introduction

This article investigates aspects of the capacity of secondary school awarded university entrance standards to predict first-year university performance for New Zealand secondary school leavers. Identifying productive secondary pathways to university success is of vital interest to educational institutions, students and governmental funding bodies. While this research addresses two years of intake and university performance of students in a single subject discipline, the approach can be applied both with other intake groups and for entire enrollment cohorts.

Employing secondary qualification scores and subsequent university performance for three first-year Biology courses, this research is intended to enhance our understanding of ‘best outcomes’ for first-year students. Such research can be used to advise secondary students, parents and staff and to consider university enrolment guidelines and expectations for success. We first introduce the New Zealand secondary education system with respect to university qualification, then outline our data sources and work through corresponding analyses, and finally offer conclusions.

Sector context

As recently as 2008, New Zealand university funding for undergraduates was based on student numbers (Engler, 2010a). In this model, the more students enrolled at a given institution, the more funding it received for undergraduate education. However, as student enrolments continued to rise, governmental reviews resulted in changes in that funding model’s ‘unlimited nature’. Caps or enrolment limits were introduced, ensuring greater certainty to the national education budget. Despite these caps, in 2008 there remained ample places for students desiring to attend university.

By 2010 the landscape had changed. Partly owing to the global economic downturn, 2009 enrolments increased markedly. Yet these were not matched by additional funding (Todd, 2010). Rather, alongside enrolment caps the government announced additional funding requirements beginning in 2012. These will tie a proportion of funding to indicators that include course passing rates, retention, program completion and progress to higher degrees. Such changes have increased attention on the need for universities to provide well-supported learning environments that enhance student success. These shifts have also triggered national debates about national university entrance (UE) requirements (Binning, 2010).

Combined, the funding changes present significant challenges to the management of undergraduate intake. Many universities are now reaching or exceeding enrolment caps, producing unfunded places and triggering governmental reviews. Consequently, some universities have implemented new enrolment criteria and curtailed or cancelled summer courses. To date, three of eight universities in New Zealand have introduced enrolment requirements based upon students’ secondary school exit qualification rankings—requirements that move beyond UE standards as set by the National Certificate in Educational Achievement (NCEA). By 2011 approximately half of all university enrolments will be so determined (Laugesen, 2010). These restrictions rely on the premise that secondary school performance can accurately predict tertiary success.

NCEA overview

Beginning in 2002, New Zealand introduced a new principal secondary qualification, the National Certificate of Educational Achievement, leading to a 2005 initial intake of NCEA students at university. Standards-based and divided into three levels, NCEA assessments are typically spread across the final three secondary school years. Students select specific subject areas or domains (e.g., Biology, Chemistry, English, etc.) and their knowledge and skills within these are assessed by a range of national standards (performance objectives). Specific standards are worth a predetermined number of possible credits, and these can be awarded at three performance levels: ‘achieved’, ‘merit’, or ‘excellence’. Yet irrespective of level of attainment, the number of credits a student is awarded for a particular standard is identical. In the consideration of NCEA performance for national standard for university entrance, both a student awarded ‘excellence’ and a student earning ‘achieved’ receive the same number of university intake credit points.

Currently, students must achieve a requisite number of credits from an approved list of subjects to attain the national university entrance (UE) standard. As of 2010, the New Zealand requirements for UE mandate that a student earn 42 credits at NCEA level 3 or higher. Of these 42 credits, at least 14 credits must be attained in each of two domains (subject areas), with the additional 14 credits earned from no more than two other domains.

Data

Two independent variables and two dependent variables have been used in the analysis contained within this article. The two independent (or predictor variables) are Best-80 rank scores and domain status. The dependent variables are students’ grade point average (GPA) and outcome grades awarded in the papers of interest. A brief definition and the derivation of each are outlined below.

The Best-80 rank scores model

The ‘Best-80 rank scores model’ is currently being employed for admissions consideration at the University of Auckland and other New Zealand universities. In the calculation of a Best-80 score for a student, the following process is employed (Shulruf, Hattie, & Tumen, 2008):

• Select the best five domains from the list of university-approved domains.
• Per domain, only 24 credits can be counted toward Best-80 score; in practice, most domains have a maximum of 24 achievement standards, but a variable number of unit standards.
• Credits awarded with ‘excellence’ have a weight of 4, ‘merit’ a weight of 3, ‘achieved’ of 2 and unit standards of 2 (if applicable).
• A maximum of 80 credits out of the five domains are counted toward the Best-80 score.

Calculating Best-80 scores is accomplished through a software-automated process that selects each domain for which a student has credits (out of a potential 40 ‘university-approved’ subjects) and then calculates the Best-80 score for each possible combination. The maximum score is then used as each individual student’s Best-80 score. Because university entrance requires only two domains of 14 credits of university-approved standards, the theoretical range of Best-80 is 56–320 (derived from a possible minimum 28 credits at ‘achieved’ level to possible maximum 80 credits at ‘excellence’ level).

Domain status in science-specific subjects

Currently, full domain status indicates whether students have received at least 14 credits at NCEA Level 3 with a performance level of ‘achieved’ or higher in the given domain. This aligns with the requirements to be awarded UE via NCEA. Students who have used some credits to make up part of their third domain are not considered as having domain status in the following discussion.

Outcome grades

Student records provide a means to examine first-year performance. Yet, as with regard to many majors, this undertaking is complicated since some students have enrolled in more than one of the three possible Biology courses and many take all three. Furthermore, some students may have enrolled in the same course in consecutive years; in such cases, only the first attempt (i.e., the initial enrolment grade) was used to represent performance. This university’s alphabetic scale marks from A+ to F were converted to numeric grades (Table 1). With respect to the cut-off percentages indicated (Table 1), these grades can be considered interval data (except with regard to tail-ends). A minor complication arising from grading policies at this university exists with the C- mark, which is locally regarded as a ‘non-continuing pass’ and is rarely awarded (only two C- grades were assigned to NCEA-qualified students in this sample).

Table 1: Letter Grades Awarded at the University of Canterbury’s School of Biological Sciences Based on the Cut-Off Percentages Indicated, and the Numeric Grade Scale They Were Converted to for This Analysis

Grade point average (GPA)

Grade point average (GPA) is also used as a dependent variable. GPAs are generated by the university. They are based on the grades a student is awarded and are weighted by the number of credits each course is worth. The GPA scale ranges from -1 through to 9 (converted to 1–11 above).

Analysis

In considering the combinations of variables and the levels of measurement of data, a variety of analysis methods have been employed. These are contained in Table 2 and detail the principal comparisons between secondary preparation and performance and first-year university performance. ‘Pathway-specific’ refers to whether the data used as independent variable (or predictor) or dependent variable (or outcome) is specific to the disciplinary pathway of interest–Biology. For example, in A below neither Best-80 nor GPA are specific to Biology, whereas in the case of C both secondary pathway and outcome grade in Biology papers have been used in analysis and both are specific to the discipline of interest.

Table 2: Variables, Measurements, Analysis and Pathway Specificity of Tests

(A) Best-80 vs. GPA (overall GPA for all courses)–Is there a relationship between the Best-80 credits secondary students are awarded and their overall performance in first-year university?

In examining whether secondary school preparation—and specifically, that preparation more directed towards university entrance standards—is associated with first-year university performance, we have considered two years of NCEA intake and first-year GPA for students enrolled in the three Biology courses. Table 3 contains correlation statistics for these students. As shown there is some association between GPA and Best-80, yet the strength of that relationship is relatively weak.

To determine if student performance in Biology across two enrolment years could be considered equivalent with respect to overall preparation at secondary level, an independent sample t test was conducted on the mean number of total credits students undertook in 2007 and 2008. Since no significant differences between the two mean scores (df = 446, t = 1.36, p = .32) arose, the datasets from the two years were merged for the following analysis.

Table 3: Best-80 and First-Year GPA (all Courses) Correlation

During these two years, neither the curriculum nor the assessments were substantially changed. Consequently, the grades in the two years are considered as equivalent, with differences in outcome grades attributable to the students (rather than any variation in assessment). When both years of enrolments are combined, N = 485, R² = 37, r = .60, and p < .001. The scatter plot in Figure 1 illustrates the spread of secondary preparation results and first-year university performance for the combined grouping of students.

Figure 1: Scatter plot of ‘Best-80’ by GPA for all 2007 and 2008 students enrolled in BIOL 111, 112 and 113

(B) Best-80 vs. outcome grade–Is there a (stronger) relationship between the Best-80 credits secondary students are awarded and their performance in specific courses in first-year university?

This step of analysis tests whether Best-80 is a better predictor of performance in specific papers (in this case BIOL 111–113, where the dependent variable is pathway-specific) rather than overall performance. Analysis uses Best-80 credits and outcome grades in the specific courses of BIOL111–113 to determine the strength of relationship. The test statistic, rs (Spearman’s rho) has been used because it is the appropriate measure of association that can accommodate ordinal level of measurement (individual grades). The Best-80 by outcome grade in the three courses (combined for each year) is shown in Table 4.

Table 4: Best-80 and Biology outcome grade correlations 

The first observation to note is that, in contrast to predicting overall GPA, Best-80 is a poorer predictor of performance in BIOL111–113. This is most likely due to the fact that GPA is a continuous variable whereas outcome grade is ordinal, resulting in more scatter in the latter case.

(C) Pathways vs. outcome grades BIOL111, 112, 113 specific x specific measure–Is there a (stronger) relationship between particular pathways + performance at the secondary level and performance in associated courses in first-year tertiary?

This third analysis factors the pathway-specific experiences for students in terms of preparation for university (domain status) and the outcome in related courses (BIOL111–113) to determine strength of relationship. In Table 5, outcome grades have been grouped into cohorts, ranging from F = failing grade through to A. The A, B and C grades include minus and plus as well as straight letter grades. Below each table section are the chi square test statistics and Spearman’s rho, using the convention rs < .2 as ‘weak’, .2 ≤ rs ≤ .4 as ‘moderate’, and rs > .4 as ‘strong’ and p values.

Table 5: NCEA Pathways and BIOL111–113 outcome grades

Apparent from the previous table is the importance of Chemistry for student success across all three papers. For BIOL 111 and 113, having domain status in secondary Chemistry significantly improves chances of a student earning B or A grades in first year, and to a lesser extent the same holds true in BIOL 112. Students who undertake secondary Chemistry may well be better prepared to engage in the first-year Biology curriculum. With respect to those students who have domain status in secondary Biology a similar but lesser effect is observed. The Biology pathway has a weak relationship with outcome in 111 and 113 and only a moderate predictive capacity for 112.

Yet this analysis at best shows a relationship of association–it does not indicate causation, and there may well be other effects involved (see also Engler, 2010b, pp. 4–5). For example, students who take Chemistry may be generally brighter than students who take Biology. Likewise there may be hidden effects from the pathway combination students take. That is, students who take Chemistry and Biology at the secondary level may be different to those that take only one or the other, as the remainder analysis examines.

ANCOVA analysis

To ascertain if domain status is a determining factor in the outcome grade, it is necessary to control for the number of credits a student takes. This was done in a two-step process. In the first step we examined the linear relationships of the Best-80 score and the outcome grade for the four different groups (summarised in Table 6).

Table 6: Pearson Correlation Coefficients Between Outcome Grade in the Biology Courses and Best-80 for the Four Groups of Students Varying in Domain Status

The non-significant results for Chemistry only in BIOL112 and BIOL113 may be due to the small size of the group. It is also worth noting that these correlations are systematically higher than those found by Brogt, Sampson, Comer, Turnbull, and McIntosh (2011), who used the total number of NCEA credits rather than a Best-80 score.

In the second step, groups that had significant relationships between outcome grade and Best-80 score for a course were examined using ANCOVA analysis, with group membership as a factor and the Best-80 score as the covariate. An interaction between the Best-80 score and group membership was built into the model to test if the slopes of the relationships between Best-80 score and grade were homogeneous. Cases where the interaction was not statistically significant (at the .05 level) were removed from the model, allowing the variance to be absorbed into the main effects. In addition, a lack of fit test showed non-significant results for all three courses, meaning that a linear fit is acceptable. The results of the ANCOVA analysis for each course are described in the following section.

BIOL111

For BIOL 111 the total credits covariate was significantly related to student outcome grades (Table 7). After controlling for the covariate, group membership was found to be significant as well (Table 8). The pair-wise comparisons between the groups showed that students who had taken Biology and Chemistry did significantly better than those who had taken Biology only. However, no significant differences were found between Biology and Chemistry vis-à-vis Chemistry only, or between Chemistry only and Biology only. The overall corrected models explained around 38% of the observed variance.

Table 7: Summary Statistics for ANCOVA Snalysis of BIOL111 

Table 8: Pair-Wise Comparisons Between Groups for BIOL111

* Means are calculated based on a Best-80 score of 212.33.

BIOL112

The total credits covariate was also significantly related to student outcome grades in BIOL 112 (Table 9). However, after controlling for the covariate, group membership was not a statistically significant predictor for outcome grade. The overall corrected model explained about 36% of the observed variance.

Table 9: Summary Statistics for ANCOVA Analysis of BIOL112

BIOL113

The total credits covariate was significantly related to student outcome grades again in BIOL 113 (Table 10). Similar to BIOL 111, after controlling for the covariate, group membership also had a significant effect on grades in BIOL 113 (Table 11). The overall corrected models explained around 45% of the observed variance. The interesting feature in this model is the significant interaction of Best-80 score * group membership. This means that the slopes of the two regression lines are significantly different. We interpret this to indicate that in the case of BIOL 113 having Chemistry as an NCEA domain has more impact on first-year student outcomes than having other subjects. The interaction highlights the importance of Chemistry in Biology.

Table 10: Summary statistics for ANCOVA analysis of BIOL113 

Table 11: Pair-Wise Comparisons Between Groups for BIOL113

Discussion

Alongside similar Australian research (Green, Brown, & Ward, 2009), this analysis indicates that, even in the cases of apparently straightforward disciplinary pathways from secondary school, predicting first-year student success is challenging, with unexpected results. For example, for Biology students at university domain status in secondary school Chemistry holds greater predictability for first-year success than other domains. In considering ‘depth’ versus ‘breadth’ of studies at secondary level, such findings may illustrate differences with research from other sectors (Schwartz, Sadler, & Tai, 2008) that notes possible counterproductive consequences for secondary ‘breadth’ with potential university Biology students.

Shulruf, Hattie and Tumen (2008) argue for a shift in emphasis on intake to quality, with university admissions based up ‘higher achievement in fewer credits’. In this data sample, mandating that students earn 18 credits of NCEA level 3 Biology and 18 of Chemistry (vs. 14 for domain status currently) could increase their chances of success in two of the first-year courses. Yet nearly 40% of current students would be excluded through such requirements. While the mean first-year passing rate of an overall intake can be closely predicted (Scott, 2008), individual passing rates are less easily predicted, even in the case of relatively straightforward disciplinary pathways.

Conclusions

All stakeholders involved—including students, secondary schools, universities and national policymakers—desire clear and evidence-based secondary pathways to university entrance and, preferably, achievement. Yet basing admissions and predicting individual success on the basis of NCEA secondary performance remains a problematic exercise. For incoming students pursuing a university Biology curriculum, there is evidence for the importance of Chemistry as a required secondary school pathway. However, there is relatively low predictive capacity for first-year performance even when NCEA domain status is achieved. Consequently, policies that institute blanket requirements based on secondary performance or specific NCEA pathways prior to university entrance may well deny enrolment to many students with a good chance of success. In the case of potential disciplinary pathway requirements, it should also be noted that some disciplines have reasonably clear pathways from secondary to tertiary (like Biology and Chemistry for Biological Sciences), whereas others do not (e.g., Psychology). If a clear secondary pathway can be identified, an analysis like the one presented in this study can be a useful tool for individual departments in advising secondary students as to the best preparation for tertiary success in that discipline.

References

Binning, E. (2010, June 18). Top brains only–varsities cut class sizes. The New Zealand Herald. Retrieved from http://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=10652668

Brogt, E., Sampson, K., Comer, K., Turnbull, M., & McIntosh, A. (2011). Using institutional research data on tertiary performance to inform departmental advice to secondary students. Journal of Institutional Research, 16(2), 26–41.

Engler, R. (2010a). Academic performance of first-year bachelors students at university. Wellington, New Zealand: Tertiary Sector Performance Analysis and Reporting, Strategy and System Performance, Ministry of Education.

Engler, R. (2010b). Are particular school subjects associated with better performance at university? Wellington, New Zealand: Tertiary Sector Performance Analysis and Reporting, Strategy and System Performance, Ministry of Education.

Green, R., Brown, E., & Ward, A. (2009). Secondary school science predictors of academic performance in university bioscience subjects. Anatomical Sciences Education, 2(3), 113–118.

Laugesen, R. (2010, April 3). Making the grade: Getting into university is becoming more competitive—and could get more expensive. Listener, pp. 26–29.

Todd, R. (2010, January 9). Canterbury University gets tough on failures. The Press. Retrieved from http://www.stuff.co.nz/the-press/news/3216892/Canterbury-University-gets-tough-on-failures

Schwartz, M.S., Sadler, P.M., & Tai, R.H. (2008). Depth versus breadth: How content coverage in high school science courses relates to later success in college science coursework. Science Education, 93(5), 798–826.

Scott, D. (2008). How does achievement at school affect achievement in tertiary education? Wellington, New Zealand: Tertiary Sector Performance Analysis and Reporting, Strategy and System Performance, Ministry of Education.

Shulruf, B., Hattie, J., & Tumen, S. (2008). The predictability of enrolment and first-year university results from secondary school performance: The New Zealand National Certificate of Educational Achievement. Studies in Higher Education, 33(6), 685–698.

 

This paper was first presented at the Annual Conference of the Australasian Association for Institutional Research, Has institutional research come of age in Australia? in Geelong, 10–12 November, 2010.

Correspondence to:
Dr Erik Brogt
Academic Development Group
University of Canterbury
Private Bag 4800, Christchurch 8140
E: erik.brogt@canterbury.ac.nz