Back to AAIR Journal Volume 16, No. 2

Using Institutional Research Data on Tertiary Performance to Inform Departmental Advice to Secondary Students

Published in Volume 16, No. 2, 30 December 2011

Printer friendly version

Erik Brogt, Kaylene A. Sampson & Keith Comer
Academic Development Group, University of Canterbury, Christchurch, New Zealand

Matthew H. Turnbull & Angus R. McIntosh
School of Biological Sciences, University of Canterbury, Christchurch, New Zealand

Submitted to the Journal of Institutional Research July 1, 2011, accepted for publication July 25, 2011

Abstract

This article examines the use of institutional research data on tertiary academic success of students in the first-year Biology program at the University of Canterbury in relation to their secondary school performance in English, Mathematics with Statistics, Biology and Chemistry. This study was commissioned by the School of Biological Sciences to examine the validity of the advice they gave to secondary students considering studying biology at university and was carried out as a joint venture between institutional researchers and departmental academics. We found that students with higher overall first-year university biology performance were more likely to also have taken Chemistry at secondary school. Controlling for overall performance, students taking both Chemistry and Biology as domains for the New Zealand University Entrance qualification (UE) did significantly better in two out of three first-year biology courses than those who had taken only one or neither subject as a domain. The extent of the advantage depended on the type of course; being greatest in the biochemistry-related course and least in ecology-related. We concluded that the advice the School of Biological Sciences had been giving students in secondary school as to the best preparation for (first-year) university studies in biology (emphasising the need to take both the subjects of Biology and Chemistry) was consistent with the institutional performance data of first-year students at university.

Context

Determining secondary pathways to success at university is of vital interest for educational providers, students and governmental funding agencies. Previous research has linked success in university science degrees with science-rich backgrounds at the secondary level. Green, Brown and Ward (2009) found that secondary school science was a strong predictor of university performance in bioscience subjects where clear connections can be made from secondary to tertiary subject matter. In the New Zealand context, Shulruf, Hattie and Tumen (2008) determined that success in university can be enhanced by the pursuit of related disciplinary pathways at secondary level. Schwartz, Sadler, Sonnert and Tai (2008) examined the ‘depth versus breadth’ question with respect to secondary Biology, Chemistry and Physics and subsequent success in tertiary science courses, finding that breadth in secondary experience did not significantly advantage students in their tertiary success, though teaching for depth was positively associated with improved tertiary performance.

Despite these findings, a substantial proportion of New Zealand universities as yet do not have disciplinary prerequisites to enrolment, even in disciplines that have obvious academic pathways at the secondary level. For example, the Biological Sciences program at the University of Canterbury encourages students to pursue Biology, Chemistry and Mathematics in secondary school, but this is not a formal prerequisite. This trend is slowly shifting, and having the capacity to advise students how to prepare for successful tertiary studies would be beneficial to students and the university alike.

The aim of this research was to find suitable minima of pathway requirement to assist academic staff in their advisory role to secondary teachers and students as they devise appropriate secondary course selection. We evaluated student performance in 3 first-year core Biology courses as a function of their previous performance in selected subjects at secondary school to determine the degree to which the advice the School of Biological Sciences is currently giving to secondary students and staff can be substantiated with data.

In the next section we introduce the New Zealand secondary education system, followed by a brief description of the first-year Biology courses under investigation. Subsequently, we discuss our data sources, analyses and draw conclusions.

New Zealand’s secondary system

In 2002, the National Certificate of Educational Achievement (NCEA) was introduced as the principal secondary qualification in New Zealand, with the first university intake of NCEA students occurring in 2005. NCEA is divided into three levels, spread across the last three years of secondary school. Students choose domains (subject areas like Biology, Chemistry, English, etc.) which are subdivided into standards (performance objectives). Each standard is worth a certain number of credits that can be endorsed with ‘achieved’, ‘merit’, or ‘excellence’. Standards can be assessed internally (by the school) or externally (via a central examination), depending on the standard. For example, in the domains of Biology and Chemistry, practical lab work is examined internally while theoretical work is examined externally.

To obtain University Entrance (UE), students must (currently) achieve 42 credits at NCEA Level 3 from an approved list of subjects. This includes 14 credits in each of two domains, with a further 14 credits spread over no more than two additional domains. In practice, many students considerably exceed this minimum requirement. For a more detailed discussion on NCEA in New Zealand, see Shulruf, Hattie and Tumen (2008, 2010).

First-year Biology core curriculum

The School of Biological Sciences of the University of Canterbury (hereafter referred to as the Department) offers three core courses in the first year: BIOL111, 112 and 113. All three courses use a combination of lecture and laboratory sessions and are required for students wishing to major in Biology:

• BIOL111 is an introduction in cell biology and biochemistry, with the cellular nature of life as the core theme. The course covers concepts of cell functions and their relation to cell structure, the structure of biologically relevant molecules and the biochemical interactions between molecules. In addition, the course covers the structure and function of biomolecules, metabolism, cell organisation and cell division.
• BIOL112 covers basic principles of evolution and genetics, including the origin, maintenance and loss of genetic variation; ecology and the determinants of the distribution and abundance of organisms, species interactions and food web ecology.
• BIOL113 presents an overview of the diversity, evolutionary relationships, structure and function of animals, plants and microbial life on earth. The first half of the course focuses on the diversity, reproduction and structure of bacteria, protists, fungi and plants. The second half examines animals and includes discussion of locomotion, respiration, and circulation, nervous and endocrine systems.

Data

Data collection was influenced by earlier focus group interviews with second-year students. In these interviews, students reported difficulties with the statistics requirements of the first-year courses and the writing requirements of laboratory reports. Students attributed the latter to insufficient preparation in English at secondary school. As a result, we decided to examine secondary student performance in Mathematics with Statistics and English in addition to Biology and Chemistry, which were suggested by the Department.

We retrieved NCEA scores for 448 individual students enrolled in the first-year Biology courses in 2007 and 2008 who had obtained UE through NCEA with at least 42 credits in UE-approved subjects. These students are a subset of the total number of students enrolled in the courses, as students can obtain UE through other means than NCEA. As can be inferred from Table 1, students routinely took more than one course, with many taking all three courses.

Table 1: Number of Students in First-Year Biology Courses With UE via NCEA

Final grades in the three core courses (BIOL111, BIOL112 and BIOL113) were taken from student records. In cases where students took a course multiple times, only the first attempt was used to most closely represent performance based on secondary school preparation. The alphabetic grade of A+ to E (based on cut-off percentages) was converted to a number (Table 2). The grades can, with the exception of the tail ends, be considered interval data, especially because the C- grade (a non-continuing pass) is rarely given (only two students in our sample received this grade).

Table 2: Letter Grades Awarded at the University Of Canterbury’s School of Biological Sciences Based on the Cut-Off Percentages Indicated, and the Numeric Grade Scale They Were Converted to For Our Analysis

Analysis

To determine whether there were differences in general academic preparation between students in 2007 and 2008 who obtained UE through NCEA, we performed an independent sample t test on the mean number of total credits taken by students in these two years. As this test yielded no significant differences between the two mean scores (df = 446, t = 1.36, p = .32), we merged the datasets from the two years. In 2007 and 2008, the curriculum and the assessments in the Biology courses were not changed substantially. Consequently, we attributed differences in outcome grades to the students, rather than to assessments.
Subsequent data analysis was split into three parts: a X² analysis; a regression and analysis of covariance; and an analysis at the subdomain level. These are described in detail below.

Influence of domains

Our initial analysis concentrated at the domain level to determine which of the four selected domains (Biology, Chemistry, Mathematics with Statistics and English) had an impact on outcome grades in first-year Biology. Cases were sorted on the basis of whether students had sufficient credits (i.e., 14 or greater) to use this domain toward qualifying for UE, and we then cross-tabulated with outcome grades in level one Biology courses (Table 3). For this analysis, we have combined the failing grades (D and E) into one category (F). Because of the binning involved, we treat the categories as ordinal, rather than interval.

Table 3: Chi-Square Analysis for the Domains of Biology, Chemistry, Math With Statistics and English by Outcome Grade in the Biology Courses

Note. Plus and minus grades have been collapsed into a single letter grade category. X² indicates the significance of grade on domain completion. Y is the Goodman and Kruskal gamma statistic and indicates the strength of the association between outcome grades and domain completion.

Counter to the anecdotal evidence provided by students, when examined at the domain level, neither English nor Maths with Statistics was a useful predictor of outcome grades in the three 100 level Biology courses. Although the X² analysis showed some statistically significant dependence for English (BIOL111 and 112) and Mathematics with Statistics (BIOL113), the strength of these associations, as demonstrated by the gamma coefficient, were weak. In contrast, all three core courses displayed significant dependence on both Biology and Chemistry domains and showed stronger correlations between having domain status and outcome grades achieved. Consequently, we have focused our attention on Biology and Chemistry in the subsequent analyses to assess their influence on success in the first-year university Biology curriculum.

Biology and Chemistry performance based on domain status

James, Montelle and Williams (2008), examining NCEA results and success in university mathematics, plotted the university mathematics course grade as a function of the number of credits in NCEA mathematics. Because pathways into university Biology are not as clear-cut as pathways into mathematics, we divided the sample instead into four groups based on whether students achieved domain status in Chemistry and/or Biology in secondary school (Table 4).

Table 4: Enrolment, Secondary Credits Passed, and Outcome Grades in BIOL111, 112 and 113 by Domain Eligibility Based on Four Categories: Students With Both Chemistry and Biology as a Domain For University Entrance, Students With Only Chemistry as a Domain, Students With Only Biology as a Domain; and Students With Neither Biology and Chemistry as a Domain

The credit range in Table 4 indicates that students who had both domains, or only Chemistry as a domain, typically had more total level three NCEA credits than students who only had Biology or had neither domain. In addition, students who took both Chemistry and Biology or Chemistry alone, obtained a higher mean outcome grade in all three core courses than those who had Biology only or neither domain. Last, Figure 1 shows a clear absence of data points in the lower right (high numbers of credits and low outcome grades) and, to a lesser extent, in the upper left quadrant (few credits but high outcome grades). These three observations are consistent with the finding that more apt students generally take more standards (Meyer, McClure, Walkey, McKenzie, & Weir, 2006) and that the domain status of Chemistry is associated with higher outcome grades.

Figure 1: Example of the distribution of outcome grades in relation to the total number of NCEA credits achieved. The solid line indicates the linear regression fit. As a result of the resolution on the vertical axis, multiple data points can be represented by a single dot in the figure.

To ascertain if domain status is a determining factor in the outcome grade, it is necessary to control for the number of credits a student takes. This was done in a two-step process. First, we examined the linear relationships of the total number of credits the students took in secondary school and the outcome grade for the four different groups (Table 5).

Table 5: Pearson Correlation Coefficients Between Outcome Grade in Biology Courses and Total Number of NCEA Credits for the Four Groups of Domain Status

* p < .05

Note that the non-significant results for Chemistry only in BIOL112 and BIOL113 may be due to the small size of the group.

Next, we examined groups that had significant relationships between outcome grade and NCEA credits using ANCOVA analysis, with group membership as a factor and the total number of credits as the covariate. We allowed for an interaction between total credits and group membership, but removed the interaction term from the model as no significant interactions were found, allowing the variance to be absorbed into the main effects. In addition, the non-significant result for a lack of fit test for all three courses indicate that linear fits are acceptable. The results of the ANCOVA analysis for each course are described below.

BIOL111

The total credits covariate was significantly related to student outcome grades (Table 6). After controlling for the covariate, group membership was significant as well (Table 7). Pair-wise comparisons between the groups showed that students who had taken Biology and Chemistry did significantly better than those who had taken Biology only. No significant differences were found between Biology and Chemistry, and Chemistry only, and between Chemistry only and Biology only. The overall corrected model explained around 30% of the observed variance.

Table 6: Summary statistics for the ANCOVA analysis for BIOL111

Table 7: Pair-wise comparisons between groups for BIOL111

* Means are calculated based on a Total credit score of 89.48

BIOL112

The total credits covariate was significantly related to student outcome grades (Table 8). After controlling for the covariate, group membership was not a statistically significant predictor for outcome grade. The overall corrected model explained about 31% of the observed variance.

Table 8: Summary Statistics for the ANCOVA Analysis for BIOL112

BIOL113

The total credits covariate was significantly related to student outcome grades (Table 9). After controlling for the covariate, group membership also had a significant effect on grades in BIOL 113 (Table 10). Pair-wise comparisons between the groups showed that students who had taken Biology and Chemistry did significantly better than those who had taken Biology only, and the overall corrected models explained around 33% of the observed variance.

Table 9: Summary Statistics for the ANCOVA Analysis for BIOL113

 

Table 10: Pair-wise Comparisons Between Groups for BIOL113

* Means are calculated based on a Total credit score of 82.14

How much Biology and Chemistry is enough?

The prominence of Chemistry as a domain in the success of student outcomes in two of the three core Biology courses immediately begs the question, ‘how many credits, beyond the domain threshold, are optimum?’ Given it is possible to obtain 24 credits in each of these disciplines, but only 14 credits are required for domain status in UE, should advice to students emphasise a focus on depth over breadth at the secondary level? To answer this, we examined Chemistry and Biology NCEA factors at the subdomain level.

The distribution of outcome grades in the three Biology core courses by number of credits received in each of the two NCEA domains of interest are shown in Tables 11 and 12 below. As multiple cells in the table have small expected values (< 5), no X² analysis could be performed, hence we present only descriptive statistics.

The column percentages indicate that students who take at least 18 credits in these domains have a substantially larger fraction in the B and A grade categories compared to the students who take less than 18 credits. While the upward trend continues for Chemistry in the third category (more than 21 credits), it flattens for Biology. The upward trend in Chemistry was expected for BIOL111, with its strong emphasis on biochemistry, but was not expected for the other courses.

Table 11: Number of NCEA Chemistry Credits by Outcome Grades BIOL111–113

 

Table 12: Number of NCEA Biology Credits by Outcome Grades BIOL111—113

The trend toward more A grades beyond the 18 credit threshold for both domains is especially worth noting, showing that students perform better in the three core courses when they have achieved more credits—irrespective of domain.

Data such as these can be useful to advise students as well as inform policy discussions regarding prerequisites, albeit with the caveats that association does not imply causation, and that setting prerequisites has implications for enrolment numbers. For example, while 18 credits of Biology and 18 of Chemistry are associated with higher rates of success in two of the courses, this does not imply that having 18 credits will yield success. In addition, such a requirement would have excluded about 40% of the students in these courses.

Using a blanket credit threshold score has two additional drawbacks: it does not take into account the endorsement of the credits, nor does it provide assistance in choosing which particular standards are most useful.

Are all standards of equal importance?

While students tend to take more than the required 42 credits, they will operate with a certain level of strategic thinking regarding which standards to take in the final years of secondary school. In providing advice to secondary students, one question that will almost certainly be asked would be ‘what standards do I have to take?’ To this end, this final part of analysis has examined the individual standards in Biology and Chemistry to see if any of them were associated with higher performance in the courses. The NCEA level 3 standards for Chemistry and Biology are listed in Tables 13 and 14, and more details on the standards can be found on the New Zealand Qualifications Authority (NZQA) website.

Table 13: Chemistry Level 3 NCEA Standards

 

Table 14: Biology Level 3 NCEA Standards

As noted earlier, standards can be endorsed with achieved (A), merit (M) and excellence (E). In the NCEA data as provided by NZQA, several other labels also exist: ‘Y’ (absent from exam but scheduled to take it), ‘V’ (did not attempt), ‘N’ (attempted but not achieved) and ‘Z’ (missing exam paper). In our analysis, all these entries were recoded into a single pseudo-endorsement ‘not applicable’ (n/a). In addition, we combined merit and excellence endorsements into a ‘merit or above’ category (M+), as only a small minority (between 5 and 10%) of students receive excellence endorsements. Finally, following the government’s merger of standards 90697 and 90689 (3 credits each) into standard 90780 (5 credits) at the beginning of 2006, we recalculated any 90697 and 90689 credits to 90780 credits using the conversions; M + A = M; N/A + A = A; A + A = A.

Table 15 shows the percentage of students who received a certain endorsement on a standard (n/a, A, M+) and who subsequently earned B or A grades in the courses. Though the actual student numbers are not shown in the table, the distribution of students receiving n/a, A, and M+ is very different for internally assessed standards compared to externally assessed ones, with many more students receiving M+ in the internally assessed standards. Moreover, internally assessed standards can be assessed multiple times, rather than once, as is the norm for externally assessed standards.

Table 15 shows several features. Firstly, for each standard, the percentage of students who did not obtain an ‘achieved’ endorsement yet still earned B or A grades ranges from 40% (Biology standard 90717 in BIOL112) to 70% (Biology standard 90714 in BIOL111). Second, there is a marked increase in the proportion of B or better grades being awarded between ‘not achieved’ and ‘achieved’ endorsements, with this effect being more pronounced in Chemistry than in Biology. Third, the increase in proportion of B and A grades for those receiving M+ is more pronounced in Biology than in Chemistry. Hence there is a role for considering the endorsement alongside the number of credits in the case of Biology.

Table 15: Percentage of Students Receiving a ‘B’ Grade or Better by Level of Achievement in Individual Biology and Chemistry NCEA Standards

This first feature of Table 15 might indicate that no one standard is absolutely crucial to first-year success. To verify this, we asked instructors teaching the 100-level courses to evaluate the details of each standard, as outlined on the NZQA website, for its usefulness in preparation for each course. For BIOL111, all Chemistry standards were considered relevant for BIOL111. In contrast, only standard 90694 (practical work) was considered relevant for BIOL112, and none for BIOL113. In addition, instructors indicated that Chemistry standards 90696, 90698 and 90700 were not necessarily relevant for the core courses, but instead would be useful general knowledge. So while the importance of Chemistry is clear from Table 15, the instructors’ comments provide a more nuanced picture about the relationship between secondary school Chemistry and success in first-year university Biology.

Conclusions

Based on our analyses, we firstly observe that there may be a mismatch between students’ secondary preparation in Mathematics with Statistics and English, and the expectations of the first-year courses. We found no association between preparation in these areas and student performance in the courses. Even students with more than sufficient preparation in those subjects identified difficulties. However, writing laboratory reports and interpreting advanced statistics are not part of the NCEA curriculum, yet they are skills required and assessed in first-year university Biology courses. University teachers will need to consider this when designing both their curriculum and student advice.
Second, our ANCOVA analysis revealed a relationship between the number of credits a student takes and their first-year university performance. Students who perform better in first-year Biology generally have taken more NCEA Level 3 credits. Further, we found that students who had taken Chemistry in addition to Biology performed better in two of the three courses, even though the first-year curriculum does not always focus on Chemistry. In addition, there appears to be a threshold of 18 credits in the domains of Biology and Chemistry, based on the distribution of grades in Tables 11 and 12. Students who obtained 18 credits also had higher proportions of B and A grades compared to students who were in the 14–17 credit band. This implies that domain status only (14 credits) may not be optimal preparation.

In this study, we only considered the total number of credits students obtained. Comer, Brogt, and Sampson (2011) examined endorsements of credits in some detail and found slight improvements in associations. While weighting credits for endorsement (e.g., Shulruf, Hattie, & Tumen, 2010) can be useful from a researcher’s point of view, at the practical level of providing advice to secondary students, it is perhaps more prudent to prioritise the accumulation of a total number of credits over more complicated metrics using weighted endorsements.

Finally, evidence suggests that for some (externally assessed) standards, improvement in endorsement is positively associated with outcome grades. The quality jump occurs first in Chemistry (from n/a to Achieved) and then in Biology (from Achieved to M+). However, based on our data, it is not possible to offer recommendations for students to take specific standards, as around half the students who get an A in the course fell in the ‘n/a’ category.

This work has identified the value of prior disciplinary learning in university success and that the Department’s original advice to secondary students can be tightened. In areas where a clear secondary disciplinary pathway exists, such as Biology, students not taking this pathway as part of their University Entrance will be disadvantaged. Based on these findings, we would give the following advice to secondary students considering studying biology at university:

• Include both Chemistry and Biology as a domain for University Entrance
• Attempt at least 18 credits in each domain
• Focus on externally assessed standards.

Acknowledgments

The authors thank John Edwards of the Planning, Information and Reporting Unit of the University of Canterbury for his assistance in obtaining and cleaning the data, the staff of the School of Biological Sciences teaching in BIOL111, 112 and 113 for their feedback on the NCEA standards and Michael Johnston from the New Zealand Qualifications Authority for his useful comments on the draft of this manuscript. EB thanks Sanlyn Buxner for her feedback about educational statistics. This project was reviewed and approved by the University of Canterbury Human Ethics Committee, proposal number HEC 2009/169.

References

 Comer, K., Brogt, E., & Sampson, K. (2011). Marked for success: Secondary school performance and university achievement in biology. Journal of Institutional Research, 16(2), 42–53.

Green, R., Brown, E., & Ward, A. (2009). Secondary school science predictors of academic performance in university bioscience subjects. Anatomical Science Education, 2(3), 113–118.

James, A., Montelle, C., & Williams, P. (2008). From lessons to lectures: NCEA mathematics results and first-year mathematics performance. International Journal of Mathematical Education in Science and Technology, 39(8), 1037–1050.

Meyer, L, McClure, J, Walkey, F., McKenzie, L., & Weir, K. (2006). The impact of the NCEA on student motivation: Final report to the Ministry of Education. Wellington, New Zealand: Ministry of Education and Victoria University.

New Zealand Qualifications Authority (n.d.). Retrieved 23 June, 2010, from http://www.nzqa.govt.nz/ncea/index.html

Schwartz, M.S., Sadler, P.M., Sonnert, G., & Tai, R.H. (2008). Depth versus breadth: How content coverage in high school science courses relates to later success in college science coursework. Science Education, 93(5), 798–826.

Shulruf, B., Hattie, J., & Tumen, S. (2008). The predictability of enrolment and first-year university results from secondary school performance: The New Zealand National Certificate of Educational Achievement. Studies in Higher Education, 33(6), 685–698.

Shulruf, B., Hattie, J., & Tumen, S. (2010). New Zealand’s standard-based assessment for secondary schools (NCEA): Implications for policy makers. Asia Pacific Journal of Education, 30(2), 141–165.

 

Correspondence to:
Dr Erik Brogt
Academic Development Group
University of Canterbury
Private Bag 4800
Christchurch 8140
E: erik.brogt@canterbury.ac.nz