Concept

We have developed a system of visualizing longitudinal data (specifically, educational data) that takes its inspiration from statistical and fluid mechanics.

Every state in the nation has been gathering detailed data on the yearly progress of students because of the standardized tests mandated by the No Child Left Behind Act.

Our research focuses on the mathematics portion of the TAKS standardized test, which was used in Texas from 2003-2011 for 3rd-11th graders. This data is comprised of 27 million unique tests given to 6.5 million unique students over this time period.

The very large numbers of student scores make it possible to apply techniques from statistical mechanics that are normally used to describe flow and diffusion of particles.

Visualizations

The visualizations that have been developed so far include:

• Snapshot plots: Show how the entire school system changed from one year to the next, separated by initial grade and initial score level

• Cohort plots: Each plot represents a single graduating class, showing how it evolved over time

• Streamline plots: Created by integrating over either a snapshot or a cohort plot; intended to follow qualitative changes in a flow plot graph

• Trajectory plots: Created by explicitly following groups of students over time to see where they ended up on average

Results

• Poverty is a more powerful influence on test scores than value added by teachers and schools.
• In Texas, there is a significant change in flow patterns starting with the graduating class of 2012. Compared to all the classes before it, low-performing students in the class of 2012 experienced large score gains in 5th and 8th grade.
• The change between the graduating class of 2011 and the graduating class of 2012 is caused by a program known as the Student Success Initiative (SSI). This program was initiated by the Texas Legislature in 1999, and caused the 5th and 8th grade TAKS tests to be high-stakes, meaning that students had to pass the tests in order to advance to the next grade. If students failed either high-stakes test, then the students were given additional tutoring/instruction and were allowed to retake the test up to two additional times. The SSI was rolled out in such a way that it only affected the class of 2012 and later (e.g. 5th grade high-stakes tests began in 2005, but 8th grade high-stakes tests were not implemented until 2008).

• Our research determined that SSI was successful in raising students’ test scores in 5th and 8th grade. The program had a successful impact on those students who receive free or reduced-price lunches and those that do not. Furthermore, the effects of SSI were not just limited to high-stakes years; test scores rose in 6th, 7th, 9th, and 10th grade. Our visualizations showed the positive and significant effects of SSI, even though we had no a priori knowledge of the program. Unfortunately, by the time we had discovered these results, the program had been defunded.

Open Research Questions

• The Python code that powers this research has been written to be open-source and extensible. Can we look at education data from other states and find similar results? We would need waivers from each state to access their FERPA-protected data, but this is an obvious extension of the methodology.

• In addition to other states, what about the data being gathered right here at UT-Austin? The university is one of the largest in the country, and has access to everything from individual student grades to teacher evaluations. Can we turn this model on ourselves to glean new insights?

• We don’t have to limit ourselves to test scores, or even education. This methodology will work for any semi-deterministic, longitudinal data set.

• Streamline plots are qualitatively accurate and only take two years to complete; trajectory plots are quantitatively accurate but take 9+ years to complete. Policy makers and legislators often do not have the luxury of waiting nine years to see if a policy is effective or not. Is there a middle ground? Streamlines become more quantitatively accurate with each extra year of information. How many years does it take before streamline plots are effectively identical to trajectory plots?

• How are educational careers affected by individual schools? If students stay in elementary school through 6th grade instead of 5th grade, how do their plots differ?

• When a different standardized test is introduced, how do the plots change? Is there something akin to a phase transition?

• A related modeling method uses first-order Markov processes. At the statewide level, these produce stationary state eigenvectors. Do these eigenvectors have any meaning or value? Can we find characteristic eigenvectors of individual districts or schools and use them to evaluate their effectiveness?

Publications

Anthony J. Bendinelli & M. Marder, ‘Visualization of longitudinal student data,’? Phys. Rev. ST Phys. Educ. Res. 8, 020119 (2012). Link to paper.