Education Science and Culture of Data Use

I've been working on several initiatives that promise to improve the way education agencies use data to drive decisions and improve student outcomes. At a recent event on the topic with some of the brightest minds in the country the topic of organizational culture came up. The idea was that even with the best information systems, best training, and highest level of data quality, the impact of data to drive decisions and actions depends to a great extent on establishing a “culture of data use”.

Alex Horniman, professor at UVA’s Darden School of Business defines organizational culture as “shared habits of believing”, and I would add “of behaving”. In institutions like education there is a great mass of cultural inertia, a strong resistance to change. This is the same kind of inertia that prevents an individual from following through with a New Year’s resolution, but in this Culture is multiplied by the tens of millions of education stakeholders and years of “shared habits of believing”. The good news is that applying the right forces to a culture, like applying a force to an object the physical world, can get the culture, or object, to move. One of those forces for education can be data, or information to the right people at the right time. And here is a great need and opportunity, to better understand not only how to use information to be able to make better decisions in education, but to understand what it will take for people within the education culture to adopt new habits of believing and behaving based on the data...to adopt changes in practice that improve the profession and student outcomes.

There are some untested assumptions about the link between good information and good decisions in education. In the book “Predictably Irrational” author Dan Ariely describes experiments by which he and his behavioral economics colleagues have proven that people will consistently respond to good information with the bad decisions and irrational behavior, based not necessarily on the quality of the information, but on any number of contextual factors. When the contextual factors were changed the test subjects would predictably make different decisions.

Moving to a culture of data use in education may require the same kind of scientific research about the context of information needed to drive decisions and behaviors that will optimize student learning. This context includes what information is presented when, how the information is presented, how it is presented relative to other information, what presentation characteristics result in positive behaviors and what information/presentation/context results in negative behaviors, etc. There are a complex set of factors apart from the data itself that impact a person's willingness to trust and take the best actions based on the data. It is easy for a person with the very best intentions and good data to make less than optimal decisions. I don't claim to be a behavioral scientist, but it seems to me that the kind of experiments needed to better understand the link between good information and good decisions in education can be done in relatively short cycles compared to other kinds of research.

A Tale of Two Learning Organizations #1

Harold looked up from his book. At slow times like during this early morning shift his manager allowed him to read. There was only one customer in the store, a middle aged man in a business suit who was in no hurry, browsing up and down the aisles and reading labels.
Harold’s job at the CVS in the New Town area of Detroit was steady work and Harold felt blessed. He knew that there were a large number of fellow high school dropouts not so fortunate. But he also knew that he would be better off if he could get his GED and he was glad he could get some studying done while earning a living.

As the customer reached the end of the aisles nearest the register Harold made eye contact and said with a smile, “Good Moring Sir. Can I help you find something?” “No thanks, I guess I’m all set, there was something else I wanted to get but I can’t think of it.” the man replied then walked up to the counter. “Do you have an Extra Care card?” “Oh yeah, here you go.” “beep” Harold scanned the customer care card then items from the man’s shopping basket. Harold noticed the man had some dandruff on the shoulders of his black suit jacket. “Twenty-one twenty-two is you total.” The man scanned a debit card and completed the sale, then the receipt printed. At the bottom of the receipt Harold noticed a coupon was printed it read “$1 off Head and Shoulders any size”. Harold thought to himself “that cash register knew just what coupon to print”. The man paused, “Can I use this coupon now?” “I don’t see why not. The shampoo is on aisle C about half of the way down.”

Little did Harold or his customer know the extent to which CVS and other retailer’s warehouse and mine data collected from customer purchases. From that data they know which customers have bought which types of products and what they are likely to buy in the future. They can identify trends and patterns in the data to optimize a myriad of decisions such as where in the store to place products and which products to place next to each other, where to open new stores and when to close a store, what products to advertise when and with what advertising media.

Successful for-profit and not-for-profit organizations today learn all they can about the customers they serve and the environment in which they operate. They understand that the customer is the core business. Successful organizations today use data for decisions that optimize the customer experience for customer satisfaction, and corporate success.

...Meanwhile, few blocks away June, a fifth grade teacher, is nervous about the first day of school. “I wish I knew something about the students I’m about to teach.” She had been given a class list but knew from experience that even that list of names might change during the first week. “This list of names is useless,” she thought, “I really need to know who these kids are, what they like and don’t like, what background knowledge and skills they will bring to fifth grade. That way I can plan instructional strategies.” June was not content to teach from the text under the traditional fixed time, variable learning model. She had had some success at differentiating instruction in previous years, using strategies like flexible grouping. It was hard to ensure that each student achieve a grade-level worth of growth for a year of study without first knowing where to start.

Unlike many high achieving organizations, schools and education agencies have failed so far to leverage technology to learn about the customers they serve (students) and the environment in which they operate. Too often the education culture and routine becomes the core business rather than student learning.

Education agencies use data primarily for management, finance, and compliance reporting. Every day, every detail about school financial transactions are collected into a centralized accounting system. Many more daily “student learning transactions” are not capture by any data system. Details of daily student work and assessment of that work are thrown away with papers, at-best capturing only summative and subjective grade book data. Educators lack the kind of detailed transactional data used by accountants, marketers, and managers in other industries to optimize decision making around their core business. Meaningful data that can inform teaching and learning decisions beyond the classroom-day are collected far too infrequently. In fact, it is not uncommon for a state to collect data about student learning only once a year. The instructional value of any data classroom data collected decreases as it loses the context provided by the teacher.

What do you think? Can our education institutions learn something from other "learning organizations"?