An informal preregistration of an investigation into preschool effectiveness
And why I think that's important
I was recently shown a post made on a mom’s group on Facebook that fascinated me. A mother of two young children asks for advice: she wants to homeschool them, but worries it is beyond her capacity. A few months into teaching her four-year-old, she is considering enrolling him in preschool, and perhaps a few further years of formal education before taking him out for some years at home.
There were over a hundred replies, and I didn’t review them, but a striking one was shown to me. Another woman reports that she has decided against homeschooling for her children, because although she has a graduate degree, she prefers to leave teaching to the professionals. She suspects that despite glamorous and ubiquitous online homeschool influencers, the most successful people are school-educated.
I was homeschooled through middle school, and my high school education was a mix of instruction by my parents, by friends’ parents, and in private online classes. I do not intend to send my young children to school either. From this unschooled vantage point, the above dialogue is jarring, and is predicated on some key beliefs about education that I had never really considered, namely:
By the age of four, formal education is necessary, and a child is at risk of falling behind if it is absent or inadequately provided.
The education of these children requires skills or training which many people do not have.
Until this moment, I had thought that for the typical child formal education began in kindergarten — or maybe not until first grade! — for the vast majority of American children. In fact, the majority are enrolled by the age of four, and it’s been this way for many years. And it’s not just a rich-person thing. Enrollment rates are only weakly positively correlated with parental income. It is, in fact, the society-wide default option.
This surprised me. It wasn’t always this way! In 1970, only 21% of 3- and 4-year olds were in school. It climbed steadily throughout the 70s and 80s, and reached its present level in the mid-50-percent range by 2000. All but four states now have a publically funded preschool program, and several have free universal enrollment. Although the ascendance of preschool has plateaued in the 21st century, this is still a pretty momentous shift in the treatment of young children. What explains it? What are the purported benefits of preschool over the alternatives?
Before a few weeks ago, my only knowledge of the effectiveness literature of preschool was a writeup I had read in a periodical of a 2010 HHS report on the effectiveness of Head Start, a low-income preschool program. The headline finding was that preschool had initially positive effects on cognitive and academic test scores, but that these differences were no longer detectable by the end of first grade. At the time, it seemed like a pretty significant blow to the program. But preschool hasn’t gone anywhere, and as always, finding the truth is much more complicated. Preschool has many ardent supporters, within and without acaedmia. It seems I had forgotten Cowen’s second law, but this is an error I have now resolved to correct.
Over the past few weeks I have begun to familiarize myself with this literature. I think my investigation will be of general interest, and I intend to write it up. But before I get much further, I think it’s useful to state my intentions in advance. Allow me to explain why that’s important.
I am both an enthusiast and skeptic of analytical methods in the social and biological sciences. You see, regular, everyday people are generally pretty good and finding the truth in highly practical matters that directly affect their everyday lives, without any reference to or knowledge of scientific methods. On the other hand, when it comes to matters that require a great deal of precision and computation, intuition fails. Regular, everyday people are not very good at building bridges, calculating integrals, or peforming thoracic surgery. This is where empirical methods and careful study are so useful.
But there is another sort of question which lies in between these two cases. The sort of question that is not highly precise or computational, but that is inherently difficult to study for some other reason (e.g. the effect of your diet on your long-term health) and/or is about something that has very little direct affect on our lives (e.g. the utility of Keynsian countercyclical fiscal policy). In these cases, we tend to believe what sounds good or is convienent for us, and then convince ourselves that the evidence supports that belief.
Empirical research methods would seem to be able to rid us of these cognitive shortcomings, but they usually don’t. The impreciseness of studying organic systems allows the people doing the research so many opportunities to insert their bias into it. In other words, our solution to this fundamental constraint of human nature, is itself constrained in precisely the same way. There’s probably a name for this, but until I know it I’m going to call it the Douglas Adams effect.
The result is seen in innumberable fields of study in the social and biological sciences. We have a conflicting literature, with academic partisans on both sides, each with their data and charts and studies, each equally convinced of the rightness of their view. For various professional, personally, and idiosyncratic reasons, very well-trained and intelligent people can look at the same data and come to opposite conclusions.
So we have to come to grips with the fact that we are all biased, and we all have an agenda. There’s no getting around that. “Just try to be objective” is good advice, but it’s not enough. What, then?
There’s no silver bullet. But the best we can do is seal as many cracks in the wall of quantitation as possible, to remove every opportunity for bias to leak in. This means an obsession with methods over interpretation, and a steely-eyed, steadfast rejection of analytic flexibility.
An increasingly popular tool to fight bias is pre-registration. Study authors will publically state in advance how they are going to collect and analyze the data, cutting off their own ability to p-hack their way to significance.
With that in mind, I am doing a sort of preregistration of my own. Here are my declarations.
Research question and definitions: I intend to survey the literature on the effectiveness of preschool over commonly available alternatives.
Significantly, there is often not much difference between preschool and daycare. This will be a key distinction. I don’t have a formal definition of preschool, because none exists, but I think the following elements are key distinctives: I) it involves specially trained teachers, II) it includes a structured curriculum with specific learning objectives, and III) it involves a student-to-teacher ratio of approximately 8:1. Now, I know that some of the studies I’m going to look at violate some of these rules (especially the last one), and that’s fine. But it’s something we have to note, because, e.g., a higher ratio may not be scaleable.
Commonly available alternatives consist of a mix of classroom-style daycare (with a similar ratio), in-home daycare, and at home care by parents or extended family. Note that these vary in their similarity to preschool itself, and therefore crossover is going to be an important consideration, athough it is not a insurrmountable problem.
I am open to different measures of effectiveness, but I expect broad concordance. Unless we have a reason, in advance, to expect a benefit on some kinds of measures but none on others, discordance between different types of outcomes (e.g. preschool lowers IQ but raises income, or improves test scores while also increasing probability of going to prison) are more likely to be due to spurious correlation than some totally unforseen and quirky mechanism of benefit.
My own priors:
I have a prior against the effectiveness of preschool in a general population. This is for many reasons, chief among them a) a universal skepticism that top-down interventions can have large positive effects, b) a narrower skepticism of formal education in particular, and c) a belief that the parent-child relationship is a major determinant of child wellbeing. However, in a low-income setting, where the child is more likely to have a bad home life and underachieve academically, I’m more sanguine.
It’s clear to me from the modicum of reading I’ve done already that a lot of people in this space have an axe to grind in favor of preschool, and this raises my skepticism even further.
Emphasis on methodological rigor: As I said above, I’m going to place particular emphasis on method, and especially sample randomness, when evaluating these studies. I see this as the best way to cut through the fog of uncertainty. Picking your outcome measures before you have a look at the data also earns bonus points, as does a prospective study design. These are all commonly agreed features of high-quality research (especially in the medical field, where I came up; less so in social sciences), so I’m not just pulling them from nowhere.
My report on this topic is going to include a review of the history of early childhood education, and some polemical commentary from me along the way, but my primary emphasis is on reviewing and summarizing the state of the literature.
I have no idea how long this will take, but you’ll hear from me eventually!
