Perlman – quality meta-analysis

A key aspect of science is replication. The findings from any one study may be driven by the idiosyncratic characteristics of the population, the circumstance, any number of things. And so when we are interested in something that we really care about, and certainly something that might have implications for the real world for policy and so on, we really need to make decisions based on a consensus of results for multiple studies. One methodology for trying to integrate findings or integrate evidence is systematic review and meta-analysis. And that entails a very sort of rigid way of searching the literature or unpublished research for that matter for findings about the topic of interest and the qualitative, more qualitative piece of this is the systematic review. It’s basically a very, very structured way to do a literature review. 

Once you’ve done that you can identify studies if the author’s provided the right statistics that are similar enough so that you can actually conduct a meta-analysis, which is a statistical analysis of those findings that have been published or shared by the researchers in some way. And what they allow you to do is quantitatively summarize whatever is out there on the topic of interest 

So we started tackling different aspects of quality. We started very, very broadly, in fact we kind of did searches of the major quality indicators that are covered in the literature.  And that includes staff child ratios, formal education, years of experience, training that’s specific to ECE so early childhood education training. 

Professional development opportunities and two measures of quality that are observational that are frequently used in the literature. One is the ECERS, one is the CLASS. And so we set out to essentially compile all of the evidence that exists for each of those quality indicators in child outcomes. 

Overall the findings have been that there are few associations and that when they exist, with one caveat that I’ll get to in a minute, they tend to be quite weak. It’s hard to interpret because what do you do under those circumstances? We know that large, a large proportion of young kids are in some form of childcare. They spend a large proportion of their waking hours in those settings. It is a context that – the level of exposure that kids get to have to childcare makes it important and it makes it really important for us to understand what about those settings can support kids? How do we design these programs in a way that supports their development? And so we’ve really struggled to try and understand the lack of associations and there are many, many methodological issues in the research that we compiled. I think there are also measurement issues, I think there are also probably conceptual issues about how we think about quality. 

There are two areas where we did find more effects and one I think meaningful effects. So the ECERS had more associations than the other pieces and especially the total score of the ECERS. Sometimes people use subscales and, you know, we have to sort through all of that to figure out what we could combine.  

But the other piece where the effects were – there were more effects and they were stronger as professional development. And to me that has very important implications because that is something that is actionable. We can develop empirically based strong professional development programs. And that is – it’s a lot easier to provide professional development opportunities to people than to require that the workforce all have undergraduate degrees, say.  Or even increasing ratios where the cost is enormous to increasing ratios. But the cost and therefore the feasibility of providing more professional development is much more achievable. And so I think – I see that as a very hopeful kind of finding.  

And there’s been an explosion in research on professional development programs for early childhood educators. I think – we’re seeing a lot more of them and we’re seeing a lot more, you know, some work better than others for sure. So we want to make sure that as we promote professional development for early childhood educators that we’re doing it in a way that’s empirically based.