Perlman - Toronto’s assessment for quality improvement

The AQI is an observational measure that’s used by the city of Toronto to assess the quality of all childcare providers that have a licence to provide care for kids who are in subsidy. So those providers have a contract with the city and they have to agree, in order to have that contract, they agree to be part of the city’s quality assurance system and part of that is undergoing an assessment using the AQI. Annually it’s done by highly trained city staff and they have a very rigorous method for maintaining reliability and so it’s done – it’s implemented I think at a very high standard.  

Versions of the measure have been around for decades and it was initially developed by city staff and partners decades ago. It was used informally for many, many years and evolved. In early childhood education settings you’ve got anywhere from more or less newborns to preschoolers and even before and after school care. And the standards for what is kind of stellar care or even if you want to think about it kind of acceptable care, look different for kids of different ages. And so you have to have measures that tap into what’s appropriate for an infant versus a preschooler. And so the city has, and always had, different versions for different ages and that’s true for comparable measures. The most common one and, I think the most similar one, is the ECERS that many people are familiar with. 

And so we tackled the preschool version first. For me that was a nature starting point because that’s the age group that has the most kids in it. We did a lot of work to create documentation to go with the AQI so that when people were trained on it, it was very clear to them what the standards were and what the expectations are. And then we tested it. We did a validation analysis where basically what you do is you take an existing measure and you test it against other measures that you expect it to be correlated to. 

We compared AQI scores to a number of different instruments – the ECERS that I mentioned earlier, the CLASS which is a measure. It’s also observational like the ECERS and the AQI but it’s much more focused on interactions. Both the AQI and the ECERS are kind of an amalgam of looking at the physical environment, looking at health and safety, looking at the materials that are available for the kids to play with but also at the kinds of interactions that they’re having with the educators.  

And so we looked at how the AQI related to those other measures and it correlated quite highly certainly at a standard that gives us confidence that it’s measuring what the city wants it to measure and what it purports to measure. 

It’s very efficient. It’s much faster to administer than say the ECERS. Like the AQI if there are two rooms, two preschool rooms, the observer bounces between them and they can two rooms in an hour and a half. The ECERS takes three to five hours per room, you know, that’s an enormous – when you’re doing it at a policy level that’s an enormous difference. Since then we looked at the infant and toddler version and the results were very similar. And now we’re starting to tackle the home childcare version. The school age version of the AQI hasn’t been validated.