U.H. Study on SES in Hawaii

The Social Science Research Institute of U.H. at Manoa conducted a study (funded by the Hawaii State Department of Education) to evaluate SES programs administered during the 2007-2008 school year.  The report was released in September.  Note:  this is the third year the evaluation has been conducted.  The study addressed three primary questions:

  1. Has the provider contributed to increasing student achievement?
  2. Are parents/guardians, and other stakeholders satisfied with tutoring services?
  3. Are SES providers complying with the terms of their contract and with state and federal rules and regulations?

Here are the facts:

  • Eighty-six schools and a total of 5,287 students participated (15% of those eligible) in SES in 2007-2008. However, this represents an increase of 2% over the previous year.
  • For the student achievement analysis, the evaluation compared students’ Hawaii State Assessment test scores before (2007) and after SES (2008) tutoring. A total of 1,593 students were included in the reading (654) and math (939) datasets.
  • To address the other research questions, customer satisfaction measures (via survey and interviews) were obtained from parents, complex area staff members, and school employees, such as teachers, principals and staff members designated to work on SES.

 Key findings: (pulled from report)

  • For student achievement, the data analysis confirmed the overall results of no significant effects of tutoring on standardized test scores.
  • Parental satisfaction:  As in previous years, parents/guardians are generally satisfied with tutoring services, responding to most questions in the range of 80-90% satisfaction.
  • School Personnel:  The results of that survey (based on 76 SES schools) indicated that providers serving more schools tended to have higher ratings. Analyses indicated a high correlation between a provider’s ratings and the percentage of respondents wanting to keep the provider on the DOE’s approved provider list.
  • Complex Area:  In general, the newer providers were identified as having the most difficulty with requirements such as invoicing, recordkeeping, notification about service capacity and communication with the schools, complexes, and parents. A few were considered to be more experienced and organized on the business management responsibilities of SES. The complex area staff recommended continuation of all current providers, yet areas for improvement were identified, most typically in terms of better communication and fiscal accountability.

So, how did IAAK fare?

  1. IAAK was largest service provider, reaching 758 students who participated in the evaluation.
  2. For reading and math, all of the students who participated in IAAK’s SES program (across all percentages of tutoring received <20%, 20-79%, 80%+) showed improvement in scores from 2007 to 2008.
  3. Only 12% of IAAK sites reported some sort of delivery/use problem.  Questions on the survey were related to SES contracted obligations, which included providing information to schools and teachers as well as planning and delivering individualized supplemental education services consistent with school and state standards. 
  4. IAAK received the highest rating from schools when asked “How would you rate this vendor compared to others you deal with?”. Survey respondents were asked to use a five-point scale (1 Way Below Average, 2 Below Average, 3 Average, 4 Above Average, and 5 Superior) to rate SES providers.  Mean ratings ranged from a low of 2.43 (between Below Average and Average) to a high of 3.53 (between Average and Above Average).
  5. 96.4% of schools rated “yes” to keeping IAAK on the list of providers.

Overall, if I don’t say so myself, a very good report card for IAAK! 

The report also presents background information on SES, including local and national statistics and evaluations.  For instance, this interesting fact is shared – 15% of Hawaii’s eligible students took advantage of free SES tutoring, which is higher than the 13% of eligible students served in the previous year but lower than the national participation rate, estimated to have been 19% in 2004.  If you have the time, peruse the report – it’s worth a review.