EVALUATION OF THE CHAI CURRICULUM

Evaluation by an outside agency, the Jewish Education Service of North America (JESNA) is an ongoing aspect of the CHAI curriculum. See below for a summary of the process thus far as well as results of specific evaluations.

Curriculum Core

1. Our most recent phase at evaluation was a first effort at assessing the impact of the curriculum on student learning. The study was limited to fifth grade students learning the Level 5 G’milut Chasadim strand. You can download the executive summary here. For a copy of the full report, please contact us.

The report notes, “The URJ must be commended for its ongoing commitment to rigorous formal evaluation and for its willingness to invest in the challenging groundbreaking process of developing tools to measure student achievement.” (Executive Summary: CHAI Impact Study, 6)

We remain committed to the process of evaluation and continual improvement. The initial study was limited (100 students from five schools completed the survey), but we can continue to benefit from the work that was done to create a tool for measuring student learning in a Jewish setting. You can download the student survey and teacher survey as well as the instructions for administration. We encourage you to try it or adapt it for your school and share the results with us. You may want to use it as both a pretest and a posttest for the unit.

2.  Another phase of the evaluation assessed how CHAI is being implemented in a cross-section of Reform congregations and to gather early indicators of perceived effectiveness and satisfaction.A team of 5 HUC-JIR graduate students conducted a total of 29 in-depth telephone interviews with key informants from a sample of 10 congregations using CHAI in 2003-2004. The interviews were designed to gather information about how the congregations learned about CHAI , how decisions were made to use CHAI, how CHAI was implemented, the perceived effectiveness of CHAI, and initial reflections on its successes and challenges, strengths and drawbacks.

We learned…

So we will…

Respondents value CHAI for focusing on enduring understandings and core values and for emphasizing depth rather than breadth, while simultaneously faulting the program for not including broad coverage of topics and facts covered in more traditional curricula.

Continue to provide ongoing training and support to better communicate the goals and design of CHAI.

Many of the perceived weaknesses are related to a lack of familiarity on the part of the teachers with the approach, and a consequent lack of confidence in implementing the curriculum.

Supply ongoing professional development, mentoring and support needed to enable teachers (particularly novice or untrained faculty) to use the materials optimally.

Anecdotally, CHAI had a profound impact on teachers who embraced the curriculum and the approach. However, some veteran and entrenched teachers prefer and rely on more traditional approaches and methodologies, and are therefore resistant to CHAI.

Include observations of teachers before and after implementation of CHAI in future phases of the evaluation, for a truer assessment of the impact of the program on teacher behaviors.
Develop a profile of CHAI teachers.

Those who favored the CHAI approach reported more excitement and engagement by students and less disruption.

Include more systematic observations and assessments of student learning in future phases of the evaluation.

Respondents generally did not perceive any relationship between CHAI and parent involvement.

Design systematic efforts and initiatives to create connections between what is taught in the religious school and the programs and offerings of the congregation as a whole.

 

Download the full report.

3.  CHAI: A Learning Initiative from a Learning Organization, by JESNA November 2003.Download the summary report.*

4.  Feedback was gathered on Levels 2 and 3 of the Curriculum Core. The summary report may be downloaded*. Based on the feedback we received, the following changes were made:

  • Beginning with Level 3, all lessons now come in a convenient loose-leaf format, enabling teachers to reproduce materials more easily. The new format will also allow for incorporation of timely updates, as they become available.
  • Each lesson includes recommended timing for each learning segment, and many suggest alternative approaches to activities, depending on the needs of the class.
  • An enhanced editing process has been put in place to ensure age-appropriateness of language and concepts in all lessons.
  • Major concepts and key words will be indicated in Hebrew, as well as in English and transliteration.
  • Published materials will refer to "levels" rather than "grades" to accommodate the needs of schools that have multi-age learning groups.
  • An "Eight Steps to a Great CHAI Lesson" has been developed to assist teachers in planning for class lessons.
  • A consolidated Materials Needed list will be included with the lessons for every Core level.
  • Summer faculty retreatsand online coursesnow provide more intensive training for teaching CHAI lessons in the classroom.

Mitkadem

1.  An in-depth implementation study of the Mitkadem curriculum was done during the 2006-2007 school year. Three different profiles were identified:

  • Traditional school implementing Miktadem in the conventional way
  • Traditional school implementing the program in an “unconventional way”
  • A one-room schoolhouse or multi-age classroom using Mitkadem

JESNA Berman Center staff conducted one-hour structured telephone interviews with four representatives from each of the three schools.

 

The pages in the final report that may be of interest to you include pages 18-19, “Cross-School Observations/Areas for Further Consideration.” Observations include:

  • Buy-in by synagogue staff, students and parents contributes to the success of the program.
  • Educational leaders from the three schools all felt that the program was meeting the goals they had for their students, both in terms of Hebrew language skills and attitudes towards learning.
  • Madrichim and paid high school-aged teachers do provide an important way for schools to decrease the schools’ staff to student ratio and increase the likelihood that their post-b’nai mitzvah students will remain connected and engaged in Jewish education within their congregations.
  • Teacher training contributes to the success of the program.
  • Providing a way to efficiently and effectively organize the variety of materials associated with the program helps teachers implement it.
  • Some schools have created additional tools to keep track of student progress, even on a daily basis.
  • Parent involvement has a positive affect on student skills and attitudes. In one school, this even resulted in the demand for an adult Hebrew class.
  • Many teachers still desire time for group instruction and work that into the program.
  • Providing a quiet space for students who need it can be helpful.

 

The full report includes detailed information about each of the three schools. On pages 20-22, you will find one-page case studies of each setting. You may use these case studies, with or without the recommendations above, as you consider using the program or assess your current implementation of the program. Study them with a school committee or staff and use them as a basis for discussion. Ask: What might improve the implementation at each school? Would we like to consider any of those ideas for our school? Is there anything that contributes to the success in these cases that we could implement in our setting? How is our school similar to or different from these schools, and what are the implications?

 

Download the full report here.

2.  In order to gather early data on the success of Mitkadem Ramah 1, a total of 19 teachers (n="4)" and school directors (n="15)" from 16 congregations were interviewed by telephone during Spring 2003. (One respondent functions as both school director and Mitkadem teacher in her setting.) Respondents were randomly selected from the list of congregations that had placed substantial orders for Mitkadem, based on the assumption that such congregations would be most likely to have introduced the materials in their programs. The congregations were located in 12 of Union's 14 regions, and included congregations of various sizes located in different sized Jewish communities (with different levels of Judaic resources and support).

The Mitkadem summary reportfrom JESNA, completed in June of 2003, may be dowloaded.* Based on the feedback we received, the following changes were made: Based on the feedback we received, the Union responded in the following ways:

  • New packaging is being designed for the manipulatives (Hebrew Helpers) that is more sturdy. The new packaging will be complimentary to everyone who ordered manipulatives previously.
  • Bulk pricing on ramot is now available on the URJ Press Web site.
  • New ramot will be released as they are ready during the school year. 
  • We developed an online tutorial(PowerPoint presentation) for teachers to guide them in structuring their Mitkadem classroom.
  • At upcoming conferences, such as CAJE and the Union for Reform Judaism Biennial, there will be opportunities for Mitakdem users to meet and share their experiences.
  • Mitkadem users are invited to sign up for the CHAIhelp listserv(choose CHAIhelp from the drop down menu), post questions, and respond to one another about implementing Mitkadem.

Parent Education
Three CHAI Parent Education Units ( Bless Me Too! Parental Favoritism & Sibling Rivalry in the Bible; Sustaining the World Through Torah, Worship & Deeds of Loving kindness; and The Power of Jewish Ritual ) were sent in early summer 2003 to 12 congregations that had agreed to pilot test the materials. Pilot congregations were asked to conduct a parent education program using one of the units (of their choice) in the early fall, and to complete and submit written feedback surveys by the end of October 2003. The JESNA analysis concludes that the parent education units were extremely well received by all of the congregations that served as pilot sites:

  • The materials and activities were appropriate, engaging and resulted in learning and desire for additional learning on the part of the participants.
  • Facilitators found the materials and activities clear and easy to use, and they enjoyed teaching the lessons. Both facilitators who had previous familiarity and those who did not found the UbD approach to be helpful.

Constructive suggestions for improvements in presentation and/or materials included:

  • presentation of source texts directly;
  • more concrete examples;
  • more opportunity for discussion; and
  • opportunities for feedback from participants.

Download the full report from JESNA.*
Feedback will be used to guide development of future units and to provide additional resources and guidance to educators and facilitators regarding implementation of existing units. 

Teacher retreats were evaluated by participants and staff in the first year and the program was altered in the second year in response to participant needs. The preliminary data shows much improved satisfaction with the retreats in the second year.

* Please note: You must have Adobe Acrobat to view this file. Download it for free.