Data Collection

Data Collection

Data collection plays an extremely important role in behavioural intervention programs. Data on the student's day-to-day performance is often needed to assess (a) whether the student is improving within programs, (b) whether and how programs need to be modified to meet the individual needs of the student, and (c) whether teachers are following the correct procedures. Furthermore, data on the overall effectiveness of intensive early intervention programs, taken before and after treatment, allow parents to make informed decisions regarding the costs and benefits of treatment. Both short-and long-term data are often required to gain or maintain funding from schools or public agencies.

The task of choosing how, where, and what to measure when providing a 40-hour-per-week program can be daunting. As mentioned earlier in this manual, a large amount of information regarding treatment effectiveness can be obtained in weekly meetings where staff and parents observe the student's progress. A well-run program also provides day-to-day supervision of teaching skills by senior aides, who in turn are directed by supervisors or consultants. Much information about treatment effectiveness can also be obtained through viewing videotapes recorded during treatment. Such tapes are often helpful when subtle problems in teaching skills need to be identified, because they allow for repeated viewing and detailed analyses.

In the course of a 2- to 3-year intervention, a student may experience several hundred different programs. Each program presents its own potential difficulties, and the subtlest teaching mistake in a single program has the potential to slow down or stop progress across all of a student's programs. Consequently, the task of assuring treatment fidelity is both critical and extensive. In this chapter, we first review how to take relatively selective data that will allow the team to troubleshoot and document progress without adding substantially to the cost and time burdens of providing treatment. The last part of the chapter describes the Log Book, which provides an efficient way of recording what particular skills within programs have been taught by a team member on any one day. This allows for each team member to instantly be updated as to what the student has been taught and how the student has performed. It also aids in decision making regarding what particular programs should be administered on a given day. Also discussed is the Maintenance Book, which provides information about what particular programs have been mastered and which ones should be rehearsed so as to be maintained.

How to Take Discrete Trial Data

Discrete trial data are primarily useful for analyzing the student's level of performance in a particular program and for troubleshooting problems within programs. Collecting discrete trial data is fairly simple. For each trial of a particular program, the teacher records the instruction given and whether the student's response was correct, incorrect, or prompted. A complete blank data sheet is provided in Appendix 33.A at the end of this chapter.

On Trial 1, the teacher instructed the student to clap, and the student responded correctly, as indicated by the circle around the letter C (for correct). On Trial 2, the teacher instructed the student to wave, and the student responded incorrectly (indicated by the letter I). On Trial 3, the teacher instructed the student to wave again, and then prompted the student to wave (indicated by the letter P). On Trial 4, the student was instructed to wave a third time, and the student responded correctly.

Note that, in addition to instructions and responses, teachers should also record on the sheet (a) the date that the data were taken; (b) the name of the teacher or aide who did the program; (c) the number of correct, incorrect, and prompted trials; and (d) the percentage of correct responses (see the Discrete Trial Data sheet in Appendix 33.A at the end of this chapter).

If the discrimination training procedure is being used to teach new skills, trials involving a contrasting stimulus should be indicated by a C on the instruction/SD line, as shown in Figure 33.2. However, trials with a contrasting stimulus should not be used to calculate a student's percentage of correct responses.

Teachers working with the same student often inadvertently develop different criteria for what defines a correct, incorrect, or prompted response. When this happens, the value of discrete trial data can be severely compromised. To prevent this, teachers' interrater reliability should be assessed once every 1 to 2 months, or whenever there is suspicion of a problem.

To measure interrater reliability, two teachers should take data concurrently during a session. Their data should then be compared and the percentage of trials on which they agree (on which they both recorded that a student was correct, incorrect, or prompted) is computed. Teachers should be in agreement on at least 90% of their trials. If they are not, they should review the criteria for correct, incorrect, and prompted responses, and practice scoring sessions together until they agree at least 90% of the time.

Weekly meetings are a good time to assess interrater reliability. Teams may also videotape a single teacher's session, and then have all the team members score the tape at their convenience. Using this procedure, a senior team member or parent can determine if the entire team, or only one or two members, are having reliability problems. New teachers should always establish their reliability with an experienced teacher before taking data on their own.

SD: 'Clap'

SD: 'Clap'

SD: 'Clap'

SD:

SD:

SD:

Tips on Taking Data

We provide the following tips on taking data to help the team members be consistent in their work with students and their recording of data.

Teaching and taking data at the same time can be difficult. If teachers work alone, they should do 4 to 5 trials before recording the student's responses (to reduce the length of inter-trial intervals). If possible, record responses when the student is occupied with a reinforcer or after the student has been excused for a break.

Taking data while teaching is particularly difficult for relatively new teachers. If possible, have a second teacher take data when new teachers are teaching.

When totalling percentages of correct responses, make sure that prompted trials are not counted as correct responses.

Let teachers know that they do not have to complete 20 trials during each program simply because the data sheets go up to 20.

When to Take Discrete Trial Data

Discrete trial data should generally be taken (a) on programs in which the student is showing slow or limited progress and (b) at monthly intervals for all programs to document the student's progress or lack thereof. A common misconception is that one must take discrete trial data on every trial of every program, every day. Taking discrete trial data is time consuming and, consequently, expensive. Therefore, it should be done in response to the specific needs of each student.

If the student is having difficulty in a specific program (progress is slow, the program causes tantrums, etc.), it may be appropriate to have all teachers take discrete trial data on the program for several days. These data may then be analyzed (as described in the next section) by the student's senior aide, parents, or behavioural consultant to see if teachers are making any procedural errors. One to 2 days' worth of data is usually enough to spot most problems. Subsequent to being analyzed, the data gathered may demonstrate that programs need to be modified or placed on hold, or teachers may need to receive additional instruction in how to implement programs.

If the student is having difficulties across several or most programs, or if there is a concern that one or several teachers are making consistent procedural mistakes, it may be necessary to take discrete trial data on all programs for several days. Again, this data should be analyzed as described in the next section.

To document the student's performance in programs that progress smoothly, discrete trial data from the last few days of each month will generally provide an adequate sample. For example, to document the student's progress in the Matching and Sorting Program, you may need to take trial-by-trial data only during the last two to three teaching sessions (approximately 30 to 45 trials) of each month.

Using Discrete Trial Data to Troubleshoot Common Problems

An experienced teacher or parent can spot a wide variety of common teaching mistakes simply by analyzing a student's discrete trial data. Following are some common problems that can be spotted by reviewing such data:

Teachers fail to follow prompted trials with unprompted trials. Figure 33.3 shows data from the Early Receptive Language Program. On Trials 2 and 4, the teacher had to prompt the student to occasion a successful response. However, the teacher never followed up these prompted trials with unprompted trials. Consequently, we do not know if the student learned to respond to the teacher's instructions or simply became prompt dependent.

Teachers allow the student to be unsuccessful on too many trials. Figure 33.4 shows data from the Early Receptive Language Program. In this sitting, the teacher used massed trials to teach the student to stand up. As can be seen from the data, the teacher prompted the student after 3 unsuccessful trials. After this, the student was again prompted after 3 incorrect responses. In this case, the teacher should have initiated the prompt sooner and then gradually faded the prompt. Overall, the student should be reinforced (and thus successful) in 90% of trials.

LSD: 'Touch nose'

2. SD: 'Touch nose'

3. SD: 'Smile'

4. SD: 'Smile'

5. SD: 'Jump'

6. SD: 'Wave'

Figure 33.3. Sample data showing teacher's failure to follow prompted trials with unprompted trials.

P P

Figure 33.4. Sample data.

Teachers spend too much time on previously mastered responses. The majority of a teacher's time in most sessions should be spent on teaching new skills. However, teachers frequently begin programs by running through all of a student's mastered responses. If a teacher's data show that he or she has spent 15 trials reviewing mastered skills and only 5 trials teaching a new response, he or she should be told to reverse this emphasis.

Teachers do too many trials of a program. Enthusiastic teachers may have a student sit for 20 to 30 trials of a program, when only 5 to 15 trials may be appropriate for a single sitting. This is a common cause of tantrums and non-responsive behaviour.

Reviewing data sheets will also reveal whether teachers are using discrimination training procedures correctly, are doing an appropriate number of programs in a given period of time, and are working on the correct target behaviours for a program.

A problem common among both new and experienced teachers is failing to vary reinforcers. It is easy for teachers to use the same food or the same verbal praise as reinforcement, trial after trial, day after day, without realizing they are doing so. Students will generally respond to this situation with tantrums, decreased attention, non-responsiveness, and incorrect responses. If there is concern that teachers are not varying their reinforcers, it often helps to have them take trial-by-trial data on the exact reinforcers they use. Figure 33.5 presents a sample format for taking reinforcement data. This self-monitoring procedure is generally effective at prompting teachers to use more varied rewards.

1. Reinforcer: goldfish

2. Reinforcer: tickles

3. Reinforcer: rocked chair

4. Reinforcer: soda

5. Reinforcer: swung upside down

6. Reinforcer: hugged

Figure 33.5. Sample data showing reinforcers used.

Documenting a Student's Progress over Time

To gain or maintain funding from school districts or other government agencies, concrete evidence of the overall effectiveness of the intensive early intervention program is generally required. Discrete trial data can be used as part of this documentation, but broader measures are also important. Keeping detailed monthly summaries of the student's progress, regularly videotaping programs, and keeping a log of the student's spontaneous language are three tools for providing thorough documentation of the long-term benefits of intensive early intervention programs.

Monthly Summaries

One of the clearest ways to document the student's progress is to keep monthly summaries of her gains in individual programs. These summaries can be brief, but they should focus on objective, concrete facts. The two most important things to include are (1) the number of items the student has mastered within each program and (2) the percentage of time the student performs correctly (this percentage should be derived from the discrete trial data described earlier).

Below is a series of sample monthly summaries for the Expressive Labelling of Objects Program:

• March 1, 2001: Joan is unable to expressively label any objects.
• April 1, 2001: Joan can expressively label 3 objects (car, bubbles, and pizza) with 84% accuracy when the objects are randomly intermixed.
• May 1, 2001: Joan can expressively label 10 intermixed objects with 80% accuracy.
• June 1, 2001: Joan can expressively label 27 intermixed objects with 90% accuracy.
• July 1, 2001: Joan can expressively label 70 intermixed objects and pictures with 81% accuracy. New stimuli are now introduced in the form of pictures or objects, and Joan learns pictures as easily as actual objects. Joan also has generalized her labels to exemplars of objects we have not specifically taught her. For example, she can label her mother's shoes even though she was explicitly taught to label only one of her own shoes.
• August 1, 2001: Joan can expressively label 95 intermixed objects and pictures with 90% accuracy. This program has been placed on a maintenance schedule.

Figure 33.6. Joan's acquisition of expressive objects labels.

For reports given to school districts or used in legal hearings, monthly summary data can easily be collapsed into 3-month, 6-month, or yearly summaries. For example, the monthly summaries just described can be collapsed into the following 6-month summary statement and graph:

Expressive Object Labels: When this program was introduced on March 1, 2001, Joan was unable to expressively label any objects. As of August 1, 2001, Joan was able to expressively label 95 objects and pictures, and had generalized her labels to novel exemplars of 3-D objects and novel exemplars of 2-D representations of objects. A graph of Joan's acquisition of expressive labels is provided (see Figure 33.6).

Video Data

Videotapes of the student's progress over a period of months or years can be one of the most persuasive and easiest ways of documenting the student's gains. The student's first day of one-on-one instruction should be taped from the very beginning. Subsequently, tapes should be made once every 2 to 3 months throughout the student's program. Each taping should include (a) all programs and items within programs mastered since the previous taping; (b) all programs currently on acquisition; and (c) baseline performances for several programs that have not yet been introduced.

A common mistake is to record only a student's mastered programs. If this is done, the resulting tape appears to show a very obedient student who does everything she is asked to do, and shows no evidence of growth. By taping programs on acquisition and obtaining baselines on future programs, the ongoing impact of programs is clearly demonstrated.

Recording Spontaneous Language

One of the most important measures of the effectiveness of a program is whether the skills taught in the program generalize to outside the program. Keeping a log of the student's spontaneous language is an effective way to document this generalization. Spontaneous language includes any verbalizations that occur outside the formal teaching environment. These may include the student's responses to questions or instructions (e.g., 'Stop it'), or the student's requests (e.g., 'I want popcorn') or statements (e.g., 'Horse!' or 'I see horse!') occurring outside of the formal teaching situation. Choose a consistent 20-to 30-minute time period to take this data. Dinnertime and the time spent getting ready for bed are good examples of situations when spontaneous language may occur. On a monthly basis, record all spontaneous verbalizations the student makes during these periods.

Summary of Basic Steps

1. Take discrete trial data on each program for two to three teaching sessions at the end of each month (for a total of approximately 30 to 45 trials).
2. Take discrete trial data on every trial of any program the student is having consistent difficulty with. Review data and give teachers feedback within 1 to 2 days.
3. Take discrete trial data on all programs if there is a concern that teachers are not following correct teaching procedures. Review data and give teachers feedback within 1 to 2 days.
4. Keep monthly summaries of the student's progress in each program. Summaries should include the number of responses mastered and the percentage of correct responses.
5. Videotape the first day of the introduction of each new program, and once every 2 to 3 months thereafter. In addition, record all mastered programs, all programs currently on acquisition, and baseline sessions for several programs that have not yet been introduced.
6. Keep monthly logs of the student's spontaneous verbalizations.

Log Book

To organize the teaching environment for the student, yourself, and the team, construct a Log Book. When set up, the Log Book provides a continuous overview of the student's current programs and the student's skill level within each individual program. To construct a Log Book, you need a ll/2-inch three-ring binder organized with separate and labelled sections for each of the student's current programs. The Log Book's first page should be a sheet listing the names and phone numbers of each teacher. The second sheet should provide the student's weekly schedule. Next in the Log Book comes the summary of each weekly meeting, followed by the student's weekly Program Checklist (See Appendix 33.A). The weekly Program Checklist should list the entire student's current programs and the dates and days of the week. (The number of programs in the student's current program schedule should not exceed 20 at any given time.) The boxes under the heading for each day should provide space for team members' initials. When a teacher performs a sitting of a program, the teacher should initial the box corresponding to the day and the program. A quick look at the initials or missing initials will indicate to other teachers on the team whether a program has been performed and how many times during a given day each program has been done.

Each individual program section in the Log Book should have an SD Sheet, which gives the particular discriminative stimulus (SD) and required response (R) for each program, as well as the kind and extent of prompt currently being used. This page must be updated at least once a week. Following the SD Sheet should be a page listing target SDs on acquisition and SDs recently mastered. Provide lined notebook paper behind these two front pages for the team members to write short informational paragraphs about the student's performance. These notes should be precise and short, and provide enough informational data about the student's progress within each program that the next team member knows how and where to start the next session. Each team member should date and initial his or her notes.

Maintenance Book

When the student has mastered approximately 20 responses within a particular program, begin a maintenance schedule for the items mastered early on in teaching. These items should be listed in a separate book, the Maintenance Book. This book should be a l-inch three-ring binder, divided into separate sections corresponding to each program on a maintenance schedule. As some of the earliest mastered responses are moved to a maintenance schedule, new SDs should be introduced (one at a time) and their progress should be recorded in the Log Book. With separate binders for items that are current and those on maintenance, you help yourself and the student remember what he has learned, and boredom due to frequent repetition of mastered responses is avoided.

Items on a maintenance schedule should be repeated as often as necessary to ensure that they are not forgotten. The frequency with which these items need to be practiced will vary from student to student. Assign the responsibility of working through the various items on maintenance to a particular but different team member each week to each month (depending on the student). If numerous programs or items within programs are on a maintenance schedule, each team member should be assigned to work through one or several of the programs each week (or more or less often, depending on the student) such that all the programs are practiced. Each team member should initial and date the sheet as the work-through of the programs on maintenance is completed. Each program's items on maintenance may be worked through separately in an informal and generalized format or intermixed with current items.