Rural arts and Culture Program


Evaluation Suggestions: A Short Guide
by Barbara Carlisle
Step One: Consider what evaluation/assessment will be appropriate for you.
A. Decide why you are doing the evaluation and for what audience.
Product or Outcome Assessment - how well did we accomplish our goals?
Process Evaluation--how effective was the process?
Who will look at this? e.g. planning committee, grant provider(s), potential future donors, participants, collaborating agencies or groups, general public, etc.

B. Decide what you will do with the information you collect.
Product/Outcome Assessment--persuade government or private agency to continue to support the activity; draw more of the target audience; improve the quality of the programming/services; improve the quality of the learning/engagement of participants, etc.
Process Evaluation--improve delivery of services, raise more money, improve communication, make changes in the programming, etc.

Step Two: Consider what you would accept as evidence that you had achieved your goal(s).
Examples:
Activities happening as planned; numbers of target audience coming; people able to name or refer to some aspect that you consider important; people having fun; people intensely engaged; people staying with the project over time, etc.

Step Three: Design the evaluation activities to collect the evidence you need.
Include evaluation/assessment in your planning and build the activities into the project from the beginning.

IMPORTANT CONSIDERATIONS FOR PRODUCT OR OUTCOME ASSESSMENT
Link the evaluation goals to your project goals. This is especially important in collecting data for donors or potential donors. (Remember: donors sometimes have donors themselves to whom they report. Give them something to report, to keep the money flowing.)
Examples:
Project goal: to improve migrant students' sense of self worth through participation and recognition in arts projects.
1. Did migrant students participate? (count them and the activities)
2. Did the project improve their sense of self worth? (Ask some pre and post questions that are related to self worth. Or make specific observations about behavior before and after. Or interview parents, or teachers, before and after. Or have students talk/write about themselves.

Project goal: to gather the community together to appreciate local or regional artists.
1. Did people come to hear local performers? (count audience and performers)
2. Did they express any sense of community about the event? (Find a way to ask questions and record the comments. Ask the artists as well as the audience.)

Project goal: to renovate a regional historical site to improve sense of local history.
1. Did the improvement/renovation happen? (Before/after photos. Within budget? Within time frame? As planned or modified?) What is good evidence?
2. Are people coming? (count them)
3. What are they saying about their sense of local history? (Ask them questions; give them a quiz to answer by finding certain objects or documents. How many could do it?)

IMPORTANT CONSIDERATIONS FOR PROCESS EVALUATION
Link evaluation to specific project features you want to improve or change.
Don't ask questions about everything: Prioritize.

You may do several mini-surveys or a single one with three or four questions.
Examples:
If you want feedback on artists or a registration process, create a checklist: “Please rate on a scale of one to five the effectiveness for you of…” Include a list and a scale of numbers to circle.
If you want to know the best way to spend your advertising money, ask “ How did you find out about this event?” with a check-off list of advertising sources.

If your project focuses on a relatively small target audience (a summer drama group, an arts camp, a workshop) ask open-ended questions on a questionnaire and get a lot of helpful information.
Examples:
Please comment on the interaction you had with teacher/artists
Please describe the best and worst experiences you had this week and why.
Please comment on the registration process, etc.


If you do this, you must provide a time within the project activities for writing and have everyone sit down and do it—if you want to get it back. Remember that the most recent events will greatly color the response--a blow-up, a big emotionally charged performance, a disappointment in funding--but that is no reason not to do it.

REMINDER - CAUTION
Don't collect information if you have no plan to use it. Don't ask questions about things you have no intention of changing, or obvious problems you know you'll fix. (e.g. the setting, the time of year, caterers didn't show up, air conditioning broke, parking problems etc.) Collecting good data gets harder the more subtle or layered you make it. Avoid annoying people and yourself by asking too many questions too often.

Step Four: Write the report and pass on the results.
Write your evaluation report in a way that links the project goals with the evaluation data. Describe briefly what you did to collect data. Always comment on the significance of data so an outsider could understand what you have learned and why it matters. If you use evaluation tools to improve your project, tell what you asked and indicate how you will use that information, or how you have already changed your project for the future.


THINGS TO REMEMBER

A. Keep records. If you have a targeted audience, find a way to count that group as part of the ticketing or entrance process. e.g. At the Chicago Art Institute they ask for your zip code to record audience by area. If you are running an arts camp, record the number of applicants, as well as the number of actual participants. If you are building a community coalition, record what groups participate in what. etc.

B. Make data collection easy and fun and integrate it into the activities.
Examples:
Have door prizes for dropping off short questionnaires or information cards.
Have audience participate in art making that has them make choices—record their choices.
Take photos at activities or have people drop beans in a jar at favorite sites.
Set up a video camera with one-minute interviews. Use a monitor so people can see themselves on TV.
Videotape audience at performance.
Have a “record your comments here” taping booth, with a free Tootsie Roll reward. (or whatever)
Have participants write two sentences on “what I did today” and collect at dinner. (or whatever you think of... )

Some evaluation might be devices to note future behavior—number coming to other arts events, larger enrollments in next year's classes, broader participation of organizations in next year's activities. Plan now to take note in the future.
Examples:
Give people coupons or buttons to serve as discounts for future use. Do next year's registrations within first project year. Keep lists and numbers to match with future lists and numbers.

C. Counting counts.
It is one of the easiest things to do and one of the most persuasive elements of assessment. You can't affect people if they aren't there. If they are, it is important to know how many. Find a way to count them, and make that number meaningful to an outside reader. Use comparative figures (last year/ this year; regulars/new people; locals/out-of-towners; percentages of different community groups; etc.) Numbers in context are better than simple numbers.

D. Collect and analyze anecdotal/interview data systematically.
(Randomly is a system. Every fifth or fiftieth person, for example.) Figure out what you want to know—not what you want to hear. Ask the same questions and the same follow-ups. Try to create questions that don't direct the answer and aren't answered by yes or no.
Examples:
"What do you remember about today's event?”
"What else do you remember?”
“Is there any experience that particularly affected you?”
“Why?”
“What got you here today?”

Have a sheet prepared for note taking. Number and identify appropriately. (teenage boy in drama group; grandmother of participant; local resident; etc.) . Don't be overwhelmed by thinking you have to have huge numbers of interviews. A few (5 - 7) well-designed interviews might be a good tool for evaluating your project.

E. Design questionnaires so you can collect and compile data easily.
Make questionnaires simple, quick and pertinent. You can color code questionnaires for age groups, or locals/visitors; you can give check off questions so they can be done quickly and you can get a significant number back. You can give a prize for turning them in. Give people pencils to fill them out. Use kids or ushers or Boy Scouts to take on that task. Volunteers can compile data.
Examples:
I... ___am a first time attender
___have attended ___ times before.
I... ___live within an hour's drive
___am an outside visitor to this area.
I heard about this event from: ___newspaper, ___ acquaintance, ___newsletter

F. Analyze the significance of all data for an outsider audience.
In interviews or self-reports for example, look for key phrases or words and analyze them. You may wish to quote a typical statement, but not the single rave review. Puffery is not evaluation. If twenty people used the words “fabulous,” or “fascinating” say that. The analysis needs to be there. Explain why certain statistics are meaningful in terms of what you are trying to do.
Examples:
30% of the participants were regular visitors to our area who had never attended an arts event here before. Our goal was to make visitors aware of local arts resources.
70% of the audience checked “I learned several new things about the history of this region from this performance.” This is above our expectations.
50% of the interviewees said something about never knowing this or that about our local history before coming to the center. Our goal was to raise awareness of local history. We consider this a significant outcome.
10 of the 15 interviewees had come before, but said they would come again and bring people visiting them from other places. Our goal was to encourage local people to bring visitors to this site. We gave everyone free guest passes for their next visitors. We will count the number of these that return to the center.

G. Whenever you can, run a system several times in a row, or across several aspects of a project, for the most convincing data. Improve upon it, but try to collect similar information over time. You will learn more from consistent evaluation and make a stronger case for your successes.

FINALLY: Be Artful
Engage the imagination as you design, execute and report your evaluation. Let people know they are involved in evaluating an arts and culture project. Let the evaluation be flavored by the heart and soul of what you are about. This might mean colorful images, historical references, creating puzzles or treasure hunts, group collage making, putting on colorful post its—processes where you learn something by the choices people make or the elements they choose, and you and they are stimulated by the process itself.