- Have any questions?
- +1(689) 256-5483
- info@myessaythinkers.com
Appropriate To Evaluation Research
This week we are learning about Evaluation Research. In your writing assignment I would like you to think of a program or policy that you think should be evaluated and outline how you would go about doing this and why. The lecture notes can be helpful in identifying a research design that you think could be the most appropriate. Remember that this is a short writing assignment, so there is no need to go into too much detail. I just want to see you apply the concepts from the readings.
The writing assignments are supposed to help you apply the class concepts to your own ideas and to practice your writing in preparation for the final research paper. Your writing assignment should be 2-3 pages in length. A couple of important points:
- Make sure to introduce any concepts from the class material you are referring to (briefly explain what they are).
- Be sure to cite, meaning use in-text citation in the (author, year) format and provide a short reference list or list of sources at the end of your write-up, using a standard citation format, e.g. APA.
E. Babbie, TheBasics of Social Researchtextbook 6th edition or 3th
WEEK 8
Evaluation Research
*
Chapter Outline
- Introduction
- Topics Appropriate To Evaluation Research
- Formulating the Problem: Issues of Measurement
- Types of Evaluation Research Designs
- Social Indicators Research
*
Evaluation Research
- Research undertaken for the purpose of determining the impact of some social intervention, such as a program aimed at solving a social problem.
For those of you who may want to study a public policy or social program, this may be a good part of a research design.
*
Evaluation Research
- Research purpose rather than method.
- Appropriate for any study of planned or actual social intervention.
- Goal is to determine whether a social intervention has produced the intended result.
- Results are not always well received.
Evaluation research is a research PURPOSE rather than a method.
In fact we use several research methods in it: Surveys, field work, interviews, etc…
It’s just like it sounds, you’re evaluating something, a program of some sort.
Not just for social science, but can be used anywhere.
A social intervention is an action take within a social context for the purpose
Of producing something.
Essentially, we as tax payers spend money on something, and we want to evaluate
To see if it works.
Of course it’s not always well received. No one likes to be told their work or project has flaws, that’s why it’s important to point out what works as well as what needs work.
*
Needs Assessment Studies
- Studies that aim to determine the existence and extent of problems, typically among a segment of the population, such as the elderly.
Typically there are three types of evaluation, at least in the social sciences. A needs assessment study is the first.
*
Cost-Benefit Studies
- Studies that determine whether the results of a program can be justified by its expense (both financial and other).
- But: Outcomes rarely amenable to straightforward economic accounting Possible escape: competition among programs
These are challenging, as there are so many things in the world you can’t put a dollar figure on.
For example, how much is a human life worth?
*
Monitoring Studies
- Studies that provide a steady flow of information about something of interest, such as crime rates or the outbreak of an epidemic.
- Often used now with “adaptive management”
With this type of evaluation, you are constantly monitoring indicators
Now organizations, whether an NGO, private, education, governmental, are often using
Adaptive management. Essentially, they try to make their management more flexible and
Dynamic so it can adjust to a program’s needs.
Common sense, but hasn’t always been used.
*
Program Evaluation/Outcome Assessment
- The determination of whether a social intervention is producing the intended result.
Again, we’re focusing on social intervention, but program evaluations are used everywhere
- The political science department, which you’re all a member of, will be conducting one next year.
Of course one of the biggest problems with program evaluation is people aren’t always thrilled
To be looked at under a microscope. It’s a lot like being audited. And let’s face it, when you’re audited
They don’t look for what you’re doing right, they want to find what you’re doing wrong or could do better. Also, there’s always the prospect of losing one’s job.
Hopefully if you do an evaluation you’ll tell people what they’re doing right too.
*
EXAMPLE PROGRAM EVALUATION
Oregon’s Property Tax Deferral Program for Seniors and Disabled Citizens
The Oregon Property Tax Deferral Program for Senior and Disabled Citizens was created to defer property taxes of seniors and disabled homeowners so that they could more easily afford to remain in their homes. Under the program, the State of Oregon pays property taxes on behalf of participating households, and the property taxes are to be paid back with six percent compound interest after the participant dies, moves, or sells the home.
Though the interest charged on the deferrals is meant to make the program revenue-neutral in the long term, a spike in enrollments and drop in repayments after the 2008 housing market crash has caused large deficits. The unexpected deficits, combined with an earlier decision to use surpluses to support other senior initiatives, nearly left the program without funds to defer taxes in 2010 and 2011.
These difficulties prompted several changes to the program, as well as an Oregon State University survey to gather more information on the people the program serves. This brief summarizes the recent difficulties in the program, subsequent legislative responses, and the results of the survey.
Created in 1963, the deferral program began with forty-two participants and less than $13,000 in deferred taxes. The deferral program grew slowly in its first decade before expanding from 262 participants and just over $144,000 in deferrals in FY 1975-76 to a peak of 13,165 cases and $19.9 million in deferrals in FY 1989-1990. The expanding caseloads of the 1980s led to large gaps between deferred taxes and repayments, but these disappeared in the early 1990s as people exiting the program and repaying their taxes began to outnumber new deferrals. The period from 1999 to 2007 saw steady surpluses around $7.5 million per year, and a drop from 9,200 to 8,500 cases. After receiving $108.8 million in appropriations between 1977 and 1995, the subsequent decade saw the deferral program’s revolving account return $78.7 million to the state’s General Fund.
The survey, which was funded by the Oregon Department of Revenue, was mailed to all program participants, including those reverse mortgage holders who were reinstated on a temporary basis. Of the 7333 surveys mailed, 2363 surveys were returned, for a response rate of 32%.
The results show that the Property Tax Deferral Program is helping a large number of long-time, low-income homeowners remain in their homes. More than 80 percent of respondents purchased their homes more than 10 years ago, and over half bought at least two decades ago. While the current annual income limit is just above $40,000 per year, 47 percent of respondents reported living on less than $15,000 in 2011, and more than 85 percent of respondents received less than $25,000. While the vast majority (over 97 percent) of respondents receive Social Security, 30 percent of respondents have no other source income. Given the low-incomes of most respondents, it is reasonable to conclude that property taxes would constitute a large monthly expense that many participants would struggle to meet.
Issues of Measurement in Evaluation Research
- Must be able to operationalize, and observe the outcome (e.g., increased public safety).
- Outcome (response variable)
- What was the program suppose to achieve?
- Experimental Stimulus (interventions)
- Population – demographic variables (within the population) as well as variables defining the population.
When evaluating a social program, you’re trying to evaluate the “Immeasurable”
If we want to measure an outcome, like reduce prejudice in schools, than how do you measure that outcome? That’s why you have to operationalize it. Is prejudice the number of times someone uses a profanity towards another person, etc…
- I mean, even taking a college class, there are learning outcomes. But how do we measure those
- Usually it’s a test, assignments.
- These are few things you may be able to measure.
- The outcome – what did grades look like?
- Experiment – how did the stimulus – Like Tyra banks in America’s Next Top Model, affect the outcome?
- This is key when you set up a program, you must have indicators of how well you’re doing.
- We use this in the economy: unemployment rate, housing sales and starts, Gross Domestic Product, etc…
- In school, it’s grades, graduation rates, drop out rates.
*
Evaluation Research Designs
- Experimental designs
- Quasi-experimental designs
- Time-series design
- Nonequivalent control groups
- Multiple Time-series designs
- Qualitative evaluations
*
Quasi-Experiments
- Nonrigorous inquiries somewhat resembling controlled experiments but lacking key elements such as pre- and posttesting and/or control groups.
Time-Series Designs
- Studies that involve measurements taken over time (e.g., Head Start Programs).
- [Head Start grants are awarded directly to public or private non-profit organizations, including community-based and faith-based organizations, or for-profit agencies within a community that wish to compete for funds.]
- Head Start is a provider of developmental and educational services to pregnant moms, babies, toddlers, preschoolers and their families, who live in poverty and struggle to meet the basic needs of life.
- Studies show it may take years to see the full benefits of such programs.
*
Figure 12-2 Three Patterns of Class Participation in a Longer Historical Period
Nonequivalent Control Group
- A “control group” that is similar to the experimental group but is not created by the random assignment of subjects.
Multiple Time-Series Designs
- An improved version of the nonequivalent control group design
- The use of more than one set of data that were collected over time
*
Figure 12-3 Active Initiatives over Time
Why Results Are Ignored
- Implications may not be presented in a way that nonresearchers can understand.
- Results sometimes contradict deeply held beliefs.
- Vested interest in a program.
Case Study: Rape Reform Legislation
(beginning in early 1970s in some states and continuing to be reformed)
New laws were formulated to:
- Redefine rape and include a series of offenses defined by aggravating conditions.
- Eliminate the requirement that the victim physically resist her attacker.
- Eliminate the requirement that the victim’s testimony be corroborated.
- Restrict introduction of evidence of the victim’s prior sexual conduct.
*
Table 12-1 Analysis of Rape Cases before and after Legislation
Social Indicators
- Aggregated statistics that reflect the social condition of a society or social subgroup.
Social Indicators Research
- Measurements reflecting the quality or nature of social life.
- Provides an understanding of broader social processes.
- Researchers are developing more refined indicators.
- Research is being devoted to discovering the relationships among variables within whole societies.
Death Penalty and Deterrence
- Social indicators help determine whether the death penalty works as a deterrent.
- If capital punishment deters people from committing murder, murder rates should be lower in states that have the death penalty than in those that do not.
*
Table 12-2 Average Rate per 100,000 Population of First- and Second-Degree Murders for Capital-Punishment and Non-Capital-Punishment States, 1967 and 1968
Ethical Issues
- Social interventions being evaluated may raise ethical issues.
- It will be difficult, and sometimes impossible, for the researcher to remain impartial.
- Due to the interests at stake, there will be pressure on the researcher.
- Evaluation research may be a mask for unethical behavior.
*
*
For those of you who may want to study a public policy or social program, this may be a good part of a research design.
*
Evaluation research is a research PURPOSE rather than a method.
In fact we use several research methods in it: Surveys, field work, interviews, etc…
It’s just like it sounds, you’re evaluating something, a program of some sort.
Not just for social science, but can be used anywhere.
A social intervention is an action take within a social context for the purpose
Of producing something.
Essentially, we as tax payers spend money on something, and we want to evaluate
To see if it works.
Of course it’s not always well received. No one likes to be told their work or project has flaws, that’s why it’s important to point out what works as well as what needs work.
*
Typically there are three types of evaluation, at least in the social sciences. A needs assessment study is the first.
*
These are challenging, as there are so many things in the world you can’t put a dollar figure on.
For example, how much is a human life worth?
*
With this type of evaluation, you are constantly monitoring indicators
Now organizations, whether an NGO, private, education, governmental, are often using
Adaptive management. Essentially, they try to make their management more flexible and
Dynamic so it can adjust to a program’s needs.
Common sense, but hasn’t always been used.
*
Again, we’re focusing on social intervention, but program evaluations are used everywhere
- The political science department, which you’re all a member of, will be conducting one next year.
Of course one of the biggest problems with program evaluation is people aren’t always thrilled
To be looked at under a microscope. It’s a lot like being audited. And let’s face it, when you’re audited
They don’t look for what you’re doing right, they want to find what you’re doing wrong or could do better. Also, there’s always the prospect of losing one’s job.
Hopefully if you do an evaluation you’ll tell people what they’re doing right too.
*
When evaluating a social program, you’re trying to evaluate the “Immeasurable”
If we want to measure an outcome, like reduce prejudice in schools, than how do you measure that outcome? That’s why you have to operationalize it. Is prejudice the number of times someone uses a profanity towards another person, etc…
- I mean, even taking a college class, there are learning outcomes. But how do we measure those
- Usually it’s a test, assignments.
- These are few things you may be able to measure.
- The outcome – what did grades look like?
- Experiment – how did the stimulus – Like Tyra banks in America’s Next Top Model, affect the outcome?
- This is key when you set up a program, you must have indicators of how well you’re doing.
- We use this in the economy: unemployment rate, housing sales and starts, Gross Domestic Product, etc…
- In school, it’s grades, graduation rates, drop out rates.
*
*
*
Figure 12-2 Three Patterns of Class Participation in a Longer Historical Period
*
Figure 12-3 Active Initiatives over Time
*
Table 12-1 Analysis of Rape Cases before and after Legislation
*
Table 12-2 Average Rate per 100,000 Population of First- and Second-Degree Murders for Capital-Punishment and Non-Capital-Punishment States, 1967 and 1968
