National social services agencies are dealing with the implications
of the changes in federal and state Welfare to Work (WTW) program
regulations brought about through the 2005 Deficit Reduction Act
(DRA).
The policies put in place by the new regulations have created an
increased emphasis on participation by welfare recipients with
dramatic penalties imposed on state and local agencies that fail to
meet federal participation requirements. Additionally many states,
such as California, are experiencing budget crises that have
seriously affected the funding streams usually available to social
service agencies for creating new programs and working with partner
agencies to better serve their recipients.
In addressing the need for effective and financially responsible
WTW activities, many local agencies are reviewing programs and
contracts with partner agencies n an attempt to show the financial
feasibility of program components. With many agencies used to
traditional budgetary practices that call for the comprehensive
spending of allotted funds with no system in place to measure the
effectiveness of the spending, there could be a lack of experience
in comprehensive program evaluation, which is what is needed. As a
result of the potential lack of experience, many local social
service agencies may be struggling with evaluation techniques and
strategies to discover how an organization demonstrates the
functional effectiveness and financial feasibility of a WTW
program, as well as the functional and financial effectiveness and
program value of a developed activity.
EZONE
The Employment Zone (EZONE) job readiness workshop was created by
the Kern County Department of Human Services to satisfy the state
and federal work participation requirements for recipients of cash
assistance. The workshop is three weeks in length, with a modified
two-week workshop created for KCDHS district offices outside of the
local Bakersfield area. The workshop includes eight days of
instruction from Mondays to Thurdays, with Fridays dedicated to job
searching. The afternoons of the second week and the entire third
week are also dedicated to job searching. The average class size is
30 participants per workshop.
Each week of the workshop focuses on specific aspects of the career
development process. The first week was designed to help people
gain the skills necessary to get a job (for example, resume
development, interview skills practice, change management, and
more). The second week focuses on the skills necessary to keep and
advance in a job (for example, organization and time management,
customer service, stress management, and money management) while to
supporting the participant's daily job search requirement. The
third week is dedicated to daily one-on-one coaching sessions with
the EZONE job developer, during which job contacts and leads are
distributed and collected and barriers to employment are discussed.
The initial results of the workshop showed that 70 percent of
participants gained employment by the conclusion of the workshop.
The results were such that KCDHS deemed EZONE a success - which led
to other county social service agencies inquiring about the
workshop specifics. The positive reception has led to EZONE being
included in "best practice" seminars in various state conferences.
A new and effective method to measure the impact of Welfare to Work
activities, such as EZONE, is critical in validating the financial
investment made on the activity by a social services agency.
Currently, the program evaluation consists of a standard exit
survey and a follow-up program that reviews the cases of class
participants who did not initially find employment. The cases of
participants who complete the activity without finding employment
are then reviewed monthly for a period of three months after the
date of program completion to learn if employment has been gained.
EZONE has been an established WTW activity for almost four years. A
comprehensive long-term study was conducted to examine participant
employment acquisition, long-term employment retention, and
post-workshop participation results for those participants who had
not yet found employment to compliment the data that was already
generated on job acquisition. The accumulation of data was
sufficient to also perform a ROI study proving the financial
feasibility of the workshop as an effective WTW activity.
Methodology
This case study utilized the Phillips ROI evaluation methodology as
the basis for data gathering and analysis. While ROI began as a
statistical evaluation tool, the methodology is growing in usage
and popularity and is currently being used to evaluate thousands of
programs in several practice areas, including training, human
resources, and healthcare, according to the ROI Institute.
Upon further research, there were few examples of ROI
implementation in social service programs and soft skill
development programs, of which EZONE is both. While an obviously
explanation for this could be that ROI is still growing and might
not have reached these business sectors as of yet, another reason
could be attributed to a number of detractors who label the
methodology as inappropriate for training that teaches attributes
of interpersonal development, such as soft skills.
Phillips asserts that ROI can be very effective when evaluating
soft skills programs where the outcome is the improvement of items
that can be measured with hard data; such as reducing turnover
through team building and communications training or compliance
with diversity or sexual harassment standards created by an
organization. Phillips attributes the misconception that ROI cannot
be used to measure soft skill programs to the misuse of the
methodology in smaller programs that do not need to be evaluated to
show a return on investment.
Study planning
The planning for the study consisted of designing and developing
tools for evaluating the functional and financial effectiveness of
EZONE and gaining approvals from KCDHS for the access to EZONE and
department statistics that are crucial to the undertaking of the
study. Tools included a post-activity questionnaire that focused on
isolating the effectiveness of the workshop by discovering past
participant views on whether the workshop was beneficial to them
finding employment. The questionnaire was randomly distributed to
participants from many of the past EZONE workshops. Additionally,
to conduct a case study involving human subjects, an informed
consent form was designed with the intent to be administered to the
study participants.
Another planning component was working with KCDHS to facilitate the
performance of the study. KCDHS was notified of the intent to
conduct the case study and was provided the details of the study
procedure. An interview with the EZONE supervisor was conducted to
learn of the history of the workshop and the causes behind its
development. Because EZONE is an established activity that utilizes
tested and presumably effective curricula delivered by trained and
experienced facilitators, there was no need to alter or modify the
delivery process. KCDHS was asked to provide access to the budget
for EZONE, as well as participation and job acquisition data
associated with EZONE to facilitate levels three, four, and five of
the evaluation process, which measured short- and long-term program
success.
The final component of the planning level was to identify possible
intangible benefits that were not able to be ascertained through
statistical data. Because of the importance of developing soft
skills - such as attitude, change management, motivation and self
efficacy to the success of the workshop - evaluating intangible
benefits was a huge component of the effectively evaluating the
efficacy of EZONE.
Anticipated benefits
Before undertaking the actual data collection, expenditures by the
department for each EZONE were estimated, as well as the savings to
the department as a result of a successful project. By estimating
the amount of money that would be saved on cash aid payments to the
roughly 21 of 30 (70 percent) participants expected to gain
employment and keep full-time jobs that paid $8 per hour, for three
months time, the amount would come to $25,200.
Factoring in the estimated expense of $3,958 incurred by the
department to run each EZONE, the expectation was that EZONE will
show a benefit to cost ratio of 6.37:1 and a ROI of 538 percent.
Intangible benefits included increased notoriety for EZONE and
possible benchmarking opportunities, and KCDHS management accepting
ROI as an effective program evaluation tool to be replicated for
evaluating other agency programs.
Data collection
When data were gathered, there were some roadblocks. Upon the
request of the necessary data and access, the researcher was
informed that due to confidentiality regulations KCDHS was unable
to allow access to any participant records that had identifying
features, such as social security numbers, names, or addresses.
Additionally, the researcher was denied access to the case
management system and could not expect any assistance from any
county employees while on county time.
These restrictions created several issues that needed to be
addressed for the study to continue:
- a lack of determination of any continued WTW participation, job
acquisition, and retention, of approximately 264 participants that
went off of aid as a result of EZONE participation
- the lack of access to client addresses meant that the intended
isolation method of a questionnaire mailed to participants would
not happen.
In addressing these issues, the researcher first looked at what was
actually provided by KCDHS. The researcher was provided statistics
of every EZONE workshop held in 2009, which included total numbers
and percentages of participants initially enrolled, the total of
participants who attended and completed EZONE, and numbers and
percentages of participants who found work during or within 90 days
of completing EZONE. The researcher was also given the spreadsheet
listing the jobs and starting salaries of every employed EZONE
participant and budgetary information regarding the running of the
workshop during 2009. The researcher determined that the data made
available was sufficient to provide an ROI that was based on the
standard scope of the workshop (0-90 days) without utilizing the
long-term statistics that would make for a more comprehensive ROI.
Additionally, to address the lack of isolation method, the
researcher created a questionnaire for KCDHS case managers that
asked for their opinions on the efficacy of EZONE. To circumvent
the restrictions placed upon employees participating in the study,
case managers were asked to complete the questionnaire at home or
on their lunch break, estimate answers to questions that asked for
statistics, and email the completed questionnaire to the
researcher's private email.
Lessons learned
Due to the limited amount of data made available to the researcher,
the study plan was modified to focus the study on utilizing the
available statistics to determine the ROI of EZONE to KCDHS. While
the actual ROI of EZONE - which included the amount of aid saved
through participants going off of aid due to jobs found as a result
of EZONE, long term participation, and a possibly more effective
method of isolation through learning what participants believed
from them directly, would be preferable - researchers expected the
the results to still show a positive ROIbecause the average
starting salary for EZONE participants was more than $1 per hour
more than anticipated, the actual cost of materials was drastically
less ($950 per year versus $3,600 per year) than anticipated, and
the amount of participants working within 90 days who completed
EZONE was not much different from the estimate (62 percent versus
70percent).
Whatever the result of the study, it is hoped that KCDHS
stakeholders will see the benefit of ROI and will want to provide
better access to records for further research. An idea that is
currently being discussed is to allow the researcher to become a
volunteer employee of the county during the data collection
process; which would give the researcher the desired access to
determine a more definitive return-on-investment.
Regardless of any future efforts, the current study has shown the
ability of a researcher to perform an ROI evaluation in a social
service agency; which will no doubt assist KCDHS stakeholders, as
well as stakeholders from other county, state and federal
organizations in determining both the practical and financial
benefits of their programs in the future.
-------------------------------------------------------------------------------------------------------------------------------------------------------------
Carl Rabun develops and facilitates trainings
for various organizations throughout California including, The UC
Davis Center for Human Services and Executrain. He is a doctoral
candidate at Capella University in the training and performance
Improvement program.