The ASWB social work licensing examinations
follow strict guidelines for development and maintenance. The end result
is a set of valid and reliable measures of minimum competency.
What are the steps?
They are the practice analysis, the linking to KSAs (Knowledge,
Skills and Abilities), the exam blueprint, and the many steps in
developing items. Then comes the work of the Examination Committee.
The testing contractor, ACT of Iowa City, then does its part, from
banking and analyzing items and pulling out questions for each
form, and finally the administration--and more analysis. After
all the number crunching, it's back to the volunteer content experts,
the volunteer social workers, for more decisions.
The practice analysis
ASWB does a practice analysis, a survey of the profession
to determine the knowledge and skills needed by entry level
social workers at each stage of practice, about every seven
years. The most recent one was completed in 2009. A task
force of volunteers served as content experts and did most
of the real work. A new practice analysis is underway.
The survey instrument is drawn up and
sent to a random sample of social workers that represent an accurate
sample of the profession. Depending on the overall response rate,
additional, targeted mailings are sometimes used to ensure that
sampling is sufficient.
Overall importance ratings for each task
statement on the survey are determined for each of four groups (corresponding
to the four exams) through the rating scales, marked for frequency and
criticality. The tasks are then linked (by volunteers, again)
to the KSA statements. KSA are grouped into content areas.
Exam blueprint
The content areas become the components for the outline
for each examination, each of which consists of 150 multiple
choice questions. The number of test items assigned to
each content area is based on the total weight and number
of KSAs in that area. The blueprint is very important,
because if a licensure candidate has a poor grasp of say,
human development and behavior, and 17 percent of the questions
are on human development and behavior, the candidate is
going to have problems passing the exam.
Each exam has its
own blueprint. In the Bachelors, for example, the content
areas are human development and behavior in the environment,
issues of diversity, assessment in social work practice, direct
and indirect practice, communication, professional relationships,
professional values and ethics, supervision in social work, practice
evaluation and the utilization of research, service delivery
and social work administration.
While there is psychometric oversight
of a job analysis, the science has to be tempered with the judgment of
social work subject matter experts. This is provided by social workers,
volunteers who are often members or former members of social work regulatory
boards. They spend time really dissecting their profession, trying to
break down a humanistic job into measurable components-once people have
done this, they often say, they do have an understanding of the exams
that stays with them.
Passing scores
Passing scores are established by the Examination Committee,
by what is called a modified Angoff method. Using a so-called "anchor exam," each committee
member estimates for each item on the test what percentage of minimally competent
social workers should get the item correct. Their responses are examined by the
psychometric experts, and minor adjustments can be made by the ASWB Board of
Directors. The set anchor exam becomes the yardstick by which all other forms
of an exam are measured. This method ensures that overall difficulty remains
consistent, even though individual exam items may change.
The test items
Once all this groundwork is done, the difficult part is to fill in
the blueprints with questions that 1. fit, 2. are pertinent to the profession,
3. are just hard enough but not impossibly difficult, and 4. do not present
unfair obstacles to candidates of different genders, ethnic groups or
geographic locations, or for whom English is a second language.
Item writers
At any one time, there are 30 - 50 people all over the country
under annual contracts to produce exam questions for ASWB. They are
chosen for a mix of demographics, trained in a three-day weekend
session, and sent home to write. Working with them are five item
development consultants, who edit, advise and sometimes reject items.
Examination Committee
The next step for the item is the Examination Committee. No matter
how hard the writer has tried, or how much additional work has been
put in by the consultant, the Exam Committee is more than likely to
find changes to make. A distractor is weak; another distractor is too
good, and may be arguably correct; the item depends on a law that is
not nationwide or too obscure to be fairly used; or someone argues
that the entire question is a giveaway. Only when there is general
agreement is the question accepted.
Pretest
New items are banked by the test contractor, ACT of Iowa City,
and are sorted into pretest blocks of 20 and put on a form of the appropriate
examination. The pretest items do not count, but they are mixed in
so candidates must respond to them as if they were part of the exam.
Once enough candidates have answered the item, it is replaced, and
ACT psychometricians analyze the ways in which people responded to
the question. If women do better than men, or whites better than African
Americans, or if people who do well on the exam generally pick a distractor
that isn't the key, or any number of other scenarios in which the statistics
are not good, the item is pulled out and returned to the Exam committee.
Otherwise, it goes into the active bank, and can be used to fill out
the blueprint on future exams.
Monitoring
The performance of the examinations and the Examination Committee
are constantly being monitored, with ACT updating the association's
staff and leadership on everything from the number of items approved
in each Examination Committee meeting and percentages of pretest items
that are successful to national passing scores. The exam changes constantly,
as does the pool of candidates taking it. What social workers are doing
also changes-a sound examination program has to reflect all this.
Inattention
by a testing company, by staff, or by the stakeholders on regulatory
boards in all jurisdictions can mean an exam that no longer does its
job.
Examination scoring
Although every jurisdiction has a set passing score
required for licensure, there is no set passing score for
the ASWB examinations.
Because the actual items change from examination
form to examination form, there is no way to establish a rigid
passing score regardless of the version taken. Some administrations
of the examinations will contain individual items that may be slightly
harder or easier than other items on other administrations not
by much, but by enough to make the establishment of one unalterable
passing score impractical, and unfair.
To compensate for these variations,
test administrations are equated, a psychometric process that accounts
for the varying difficulties, and moves the passing score up or down
accordingly. As a result, overall difficulties remain the same from test
to test.
Candidates receive either a "pass" pr "fail" score. Failing candidates receive diagnositc informaiton on their performance on each content area.
|
