Examination Development

The ASWB social work licensing examinations follow strict guidelines for development and maintenance. The end result is a set of valid and reliable measures of minimum competency.

What are the steps?
They are the practice analysis, the linking to KSAs (Knowledge, Skills and Abilities), the exam blueprint, and the many steps in developing items. Then comes the work of the Examination Committee. The testing contractor, ACT of Iowa City, then does its part, from banking and analyzing items and pulling out questions for each form, and finally the administration--and more analysis. After all the number crunching, it's back to the volunteer content experts, the volunteer social workers, for more decisions.

The practice analysis
ASWB does a job analysis, a survey of the profession to determine the knowledge and skills needed by entry level social workers at each stage of practice, about every seven years. The most recent one was completed in 2003. A task force of volunteers served as content experts and did most of the real work.

The survey instrument is drawn up and sent to a random sample of social workers that represent an accurate sample of the profession. Depending on the overall response rate, additional, targeted mailings are sometimes used to ensure that sampling is sufficient.
Overall importance ratings for each task statement on the survey are determined for each of four groups (corresponding to the four exams) through the rating scales, marked for frequency and criticality. The tasks are then linked (by volunteers, again) to the KSA statements. KSA are grouped into content areas.

Exam blueprint
The content areas become the components for the outline for each examination, each of which consists of 150 multiple choice questions. The number of test items assigned to each content area is based on the total weight and number of KSAs in that area. The blueprint is very important, because if a licensure candidate has a poor grasp of say, human development and behavior, and 17 percent of the questions are on human development and behavior, the candidate is going to have problems passing the exam.

Each exam has its own blueprint. In the Bachelors, for example, the content areas are human development and behavior in the environment, issues of diversity, assessment in social work practice, direct and indirect practice, communication, professional relationships, professional values and ethics, supervision in social work, practice evaluation and the utilization of research, service delivery and social work administration.

While there is psychometric oversight of a job analysis, the science has to be tempered with the judgment of social work subject matter experts. This is provided by social workers, volunteers who are usually members or former members of social work regulatory boards. They spend time really dissecting their profession, trying to break down a humanistic job into measurable components-once people have done this, they often say, they do have an understanding of the exams that stays with them.

Passing scores
Passing scores are established by the Examination Committee, by what is called a modified Angoff method. Using a so-called "anchor exam," each committee member estimates for each item on the test what percentage of minimally competent social workers should get the item correct. Their responses are examined by the psychometric experts, and minor adjustments can be made by the ASWB Board of Directors. The set anchor exam becomes the yardstick by which all other forms of an exam are measured. This method ensures that overall difficulty remains consistent, even though inidividual exam items may change.

The test items
Once all this groundwork is done, the difficult part is to fill in the blueprints with questions that 1. fit, 2. are pertinent to the profession, 3. are just hard enough but not impossibly difficult, and 4. do not present unfair obstacles to candidates of different genders, ethnic groups or geographic locations, or for whom English is a second language.

Item writers
At any one time, there are 30 - 50 people all over the country under annual contracts to produce exam questions for ASWB. They are chosen for a mix of demographics, trained in a three-day weekend session, and sent home to write. Working with them are three item development consultants, who edit, advise and sometimes reject items.

Examination Committee
The next step for the item is the Examination Committee. No matter how hard the writer has tried, or how much additional work has been put in by the consultant, the Exam Committee is more than likely to find changes to make. A distractor is weak; another distractor is too good, and may be arguably correct; the item depends on a law that is not nationwide or too obscure to be fairly used; or someone argues that the entire question is a giveaway. Only when there is general agreement is the question accepted.

Pretest
New items are banked by the test contractor, ACT of Iowa City, and are sorted into pretest blocks of 20 and put on a form of the appropriate examination. The pretest items do not count, but they are mixed in so candidates must respond to them as if they were part of the exam. Once enough candidates have answered the item, it is replaced, and ACT psychometricians analyze the ways in which people responded to the question. If women do better than men, or whites better than African Americans, or if people who do well on the exam generally pick a distractor that isn't the key, or any number of other scenarios in which the statistics are not good, the item is pulled out and returned to the Exam committee.

Otherwise, it goes into the active bank, and can be used to fill out the blueprint on future exams.

Monitoring
The performance of the examinations and the Examination Committee are constantly being monitored, with ACT updating the association's staff and leadership on everything from the number of items approved in each Examination Committee meeting and percentages of pretest items that are successful to national passing scores. The exam changes constantly, as does the pool of candidates taking it. What social workers are doing also changes-a sound examination program has to reflect all this.

Inattention by a testing company, by staff, or by the stakeholders on regulatory boards in all jurisdictions can mean an exam that no longer does its job.

Examination scoring
Although every jurisdiction has a set passing score required for licensure, there is no set passing score for the ASWB examinations.

Because the actual items change from examination form to examination form, there is no way to establish a rigid passing score regardless of the version taken. Some administrations of the examinations will contain individual items that may be slightly harder or easier than other items on other administrations not by much, but by enough to make the establishment of one unalterable passing score impractical, and unfair.

To compensate for these variations, test administrations are equated, a psychometric process that accounts for the varying difficulties, and moves the passing score up or down accordingly. As a result, overall difficulties remain the same from test to test.

Once equated, the scores are provided to social work boards as scaled scores. Scaled scores are basically raw scores (how many questions answered correctly) translated into the testing language of the particular licensing jurisdiction. If, for example, a jurisdiction's licensing law requires a score of at least 70 out of 100 in order to become licensed, the score on a 150 item test won't translate directly into that system, especially after it has been equated. Raw scores have to be adapted to the scaled scores system used by particular jurisdictions.

Beginning in 2008, some jurisdictions opted to use a pass/fail system for scoring. In these jurisdictions, candidates receive results as a “pass” or “fail” with no accompanying score. Unsuccessful candidates receive diagnostic information on their performance in the test content areas.