Introduction
Assessment has become a central focus in advising, as well as in higher education literature and practice (Campbell, Robbins, & Nutt, 2004). Wes Habley, director of the ACT Office of Enhancement of Educational Practices, during a National Academic Advising Association national conference keynote speech in 2000, called upon the advising community to make assessment a priority (National Academic Advising Association, 2005). While each campus is unique, thinking about assessment as a way to improve the overall student experience, both in and out of the classroom, can strengthen any institution's advising program (Light, 2004).
Effective assessment programs measure behaviors and outcomes of a program in relation to its goals (Campbell, Robbins, & Nutt, 2004). According to the University Advising Council of Penn State University (2005), “assessment should focus on behaviors and outcomes, not merely on satisfaction ... should assess the relationship of adviser and advisee, including both self assessment and assessment of the other ... [and] be used to improve a system, not just individual behavior” (para. 2). A thorough assessment of an academic advising program should utilize a variety of qualitative and quantitative assessment tools; however, most assessment programs rely heavily, sometimes solely, on student satisfaction surveys (Hurt, 2004). A student focus group, in addition to a satisfaction survey or as a part of a program employing a variety of assessment tools, can provide data concerning student learning from the advising experience and from the overall advisement process of the institution. A focus group, with carefully composed questions that engage students in a conversation reflecting upon what they have learned from the advising experience, has the potential to be a driving tool in an outcomes-based assessment. The data collected from an informative, well-executed focus group can be helpful in generating topics for surveys and other assessment tools.
Getting Started: Determining Size, Frequency, and Composition
Researchers have a number of important decisions to make as the planning of an advising focus group commences. Commonly, focus groups have six to ten participants and may be conducted three to five times per project (Morgan, 1997). It is essential to consider the particular needs of your institution and advising program when determining size and frequency. For smaller institutions, one or two focus groups may be sufficient whereas for larger institutions a number of focus groups will give a clearer picture of the student experience. In general, keep the number of participants between six and twelve so that the group is small enough that all participants feel comfortable speaking and transcription of the session is manageable. Early on in the decision-making process, researchers should determine if the focus group session will be taped. Using an audio recording device is helpful when transcribing the session, but permission must be obtained.
Assessment of an advising program should be a collaborative process (Campbell, Robbins, & Nutt, 2004; University Advising Council, 2005), and the formation of student focus groups should involve collaboration with established student groups, such as a student government association or commuter student association, and with departments, committees, or other university groups that can recommend students for participation.
Researchers must decide whether they would like to create a homogenous group or a diverse group. If a program is planning to run only one or two focus groups, it is advisable to control the composition of a focus group by recruiting participants from purposefully chosen categories that attempt to mirror the student population. For example, if a student population is 60 percent female and 40 percent male, a group of ten participants should have six females and four males. If a program is planning a series of focus groups, it might be useful to create different homogenous groups for each focus group session: one group specifically for commuter students, one group specifically for residential students, one group specifically for honors students, for example. The benefit of a homogenous group is that students may feel more comfortable speaking among peers with similar backgrounds (Morgan, 1997). A first-year student, for instance, will most likely be more comfortable discussing his or her advising experience in a group of first-year students than in a group comprised of mainly upper-division students.
Writing Questions and a Script
Composing the questions to be asked in the focus group is perhaps the most significant part of the design process, as it will determine the type of information collected. To obtain information greater in breadth and depth than that which is obtainable from a satisfaction survey, it is imperative to design questions based on desired student learning outcomes of the advising experience. Questions should focus on behaviors, not just satisfaction, and require the student to be both reflective of the advising process and self-reflective: for example, asking students about their preparation process for an advising session (How much time do you spend preparing your schedule before meeting with an adviser? What do you do beforehand to prepare yourself for a meeting with an adviser?); asking open-ended questions that would not work on a survey utilizing a rating scale (How do the student and adviser share responsibility for advising?); and asking questions that examine the adviser/advisee relationship (What could students do to improve the advising experience? What could advisers do to improve the advising experience?).
A carefully composed script should be written before the focus group session. First, be sure to thank participants for their time and willingness to participate. It is customary to offer something to the participants in exchange for their feedback. For a student group, a free lunch or university garb is usually greatly appreciated. Let students know that at the end of the session they will receive something as a token of the department's appreciation. Since students may be unfamiliar with the purpose and practice of a focus group, it is helpful to explain the purpose of focus groups in general as well as the specific purpose of the advising focus group. Students appreciate being treated with professionalism and seriousness as it shows them that their opinions and experiences are important to the department. It is important to let students know the goals of your assessment and to tell them that you are asking for their help and cooperation in meeting these goals. To create a comfortable atmosphere respecting student privacy, the moderator can solicit demographic information through a sign-in sheet without requiring students to give their names or specific personal information.
The focus group planners should discuss beforehand whether the moderator will strictly adhere to the script or will have freedom to ask follow-up questions for clarification of student responses. Choosing an objective moderator—perhaps an individual from another department or one who is not an adviser or advising professional—may encourage students to speak openly about their advising experience. An objective moderator should refrain from responding to student comments and explain to students that he or she is there to lead the session but not to pass judgment or offer opinions.
Before diving into the most essential questions, it is advisable to start with an icebreaker question in which each participant gives an answer (Morgan, 1997). In the script, indicate the questions that should be asked to initiate open table discussion and those meant to solicit an answer from each individual. To keep all participants involved and motivated, it is helpful to alternate between questions intended for individual responses and open table discussions.
One last thing to keep in mind as the script is composed is the number of questions and the length of the sessions. Remember that busy students are donating their time to your research. Also, in an hour session in which all participants are responding to questions and engaging in conversation, keep in mind that you may finish only four or five questions.
Understanding and Using the Information Gathered
Creating a transcript of the focus group(s) is the first step in understanding and using the data collected. There are many ways to interpret focus group data including coding and creating a quantitative analysis of the qualitative data. At Adelphi University where I administered an advising focus group, simply creating a summary of student responses met the department's needs. When large amounts of data are collected from multiple groups, the primary focus of the interpretation and analysis may be more complex, requiring group-to-group validation. Ultimately, the type of analysis chosen should be based on the department's individual needs and assessment goals.
In many instances, particularly at the onset of a new assessment plan, nothing compares to simply hearing students in their own words reflect upon their experiences and behaviors. This opportunity can help generate ideas for the creation of further assessment tools such as surveys or interviews and pinpoint how to measure student behaviors and outcomes.
Finally, maintain the collaborative spirit of the assessment process by sharing information collected and analysis with all parties involved in the assessment. Together devise an action plan to improve the quality of advising based upon the information gathered and determine which direction to take your assessment plan next.