Working with Qualitative Data in Student Affairs Assessment

For many, the word assessment equates to surveys, partly because surveys give us numbers (you know, data). This might be over-exaggeration, but there's some truth in there. Think about the assessment conversations that take place in your department and division. How many of them revolve around survey administration, response rates, and survey data? When I worked in student affairs, these topics dominated most assessment conversations I had. If all we're doing is looking at survey data, then we're missing a big part of the assessment pie.

One of the challenges that student affairs faces when it comes to assessment is that so many of us have gotten used to telling stories, and those stories have been the data we've known. All too often, though, we told the same stories over and over again, and they were often of some shining-star student leader. But what about the stories without the happy endings? What about the painful stories from which we learned valuable lessons about our work? I don't want to diminish the value of stories. Stories (for many student affairs professionals) are central to how we discuss our work. What we need to do is start looking at the stories differently.

Stories are data. Just as results from a single survey participant can't tell us much, neither can a single story. We need to be systematic about collecting and analyzing the data from student stories, just as we would with survey data. For the remainder of this post, I offer some how-to suggestions for working with qualitative data in student affairs assessment.

Write down your stories. Think about the stories you share - of the shining stars that you want the president to hear; and the darkest hours that you reveal to colleagues when you're at conferences. Ask your colleagues in your department to do the same thing. Take a look at the words on the pages. Are there common elements or patterns? Do the stories leave you with more questions? Both of these questions lead to -->

Ask open-ended questions about your work. We need to have a mechanism for answering the how and why questions about our work, and about our students. You need to identify questions that surveys can't answer, and that you've probably hinted toward these kinds of questions in your stories (you'll see them when you look at your stories with a wider lens). Qualitative methods are like any methods in assessment - everything begins and ends with questions.

Identify who you need to talk to in order to answer those questions. In order to move beyond the stories we tell, we need to collect data from various people with various perspectives. Then we need to develop means for analyzing data from those varied perspectives (more on that in a moment).

Qualitative data doesn't begin and end with interviews and focus groups. Look around your office, and your departmental space. What sources of data could there be about your unit, your students' experiences? Are perspectives about your work and students' experiences shared in the student newspaper? Have you ever thought to look at the agendas and minutes from meetings as sources of data? There are a number of potentially rich text-based sources of data available on your campus.

What do you do with all of this amazing data? This is where I've heard some discuss qualitative data analysis as being as much art as science. If you're reading this, you're likely more interested in the science side, which could be accompanied with a handy how-to list. So, here's the how-to list:

1. Get all of your data of spoken word into text form. If you've recorded interviews and focus groups, transcribe them. Word for word. You have to decide whether you transcribe all of "ums" and pauses. Depending on the topic, those elements could end up tell you a great deal.
2. You need to know the lens through which you'll view the data. Conceptually, what is the project about? Does it relate to engagement? If so, bolster your understanding of how engagement is framed in scholarly literature. Or, are you viewing the data through your student learning outcomes? Know the lens through which you need to view the data.
3. Read every transcript. Now read each one again.
4. Read every transcript, this time with your conceptual lens (from Step 2) in mind. Mark the passages that related to elements from that conceptual lens.
5. Look for commonalities. Which elements do you notice more often? More importantly, based on the data itself, what seems to be most significant? Qualitative analysis isn't about counting. It's about you interpreting the data through that conceptual lens, trying to make sense of perspectives from varied sources.
6. If you have someone you can call on for help, have that person complete steps 3-5. Compare notes, and discuss any discrepancies. Develop a plan to work through those discrepancies by developing a plan for re-analyzing the data.
7. When you've identified these areas where commonality and significance meet, summarize them in your own words. Pull poignant quotes from those passages you marked in Step 4. What you're doing is telling a story with the voices of multiple people, from multiple perspectives.
8. Share what you've found and engage people in conversations about what the findings mean and what actions should be taken.
9. Engage in conversations about the questions you have now that you've embarked on this qualitative journey. What more do you want or need to know?

I hope this isn't an over-simplification of the process for qualitative data analysis. Like many things, it takes practice and making mistakes in order to learn the ins and outs. I would love to have conversations about using qualitative approaches in student affairs assessment, so please reach out to me on Twitter @drbbourke, or post a comment or question below.