Working with Qualitative Data in Student Affairs Assessment

For many, the word assessment equates to surveys, partly because surveys give us numbers (you know, data). This might be over-exaggeration, but there's some truth in there. Think about the assessment conversations that take place in your department and division. How many of them revolve around survey administration, response rates, and survey data? When I worked in student affairs, these topics dominated most assessment conversations I had. If all we're doing is looking at survey data, then we're missing a big part of the assessment pie.

One of the challenges that student affairs faces when it comes to assessment is that so many of us have gotten used to telling stories, and those stories have been the data we've known. All too often, though, we told the same stories over and over again, and they were often of some shining-star student leader. But what about the stories without the happy endings? What about the painful stories from which we learned valuable lessons about our work? I don't want to diminish the value of stories. Stories (for many student affairs professionals) are central to how we discuss our work. What we need to do is start looking at the stories differently.

Stories are data. Just as results from a single survey participant can't tell us much, neither can a single story. We need to be systematic about collecting and analyzing the data from student stories, just as we would with survey data. For the remainder of this post, I offer some how-to suggestions for working with qualitative data in student affairs assessment.

Write down your stories. Think about the stories you share - of the shining stars that you want the president to hear; and the darkest hours that you reveal to colleagues when you're at conferences. Ask your colleagues in your department to do the same thing. Take a look at the words on the pages. Are there common elements or patterns? Do the stories leave you with more questions? Both of these questions lead to -->

Ask open-ended questions about your work. We need to have a mechanism for answering the how and why questions about our work, and about our students. You need to identify questions that surveys can't answer, and that you've probably hinted toward these kinds of questions in your stories (you'll see them when you look at your stories with a wider lens). Qualitative methods are like any methods in assessment - everything begins and ends with questions.

Identify who you need to talk to in order to answer those questions. In order to move beyond the stories we tell, we need to collect data from various people with various perspectives. Then we need to develop means for analyzing data from those varied perspectives (more on that in a moment).

Qualitative data doesn't begin and end with interviews and focus groups. Look around your office, and your departmental space. What sources of data could there be about your unit, your students' experiences? Are perspectives about your work and students' experiences shared in the student newspaper? Have you ever thought to look at the agendas and minutes from meetings as sources of data? There are a number of potentially rich text-based sources of data available on your campus.

What do you do with all of this amazing data? This is where I've heard some discuss qualitative data analysis as being as much art as science. If you're reading this, you're likely more interested in the science side, which could be accompanied with a handy how-to list. So, here's the how-to list:

1. Get all of your data of spoken word into text form. If you've recorded interviews and focus groups, transcribe them. Word for word. You have to decide whether you transcribe all of "ums" and pauses. Depending on the topic, those elements could end up tell you a great deal.
2. You need to know the lens through which you'll view the data. Conceptually, what is the project about? Does it relate to engagement? If so, bolster your understanding of how engagement is framed in scholarly literature. Or, are you viewing the data through your student learning outcomes? Know the lens through which you need to view the data.
3. Read every transcript. Now read each one again.
4. Read every transcript, this time with your conceptual lens (from Step 2) in mind. Mark the passages that related to elements from that conceptual lens.
5. Look for commonalities. Which elements do you notice more often? More importantly, based on the data itself, what seems to be most significant? Qualitative analysis isn't about counting. It's about you interpreting the data through that conceptual lens, trying to make sense of perspectives from varied sources.
6. If you have someone you can call on for help, have that person complete steps 3-5. Compare notes, and discuss any discrepancies. Develop a plan to work through those discrepancies by developing a plan for re-analyzing the data.
7. When you've identified these areas where commonality and significance meet, summarize them in your own words. Pull poignant quotes from those passages you marked in Step 4. What you're doing is telling a story with the voices of multiple people, from multiple perspectives.
8. Share what you've found and engage people in conversations about what the findings mean and what actions should be taken.
9. Engage in conversations about the questions you have now that you've embarked on this qualitative journey. What more do you want or need to know?

I hope this isn't an over-simplification of the process for qualitative data analysis. Like many things, it takes practice and making mistakes in order to learn the ins and outs. I would love to have conversations about using qualitative approaches in student affairs assessment, so please reach out to me on Twitter @drbbourke, or post a comment or question below.

Aligning #sagrad Programs with Professional Competencies

I generally find inspiration for blog posts from conversations and observations on Twitter, and this post is no exception. One common strand of Tweets marked by #sachat include statements or questions about the shortcomings of master's degree programs in higher education and student affairs. These statements and questions about #sagrad have prompted me to think a lot about aligning #sagrad program curricula with professional competencies. There are currently two sets of competencies I'm aware of: those offered by ACPA/NASPA, and the CAS Standards for master's programs. My focus in this post is the set of professional competencies by ACPA/NASPA.

On the surface, it looks pretty easy to map most competency areas to individual courses. However, instead of looking at the connection between competencies and individual courses, we should view the master's program in higher education and student affairs through an integrative lens. The learning outcomes associated with each competency should be addressed across the curriculum. Don't we want student affairs professionals to approach their work through the integration of multiple frameworks?

Rather than leave the reader with a question to ponder, I offer some ideas on what would be required to work toward an integrative approach for faculty, students, and professional associations:

What does this require on the part of faculty?

As faculty members in graduate preparation programs, we should always have a finger on the pulse of the field. Not only do we need to keep our course content current and relevant, but so too should our approaches be responsive to changes in the field. Within our individual programs, we need to work with our colleagues to map the curriculum to professional competencies. This can and should be part of regular self-studies that fit into larger assessment plans. We need to know that our curricula meet the needs of the field, while leaving room for individual program identities and niches to emerge.

What does this require on the part of students?

Recognize your responsibility in your learning and preparation for the field. The process of formal education and learning isn't simply the transmission of knowledge to willing learners. Preparation for the field of student affairs through a graduate preparation program requires students to be active in their learning and preparation. Students need to think critically about their learning in the field beyond coursework. How will they make sure they're getting what they need to address professional competencies?

Faculty and students alike should refer to the rubric for professional development published by ACPA that is a companion for the ACPA/NASPA Professional Competencies. For faculty, the rubric can be useful in distilling learning outcomes from the professional competencies. For students, the rubrics can be seen as a road map for professional development, and as a self-assessment tool of competency.

What does this require on the part of professional associations?

This line thinking has also been spurred on by a recent #satech summit hosted by NASPA that doesn't appear to have included faculty from any graduate preparation programs in our field. I'm excited about conversations about professional competencies, but if we are going to hold #sagrad programs accountable for the training and readiness of new student affairs professionals, then faculty should be part of these conversations from the beginning. There isn't a complete disconnect between professional associations and graduate preparation faculty. Since finishing my master's degree in 2000, I have seen more efforts to reach out to faculty on the part of multiple professional associations in student affairs, particularly in the time that I've been a faculty member (since 2007). Something that I've been wanting to see (and I do see in pockets) is greater interaction within associations beyond face to face conferences. We have so much technology at our disposal. Let's use it to connect and engage in discussions about our profession.



All of us engaged in the practice and teaching of student affairs have a vested interest in the future of graduate preparation. Clearly, I want to be part of these kinds of conversations. I want to have the conversations with faculty colleagues from other higher education and student affairs programs; with leaders of professional associations; with student affairs educators at all levels of experience. I'm looking forward to the NASPA session featuring Larry Roper and Pat Whitely (Tuesday March 18 at 8:45 AM in Convention Center Ballroom I & II) as they share their thoughts on the future of graduate preparation programs.

I'd love to read your thoughts on what faculty (either collectively or individually) can or should be doing differently. If nothing else, I want to align my practice with the current realities of the field.

Student Affairs' Role in Online Higher Education

I became active on Twitter in late 2012, and since that time I have participated in conversations that keep leading to a basic question:

What is the role of student affairs in online higher education?

After reading Keith O'Neal's blog post about community in online courses, the answer to the above question came to me as two interrelated concepts:

• Community building
• Engagement

These two things are central to the work of most student affairs educators who work with students in face to face environments. Thanks to our #sagrad programs, we are all well-versed in approaches and techniques in helping students develop communities that are supportive of meaningful and purposeful engagement. We should be able to take those concepts that we rely on for face to face interactions and settings, and simply duplicate them for online students, right?

Not necessarily. Hopefully by now you've come to recognize that most of our students do not fit a narrowly conceived definition of "traditional" college student. And, hopefully you've adapted your approaches to meet the needs of an increasingly diverse student population that is stretched thin thanks to myriad responsibilities beyond those of a "traditional" college student. Online learners require this same type of recognition and mindset. If you'll bear with me, I have some suggestions of how student affairs can support community building and engagement of online learners.

1. Check your ego at the web browser. We have to remember that helping students build community and become engaged isn't about us. We should be seeking to help online learners because we're experts in community building and engagement, just as faculty are experts in their content areas. We have different roles in supporting online learners.
2. Talk to faculty who are teaching online. Chances are, their online class structure is the one place online students interact. Find out what the interactions are like. Are the faculty engaged? Do faculty rely on discussion boards for course-specific topics? We need to know what types of efforts (f any) exist to help students connect and interact.
3. Assess. Talking with faculty can and should certainly fit into a broader assessment approach, but there needs to be a broader assessment approach, and an assessment plan. It wouldn't make sense to charge ahead to serve online learners without first assessing their needs.
4. Remember that online learners are online for a variety of reasons, and there's a chance that building community and engagement are not on their radar. But, thanks to decades of research, we know that these two elements go a long way to fostering student success.

It is time for student affairs to assess its role in promoting community building and engagement for online learners. Not all functional areas will have an active role with online learners, but we all have a vested interest in supporting these students. Why? Because they're our students, regardless of where or how they learn and engage.

Please feel free to share your thoughts in the comments. You can also find me on Twitter @drbbourke