Download PDFLocal Assessment Design
On the first day of my assessment course, we were asked to start brainstorming something to assess. My instructor emphasized that our assessments could be implemented for real, but they did not have to be. I’m a bit of a doer so I imagined my assessment as real from that moment. It occurred to me that I could use this as an opportunity to provide my department (and more specifically myself and my peers) with some useful information for coordinating support for graduate students. Assessment is meant to be functional and local, and it’s most useful when it does something, when it has influence (Schuh et al., 2016; Sun, 2024). In other words, my purpose here was never to use my graduate student colleagues for research. It was never to make claims about the state of wellbeing for graduate students. I pursued this work because of the opportunity I saw to better support people I know with more information about how they were experiencing life in our department.
This has implications for things like Institutional Review Board (IRB) review and the ways I collected, reported, and shared data. Assessment does not typically fall under the purview of IRB because it is not considered research. Again, the purpose of assessment is not to produce generalizable data (Spoiler alert: This means I don’t share my survey results here). Knowing I would not be seeking IRB approval for this project set the tone for my work in that ethical considerations were my top priority. Having the guidance and structure of the assessment course I was in helped with this approach, which incorporated many opportunities for stakeholder input and offered support in ethical survey design, ranging from the wording of the questions to the accessible design of the survey interface. When pursuing this kind of work, assessment designers should deeply consider their purpose and how they will protect graduate students in the process. In my context, this meant deciding to coordinate a predominately quantitative survey, which would limit the amount sensitive information the survey would ask students to produce and the amount of sensitive information I would have access to as the person analyzing the data. However, before I could really think about the survey design, I had to establish a sense of the context in which the survey would take place.
I started with a case analysis of my department, beginning by situating it within the larger schema of the university. I’ve included excerpts of this case analysis in italics to illustrate the kind of information that was most pertinent. This kind of preliminary work not only helped to ground my thinking in my local context but later was useful for creating a comprehensive report that could be shared with stakeholders. In the excerpt that follows I situate the Department of English within the university, describe the various graduate programs within the department, and characterize the labor expectations for graduate students.
The Department of English is the sixth largest department at the University of Wisconsin-Madison and one of the largest departments housed within the College of Letters and Science. Home to the renowned University of Wisconsin-Madison Writing Center, the program of English as a Second Language, and two courses fulfilling the university’s general education communication requirement (introduction and intermediate composition), the English department provides crucial services to the university broadly. The department offers a graduate study program with four distinct tracks: Literary Studies (MA and PhD), Creative Writing (MFA), English Language & Linguistics (PhD), and Composition and Rhetoric (PhD). In addition, they offer administrative services for the graduate program in Interdisciplinary Theater Studies (ITS), though students in the program are not considered part of the English department. At the time the survey was conducted, approximately 136 graduate students were enrolled across the four tracks in English, excluding those enrolled in the ITS program.
Graduate students in English make up a dominant instructor body for English 100: Introduction to College Composition, English 201: Intermediate Composition, and an array of literary course offerings. The writing center also relies on graduate students as writing tutors, supporting undergraduate and graduate students across disciplines. In addition, graduate students in English work closely with faculty in project or research assistant roles, often in addition to their role as a course instructor or writing center tutor. Graduate students may also apply for select leadership positions for the English 100 program, English 201 program, and English 100 Tutorial (a small-scale writing center serving English 100 students), Writing Center, or Writing Across the Curriculum (WAC) program. Importantly, unless a student has been granted a fellowship opportunity, they will be expected to work in at least two of these roles to earn their guaranteed stipend. Graduate students in English navigate these various positions, including their corresponding orientations, staff meetings, and professional development, in addition to traditional graduate-level course work and degree milestones like preliminary portfolios, dissertation proposals, and oral defenses.
In addition to this more general information, I completed a document analysis of two department produced statements, a departmental values statement and a statement on excellence in teaching, which helped me to understand the disciplinary expectations of the department community. Again, from the aforementioned case analysis:
The Department of English has been working since 2022 to establish a Core Values document. This work has taken place over many department meetings and revising sessions, and the Core Values were approved by the department in April 2024. The values include Intellectual & Professional Growth, Dynamism & Diversity, Equity & Justice, Transparency & Shared Governance, Community & Mutual Responsibility. The department has collaborated to provide descriptions of how each pair of values will be demonstrated by the department. In addition to the Core Values, the department has developed the Excellence in Teaching statement and assessment guide. The statement and guide were compiled following the push for racial justice catalyzed by the murder of George Floyd, and a summary of the document highlights that “the department recognizes that excellent teaching must be inclusive in nature.” In the statement, the department identifies five “domains of teaching excellence,” outlining various pedagogical areas with outcomes for measuring each of the domains. The Core Values and Excellence in Teaching documents illustrate the department’s interest in providing equitable teaching and learning experiences for department participants. I recognize these documents to be a result of a long, deliberative process, where department members have worked to reach consensus about their shared values and aspirations within the department. Graduate students were invited to participate in both documents and common feedback reflected concerns about how the department would be able to meet the document’s intended goals. In thinking about the value of a survey like this, I believe the data could support the department in operationalizing the Core Values and Excellence in Teaching.
This case analysis document also gave me a starting point, particularly as a graduate student, to talk with department leadership about implementing the survey because I believed we were working under shared goals. One of the core practices for assessment development is constituent involvement because when stakeholders are thoughtfully included, they may be more likely to support the implementation of the survey and more likely to take up changes in response to survey data (Schuh et al, 2016; Sun, 2024). This meant even as I felt nervous about engaging senior faculty about a topic like this and didn’t want to appear overly critical of the department or my program, having their support was crucial. I began with an informational meeting where I sought to gain historical context for surveying in the department from the Director of Graduate Studies (DGS). At the end of that meeting, I proposed what I called the Survey on Graduate Student Wellbeing (SGSW) and outlined a timeline for data analysis. The DGS was immediately supportive, and I felt confident in moving to the next steps of the assessment development process. Building trust with stakeholders and emphasizing the value of assessment are crucial components of local assessment design. For me, this meant approaching the DGS from a generous and collaborative frame of mind, in a decidedly brief meeting, with a comprehensive plan and clear and specific asks for department leadership.
With support from the DGS, I returned to the model I was introduced to in Morris’s class, PREMISE, to start thinking about how the survey itself should be designed. PREMISE provided an articulation of seven nameable, measurable aspects that I could use to talk with graduate students about their wellbeing. Cochran (2022) describes the framework as “a synthesis of PERMA and flourishing (Seligman, 2011), Ryff’s six-factor model for wellbeing (1989), and Neff’s self-compassion theory (2003)” (p. 28). He writes that each of these models and theories alone has much to offer but combined provide a more holistic way to measure wellbeing within education spaces (p. 32). There are existing quantitative measurement tools based on two of the previously mentioned models: the PERMA profiler and the Ryff scale. However, Cochran (2022) notes that neither address identity formation nor efficacy, which he identifies as crucial for wellbeing measurement in the educational context (p. 36). With this in mind, I concluded that new assessment questions should be developed with the purpose of more comprehensively measuring PREMISE within the unique, local context of my department.
Cochran (2019) provided descriptions of each aspect of wellbeing, shown in Figure 1 which has the PREMISE category in the left column and the description of the category in the right column.
Figure 1
| PREMISE Category | Description |
| Positive Emotions | “Both situational positive emotions (i.e., a ‘good mood’) and long-term positive emotions (i.e., temperament).” |
| Relationship with Others | “Healthy social relationships that are supportive, encouraging, loving, kind, and authentic. Measures of healthy relationships include to what degree members in a relationship feel a mutual sense of secure attachment toward each other and feel supported by one another.” |
| Engagement | “The attachment an individual feels toward a community that could be measured by the individual’s commitment to stay in the community and interact with other members of the community. Educational engagement encompasses how involved an individual is in their work, their persistence in the face of challenges, and their delight in learning.” |
| Meaningful Experiences & Goals | “Discovering meaning in life is a process that can be developed through action, experience, writing, and reflection. Three pathways to discovering meaning in life (Frankl, 1956) include: 1) creating something, doing a deed, accomplishing something, 2) connecting with nature or with others through altruistic love, and 3) recontextualizing the suffering of self and others by exploring the meaning of suffering.” |
| Identities (Autonomously Endorsed) | “How we perceive ourselves and how we describe who we are (see, for example, Marcia's Identity Status Theory and Erik Erickson's Psychosocial Development Model). What are our strengths, our (often nonconscious) limitations, and how do we explore and more accurately understand both? We position that identities are optimally a composite of strengths that depend on social, personal, and emotional intelligences. For many humans, our identities may be projections at least partly informed by our nonconscious processes; we argue that the pathway to self-actualization comes by way of encouraging individuals to autonomously and consciously shape their own nonconscious processes, rather than by introjecting others' values and sensibilities.” |
| Self-Compassion | “A theory (Neff, 2003) for replacing self-limiting thoughts and behaviors with self-kindness, common humanity, and mindfulness. In educational contexts, developing self-compassion is necessary in order to move from fixed mindset to growth mindset (Dweck, 2006) and benefit mindset (Buchanan, 2015) to cultivate intrinsic motivation, self-growth, and generativity.” |
| Efficacy and Environmental Mastery | “Efficacy is the perception we hold of our abilities to achieve a desired goal. Ryff (1989) describes environmental mastery as the ability of an individual to create and change conditions in one's life suitable to their own psychological needs. Encouraging students' locus of control is essential when developing learners' self-efficacy and environmental mastery.” |
While quality assessments are designed around set outcomes (Sun, 2024), translating each aspect of PREMISE into measurable outcomes proved to be difficult in that so much of wellbeing is influenced by factors outside of the university space which vary widely among people. I needed to focus on the ways life in the department specifically impacted these aspects, and I knew my perspective would be limited by my positionalities. For this reason, I determined it was necessary to conduct a focus group on the intended outcomes, soliciting feedback from graduate students of a broader range of perspectives. I mostly relied on convenience sampling, inviting students who I had at least some previous interaction with and who had demonstrated interest in improving dynamics in the department either through their participation in department committees or other organizing efforts. Three students from the Composition & Rhetoric program and two students from Literary Studies attended the focus group meeting. During the focus group, I introduced the PREMISE framework and described each of the aspects using the descriptions in Figure 1. Participants had time to reflect on each aspect and their expectations for the department before the group reviewed the proposed outcomes together. Then, participants offered their experiences in the department, and we worked to revise and expand the outcomes as I’d drafted them. The result was greater specificity. In general, participants agreed with what I had written, but brought more specific language and sought to hold the department more accountable. I felt the value of collaborating with my peers and they expressed gratitude for the opportunity to think through wellbeing in our department in an actionable way.
Once we had articulated outcomes, I wrote questions that went through several rounds of drafting and revision based on feedback from various audiences. I knew I would get feedback from my classmates in my assessment class and from my instructor, but it was also important to solicit feedback from direct stakeholders, so drafts were sent to department leadership and a graduate student who offered a cognitive review. The result was a 45-question survey with 44 multiple-choice, Likert-scale questions and one open ended question. Graduate students can have more limited capacities on their time and energy, and it was important to me that the survey design explicitly and implicitly acknowledged that. In addition to prioritizing intentionality and concision in the question-writing, I made sure to include the projected time for completion (less than 10 minutes) on all the promotional materials and the survey itself. Faculty expressed concern that the length of the survey may affect the rate of completion or the number of survey respondents who would start the survey but not finish. However, graduate students I spoke with suggested the value of the survey data justified the total length.
Protecting student information and preventing potential harm caused by the survey questions were necessary priorities. While the survey was not anonymous in that I knew who the survey was being sent to, the data was anonymized through Qualtrics. This meant the survey was sent to individual students with a unique link to ensure validity of the data, in addition to confidentiality. Some students emailed me with questions related to the differences between confidentiality and anonymity; others had general concerns about how their data would be protected. I responded to each email individually, and in addition, created a frequently asked questions sheet on Google Docs where participants could read more about the survey design, data protection, and data usage. I saw this as another opportunity to establish trust with graduates who I wanted to feel confident they could complete the survey without harming their standing in the department or with faculty.
I completed analysis of the survey data (which included responses from nearly half of all enrolled graduate students in the department) using Qualtrics to identify significant statistical relationships for the multiple-choice questions. Then, I completed two rounds of inductive coding for written responses left by a third of graduate students who participated. I compiled this information into a report, which I first distributed to graduate students alongside an anonymous Google Form for submitting feedback or suggestions for how we should use the report. Comments expressed gratitude for the work that went into the survey, suggested the report be sent to department leadership, and that it be presented to the department broadly. Again, it felt important that graduate students had an opportunity to make suggestions at each stage of the process, particularly for how their data would be used. This was intended to increase graduate students' sense of agency and trust in the assessment.
In thinking about some of the limitations and further considerations for a survey like this one, I am excited about the opportunity to continue refining the process of assessing wellbeing. For example, I learned that the graduate students in my life were motivated by the opportunity to share about their experiences and further motivated when they had tangible data available to organize around. The survey also gave me an opportunity to collaborate with graduate students in other programs, which felt generative in my literary-studies-dominated department where the dynamic can sometimes feel contentious. While I saw that improvements could have been made to some of the questions to more accurately capture student experiences, I also recognized this as a recursive process. Results of the survey provided us with information about our department and wellbeing, but also about the survey design itself. In the future, I hope to continue thinking with stakeholders about various aspects of the PREMISE framework and the measurability of wellbeing, which is not (and should not be) limited to the kind of survey I facilitated here.