“Conversation at the Boundaries Between Communities”: An Examination of Tutor and Peer Review Effectiveness Based on Commenting Practices

Download PDF

Contents

Introduction

Contextualization

A Review of Existing Scholarship

Methodology

Results and Discussion

Conclusions

Acknowledgments

Works Cited

About the Author

Results and Discussion

Overview of Findings

My analysis of comment effectiveness confirms my hypothesis that comments made by Writing Fellows are generally more effective in terms of higher-order emphasis and specificity than those made by members of the Rose Writing Studio, tending to point out specific higher-order issues and explaining the reasoning that warrants reconsidering the issue. For two out of the three drafts, the experienced Fellows’ average comment effectiveness score was higher than the respective Rose member’s score. In the two drafts where the Writing Fellow scored higher, the score of the Rose member was approximately one point lower than the Fellow. In the comments made on Sydney’s draft¹, the Writing Fellow, Claire, received a score 0.77 points higher than the student in the Rose Writing Studio, John, and in the comments made on Dave’s draft, the Writing Fellow, Sam, received a score 1.23 points higher than the student, Penelope. In contrast, in the one instance in which the Rose Writing Studio member received a score higher than the Writing Fellow, the member’s score was only 0.15 points higher than the Writing Fellow’s score. This emphasizes that, in general and among my small sample size, the Writing Fellows’ comments were more effective, though both kinds of review offered generally helpful suggestions and, in particular, prioritized higher-order concerns. Even so, the scores of the two Writing Fellows and among the Rose members varied considerably, highlighting that review is a highly individualized process for both writer and reader.

Higher-Order Concerns and Specificity

Writing Fellows and members of the Rose Writing Studio alike were apt to point out issues with higher-order elements of the drafts, underscoring the value of both forms of review. In Anna’s draft, for example, both the Writing Fellow and the Rose member recognize that the argument is lacking a “so what,” highlighting that her thesis, a higher-order concern, is underdeveloped. Sam, the Writing Fellow, comments that a “so what” would be good to include, perhaps by “discussing the ramifications of those examples in the present day.” A Rose member, Jill, similarly comments that “the ‘so what’ could be strengthened” to demonstrate why the topic “continues to matter today.” Just as both the Writing Fellow and the Rose member highlight a specific higher-order concern in Anna’s draft, so too in Dave’s draft do both Sam and Penelope recognize that the thesis fails to incorporate important elements of Dave’s argument that he engaged with earlier in the introduction. Sam, the Writing Fellow, asks Dave, “Are technological determinism or social constructionism important parts of your argument as well? Could you incorporate those ideas into your thesis?” Similarly, Penelope encourages Dave to “make it clear in this thesis that you do not support the idea of technological determinism.” Both reviewers recognize that the thesis is missing crucial elements and prompt Dave to edit the thesis, a higher-order concern, in his next draft. However, Penelope notably fails to address her concerns with Dave’s thesis in her end note, whereas it is the first element for Dave to address on Sam’s end note. This suggests that, while both forms of review tend to stress higher-order concerns, the Writing Fellow ultimately engages with higher-order concerns more effectively than the Rose member. This also reinforces the importance of end notes, which synthesize the marginal comments, making it all the more important that they emphasize higher-order concerns.

Though Writing Fellows and Rose members alike engage with higher-order concerns, the comments made by Writing Fellows adhered more consistently to Willingham and Sommer’s findings that comments must be specifically applied to the writer’s situation. In Sydney’s draft, for example, one lengthy paragraph lacked proper development, inundating readers with information and evidence without applying it to her larger argument. The difference between comments made by the Writing Fellow and Rose Writing Studio member are characteristic of the Writing Fellows’ general use of specificity: Claire comments that the paragraph “raises a really interesting point,” but it requires expanded analysis on “why HBO Max is not global,” the argument made in Sydney’s paper, and “what this means.” In contrast, the Rose Writing Studio member, John, asserts that Sydney provides “great examples” but that “adding some of your personal input (like your thoughts, reflections, and evaluations) can add immense value to your work.” While both comments make an effort to reinforce the positive elements in Sydney’s work, an important goal when working with student writers, Claire more efficiently ties the discussion to specific elements of Sydney’s argument, thus providing effective specificity to “guide [her] when editing [her] work” (Willingham 10). In contrast, John understands that Sydney’s paragraph lacks analysis but fails to engage with her specific argument, thus falling short of providing her with enough information to guide her editing for the next draft. Thus, Writing Fellows are more likely to offer the student-writer the “why” of their comment, helping the writer understand the “process” and not merely mend the “product,” to use Sommers’s terminology (156).

Lower-Order Concerns and Directivity

Though Rose members lacked specificity in their comments addressing higher-order concerns, they were notably meticulous when addressing lower-order concerns. In that way, members of the Rose Writing Studio acted as grammar and citation “convention informants” for their peers, a role that Writing Fellows comparatively avoided. The lower-order concerns addressed by members of the Rose Writing Studio involved wide-ranging grammar usages that they perceived as issues, from period placement to the use of prepositions. When citing a source in MLA, Penelope frequently encourages Dave to place his periods after the parenthesis; she also comments that his use of quotation marks is “somewhat distracting here.” John similarly informs Sydney to use prepositions such as “in” rather than “within” and “since” rather than “by,” as well as points out phrases in which the verb tense is incorrect, spelling is wrong, and spaces are needed. On Sydney’s draft, this results in John commenting eighteen times about grammar and usage out of his twenty-eight total comments. In both cases, comments made by Rose Writing Studio members stress grammar and citation convention at the expense of addressing higher-order concerns, instead inundating the writer with suggestions that point out genuine mistakes but fail to increase the draft’s readability overall. Based on this pattern, it appears that Rose members view themselves as convention informants for their peers; as freshmen, each of the members of the Rose Writing Studio has some knowledge of collegiate writing but is far from an expert. Thus, when the students find an aspect of convention that they believe to be true, they inform their peers of it, even when it detracts from their comments’ more nuanced engagement with higher-order concerns.

In contrast to the Rose Writing Studio members’ roles as convention informants, Writing Fellows tended to assume a more nuanced approach toward criticism, especially employing the use of questions. On average, Writing Fellows asked six questions throughout their comments, whereas Rose members only asked one; throughout the course of their comments on the three drafts, Claire and Sam asked eighteen questions, while John, Penelope, and Jill only asked four. Both groups used similar phrases when offering suggestions, such as “I would consider…” or “I suggest…”, but Writing Fellows seemingly viewed their comments as a form of discussion, offering inquiries that the writer might answer while moving forward with revision.

Addressing the Outlier

Though trends such as the use of questions and specificity emerged while analyzing the difference between comments made by experienced Fellows and Rose members, that is not to imply that the comments made across each group were similar in every way. Such difference materialized most prominently in the finding that one Rose member, Jill, received a higher comment effectiveness score than the experienced Fellow commenting on the same paper, Sam. Both scores, falling above 3.5 points, highlight that the commenters made overarchingly productive comments on Anna’s draft. Jill’s comments were more productive because, while she only commented eight times, five out of the eight times the comments included a very specific reference to the draft’s content, a higher-order concern that emerges, and the reason why a change would be helpful. For example, in regard to strengthening the draft’s “so what,” Jill relates the prompt to her recommended alteration to the thesis based on the paper’s topic, encouraging Anna to demonstrate “why colonialism continues to matter today, even if ‘the continent went from being under colonial rule to facing widespread independent movements,’ as the prompt describes.” While Sam offers Anna much more extensive feedback, it does not all relate to higher-order concerns, fomenting his lower overall score. For example, Sam comments extensively on word choice concerns, such as the improper use of the word “mocking” in one paragraph and the order of the words “markets” and “land” in one sentence in Anna’s introduction. In these instances, Sam’s comments address lower-order concerns and ultimately hinder his comment effectiveness score, prompting the peer reviewer, Jill, to achieve a higher score than the experienced Fellow.

As evidenced by the comments on Anna’s paper, a commenter’s effectiveness hinges on the individual and their response to a specific draft; just as comment effectiveness tends to vary from person to person, so does the effectiveness of comments by the same individual vary from draft to draft. The comments offered by the experienced Fellows, Claire and Sam, differed from one another, as did the comments made by Rose members, John, Penelope, and Jill. This finding highlights that, while it seems that, at least among this small participant pool, Writing Fellows are generally more effective commenters in terms of higher-order concerns and specificity, this conclusion must not be utilized as a reason to neglect the importance of peer review. Indeed, the Writing Fellows Program offers an elevated form of peer review due to the education of peer-tutors, yet even students unaware of writing center pedagogy can provide productive feedback as lay-reviewers.

Methodological Limitation Based on Findings

Upon discovering that Rose Writing Studio members often pointed out lower-order concerns in their peers’ drafts, I was prompted to reconsider my methodological focus on higher-order concerns. Among the guiding principles of writing centers is the development and long-term improvement of the writer (Harris), hence prompting my emphasis on higher-order concerns in my comment evaluation scale. However, it is important to note that there are ongoing conversations within the writing center community about addressing lower-order concerns, especially for English Language Learners (Nan, Teo, Min). While none of the drafts I analyzed were written by self-identified English Language Learners and because none of the authors requested that lower-order concerns be addressed in their intake form, I rated comments on grammar and usage as ineffective. While such an approach was, in this case, fitting, it might not be so when analyzing comments made on other drafts.

Another limitation of my method involves its emphasis on specificity in the rating scale. While specificity is important as a student revises their draft, Willingham also notes that comments “should not [be] so specific that students simply implement the instructor’s suggestions” (10). However, it was only upon analyzing my results that I recognized this trend. For example, Jill’s comments on Anna’s draft, while praised above for being specific, were sometimes detrimentally so. When commenting that Anna should include a sentence applying her evidence to her topic sentence, Jill writes that the evidence “perhaps…highlights disparate responses based on whether one was favored or not, a tradition that continues today with the paternalistic rule in many African countries.” She then goes on to offer Anna a scholarly article that she recently read in a class that could be beneficial for her argument. Such advice incorporates a higher-order concern, including argumentation, and is particular to the context of Anna’s draft, so it received a score of five. However, its specificity discourages the writer from critically engaging with the feedback, instead allowing her to restructure her application with the persuasive connection offered by Jill. A similar trend arose in John’s comments for Penelope’s draft, as he effectively engaged with the paper’s ideas but, in doing so, sometimes offered Penelope words and phrases that could be implemented as her own in a future revision. As such, while particularity in addressing one’s comment to the context of the paper is crucial, it is also important that the comment engages the writer’s critical thinking, yet my study failed to account for that fact. That said, it is worth noting that directivity is helpful in some contexts. In Jennifer E. Staben and Kathryn Dempsey Nordhaus’s “Looking at the Whole Text,” for example, the authors argue that a balance between directive and non-directive may be most helpful when working with English Language Learners.

Therefore, I believe that drawing a clearer line between specificity and directivity would likely not have significantly altered my findings, as a comment that is too specific, bordering on directivity, is, if anything, more helpful to a student writer than a tutorial is intended to be. A directive comment may still foster meaningful higher-order revisions to a student’s drafts, but it detracts from the overall collaborative experience because it dissuades critical thinking. Because I hinged comment effectiveness on emphasizing higher-order concerns and specializing one’s comment in the context of the draft, a consideration of the critical thinking that is involved in the revision process was not included. Hence, while the failure to distinguish specificity and directivity is unfortunate, it is not detrimental to my findings. Finally, it is important to note that I alone assigned values from the numerical scale to each comment. While I created this scale, in part to control for bias, this lack of triangulation may, to some, cast the findings in doubt.

^[1] Pseudonyms are used for all parties involved, including the authors of the drafts being commented on, the Writing Fellows, and the Rose Writing Studio commenters.