Framing Research Assessment for the Social Sciences and Humanities: A Community Validation Workshop

In an effort to translate Open Science principles into practical guidance for research assessment, OPERAS, the lead of the GraspOS Pilot on Social Sciences and Humanities (SSH), and SCIROS brought together SSH experts for a collaborative on site workshop aimed at validating draft guidelines and recommendations. The session embodied a fundamental principle: a meaningful reform must be shaped by the communities it serves. Rather than presenting finished products, the workshop functioned as a structured dialogue; an opportunity for participants to review ideas, identify gaps, and ensure that guidance reflected the views of the community.

What Happened

The day unfolded in two main sessions. The morning focused on high-level guidelines for how SSH communities might approach research assessment, while the afternoon shifted to practical recommendations, specifically how we can adapt the SCOPE Framework for Research Evaluation to SSH contexts. We opted for an active approach, with hands-on exercises that required participants to think critically and offer concrete feedback.

The first exercise, “Traffic Lights,” used colour-coded post-its to quickly flag content sections as strong, needing improvement, or inadequate. This was followed by more targeted work in the second exercise, “Adopt one section”: participants split into small groups to examine individual guidelines in detail, assess the narrative of the documents, its consistency, critical missing points and formal issues through multiple rounds of peer review.  Similarly, the third exercise, “Inside SCOPE” allowed a detailed analysis of the recommendations drafted within the GraspOS pilot tailored to SSH based on the SCOPE framework. Beyond the key documents, we also developed a checklist that can be used as a guide to summarise the key concepts to take into account in an evaluation. The fourth exercise, “Checking the checklist” aimed at improving this document. Throughout, the focus remained on receiving  actionable suggestions to improve the guidelines and recommendations.

Image 1. Participants of the workshop, Univeristy of Coimbra, October 2025.

Four Core Insights

As the feedback accumulated, four themes became evident across all exercises. First, participants called for greater clarity and precision. When documents aim to serve diverse audiences across the SSH domain, definitions matter enormously. What exactly is “qualitative assessment”? How should we define “local community”? The guidance needed sharper language and less ambiguity.

Second, they emphasised the importance of articulating the ‘why’. Readers needed to understand the reasoning behind framework selections, tool recommendations, and other design choices, not just be told what to do. This builds trust and helps users adapt guidance to their own contexts.

Third, structure and integration emerged as priorities. While initially we had two documents (one for guidance, and one for recommendations), participants felt that currently standalone resources could be merged, with stronger transitions and internal cross-referencing to create a coherent narrative rather than a collection of separate outputs.

Finally, participants stressed nuance and practical applicability. Guidance should acknowledge emerging challenges such as: preventing gaming-related practices is complex, the emerging role of AI in assessment and how it may affect writing and assessing Narrative CVs, and the tension between openness and quality assurance. 

Next Steps

These insights and many others translated directly into changes in the key documents. As examples, for the guidelines, contextual examples were added to clarify when qualitative assessment applies, strengthened transitions were created in key sections, and emerging concerns like AI in CV evaluation were introduced in the narrative. We also heard the need for explicit rationale statements explaining why certain tools and resources were selected.

For the SCOPE Framework itself, the editing centered on three interconnected improvements: conducting a thorough review of all unclear concepts and ensure consistency in terminology, clarifying with concrete examples how the experience in the pilot helped shape each recommendation for SSH, and refining the introduction and conclusion to better orient users to the framework’s purpose.

Image 2. Guidance and recommendations for research assessment in SSH: Lessons learned in the GraspOS project, p. 17; https://zenodo.org/records/17951789

Collaboration is key

SCIROS operates at the intersection of infrastructure, theory, and practice in Open Science for the SSH. Research assessment touches all these dimensions, which is why OPERAS sought to engage with SCIROS and participate in their study visits, to gather insights and feedback coming from actual SSH communities. The workshop proved invaluable. Participants came genuinely invested in contributing to guidance that would work in their own contexts. The structured exercises created the right conditions for that: formal enough to ensure an organised and oriented practice, but flexible enough to surface unexpected ideas. Guided by our colleagues, we were able to complete the guidelines and recommendations. 

The result of three years’ worth of work can be found here.

By validating these guidelines through community dialogue, OPERAS and SCIROS demonstrated a fundamental principle: the best guidance emerges not from expert judgment alone, but from collaboration with the people who will actually use it. This embodies what the SCOPE Framework calls “Evaluate with the Evaluated”, a principle that proved essential in practice.

What struck us most, however, was the energy and depth of the participants’ engagement. The feedback was excellent and actionable, but more importantly, the process itself reminded us why this work matters. The passion, creativity, and insights people brought improved our work in ways we genuinely didn’t anticipate. That’s precisely what community validation should do. We’re deeply grateful to SCIROS for their openness, collaboration, and generosity in hosting and supporting this workshop.

Note: This post was written collaboratively by Fotis Mystakopoulos & Carol Delmazo, project members of GraspOS on behalf of OPERAS Research Infrastructure.

Relevant Links: 

Delmazo, C., & Mystakopoulos, F. (2025). Guidance and recommendations for research assessment in SSH: Lessons learned in the GraspOS project. Zenodo. https://doi.org/10.5281/zenodo.17951789.Delmazo, C., & Mystakopoulos, F. (2025).

Designing Responsible Assessment in SSH: A Checklist. Zenodo. https://doi.org/10.5281/zenodo.17989484.

---

The text only may be used under licence Creative Commons Attribution 4.0 International. All other elements (illustrations, imported files) are “All rights reserved”, unless otherwise stated.

---

---