Big Open Science Podcast | S01E05: What If Research Evaluation Supported Openness? Reflections from a Collaborative Workshop #RESSH2025

Transcript below

[00:00:08.400] 

Hello and welcome to this special episode of Big Open Science Podcasts featuring a recording from the workshop A Reforming Research Evaluation in SSH Embracing Open Science for Diversity, Interdisciplinarity and Inclusivity, which took place at the RESSH Conference 2025 in Helsinki. My name is Marta Świetlik and I work in Digital Humanities center as an UX expert and community manager. I hold a PhD in digital humanities and I am trained as a user experience researcher, art historian and art curator.

This workshop format was first developed and tested during OPERAS 2024 Conference in Zadar. It proved to be effective and we shared the full methodology and reflections in a dedicated blog post. You will find all the details and links in the episode. Description in this workshop we explored how two international initiatives, the Coalition for Advancing Research Assessment and the Helsinki Initiative on Multilingualism, are driven evaluation reforms. Both explicitly align with open science principles such as transparency, accessibility and inclusivity to ensure that research in the social sciences and humanities is evaluated fairly and broadly, not just through narrow metrics. The workshop had four main to identify limitations in current research evaluation systems to gather stakeholder driven good practices that support diverse research outputs interdisciplinarity and multilingualism to encourage discussion on aligning assessment practices with COARA principles and to facilitate knowledge exchange on the practical implementation of reform strategies.

[00:02:04.470] 

As participants arrived, they were invited to self select one of four key stakeholder roles that best represent their background and perspective. First, researchers second group founders, policymakers and evaluators third group journal editors and publishers fourth group librarians and open science advocates. This approach helped us structure the conversation and ensure diverse voices were represented throughout the workshop and for the Helsinki edition, I wanted to take it one step further. Not only did we facilitate the workshop, but we also recorded the participants input and feedback. The goal is to better design similar events in the future and to support others who might be interested in running such workshops themselves. The format is of course open. If you would like to use it, please do. We warmly encourage you to adapt and build on it and if you do, let us know how it went. Share your thoughts and experience with us, because if we are to create meaningful knowledge, we must create it openly. Only then does it truly matter. The workshop began by highlighting key challenges in evaluating research in the social science and humanities. Traditional metrics like journal impact factors and citation counts often fail to capture the full range of valuable SSH outputs, especially those communicated in local languages or non traditional formats that better reach society.

[00:03:41.720] 

Since nearly half of Europeans do not speak a foreign language research limited to English risks excluding many audiences and relying on translators. So the session then introduced two important international initiatives driving reform. First, the Coalition for Advancing Research assessment, with over 700 signatories worldwide, promotes evaluation based on qualitative peer review and responsible metrics. CoARA agreement on reforming research assessment calls for recognizing diverse outputs like books, software, and public engagement and explicitly supports linguistic equity ensuring research in any language is fairly assessed. Second, the Helsinki Initiative on multilingualism, launched in 2019, focuses specifically on promoting multilingual scholarly communication. It urges institutions to support publishing in multiple languages, protect local language infrastructure, and ensure peer review and metric systems fairly value work regardless of language or publication channel. Both initiatives strongly align with Open Science values, which emphasize transparency, inclusivity, and accessibility of knowledge. They call for broadening what counts as scholarship to include diverse outputs and encourage interdisciplinarity, collaborative and locally relevant research. By adopting these principles, research evaluation in SSH can become fairer, more inclusive, and better suited to today’s complex scientific and societal challenges. This leads us to the core question we explored together in the workshop.

[00:05:36.570] 

How do current evaluation systems hinder the adoption of Open Science principles and how they can be changed to support openness, transparency and diversity in research outputs? During the workshop, after extended discussions, participants engage in a hands on group exercise called Designing Solutions. The exercise centered around the Evaluation Reform Blueprint, a participatory tool designed to help groups collaboratively develop practical, stakeholder informed responses to real challenges. Research Evaluations the blueprint encourages moving beyond discussion toward actionable planning. Originally developed and tested by the Operas Advocacy SIG during other advocacy workshops, this method guides groups through a structured process documented on a printed template. Each group use the blueprint to outline the following key elements. First, challenge description a clear explanation of the scientific research evaluation problem and its impact within the social science and humanities. Second, good practices, current good practices if they exist, or activities addressing this challenge in their own environments along the reflection of their effectiveness. Third, proposed solution the most important part so concrete actions proposed to tackle the challenge emphasizing alignment with Open Science principles, principles and reform goals for stakeholders involved so identification of key players and institutions who should participate in implementing the proposed solution and finally, risks and mitigation strategies so potential obstacles or risks anticipated during implementation and ideas for overcoming them.

[00:07:31.080] 

Groups documented their ideas directly on the blueprint templates, which serves both as a working tool during the exercise and as a foundation for the presentations. In the next part of this episode you will hear the presentations from the four groups each sharing their insights, proposed solutions and reflections based on this collaborative exercise, the first presentation comes from the researchers group represented by Agnieszka Szulińska from the Institute of Literary Research of the Polish Academy of Sciences. Agnieszka will share their group’s identified challenges, current group good practices and proposed solutions related to research evaluation from the researcher’s perspective.

[00:08:16.780] 

So researchers here. And of course we identified a lot of challenges as usually, but we decided to focus on or narrowed our observations to peer reviewing, because in our opinion, in general it is not very well recognized still as a scholarly activity. I mean, it is not rewarded financially or any other way enough, let’s say. And it’s usually too little time to do that. If you receive something for peer reviewing and there is not much open resources, how to be a good, let’s say in general reviewer, but specifically peer reviewer too. Sometimes. Of course, some journals provide some training, but not all. And some tools also provide peer reviewing recognition in a form like peer review, tracking how many peer reviews you’ve done this year or something. But it is not a common thing to find that information. And our proposed solution in that area are. For one thing, there should be a rotation of peer reviewers, because sometimes if you go on a journal or any other kind of website with peer reviewers lists, some people are already dead for decades and it’s not updated. Second thing is that of course, a publishing house or the institution interested in peer review should rather pay the person.

[00:09:48.060] 

That will be easier to assign time when you know you will be rewarded. And maybe this will be also beneficial. And also institution institutions like those affiliated with scholars with researchers should provide training for research assessment. I think peer reviewing is itself enough like specific to do a training solely on that. But from my experience, because I hope I will defend my PhD this year. Hopefully reading the PhD reviews is also something that is not. You can see that some people are not trained in that, let’s say. And of course, when it comes to stakeholders involved, we are counting on publishing houses and publishers in general. And for example, publishers might not only rotate the list of people who will perform this activity, but also may prepare a list of already done activities by a single person. For example, this professor did 14 reviews for us for two years, even with specific list or links. But the risk might be if it should not go into the past, because once it was decided that it’s a blind process, it should remain that for always. Not that oh, two years ago it was a blind process.

[00:11:18.280] 

Now we will go and go full transparency and we will publish who will review after a certain date. We go with an open process. And at least so I think this is mostly one more thing. When stakeholders involve, of course, funding bodies, it would be nice if not only, for example, when we are writing a proposal to get funding for our research project, it would be nice that our peer review activities, or review activities in general are also taken into account, not only, for example, our publications and so on. So I think that will be all. Or the group would like to add something. Thanks.

[00:12:01.470] 

Next we hear from the founders, policymakers and evaluators group with their presentation delivered by Ines Almeida from the University of Coimbra. This group focused on evaluation reform, challenges and solutions from the viewpoint of those shaping policies and founding decisions.

[00:12:22.220] 

Okay, so we are the funders, policy makers and at the same type, evaluators. So as challenge, we identified how to define evaluation criteria oriented to the output. Sorry, that usually the evaluation is oriented towards the output and not to the process itself. And so we need to ensure quality of the output. We need to look at the process and how the output was created, because in the end we want to ensure quality and we want to evaluate quality. But this is the big challenge. How can we do it? So what kind of existing approaches do we know? So we have in course some programs with stakeholders from the university to rewards the implementation of specific practices and outputs. And we are designing assessment criteria together with these people that are assessing and being assessed, such as in this case, graduate students, some funders providing researchers with support, for instance, data stewards, to ensure quality of what is produced in terms of the output data. And as proposed solutions. We think that we need to train the trainers on best practices so that they can evaluate the practices and outputs produced. It’s also important to have some guidance, high level guidance at the national level that can be then translated and adapted to the different institutions.

[00:14:37.790] 

And we should ensure some basic checks, such as, for instance, apply the FAIR principles to all types of research outputs, not only to data, because this will ensure at least that some steps in the creation of the outputs are ensured. And then you have information to evaluate the output produced. Who should be involved? So the evaluators, they need to be trained, the policymakers and even people at the universities, the different stakeholders, for instance, researchers that need to translate the FAIR principles to their own field. So we need people that know about this at a very high level, but also how can this be translated at the discipline specific level. Risks and MITIGATIONS we mostly identified the risks. The mitigations are still to be planned, but for instance, high bureaucracy. This may increase the time and effort that the people need to deposit on the production of their outputs without knowing that they will be somehow rewarded by that. And so that may lead to the motivation. And there’s also the risk of the discipline. Differences may pose some challenges. How do you ensure that this is applied across discipline, but in a similar way? So going back to proposed solutions, we also need to be sure that we can reward this type of practices, the implementation, not only the training of the people, but how they actually implement.

[00:16:44.030] 

And if this cannot be done at the high institutional level, we can at least try to develop parallel pilot programs that start to test how this can be done.

[00:16:58.920] 

Following that, the Librarians and Open Science Advocates Group, presented by Fotis Mystakopoulos, representing operas, will discuss their collaborative insights. Their presentation highlights practical approaches to support open, inclusive and diverse research assessment systems.

[00:17:16.920] 

So I think what we tried to address here is, and I volunteered some of my experience as a librarian in the past, where I think we look at it from the perspective that it supports staff mostly, so we didn’t try to influence in our discussion like policy or something like that, but mostly when it comes to it, when it comes to research assessment, what kind of activities librarians may or may not do or support or help. But we identified a very particular issue that maybe it’s not the most obvious thing to say here, which is the lack of actual human resources and expertise in supporting activities of research assessment. And you mentioned, for example, earlier, the focus on bibliometric analysis sometimes, which is not necessarily priority, specifically for example, citations, despite all the different activities that we have with normalization between disciplines and everything. But we also mentioned the research outputs, like we have diverse research outputs. How do you make sure that this information is out there visible? And I think the more specialization we seek in describing new outputs, I think it’s easier. Everybody knows about research articles, but when you go to the nitty-gritty stuff and you want more detailed analysis, I think this is where there might be an issue with what people know in terms of these activities.

[00:18:41.750] 

And it’s not about looking at a privileged group that really participates in these activities, but we’re talking about wider participation across universities, let’s say in the European research area. And I think by having more staff and by having more skilled staff or expertise around the issues that SSH is facing, which is not always obvious, then I think you can support services that provide services in relation to research assessment, as we discussed. So that was our thesis, let’s say, to begin with. And I think one of the second question I Think we addressed it more like not from what is happening right now as good practice, but also from the limitations that we see. That’s, I think, the journey that we took, the narrative. So I highlighted how sometimes librarians can have a narrow focus, like they focus on one discipline and they dive into it. And then we had like this conversation about how can we make it more interdisciplinary or have people who can work across different communities in a university, for example, if we’re talking about librarians in particular. So the narrow focus maybe lack of technical background, because librarians don’t traditionally, I think start with.

[00:19:58.370] 

At least from my degrees, the focus is not always technical. It’s more like how to manage collections. Of course, you get into some discussions around metadata or other ways of managing bibliographic information, but it’s not the developments that open science is taking from digital point of view. I think librarians still need more to gain, let’s say, more background on that of that information. And perhaps I think what we say here, lack of training. And I think what we were trying to say, that there’s no awareness a lot of the times on these topics, which more or less it’s part of the challenge, as that we mentioned earlier. So proposed solution is that of course we want more stuff on libraries to be able to support this kind of activities. They can be understaffed a lot of the times, I think depending on the size of the university or the funding that it receives. So this would be ideal. But also we would like new stuff with the technical aspects and the technical background that would actually support new solutions in, as we said, maybe discussing more SSH related outputs and how to make this more visible in the digital environment and beyond.

[00:21:17.580] 

And training of personnel, as we said, of people who are already there, existing. Yes. So yeah, and training people into, you know, how to manage the outputs from SSH stakeholders, management, of course in investors to get more funding to hire people sometimes. But this is difficult. Consortium to support us with training and sharing of resources and policymakers, institution, international organizations that can help drive the kind of changes that we need. And risks and mitigations not get funding, political instability, competition from other industries. If you want people with technical background. And of course then the strategy for mitigation would be to something I mentioned just now, community. Like get the community to support in terms of solutions and share the sources. And that was our mitigation, our main mitigation strategy for all of these issues. Thank you.

[00:22:18.780] 

Finally, the journal editors and publishers group with their presentations led by Zofia Smolarska from The University of Warsaw’s Center for European Regional and Local Studies will share key points from their discussions, including challenges and actionable proposals related to publishing and editorial perspectives in SSH evaluation reform.

[00:22:41.530] 

So our topic was exactly multilingualism.

[00:22:45.130] 

So.

[00:22:48.410] 

We chose the challenge of publishing in local and indigenous languages in open access and get high citations at the same time and get to Scopus and WC at the same time and recruiting reviewers in both languages or the one that know English and our local language at the same time. So that was the challenge. And then we talked about actually very many similarities between Poland and Peru, which was really surprising and eyes opening that we have similar challenges and similar ways to mitigate the issues. So the existing approaches are that publishers push the text back to the authors and ask them to translate them into English, which is not ideal of course, because first we don’t want maybe to publish only in English. So that’s one thing. And the other thing is that why authors should pay for such solutions in the first place. Not all of them have funding, not all of them are affiliated. What about the independent authors? But this is very, very common among publishers, just to push it to the author side. And also the other solution is also in Poland and in Peru get to get funding, governmental funding for translations.

[00:24:28.880] 

And well, the proposed solution would be first we were thinking that the rules should be changed for the ranking of the journals, first of all, so that for example, in our system the journals that are indexed in Scopus are automatically higher ranked in the ministerial system than than the journals that are not there. So maybe the policies should change. But then it turned out that we didn’t propose a solution on our side.

[00:25:02.770] 

What we can do, but what should.

[00:25:04.130] 

Be changed in the system. So we were thinking then at the very last moment what we can do and then the push the policies to change. So we should create a network of journals that wants to publish works in indigenous or local languages, maybe along with English versions, maybe not, and create this support network and make a pressure on policymakers to include the journals that are very important for the local community, for example, and put them higher in ranks according to the quality, not according to if there are or not in Scopus or in which language they publish. There was one solution with AI that we should maybe apply AI to make a rough translation, a sketch translation of a text for the reviewer if there is a language barrier, for example. Yeah, something like this. Yeah.

[00:26:16.570] 

We are coming to an end now. Thanks so much for listening. In this episode we shared recording from the workshop Reforming Research Evaluation in Embracing Open Science for Diversity, Interdisciplinarity and inclusivity held at RESSH 2025 conference in Helsinki. You heard about the workshop setup, the stakeholder roles participants choose, and how they use the Evaluation Reform blueprint to tackle real challenges. Then four groups presented their ideas and solutions from different perspectives researchers, founders and policymakers, librarian and open science advocates and journal editors and publishers. If you want to dive deeper in the topic, check out the upcoming blog post on skyros.nothesis.org all the links you need, including the blog, the blueprint template and the original workshop methodology by Operas advocacies are in the episode description. We’d love to hear your thoughts. Let’s keep working together to make research evaluation more open and fair.

[00:27:25.140] 

Stay connected and follow SCIROS. For more insight, find us at our blog skyros.nothesis.org Skyros project is made possible thanks to the support of the Polish National Agency for Academic Exchange under the Strategic Partnership Program. And remember, science is best when it’s open. See you next time on Big Open Science Podcast.

---

---