Big Open Science Podcast | S01E13: Measuring the Impact of Open Science: CSIL, PathOS, and the Cost–Benefit Perspective

Speaker 1

Welcome to the next episode of the Big Open Science podcast, where we explore the present and the future of open science in the social sciences and humanities. I’m Cesare Roschinski, and today we are recording from CSIL in Milan. In this episode, we’ll be talking about how the impact of open science can be understood, measured, and evaluated. From a policy and economic perspective. We will focus on the Pathos project, which set out to develop concrete frameworks and tools for assessing what open science actually delivers to research, society, and innovation. To discuss this, I’m very pleased to be joined by Louis Colnaud from CSIL. Louis, welcome to the podcast. To set the scene, could you start by telling us a bit about yourself and your role at CSIL?

Speaker 2

Hello. So thank you very much for having me on the podcast today. It’s a pleasure to be here. As you’ve said, my name is Louis Colnot. I’m a senior researcher here at CSIL. I have a background in social sciences in general, mostly with a specialization in political sciences and economics. And here at CSIL, I mostly started by working on issues related to regional developments. So as some of you listeners may know, cohesion policy, which is very big, especially in Eastern and Southern member states. And gradually I’ve got more and more interested and engaged in projects related to science policy, trying to see how scientific research connects to policymaking, economic development, and integration of citizens.

Speaker 1

Excellent. That was, that was perfect. And we are very happy having you here and having CSIL in Skyros project. So we’ve mentioned CSIL a couple of times. So let’s zoom out for a moment. Could you tell us about CSIL itself, what the institute does, what its mission is, and how it operates within research on science policy and innovation?

Speaker 2

Yes, of course. So we are a cooperative company which was founded quite some time ago in 1980. And basically it started as a company more oriented toward market research, especially in fields that are very important for the Italian economy, namely the furniture market. So delivering advice to companies active in this field to understand the trends and everything. And gradually the focus of the company expanded towards the analysis of public policies. So we started in different fields, notably as I mentioned before, cohesion policy, regional development, and so on and so forth. And we have a variety of different backgrounds of people in the team, including mostly economists, but also sociologists, political scientists, and so on and so forth. And basically we have some kind of common toolset of methods that we can apply to try to understand public policies in different fields. And we were always very interested in innovation, which connects to economic development, but also to scientific research and so on and so forth. So over time, we got more and more engaged into this research applied to scientific research. Basically, especially at the beginning, it was mostly focusing on infrastructures. So let’s say more the hard and tangible part of research.

Speaker 2

Trying to understand how infrastructures are developed, operate, what kind of impact they have. And we expanded over time our focus. We now regularly assess different research programs on behalf of clients such as European Commission, for instance. We are supporting, for instance, the emergence of new schemes, of new research infrastructure. So we always try to connect the parts linked to analysis with recommendations in order to improve things on the field. So basically, science policy is an emerging field for us. We’ve been working for several years now, but it’s definitely something where we are pushing forward.

Speaker 1

Okay. So this is a very interesting set of elements, and we are happy that we can bring them together into the topic of science. And break a bit a silo of academia by connecting our experience. So, to focus on science policy naturally connects to the PATHOS project, which CSIL was involved in. What were the main objectives of PATHOS and why is the question of open science impact particularly important at the European level? Building on that, what role did CECIL play in the project? So basically, what were your team’s main tasks and responsibilities?

Speaker 2

Yes, of course. So maybe just to give you some general framework background about the PATHOS project. So it’s a Horizon project, so formally a research project involving the team from different institutions across the EU. And basically the idea was to try to analyze the situation regarding the impact of open science, what could be said in the existing literature, and so on and so forth, what kind of tools are present, and to go beyond that, to try to develop new methodologies, new approaches to try to understand better this kind of potential impacts. We know that open science can have positive impacts, it can have unintended consequences, it can have cause or even negative impact in some cases. So the idea was really to take an objective stance on the situation and to develop this new set of tools that could then be applied by researchers, by policymakers to understand better what’s going on and to try to improve over time. So this was the general spirit of the project. And as I’ve mentioned, there were lots of different partners involved. Different specialties. One of the key elements that we contributed to during this project was the identification of what we call impact pathways.

Speaker 2

So it’s a, it’s a fancy word, but it actually, it’s fairly simple. An impact pathway is really a chain of causes and consequences, but start from the very basic activities you may do in an open science project. So for instance, organizing a workshop, having some kind of repositories where you put papers and so on and so forth. And you see how these activities are supported by resources like funding or having skills, specific skills and so on and so forth. And you track down these activities up until the very indirect effect they can have. So for instance, if you have a repository holding different scientific papers, one of the impacts in the long term may be that science is more accessible to citizens and they have some better understanding of the situation and improvement of scientific culture and so on and so forth. So basically a big part of the work was to try to reconstruct these potential pathways. And of course, they are different depending on the type of open science project we are mentioning. Given the huge variety of things you can do with open science, we have expectations that we may lead to different types of consequences.

Speaker 2

So the idea with Pathos was to try to put the spotlight on some of the major pathways that could then be applied in different contexts, but also of course always to be refined. So we contributed to that and one of the other major a contribution of CSIL in the framework of this project was building a framework for cost-benefit analysis, which is a specific methodology, but we can discuss later, I guess.

Speaker 1

And yes, so I like the idea of impact pathways, and it gives you a perspective that we don’t act in a void. So there is a bigger background and causality of our actions, and it’s nice to know that we can We can measure that. And speaking of cost-benefit analysis, so one of the distinctive elements of Pathos was its analytical approach. Could you explain what the CBA framework is and how you adapted it to capture the specific dynamics of open science?

Speaker 2

Yes, of course. So cost-benefit analysis is a fairly well-known approach in economics. And basically the idea is to try to compare the costs and the benefits of a project, an intervention. It can be really different things. For instance, you’re setting up a new research center or you have a big portfolio of scientific projects in one field. And basically you try to understand what are the costs of this intervention you’re trying to give a value to and its benefits. And the basic rationale of cost-benefit analysis is that if the benefits are greater than the costs, it means that it’s desirable to have this kind of intervention. So it’s kind of a systemic approach to try to value projects, interventions based on their potential consequences. And it’s called cost-benefit analysis, but it doesn’t mean that it’s only about money. We’re not that obsessed about money. It’s just that basically in economics, we are trying to understand welfare. And welfare is a little bit like satisfaction, what kind of good you can get from a project. And we are using basically money as a way to compare different things. For instance, you have financial costs, of course, but you have also different kinds of costs.

Speaker 2

For instance, if you’re spending time studying something, it’s time you’re not doing something else.

Speaker 1

Okay.

Speaker 2

There are also benefits that do not have a financial value but are still important. For instance, the typical example is pollution. Or if your action actually saves pollution, it may not give results to an observable benefit in terms of finance, but it has a value. So basically, the idea of cost-benefit analysis is to try to be comprehensive, to track the financial aspect, but also the non-financial aspects, and to compare them all together in a big mix to try to understand if the project, the intervention, is worth it or not. Basically, this approach has a lot of different advantages compared to other frameworks. First of all, it’s really an approach that tries to have a long-term or at least an adapted time perspective. It’s very important in research because, for instance, you may— let’s imagine you have a project where you want to digitize archives. For them to be used by researchers later on.

Speaker 1

I like this scenario. I can imagine that.

Speaker 2

Yes, I’m sure you can. At the beginning of the project, you may have a lot of work in trying to actually pull the documents from the archives and so on and so forth, trying to digitize them. So it may actually cost a lot of resources, a lot of time. And for, I don’t know, maybe 2, 3 years, you have only costs. And of course, you cannot say this project, I will evaluate it in 3 years because then you are only costs and no benefits. So basically, CBA is saying you need to have an analytical period that is consistent to what the project is trying to achieve.

Speaker 1

Okay.

Speaker 2

So for instance, I don’t know, for this kind of project, it can be 10, 15 years where you can actually see the costs and the benefits and compare them on a level playing field, let’s say. So this is one of the important elements. Another key aspect of CBA is that we are interested in what we call an incremental perspective. What it means is that you do not only observe costs and benefits that happen in real life, but you need to compare them to what would have happened without the project, or if something else would have been done. Because it’s clear that in some cases, some of the benefits may have happened without the project. For instance, I don’t know, if you have an open science project that leads to more publications, some of them may have happened even without the project. So when you’re considering the cost and benefit, you must always try to compare what would have happened with the reality. Of course, it’s very difficult to do, and it implies that you have to engage with the stakeholders, with the scientific community to understand what is realistic, what could have happened, what is subject to debates, and so on and so forth, to try to reconstruct these scenarios, let’s say.

Speaker 2

And this is often something that is not done when doing impact assessment analysis because it’s quite tricky. It can lead to resistance and so on and so forth. So it’s an important part of the process, let’s say.

Speaker 1

Okay, so it’s interesting. What I am taking from what you said, it’s like CBA It’s partly like a long-time investing. So you need a time frame to see also benefits, not only the costs. And some kind of multiverse or alternate histories that you can see several pathways what can be done. Okay, so how you adapted CBA to capture the specific dynamics of open science then?

Speaker 2

Yes, it’s a very good question because traditionally this type of analysis was applied in things like evaluating transport infrastructure, things like that. So very tangible, repeatable investments. And when you’re dealing with science in general and open science in particular, you have some specificities that make it particularly challenging. So as I’ve mentioned before, in the past at CSIL, we basically pioneered the use of CBA for research, especially in terms of research infrastructure. So for instance, we worked with CERN in Switzerland to evaluate their colliders and so on and so forth. So we had this experience of trying to apply CBA into, let’s say, a more standard infrastructure in research. And this already has some major specificities because you’re dealing with science. So you have the fact that the time horizon is quite long, the fact that the results can be uncertain because before starting the scientific project, you may not know exactly what you’ve been discovering. It’s basically the purpose of it. You’re aiming for something, but you don’t know exactly when and what you will discover. So this is an important issue to discover. But basically, in the case of open science, there is an extra layer of complexity that should be taken into consideration.

Speaker 2

So first of all, I would say that compared to, for instance, infrastructures, Open science can cover a lot of different things. It can be infrastructures also, but it can be practices, it can be specific projects. So the unit of analysis is quite broad and heterogeneous. So it must be taken into account. Then another major point is that governance is pretty complicated in general in open science projects. Because you may have different providers, different stakeholders. They may be based in different countries. They may be from different communities. There is a lot of not only formal but also informal exchanges. So making sense of that can be a little bit tricky. And in concrete terms, it can have implications also for the tracking of use. Because often open science, by definition, the idea is to make it accessible often freely to users. So in many cases, it means that you do not have the tools to track people and what they’re doing because you may not record, for instance, very concretely. You may not need an account to download data or papers or anything like that. So it’s then challenging to understand who are the exact users and what they’re doing with your services and so on and so forth.

Speaker 1

How to measure it.

Speaker 2

Exactly. Yes. So basically it’s an important issue because for CBA you need to understand who the stakeholders are and what they are doing and basically how they are benefiting from this perspective. So it’s quite a challenging issue. And then, which is also related to that, you may have a broader pool of potential beneficiaries compared to traditional science. For instance, the thing I was mentioning before, if you’re making publications or archives available for anyone, you may have, of course, historians or the researchers benefiting from it. But you can also have journalists taking a look at it or curious citizens and so on.

Speaker 1

And other unexpected users.

Speaker 2

Exactly. And it’s particularly the case for open science projects. So when OPATO started, there was no really framework for taking these aspects into consideration and adapting a CBA background in order to spot in particular the potential costs and benefits that could be typical from these kinds of projects.

Speaker 1

Excellent. So we, I think, in the project partly know it, in Skyros project, because we have 3 pillars for infrastructure, practice, and theory of open science. So it is in fact very, very complex. But going back to CBA framework, it was applied to concrete case scenarios.

Speaker 2

Yes.

Speaker 1

So for example, Pathos tested its tools on RECAP, What were the key conclusions to emerge from this case? Did the analysis reveal anything unexpected, such as effects on innovation, collaboration with industry, or cost reductions?

Speaker 2

Okay, so first I should maybe briefly present RCAP to our listeners. Basically, RCAP is a Portuguese network of institutional repositories. So for people who may not know exactly what it is, it’s basically places where researchers from universities or hospitals or the kind of research organization can actually put their publications, their papers, and they are basically archived in a way that can be accessible for other researchers, but also to the wider public. And in Portugal in particular, we have this service which is called CAP, which basically means that you have a common portal when you can search for all the repositories. So you have only one place to go to find the papers from the University of Lisbon, of Coimbra, and so on and so forth. So it’s the part that people see. And then you have also the part that people do not see, but is also very important. Which is? Is the fact that CAP at the central level, so as an actor itself, provides to universities or other stakeholders the ability to host their repository. So basically, they have a common server, a common infrastructure, a common software. And if you are a university, you can go to them and ask to open your repository, and they take care of most of the technical parts.

Speaker 2

And as an institution, you can focus on organizing your repository, defining the sections, defining your policies related to that. So you’re more about the, let’s say, strategic and practical aspect and not so much about the IT. Technical one. Exactly. And of course, it’s very important, especially for smaller institutions who may not have the resources to try to open and manage on a daily basis this kind of service. So basically, this is what Trecap is. And we’ve decided to apply the CVA to it with the collaboration of our partner in Portugal from the University of Minho, from RCAP itself, from the funding agency. So we got a lot of different people involved and we were able— Good partners. Yes, absolutely. And we were able to connect with a lot of the institutions taking part in it to understand the realities of the system. And basically what we were able to uncover with the CBA was the importance of this more, let’s say, backend side to open science. So not only the visible parts like researchers’ engagements, research activities, but only the fact that there are institutions and people like librarians or IT specialists working in the background to make open science possible.

Speaker 1

What is behind the scenes.

Speaker 2

Exactly, exactly. And it’s actually a big deal because one of the main benefits of the system was the fact that you have some kind of pooling of resources because instead of having, I don’t know, 50 different universities setting their server, having to hire people, having to design something from scratch, you have one central team who can help. And we can provide free of charge for the university service. So you have some kind of savings due to the fact that you are basically collaborating in order to achieve this goal.

Speaker 1

One solution for many needs.

Speaker 2

Yes, while retaining the ability to be flexible, because one of the key parts is that you have basically different levels of repositories with different, let’s say, grades of customization. And for institutions that may be a little bit bigger or maybe have their own resources, they can also choose to set up their own repository and connect it themselves to the platform. So you have this possibility and it’s delivering some benefits, especially to smaller institutions. But you also have this, let’s say, open access philosophy to it.

Speaker 1

But defining is extremely important. Open science needs infrastructure. Can I state it that way?

Speaker 2

No, yes, I think it’s a very good and valid point. And how should I put it? It’s always a part that is, let’s say, less fancy, less visible to outsiders at least. And it’s also helping researchers a lot. In their daily work because for instance, if you have a team of librarians working to improve the metadata of your publication, they are maybe taking care on behalf of researchers of this part of collecting the evidence and so on and so forth. It’s actually typically perceived as an administrative task. And of course nobody likes them, researchers hate them in particular. Not to overlook it. Exactly. It’s very important. Exactly. And it’s very valuable then for the research itself because This saves time because researchers don’t have to spend so much time on that if the system is done well. And it’s clearly a big value. And also then to have this comprehensive, high-quality pool of papers made available to everyone.

Speaker 1

But it’s excellent to hear that we have already working good structures that can handle these tasks. So looking at these results more broadly, the project The Pathos project has now reached its conclusion. What does the end of Pathos mean for CSIL? What are the next directions for your research and how do you see these tools being applied in the future?

Speaker 2

Yes, that’s a very good question because as I’ve mentioned before, prior to Pathos, there was no, let’s say, adapted CBA framework, for instance, to the realities of open science in particular. And so with this project, we’ve deliver some kind of standard methodology that is made available. It’s on the node and so on and so forth. So for everyone to read and to potentially use. But it also means that it’s only the beginning because we scratch the surface and there may be additional types of costs and benefits, additional ways to assess them because once you have identified the different costs and benefits, you need to actually convert them into numbers, which is also one of the challenging parts, of course. And we’ve applied different methodologies depending on the cases on the projects we’ve evaluated so far. But we know there are things that could be done even better. For instance, in terms of bibliometrics. How do you actually evaluate the value of a publication beyond the standard measures that may have pitfalls?

Speaker 1

This is a very hot topic.

Speaker 2

And we are aware of these risks. And potentially in the future, we could try to develop further and more refined approach, taking into consideration, for instance, what’s actually in the paper, which implies having access to the full text, which implies the value of open science itself.

Speaker 1

Of course.

Speaker 2

So all this kind of expanding methodologically, refining the tools in order to have even better, even more robust results is definitely something we’ll try to do. Because what you have to understand in assessing the impact In general, and in CBA in particular, the key point is the process you’re following. The numbers, of course, are important, but depending on what data is available and a lot of different factors, you may not get the results up to the 0.1% of accuracy. What is really important is to do the best you can with the data you have, the resources you have, and the information you collect from the community so that you can trust the process in order to approach the more, let’s say, robust and credible estimate. So definitely we would like to improve in that direction. And in terms of next steps, concretely, we are very interested in trying to support stakeholders in policymaking, in academia, to try to understand better their impact and to try to support them for different reasons. First, because it’s a way to support the development of science. We are in a time period where you have some constraints about resources and so on and so forth.

Speaker 2

And it’s sometimes difficult to make visible the impacts of science to justify it compared to other priorities. So it can definitely be some value of the work we are doing to try to help stakeholders to make these elements more visible. Visibility is very important. Including in the discussions with policymakers, funders, and so on and so forth. And we also think that even without this more, let’s say, external motivation, it can also be an interesting opportunity for researchers themselves to try to consider this aspect even in the early stages of the research because it may also be an opportunity to, let’s say, refine the understanding of what we are trying to achieve. So to give you an example, we are currently supporting the development of different research infrastructures in different fields. And if you are involved from the very beginning in trying to understand your impact, how you will actually monitor them and so on and so forth, you’re actually helping the design of the infrastructure because it helps people, researchers, institutions and so on and so forth to try to pinpoint the exact value of the infrastructure compared to what is already existing and to avoid the fact that you may duplicate some services or some equipment.

Speaker 2

So you actually try to guide the process towards something that is more efficient for everyone.

Speaker 1

Yes, this is interesting to have a clear design, clear model, and to perform better in the task. This is extremely interesting. So finally, bringing this back to a more personal level, if I may. How was working on Pathos and on open science more generally influenced your own perspective on the role of research in society?

Speaker 2

Yes, I must admit I’m a little bit biased because I think that scientific research has a lot of value and potential, including for more fundamental research that may not have direct applications or consequences. But I think what Pathos brought was the fact that going a little bit against this bias, it’s very difficult to try to assess and to substantiate the impact of science, including open science, because of all the issues I’ve mentioned before, the fact that there are also some some resistance, some pitfalls, some costs to open science practices. So I think it gave me a more realistic outlook of the situation, but also spotted some opportunities to try to address the situation so far and to address these existing limits. Okay.

Speaker 1

The more difficult, the more fun.

Speaker 2

Yes, I guess you could say that. It’s always a work in progress. And I think that’s definitely compared to 5, 10, 15 years ago, there is some progress in trying to understand how open science can deliver some results. Not only in terms of access to publications, but also to the more qualitative practices that may be present and also have some interesting values. To make people cooperate more, to try to engage with different countries that may be a little bit on the margins so far of traditional sciences. So I think it’s definitely some of the potential for the future. Of course.

Speaker 1

So thank you, Louis. This has been a truly insightful conversation, shedding light on how the impact of open science can be understood and assessed. In practice. It was a pleasure, and thanks for sharing CSIL’s experience with the Pathos project and for walking us through the cost-benefit perspective of open science.

Speaker 2

Thank you very much, it was a pleasure. Stay connected and follow Skyros for more insight. Find us at our blog skyros.hypothesis.io. Ok. SCIROS project is made possible thanks to the support of the Polish National Agency for Academic Exchange under the Strategic Partnership Program. And remember, science is best when it’s open. See you next time on Big Open Science Podcast.

---

---