Meeting: Driving Institutional Change for Research Assessment Reform

Meeting: Driving Institutional Change for Research Assessment Reform

January 4, 2020 0 By Kody Olson


I’m Boyana Konforti. I’m director of scientific
strategy and development here at HHMI. I am really, really excited
to welcome all of you here. This has been a meeting
that’s been about a year in the making. And it began with a kind
of informal conversation between DORA and
HHMI about how we could change how science and
scientists are evaluated. And there are many people
that we need to thank or I need to thank before we
get this meeting started– Erin O’Shea, who’s
HHMI’s president; Bodo Stern, chief
strategic officer; Erika Shugart, who’s
the CEO of ASCB; Stephen Curry from Imperial
College London and chair of DORA’s steering
committee; and of course, all the extraordinary
HHMI staff here– the conference services,
communications, AV. But really, the driving
force behind this meeting has been Anna Hatch. [APPLAUSE] She’s really had
the unenviable task of taking suggestions, comments,
and ideas from me, Erika, and Stephen, and somehow
performing some magic and coming up with consensus,
which we couldn’t even imagine. So we’re very grateful
to her for that. And I’m also really
looking forward to meeting and working with some of you
so that we can actually create the change we want to see. And in order to do that,
I want to make clear that HHMI welcomes the
participation of persons of all identities
and all backgrounds, that we expect attendees to
treat each other with courtesy and respect, and that
if you experience any inappropriate behavior,
speak to your event organizer. That would be me. And Bodo is also part of
the event organizing team and that all reports
will be investigated. And we will take
appropriate action. And so here is our information,
should you need it. But it’s also going to be
outside right beside conference services desk on a screen there
for you to have our contact information. And now, I’d like to welcome
Erika Shugart, who, as I said, is CEO of ASCB. She’s going to tell us something
about the results of the survey that I hope all of you took and
also the goals of the meeting. So let’s get this party started. [APPLAUSE] Well, hello. Well, I was going to do
some thank you’s, too. And I think Boyana, because
she was standing up here, couldn’t thank herself. But it was really a team
with Boyana and Anna that put this together. And I just am so excited
to be here with all of you for the next couple of days. So I am here from a
scientific society, the American Society
of Cell Biology. And I’m really happy to
see a number of our members in the audience. It’s always nice to
reconnect with them. But I like to think
about, you know, with the purpose of a scientific
society, what are we here to do? So I think a lot about this. We’re really a
community of scientists. And we’re here to help
advance our members’ professional career. We’re here to
advance the community and represent them
in all sorts of ways, whether it’s advocating
for them on their behalf or helping them work together
to achieve their goals. And so I often
talk to scientists about research assessment. And it’s interesting
conversation. We always of start out
and as an acknowledgment of all the
challenges– you know, the fact that in the
area that I represent, it’s a hypercompetition. You know, there are more
qualified candidates than there are
academic positions. And we talk about biases. And we talk about the
disadvantages people have and the advantages that
other people may have. And so we have
these conversations. And then, there’s often
a moment– not always– but there’s often
a moment, where, then, the person– the member
I’m talking to turns around and says, but science
is a meritocracy. [LAUGHTER] And I’m like, well,
I’m sure that everyone that makes it through an
academia is highly worthy. I get to meet them every day. They’re amazing human beings
doing absolutely amazing work. But I also know that there is
a group of people that are also very worthy that perhaps
are not doing that path, because they were not able to
do that path, which is fine because there are many,
many career paths. But nonetheless, I think unless
we are thinking about what kind of community we
want, then we really aren’t doing what we need to
do, because if we continue to make just individual
decisions about people and what we view as
their merit, then we’re not going to build
a community actively. So we need to
really think about– and I think this is one of the
challenges that I throw out for us to do– is sort of, how do
we get the diversity that we need to have
a strong community? How do we ensure that we
can think about what we want to see when we look out? And I think that’s one of the
things we want to think about, because it really comes
down to research assessment. It’s not a bunch of
individual decisions. It’s really about
forming a community. It’s really about
growing that up. So that’s what I think about. And I’m really looking
forward to thinking about it with all of you
over the next couple of days. But I am up here to
talk a little bit about a survey that was done. And I’m doing this on behalf
of Anna, who put all of this together. So we did an informal survey. This is by no means
a scientific survey. It’s not a representative
sample at all. There will be no error bars
on anything that you will see. [LAUGHTER] But it was a good
opportunity for us to get some community
feedback beyond those that can be here today and
ask some questions to give us a little sense of what
people are thinking about out in the wider world. And we did get 190
responses total. And who were they from? So here’s sort of
a pie chart that represents the career
levels of the folks that were able to respond. And what you see in sort
of the darkest color down there at the
bottom is the ones that I think have been through
the majority of the hurdles. You know, they’re
tenured professors. So they’ve been
through the hiring. They’ve been through, you
know, at least one round of promotion and tenure. They still are going to be
looking at research assessment and grants and as they move up
the– you know, as they move up in their career. But I sort of think,
you know, that that was– that’s the biggest group
is those tenured professors. We then have a number
of other folks that– professors,
postdoctoral fellows, and graduate students,
professors pre-tenure, I should say,
postdoctoral fellows, and graduate students–
but still have a pretty major hurdle
in front of them. You know, it’s like
they’re either still looking to be hired or they’re
still looking to, like, get that tenure. So that was another group. The other group that’s up there
in kind of the lightest color is generally
speaking people that are either emeritus or people
that are outside of academia. And then, we do have
a nice representation of staff scientists, as well. So the first question we asked
was about mission statement and whether it is aligned to
the practices that people see and the promotion and
funding decisions. And what you see is in
the middle gray there, a lot of people don’t know. This is not unexpected. Many people cannot recite
their institution’s mission statement. So they really may
not know about this. Then, we have about
40%, 41% said yes. And then, a pretty
sizable portion, 21%, said no, which is kind of highly
unfortunate, because one would hope that the values
of the institution reflect the mission
of the institution. But as we all know, that
does not always happen. So the next question we asked
was about journal-based metrics and whether they were
acquired in certain areas. And this was the question, when
you read the comments, that actually generated
probably the most debate about the
phrasing of the question, because most people
are like, well, you’re asking about required. It’s not required that
everybody uses it. So I think what we
see here is, you know, if people are asking
sort of, is it required, and then whether it’s used as
sort of a different question. So I would just
generally say speaking of seeing the comments that
probably all of the yeses, if we’re asking about
used, would actually be quite a bit larger, because
most people are like, well, it’s not required. But we still use it. So what we see in
hiring decisions– it’s almost evenly
split between people that do use metrics and
people that don’t, with a, you know, a portion
that I don’t know– fairly similar in the
annual performance reviews. It’s really in
promotion and tenure that we see that
journal-based metrics are used by more than a majority of
the institutions, or required, I should say–
excuse me– required by the majority of institutions
according to our surveyors. So then, we sort of
moved on from what is going on to asking our
respondents to tell us a little bit about what
they think should be valued. So we asked first about hiring. And then we asked about
promotion and tenure. So first, you sort
of see a list here of how important people thought
the following items were when making decisions for hiring. And what you see in kind of
the top four, I think, are– and you’ll see something very
similar when we move over to the promotion and tenure– is, you know, they care about–
people care about research. They care about whether you are
going to be able to get money. They want to know
you and then, what the content of the letters
of recommendation, which I do think is interesting. You look a little
bit farther down. So then, the one that’s
just underneath that is prestige of the journals. So that’s kind of one
that I know generally speaking DORA is not as excited
about and not too keen on– OK– so teaching and mentorship. And then, we have
reputation of individuals, which would can often mean
kind of like, where did you come from? How do I know your academic
genealogy– so again, one that has raised
some concerns in some of the assessment. And then, down at
the bottom is a lot of those more
service-oriented kinds of approaches– public
engagement and open science. We can see a very,
very similar picture over and in the
promotion and tenure. So this is the same very
similar list, but instead asking about making– you know, what is important
for promotion and tenure? It’s a little bit
shorter, because we’re not asking the questions
about familiarity with institutions and things. They’re all in the
same institution. So again, the top three parallel
what we saw in that other one. Can you get money? Do I think your
research program is, you know, what we want it to be? And what do people
have to say about you? So we see the same
top ones, and again, and you know, that middle of the
same with teaching, mentorship, and then down at the bottom,
again, some of the more service oriented and things around
reproducibility and open science. So you can kind of take a look
there and I think think about, you know, because
again, this gets back– I think we’re
going to be talking a lot about how do
people’s values intersect with the research assessment. So the next one was asking–
and I think this is a little bit of a loaded question, I
will admit on this part of the survey– [LAUGHTER] –you know, what is more useful
to evaluate your research quality– metrics or written
description of significance? And given that most people
that saw this probably are somewhat familiar with
DORA and maybe were like, well, I know the right answer
for this question– [LAUGHTER] –which, of course, is
the written description of significance, right? But nonetheless, even knowing
that, still 12% said metrics. So and it’s not that
metrics are bad. I’m making a joke, because
DORA is kind of known as being anti- impact factor. And it’s because that’s
really the misuse of metrics. It’s really taking a metric
that’s designed for one thing and using it for another. And it’s not– we’re not
necessarily anti-metric. But we do have some hesitations
is probably a better phrase. All right. And then, the last
question we asked was really a question–
so we’re going to be talking over the
next couple of days about institutional change
and change in organizations. And there are
different approaches that we can be taken– taken to that. And so this last
question is really asking about kind of a top-down
versus bottom-up approach. It’s like, what are
the interventions that are most likely to
decrease your institution’s reliance on journal-based
metrics and research evaluations? And so you know, the
top one right there– total top-down kind
of thing– it’s like mandatory requirements
from funding agencies. Updated review and promotion
and tenure policies– that can kind of come from a lot
of different directions. It could be a top-down. It could be bottom-up,
but nonetheless, having clear and stated
policies, then, training people on how to hopefully
use those policies. So they align– you
know, the practice can align to the policy. And then, we have
one, which I know we’ll be talking about over
these next couple of days around working
groups– you know, creating working groups
to really evaluate. That’s kind of much more
about a bottom-up approach. So that’s getting the folks that
are involved thinking about, how can we make change
in our organization? And then, you can see
as we go on down a few of the other kinds
of topics there. So that’s a little
bit of the survey just to give a little taste,
a little bit of thinking about what we might want to do. But now, I want to just
talk about the goals. So hopefully, you all
saw them on the website, but just to remind everyone–
so what we’re here to do, 1, is to encourage
establishment of working groups at institutions in order to
evaluate research assessment practices, look at how those
practices align with policies, and then hopefully use those
working groups to improve the policies and the practices. The second is– and
this is one of things I’m really excited about over
the next couple of days– is that we’ve brought in people
with expertise in culture change and human design
and fields that are quite different than, you know, the
science that I’m familiar with and to help us think about,
how do you actually do this? What is some of the
evidence-based, you know, research that can help inform
our policies into practice? And then, finally, the
room is really exciting. I don’t know about you,
but I have had a chance to talk with people from a
lot of different backgrounds already. And so we hope that
everybody will walk away with new collaborations
around research assessment, be able to meet some new folks,
think about things in new ways. And to kind of help that, I
wanted to sort of give a little poll– a little hand poll– and
I know if the folks out there on the internet– you can raise
your hand when I say your type, too– [LAUGHTER] –it’s just get a little
sense of who is in the room. And you may raise your hand to
more than one of these up here. So first, who is a faculty
member in the room? OK. We got about a lot
of faculty members. So maybe, you know, 80% of the
room is raising for faculty. What about university
administrators? Oh, we have a sizable number
of university administrators, a little bit of
doubling up there. But yeah, it’s like
at least half the room looks like administrators. What about a culture
change experts? OK. Smaller change–
smaller number of hands, and they’re a little bit shy. [LAUGHTER] So but they’re out there. All right. So early career researchers– excellent. So we have a good smattering of
early career researchers here. What about funders? Do we have our funders? Oh, we’ve got some
funders here– yeah, quite a few funders– excellent– because you need to
do those mandatory things that tell us what to do. What about scientific
societies staff? We got some proud over there
in the corner and up here in the front. Librarians– oh, look
at our librarian crew. We got a good representative
of librarians in the back. And then, the catch-all other
category of representatives from other nonprofit
initiatives– [LAUGHTER] –and, again, we have about
maybe 10% to 15% of the room was those. So you can see that it’s a
really great group, really looking forward to
meeting with all of you. And now, I’d like to welcome
to the stage Dr. Stephen Curry. He is the chair of DORA’s
steering committee, as well as the professor
and the assistant provost for equality, diversity, and
inclusion at Imperial College London. And he will be
introducing our keynotes– Stephen. [APPLAUSE] Thank you very much, Erika. And let me welcome you myself,
all of you, to this meeting. I, too, am very
excited about it. Coming from the
United Kingdom, we don’t get very excited
about very much. OK. [LAUGHTER] We tend to play it cool. And you know, things are OK. But I am really looking forward
to the next two or three days. And we have a great
keynote session tonight to kick things off and
set the ball rolling and get us in the mood before
we head for the bar at 9:00. And so we have a slight
change to our program. So Erin O’Shea,
unfortunately, is not able to join us this evening. So we have a surprise
guest speaker, who is coming in third
place at the end. And so we are going to
start off initially, then, with Professor Erin McKiernan. And Erin is a professor in
the department of physics. So I’m from physics
myself originally. So I am very glad to see that–
at the National Autonomous University in Mexico. So I met Erin for
the first time today. We have known each
other through Twitter. That’s how I know most of
my social circle these days. [LAUGHTER] But I know Erin to be a
fearless and tireless advocate for open science. She is the founder of the Why
Open Research project, which is an educational site
for researchers to learn how to share their work. And we are very glad to have her
as an extremely valuable member of the DORA steering committee. And Erin, as well as being a
physicist and biophysicist, does a lot of really important
work actually looking at the nuts and bolts
of research evaluation. And she’s been involved in
the ScholCommLab’s Review, Promotion, and Tenure Project
and has authored quite a number of works looking at the use and
misuse of the journal impact factor. So Erin is going to talk
to us this evening about how the journal
impact factor is used in review, promotion, and
tenure in the United States and Canada. Erin. [APPLAUSE] All right. Can you all hear me OK? Yes. Super. OK. Well, thanks so much. I’m super excited to be here. Kind of one of the
areas that I’m really focusing on these days
is research evaluation. So this is a perfect
forum for that. So yeah, I want to talk to
you guys today about some of the work we’ve been doing. This is a collaboration. I’ll talk about some
of the people that are involved in this project. And basically, the idea is
we need more information. If we’re going to reform
the evaluation systems– academic evaluation systems,
we need more information about actually what is
going on in these systems– what are we measuring? What are we focusing on? So that’s the focus
of this project. Before we kind of
dive in, of course, this presentation
is openly licensed. Feel free to take
pictures, tweet– whatever you want
to do with that. And I’ll post the slides
on Figshare afterwards. So everybody has
access to those. All right. So let’s talk
promotion and tenure. And I just want to
give a shout-out to my dad, who does these
beautiful cartoons for our Why Open Research site. [LAUGHTER] So yeah, promotion and
tenure– we’re talking research evaluation, right? Obviously, promotion
and tenure is not the only place, where
research evaluation occurs, but it is a large
part of how academics are getting evaluated. And it’s kind of this
tenet of the academic world that if you work hard enough and
you check enough of the boxes, that you’re going
to move up the ranks and eventually be awarded
this job security, right? And so that’s what a lot of
academics are shooting for. And it’s also
understandably what is driving a lot of
faculty behavior. And in particular, if we’re
interested in open science, when we survey faculty,
when we survey researchers, a lot of researchers
cite concerns about promotion and tenure,
about moving up the ranks, about getting those points
and checking those boxes as a top reason why they don’t
end up sharing their work. So they say, yeah, I
think open access sounds like a great idea. I think sharing my data
sounds like a great idea. But I’m not going
to do it, because I need to publish in those
high-impact factor journals. Or I don’t want to
get scooped, right? So this is a huge driving
force behind how faculty behave and how they decide to share
or not share their work. So the questions here are,
at the university level, what are universities valuing? And also, this relates back
to this survey question about mission statements, right? Is that lining up with
their mission statement? What are universities rewarding,
because these, unfortunately, might not be the same thing? And what specifically counts
towards promotion and tenure? When we were getting
together initially– Juan Pablo Alperin and
Meredith Niles and I– so those are my collaborators– we realized that there’s a
lot of anecdotal information, right, about what counts
towards promotion and tenure. But we didn’t have
many solid numbers, data on what’s actually
in these documents and what’s getting rewarded. So that was the motivation
behind this project to kind of find out
what was in there and what’s counting when
we’re evaluating faculty. So ranking, ranking,
ranking, ranking, right? So we know that a lot of
these evaluation systems involve assigning points
for different things, and especially at the
hiring level and maybe even at the promotion level ranking
faculty and saying, OK. This one has more points
than this one, right? But what are we actually
awarding points for? And are we awarding
points for the things that maybe we think we should
be awarding points for? Are we awarding
points for things that will promote the
university’s values and the university’s missions? There’s a lot of
this going on, right? And I think I don’t
have to sell this here. So there’s a lot of impact
factor worship going on. There’s a lot of faculty
hiring committees or faculty evaluation promotion
and tenure committees looking at the impact
factor of the journals that faculty are publishing
in and saying, OK. This article is worth
more than this article. This gets more points than that. And so we know this kind
of from talking to faculty. But again, a lot of
this was anecdotal. We talked to faculty. They say that impact factor
is really a driving force. It’s something that
comes up again and again in their evaluations. But we didn’t have numbers. We didn’t know the answers
to these questions, right? How often is the
journal impact factor actually mentioned in review,
promotion, and tenure documents at universities in
the US and Canada? What is the nature
of these mentions? So it might be that
the journal impact factor is mentioned
in these documents, but mentioned in
such a way that’s cautionary or neutral
or supportive. We didn’t know what the
nature of those mentions were. And also, what do these
documents say they’re measuring with impact factor? So they’re using it as a metric. But what do they think
it actually indicates? What does it indicate to the
promotion and tenure committee? So these were the questions
that we wanted to answer. This was part of a larger study. So what we did was
collect review, promotion, and tenure documents
from institutions in the US and Canada. And we started with
those countries, because it’s easier to
obtain the documents. And I say easier, not easy. This was actually
the hardest part of the project
was just obtaining the sample of documents, because
a lot of them are not public. They’re not always in the
same place on websites even when they are public. So there was no way to
kind of automate this. And even when we contacted
people– faculty, even department chairs at the
university, a lot of the time, we got a response something
like, we don’t actually know if those
documents exist, right? So it was difficult
to obtain these. But in the end, we
did get a sample of over 860 review, promotion,
and tenure guidelines. So these could be anything
from a presentation that was given to faculty telling
them how to kind of prepare their dossier or
something more formal, like an institutional-level
guideline. We got documents from
129 universities. And this I should mention is
a stratified random sample. So it’s a representative sample
there from the US and Canada, and then 381 academic units from
within those 129 universities. We divided these up, because not
all universities are the same. So we divided these
up into R-types, which are the research-intensive
and doctoral-focused institutions; M-types,
which are primarily master’s institutions; and then,
B-types, which are primarily undergraduate institutions. And again, there’s a
varying level of research going on at those
institutions with R-types being the most
research intensive. And then within the R-types– and we focused just
on R-types here. We didn’t have a
large enough sample to really do that for the
other institution types. We divided their
academic units up into the life sciences, physical
science and mathematics, social sciences and humanities,
and then multidisciplinary. So we might have College of
Arts and Sciences, right? So we would classify that
as multidisciplinary. So this is our sample. And I should mention that all
the data from this project is openly available, not
the documents themselves, because those belong
to the institutions. But all the analysis that
we did on these documents– all that is is openly available
here on Harvard Dataverse. So you can go through,
play with those numbers, check our analysis, et cetera. All right. So with respect
to impact factor, we first had to go
through the documents and kind of do a
general reading, because it wasn’t so
simple as just saying, OK. Impact factor, right? There are actually multiple ways
in which the impact factor gets mentioned in these documents. So first, we did a screening,
read through them briefly, and identified some terms that
we really wanted to focus on. So this top ring
here are the terms that we considered to
be directly referring to journal impact factor–
so impact factor, impact score, impact metric,
and impact index. Is anyone taking a
count of how many times I say the word
“impact” in this slide. [LAUGHTER] And then, the second
circle are terms that are some way referring
to journal impact– so high-impact journal, impact
of the journal, journal’s impact. Those are the two circles– those are the set of terms
that we focused on analyzing. There’s a larger set of terms
that we found throughout these documents that we think
are indirect but probable references to journal
impact factor– things like top-tier
journal, major journal, high-impact journal, right–
or no, high-impact journal– sorry– I said that,
but highly-ranked journal or significant
journal, prestigious journal. We think these are probably ways
in which people are referring to the journal impact factor,
but since they’re indirect and we’re not
necessarily sure, we didn’t include those
in the analysis. So we’re only analyzing the
terms from those top two rings. All right. So what did we find? First of all, very
basic question– how often is journal impact
factor getting mentioned? In what percentage
of institutions are we seeing explicit
mentions of this– of journal impact factor or
those closely related terms? So the overall number
is 23% of institutions in our sample mentioned the
impact factor or related terms in at least one of their
review, promotion, and tenure documents. We expected that number
to be a little bit higher. But I’ll talk in a few minutes
about why that number maybe wasn’t as high as we
might have expected. And it has to do with
those group 3 terms, right? And when we break this
down by institution type, we see a different picture. So that number goes
way up for R-types. So it’s 40% of
R-types are explicitly mentioning journal impact
factor in their documents, but only 18% of the M-types
and 0% of the B-types. So the more research
intensive, the more doctoral-focused institutions
are mentioning the journal impact factor much more often. When we break this down
by R-types when we look at discipline, it
was pretty similar– so 33% of life sciences,
29% of physics and math, 21% of social
sciences, humanities, and 17% of multidisciplinary. We didn’t have a
large enough sample to kind of do a statistical
comparison there, but pretty close across those disciplines. All right. So then, we looked at the
actual mentions themselves. So instead of just
counting, we went through, and we read each one of these. And we classified them as
either supportive, cautious, or neutral,
depending on how they talked about the impact factor. And just as a
methodological issue, we had two evaluators go through
and independently rate these. They agreed on over 80%
of the classifications. And then, those which
were in question– we had a third person evaluate. And we took the agreement there
between the two reviewers. So when we look at
that, overwhelmingly the mentions of
journal impact factor were supportive of its
use in evaluations. So 80%– 87% of these
institutions that had a mention were supportive. Only 13% of those institutions
that had a mention expressed any type of caution
about using this measure. And that could have been very
lukewarm, like maybe we don’t think this is such a good idea. None of them heavily criticized
the journal impact factor or prohibited its use in
evaluations– not a single one. And then, we had the 17%
that had at least one neutral mention,
something like, you can list journal impact factor. But we didn’t consider
that to be necessarily supportive or cautious. It was just you can list it– so overwhelmingly,
supportive– these mentions. We’re here, at least in
part, because of DORA. So we have to look at
that in our sample– not a single DORA signatory in
our sample, no mentions of DORA in any of these RPT documents. And we have documents
going up to 2016. So this is not an issue
of a time lag there. I think it’s important to note– I don’t have the
exact numbers here. But there are very
few universities from the US and
Canada, who have, at an institutional
level, have signed DORA. So I think that’s what
we’re seeing here. We have a relatively small–
it’s a representative sample– relatively small, and we
have a very small number of institutions. So we just didn’t capture those. It would be really interesting
to study institutions that have signed DORA
and then see how that’s reflected in future
revisions of their RPT docs. OK. And then, so we looked
at how many mentions. We looked at the nature
of these mentions. And then, we wanted to go one
step further and figure out what it is they think they’re
measuring with the journal impact factor and how they’re
using it in evaluation. So again, we read
over these quotes. And we kind of picked
out reoccurring themes. First of all, a lot of
them are unspecified. So if you read, they mention
the journal impact factor. They might be very
supportive of its use. But they don’t say
anything additional about how they’re using it or
what it’s intended to measure. So that was a huge percent. So 77% of these
institutions with mentions– they just didn’t specify what
they’re using the metric for and that breakdown
by R-type and M-type. For the ones that did specify,
the most common association we observed was that made
between journal impact factor and quality. And this could either be quality
of the journal or quality of the research itself. So 63% of the
institutions with mentions associated this metric
as a measure of quality. And then, you can
see the breakdown by R-types and M-types. So I’d like to show you
just a few examples. And this is a particularly
spectacular one. It says, we will employ the
metric article impact factor, which is the journal
impact factor multiplied by the number of citations
for that article. Yes, collective groan– [LAUGHTER] –journal-level metric
multiplied by an article-level metric– sigh, but and the
quite spectacular thing is they think that this
metric is not only great, but as it says,
employing this metric, faculty have incentive to
publish in the highest quality journals, which will
increase the JIF, and simultaneously produce
the highest quality research manuscripts,
potentially increasing the number of citations
and increasing the– yes, thank you. [LAUGHTER] All right. So this is the level of– yeah, I don’t have
kind words for this. This is what we’re seeing
in these documents. And it’s incredibly
depressing, because you’d think that as scientists,
we would know better, first of all. And you’d think that with
all the initiatives like DORA and other research
evaluation reform initiatives that we had moved beyond
this, but unfortunately, not the case. So there are a
lot more examples. And in fact, as part
of our open data, we have a supplemental
data set online. You can read all the quotes. So and they’re color
coded like this. So you can see how many times
they mention quality and impact and importance, et cetera. So have fun with that. [LAUGHS] The second most
common association was between the journal impact
factor and impact importance or significance of
the research itself. So 40% of institutions
with mentions made this type of association. As an example,
candidates will be encouraged to submit
a statement that explains the importance of the
publications, which may include factors such as
journal impact factors and also note publication in
journals with low acceptance rates. Apparently, that indicates the
significance or the importance of the research– so again, quite common. And then, the third most
common association that we saw was this between the journal
impact factor and prestige or reputation or status
typically of the journal or the publication outlet. And so that’s 20% of
institutions with mentions made this particular association. And so here– and
I’ve highlighted things that are in contrast to
DORA’s message in particular. Publication in respected
highly-cited journals counts for more than publication
in unranked journals. We are explicitly saying
we will award more points for this type of publication
than for that type of publication independent
of any type of valuation of scientific merit, right? And it says, yes,
we will consider which are the top journals
based on their ranking in the Thompson ISI
Citation Database, which generates the
well-known impact factors. In this last one, we’ll talk
about driving faculty behavior. In general, it behooves faculty
to be aware of the prestige rankings of the field’s
journals and to publish in the highest-ranked
journals possible. So this language is all
over these documents and is particularly depressing. [LAUGHS] All right. But is there any good news? So there were some mentions that
didn’t support journal impact factor use. So this is one example. It says, impact
factors of journals should not be used as the
sole or deciding criteria in assessing quality. On the downside, again,
there’s that association between journal impact
factor and quality. And they’re not saying
they won’t use it. They’re just saying it
won’t be the sole criteria. So this is a very lukewarm kind
of, we don’t really like this. The other cautious messages
we saw related to discipline. They said, well, we don’t think
journal impact factor really is relevant for our
discipline– might be relevant for other ones,
but not so much for us. But it wasn’t about
the metric itself. It was about disciplinary
differences– so very lukewarm and again, no language
that looks anything like DORA language and no
kind of strong statements against use of the
journal impact. So some conclusions from this– we now have the numbers to
support that anecdotal evidence coming from faculty that
journal impact factor is featuring in evaluations,
perhaps less prominently. Yeah, we didn’t see it
explicitly mentioned in as higher percentage
of institutions as we might have imagined. Why might that be? I suspect we have a
situation like this. So we were probably
underestimating the use of this metric. We were looking at this
tip of the iceberg, which is explicit mentions or
very closely related terms. And we’re leaving
out all these terms, which is a much larger set
that are probably indicating to faculty members, who are
sitting on these committees, oh, yes, you should
pay attention to journal impact factor
without explicitly saying it. So I suspect that it’s a much
larger percent of institutions. And like you said in the survey
data, well, do they require it? Maybe not, but they’re using it. So we don’t actually
know what’s happening inside those committee meetings. I suspect there’s a lot more
use than we’re estimating just from that raw percentage. There’s another limitation. And that is precisely that
we’re not talking to faculty. We’re looking at a text analysis
of what’s in the formal RPT guidelines, right? So how is it the
faculty are actually thinking about the
journal impact factor? And do faculty think it’s
important independent of what’s written in those documents? Do they think it’s
an important metric? And do they think it’s
one that’s being used, whether or not it’s required? So we have a little
bit of survey data. We’ve surveyed faculty at those
same institutions for which we have RPT guidelines. And we’re just in the process
of getting some of these results out. But basically, there’s a
couple interesting findings. First, we asked
faculty, what are of the most important
factors for you when deciding where to publish? And then, we asked them, what
are the most important factors for your peers when they
decide where to publish? And so we saw this
interesting effect that they see in social
psychology, which is people think they’re
better than their peers. [LAUGHS] And so they said,
for me, readership– I care about who’s reading
that journal, right? What scientific
audience am I reaching? And that was the factor that
ranked the highest for them– so has a readership I
want to reach and then also here journal publisher
venue that I regularly read. So they were focused
on readership. And impact factor came
in here fifth, right? But when we asked them what are
your peers concerned about– [LAUGHTER] –they think their
peers are primarily concerned with prestige
and impact factor. So they said here, top factor– overall prestige– impact
factor moved up to fourth. Citations here is in third. So all of this relates to
prestige, journal venue, and impact. So it’s interesting,
because the results suggest this disconnect between what
academics are valuing or saying they value, and what they
think their peers value. And in particular, also,
we saw an age effect. So those, who are actually
more likely to sit on faculty committees,
evaluation committees, the– by age and by academic level,
whether they were tenured, the older and tenured
faculty were actually less concerned with
impact factor or prestige. But the younger
faculty and pre-tenured were more concerned
with it and thought that everybody else was more
concerned with it, too, right? So one important thing here–
and I saw somebody write it on the list outside– if you could change one thing
about academic evaluation, it would be talk to your peers. Are we communicating enough
with each other about actually what we value,
right, because we’re thinking everybody else values
impact factor and prestige. But actually, we’re
all saying we don’t. OK. So there is this disconnect
here and maybe just a little bit more communication. I’m not saying
it’s the solution. But it might actually help. Another result– conclusion–
our results raised concerns that this journal impact
factor, this metric is being used to evaluate
quality and significance of research, despite
the numerous warnings and numerous studies
we have showing that there is no relationship
between this metric and quality, right? And in fact, there
might sometimes be a negative correlation
between this metric and quality. And in the paper, we
have a whole discussion about those various studies. Again, we don’t
seem to be relying on the scientific evidence here. So in that spirit, I want to
close with this quote, which is, it is curious that we
would choose to rely upon such as non-scientific
method as the impact factor to evaluate the
quality of our work. More curious is that we would
do so as unquestioningly as we have. Why we have done so
is not entirely clear, but that we need to stop is. [LAUGHTER] So with that, I will close. I just want to
thank a few people– Juan Pablo Alperin, who’s
the PI on this project. Meredith Niles is the co-PI
together me, Lesley Schimanski, Carol Munoz Niees,
Lisa Matthias, who all worked on this
part of the impact factor part of this project, and then
our funding from Open Society Foundations. And with that, I’m
three minutes over. I will take questions. Thank you. [APPLAUSE] And I’m going to go back
to this quote, because I think it’s fantastic. Yeah. [INAUDIBLE] Are we going to–
oh, we’re going to do questions at the end. See? Sorry. I don’t even know
what’s going on. All right. Thanks [INAUDIBLE] Thank you very much. So we will have–
if there is time in the time to go for
questions at the very end. So we have to see some
data and kicking us off. So next up, we are delighted
to welcome Professor Paula Stephan. And Paula is professor of
economics at Georgia State University. And her research focuses
on the economics of science and the careers of
scientists and engineers. And she is the author
of a book called How Economics Shapes
Science, which I have read and which I would
recommend heartily to this entire audience. I think it really explores many
of the issues that we are going to be discussing this
evening and through the rest of the meeting. So she is also a Fellow of
the American Association for the Advancement of
Science and a member of the board of
reviewing editors at a journal called Science. I don’t know if anybody’s– [LAUGHTER] –heard of it. And Paula is going to talk
to us this evening about how the journal impact
factor is used– oh, sorry. No, she’s not. [LAUGHTER] That was just a joke to
put you on your toes. How can we change the
research assessment culture? So this is going to be a
really exciting 20 minutes. [LAUGHTER] Or have you changed your title? Please– [INAUDIBLE] [APPLAUSE] –Paula. I really– this title
was on the program. And I decided that it
was very close to what I wanted to talk about. So I left the title. So thanks, Stephen. And it’s great to be
here and to spend time talking about these issues. And it’s also lots
of fun to reconnect with alums from the next
generation, the National Academy Committee,
that we had a committee report on the next generation
of biomedical researchers. And some of the people, who
were very active in that are participating here. So let me begin. I think it’s very
helpful to lay out some facts about the
biomedical workforce. And the facts I’m
going to lay out relate to the United States. I don’t have good data for
Europe or other countries. But I assure you
that if I did, they’d look very– we have very,
very similar trends here. So first of all,
the first fact is there is a large and
growing number of PhDs are produced in the biomedical
sciences in the US annually. It was 8,500 in 2017, the
last year NSF has data for. And that was up from
7,200 10 years earlier. And then, a large percent do
not have definite commitments at the time they graduate. So this is excellent data that
the National Science Foundation collects through a survey
called the Survey of Earned Doctorates. So the SED has a response
rate of over 96%. And the reason why
is on most campuses, people think you
have to fill it out if you’re going to graduate. [LAUGHTER] OK. And that’s not true. But the data’s there. And what you can
see from this is that this is the
percent of people that have definite commitments
for employment and at doctor award among US
doctoral recipients. And you see that NSF presents
it for different disciplines. And you see that the discipline
that ties here with engineering is the life sciences in
which it’s been going down. It’s just ticked up just
a wee bit, just under 60%. And then, the next is
the largest percent with definite commitments,
or definite commitments for postdoctoral positions. OK. And you can see
that in this data. This is the postdoc rate
for US doctorate recipients. And of course,
this is for people, who have definite commitments. And if you look at,
it’s the highest here in the life sciences. It’s peaked at about
70% of all the people with definite
commitments for going into a postdoc position
in the early 2000s and came back up
with the crisis. And now, it’s down to
about 62%, but pretty high. And the next fact is that a
large percentage of postdocs aspire to tenure-track jobs. I don’t think that’s a
surprise to us at all. Here’s some data that
Henry Sauermann and Mike Roach put together
that some of you may have seen an
article about this that was– there was both
an article in PLoS and an article in Science
analyzing some of this data. It’s the survey of people
at 39 institutions. And they ask across
fields, if you planned on taking a postdoc– that’s what PD refers to here– or if you didn’t plan
on taking a postdoc. And in the biomedical
life sciences, you’ll see that this dark red
is faculty research positions. And 61%– you can have a
multiple responses here. So these things
don’t add up to 100%. But 61% of the sample
listed that they had an aspiration or at least
one of their aspirations was to be a faculty in
a research position. And then, a very small
percent will actually obtain tenure-track positions. I think most of us
in here know that. If you look at some recent
NSF data, fairly recent, NSF says that for people, who’ve
been out three to five years, 10.6% of the biological,
agricultural, and environmental
scientists will have a tenured or
tenure-track position. And that was down from 17.3%
in 1993, 20 years earlier. So it’s about a 40%
decline from that time. And you can see that it’s the
lowest across disciplines, although there’s some
pretty low rates there from other disciplines. And then, I referred to the
Next Generation National Academy Committee and one of the graphs
that we have in the report– and I would urge
everyone in here to look at some of the things
that are in that report– is that we looked at over
time, what types of positions did people getting PhDs
get based on their cohort? So there are lots of things
going on in this diagram. So let me just explain. This is people, who’ve
been out one to five years. So we can go back to data. And in 1993, we
can say, if you’ve been out one to five years,
what sector are you in? And we divided that
into these sectors. So red is tenure or
tenure-track position. OK. And the ones I particularly
want to focus on here are tenure or
tenure-track position. And purple is
non-academic, non-research. So this is people, who’ve
been out one to five years. And you see the percent
in tenure-track positions for people, who’ve been out
one to five years declines with each cohort. The last one we had the data
for for that report was 2013. And then, look at–
this is people, who’ve been out 6 to 10 years. And we see that it was
about 30% for in 1997 for people who’d been
out 6 to 10 years, had a tenure-track position. And this is the percent, who are
in non-academic, non-research jobs. And then, this is people,
who’ve been out 11 to 20 years. And we see that people,
who’ve been out 11 to 20 years in 1993– about 37% had tenure-track jobs. And a very much
lower percent had non-academic, non-research jobs. And then, look at
these halcyon days for people, who have been
out more than 20 years. In 2000– in 1993, almost 40%
were in tenure-track positions. And look at even
for that group it’s declined remarkably over time. And the purple line is
non-academic, non-research. Now, this graph was first
produced for the NIH Workforce Committee in 2012. And it’s done again, for– at the University of Kansas. They put a lot of
work into doing this. And Shirley Tillman was one
of the co-chairs of the NIH Committee. And I mean, we really thought– I was a member of
that committee also– we thought this graph was just
very telling, in particular, the red line really going
down for every cohort, and that purple
line really going up for many cohorts, which is
a non-research, non-academic jobs. And so the final one, which
is absolutely no surprise to anyone here, is
that bibliometrics play a really large
role along the way. And one of the things
I always like to say– some of you in here have heard
me say this many times before– that publication in a top
tier journal, which generally, for many people,
if you ask them, means Science, Nature, or
Cell is the “get out of jail” card of postdoc,
of being a postdoc. And I was also on the National
Academy’s postdoc committee. And one of the things
we heard again and again were people saying
they were staying in their postdoc position until
they hit one of the big three as for software, and that would
be their exit or their “get out of jail” card. So more generally–
and as everybody here knows–
bibliometrics, especially short-term bibliometrics
play a large role in hiring, in third year review,
in evaluation for grants, and as we just
heard, in promotion. And I’ve been particularly
interested recently– and Anna asked me
just to mention this– in how the heavy reliance
on short-term bibliometric measures arguably
encourages risk aversion when it comes to funding
individuals, as well as hiring and promoting people,
who take risks. And so you can see
this with some work I’ve done with two colleagues. One of the colleagues
is Jian Wang, who’s at the University of Leiden. And the other colleague
is Reinhilde Veugelers, who’s at KU Leuven. And we had an article
in Research Policy. And then, there was a comment
that summarized this research in Nature with this graphic. But Nature designed,
not us, I have to say. OK. So the premise of this work
is that novel research that combines streams of
knowledge that have never been combined before is
riskier than research that doesn’t do that. And it’s more likely
to be frontier shifting and also more likely to fail. And we have operationalized
this by looking at the number of first ever
journal pairs in references of papers published in 2001. And we find that novel
research papers that combined streams of research
that have never been combined before are very rare. 89% of all publications
in 2001 don’t do that. 11% do. But generally, they only
make one or two combinations. But there’s the
tail of this, which is not surprising in which we
see a large number of novel combinations. And we call those highly novel. And novel research, we
find, has higher citations. But it also has
much more variance, which we see this
as a measure of it’s being correlated with risk. And it’s more likely
to be top cited. But it’s also more
likely to, quote, “fail” and get virtually
no citations at all. And I think what
really struck us, particularly when you
think about promotion and when you think about
hiring to begin with, is that novelty takes
time to show up at all. And so the aqua line here
is highly novel papers. This is the probability that
it will be a top cited paper. And the flat line here
is non-novel papers. And what we see is that in
these first couple of years, the highly novel
papers, which end up being much more likely
to be highly cited, are just not showing up at all. And that’s so important,
I think, because you know, we’re putting so much emphasis
for early career people on things that happen right at
the beginning of their career. And this type of
measure suggests that we’re not even going to
see what they may be up to. And then, not a
surprise to anyone here, I think there’s a negative
correlation with journal impact factor. Now, why? We don’t know whether it’s
selection– self selection, where you submit the
paper, or whether it is an editor’s decision. But there’s a very, very
strong negative relationship. So I mean, I think
the implications are fairly straightforward. It suggests that heavy reliance
on short-term bibliometrics, such as impact factor and
recent citations, biases, decisions against investigators,
who are doing novel research. And we think some
of this is highly or more likely to be risky. And yet, we know that
funding agencies– there are certain funding agencies
that absolutely require that you report your citations
and the journal impact factor in your application. And then, there are
funding agencies that don’t say you have to do it. But when you submit your
grant through the university, they have you add it. The European Research Council– I’ve served on panels there. Everybody’s vita
has impact factors associated with their CV. And it’s not that
they require it. It’s just your university
says, oh, you better put it on if you want to
be competitive at the ERC. And you say similar things
in many other places. And then, hiring committees
look at such metrics. And I don’t know. One of the things I’m interested
in in your research, Erin, is that at least in my
university in many ways, not talking about
hiring so much, but talking about third
year review and promotion, the person, who really
pushes here is the provost. The provost says, I got to
see those high-impact factors, or I’m not going to let
this person go through. And that really trickles
back down to the department. And the department starts taking
what the provost is saying. And so it really seems to be
at least in some universities, I think, fed somewhat
from the top. And I think artificial
intelligence really encourages a lot of
this, particularly in hiring decisions and going
through lots and lots of CVs and being able to sort it out. I should tell you that
just very recently– and it’s very preliminary– we’ve extended this research
by examining the relationship of novelty to getting funding
from the European Research Council. So for those of
you who don’t know, the European Research
Council is the major funder– a bottom-up funder in Europe. It has a very large budget. And it has 27 different panels
that it funds people for. And it funds starter grants–
those are people just out– advanced grants. And occasionally,
it has a program called consolidator grants. And anyway, we were able to
get all successful grantees and a very– and
67,000 applications have been made to date. So we got all successful
funded people. And we got a very
large sample that seems to be random of non-funded. And we distinguish
between those, who had submitted
a proposal, who’d done novel research in
the past versus those, who had not to look at what
their funding decision looked like. And we found that those with
a history of novel research are more likely to be– that those without a
history of novel research are more likely to be funded
than those with a history. But and I think that
really speaks to the fact that review panels act
like insurance agents. And all they want
to look at is loss. They sit there worrying
that if they let this by, they might really
have made a mistake, you know, and have
squandered resources. And so they look at negative
tale I really think. But interestingly
enough, risks seem to be more tolerated for
senior investigators. The people, who are really
getting penalized here are the starter grants. Those are the people–
so once again, it’s these bibliometrics
really having, I think, a very adverse
impact on early career people. So what can we done,
or what should be done? And I have to say I’ve been
on so many National Academy committees. [LAUGHS] And I’ve had so
many recommendations with a lot of
colleagues in this room. So I think what should be
done and what can be done– it’s a hard road here. But let me say it again. OK. I think most
importantly, we’ve just got to stabilize the
biomedical workforce. PhD programs and
postdoctoral programs are used as inexpensive ways
to get the research done. And I think this
creates an unreasonable amount of competition for a
limited number of research positions and that these
are being allocated on bibliometric measures. And I just think it’s
just been killing the system for a long time. But now, we’ve got all
these bibliometric measures and AI, et cetera. So it just exasperates
the problem that we’ve known that it’s
been there for a long time. And I want to remind all of
us that there are other ways to get the research done. I really have strongly
felt and continue to feel that it’s time to create
an increased number of staff scientists positions and to
quit growing the supply of PhDs for research positions that are
just unlikely to be out there. What can be done
at universities? Well, I mean– and I think
everybody here really believes this– that we need to
dedicate more resources to evaluating by reading and
not evaluated by counting. I think, Frank, you’ve said
something very similar to that. And we’ve just got to
educate deans and provosts to the downside
of bibliometrics. I really think
that these people, at least in some institutions,
are under lots of pressure between departments. And they’re under pressure. And bibliometrics are an
easy fallback for them. So I think in terms
of what we can do, we’ve really got to
think about that. I think we’ve absolutely
got to require applicants for positions and
promotions to submit packets of their best and not
everything and be selective. And I think we’ve got to
stress the research portfolio as an important criteria. And I believe that
for panels also. I mean, one of the things that
I think has really gone amok is that review panels,
review proposals one by one, and they never think about
what their portfolio is. They never think, you know. And we, as investors, you
don’t buy a stock one by one. Hopefully, you don’t. You should buy stocks
and think about how it relates to your portfolio. OK. Well, you need– we need
to make funding decisions. We need to make promotion,
hiring decisions and thinking about the portfolios we have. And not everybody’s going to
do exactly the same thing. We need some high-risk people. We need some probably
low-risk people. We’ve got to get this done. But we don’t want to
have it all be the same. And more generally,
I just think we’re missing so much by
not having a place, where failed research
gets published. People don’t learn from other
people’s mistakes or things that didn’t work
out or whatever. And people should also
get credit for that. And I’m very, very keen on
deescalating the arms race. I think rankings are
absolutely killing the research enterprise. I really do. I think they were already there
before the Shanghai Rankings. But I think the Shanghai
Rankings– particularly, you can see the
effect in Europe– have just– have been
very, very detrimental to building strong programs. And instead, everybody’s
worried about counting, because that’s what places
like the Shanghai Rankings do. They count. They count high-impact factors. They count Science. They count Nature, et cetera. And that’s just
killing our enterprise. And that’s a very
hard thing to change. But institutions all across
the United States really spend so much of their
resources and so much of their efforts
thinking about where they stand in the rankings. And then, I’m a firm believer
that we have over-relied on the university
model as the place to get all our research
done in the United States. I think we need a
little diversity here. I think we need more
institutions, where people aren’t engaged in training. I think training needs research. But research doesn’t always
have to come with training. And I think it’s important to
have other models out there and to have a little
competition going on and to also have
research going on, where we’re not
producing more PhDs. I once said that birth control
is the most effective– I mean, that abstinence
is the most effective form of birth control. And I really believe– [LAUGHTER] –that in terms of research
institutes that can’t give PhDs are not going to be doing their
research by hiring cheap PhDs. OK. And then, I think we really
need to start thinking about different funding models. I’m not at all convinced a
lottery is the way to go. But I have to say, there’s some
interesting research out there. And I would really
suggest that people go look at Carl Bergstrom. He’s at the University
of Washington. He’s an evolutionary biologist. And he’s done just some really
interesting work on lotteries. And I would encourage you to
look at his website and some of the things he talks about. And for those of
you who don’t know, the Volkswagen
Foundation in Germany is now funding most of their
proposals through a lottery. And also, the
Swiss Foundation is funding a lot of its
proposals through a lottery. So it’s not something
that people are just whistling about. There’s a little
bit going on here. And then, I really think–
and I’ll close with this– I think we need to generate
good, reliable placement data. I think this goes
back to my first point that we’ve got to stabilize
the biomedical workforce. I just really think
that undergraduates need to know career
prospects before they go to graduate school. They don’t need to just
get their information from one faculty member,
who hopes to recruit them to work in their lab. OK. And in this respect, I think
some progress is being made. And the progress here– I really want a
shout-out to Ron Daniels. Ron Daniels, many
of you may know, is the president
of Johns Hopkins. And he chaired the Next
Generation National Academy Committee that some
of us in here were on. And one of the things that he
did while the committee was meeting, not after
we wrote our report, but while it was meeting,
he formed the Coalition for the Next Generation
of Life Scientists. And he got 37
universities by the time the report went public
to commit to making career data available in
the biomedical sciences. And I would encourage– let’s
see if the mouse is going to work– oops– I don’t think this mouse works. Maybe somebody– [INAUDIBLE] Oh, it is working. It just doesn’t
work on my screen. So this is– you
can– this database, and this tells you all
the institutions that have agreed to collect data
both on PhDs and on postdocs. And it’s growing over time. And let me just give you
one example here of UCSF. They have these dashboards. But I mean, one of the things
that really appeals to me is that we can look at
the year of separation, like, let’s say 2015– this is for postdocs. And we can find out what percent
went on that separated in 2015 are in further
training, or they’re in primary research,
primary teaching, et cetera. And then, you can see
the sector they’re in. And then, you can see the
job function they’re in. And one of the things
that really, I think, is striking about this– this is data we’ve been trying– I mean, trends in
the early careers of life scientists– in 1996,
we made this recommendation. But we simply didn’t have a
good way of getting the data. Now, through LinkedIn
and lots of things, you can get this data. And I’ve gone through
a lot of these. And for almost
every institution, they’ve been able to find
everything that 11% to 12% of the people. So with that, I
will call it a day. Thanks. [APPLAUSE] Thank you very much, Paula. I’m very glad to see someone
calling for staff positions. I think that’s a real key
to change here and also another ally in the fight
against the cancer that is university rankings. Oh, I think my
microphone is gone. Hopefully, it will come back. So we move on now to the
third speaker of the evening. And we’re very lucky to
have with us Dr. Bodo Stern. So Bodo is chief of
strategic initiatives. And in that capacity– here at HHMI– and
in that capacity, he works directly
and very closely with HHMI’s president,
Erin O’Shea. And thanks to that
close relationship, he’s able to step into the
breach to give Erin’s talk and to tell us what– how HHMI
is evaluating researchers. So Bodo joined HHMI in 2015. Prior to that, he
was at Harvard. And before he joined Harvard, he
was a senior scientific editor at a journal called Cell– [LAUGHTER] –which– I don’t know. I’ve never heard of it. But anyway– [INAUDIBLE] [LAUGHTER] Over to Bodo. Thank you, Steve. So Erin sends her regrets and
apologies that she cannot be here tonight. She had to take care of an
emergency that just came up. I hope she will be able to
join us later for the meeting. And I also hope that these
slides are self-explanatory and really good so that my
commentary is supportive, but not essential– [LAUGHS] –because I don’t see the– hang on. Where are they? OK. I want to start by
thanking Boyana and Erika for your opening remarks. I want to thank
Anna, who was really the master organizer of this
meeting with help from Boyana. I want to thank Stephen for
co-sponsoring– from DORA for cosponsoring this meeting. And I also want to give a
shout-out to our conference services that will sustain us
with excellent food throughout our really energy-consuming
discussions here over the next two days and also communications
and A/V for making sure that we always have slides here and
that we can also send out the slides– I mean, that people from outside
can join our discussions. Before I dive in, I
just want to give you a few words about HHMI. HHMI has a long history of
supporting discovery science. And we are proud of it. But at the same time, we
know that we can improve. it’s not just enough
to do good science. It’s not just enough to
focus on what science we do. It matters also how we do it. Yeah. And for that reason, we have to
find some strategic priorities that are supporting our
efforts in discovery science. And I’ve listed them here– diversity, equity and inclusion,
a healthy academic ecosystem– and under that umbrella, we
cover scientific publishing and academic incentives– and public engagement. And with this new framework,
we also looking at how do we select our scientists. How do we assess our scientists? And I will give you, you
know, some more information on how we are assessing our
scientists when they come up for renewal. Now, we know there
is improvement possible and necessary
also in our processes. And you know, we all
hope that we will learn from these discussions. That’s why we are
here, yeah, that we can learn from each other
how to improve assessment, but especially the review of our
HHMI investigators for renewal. These processes have
been honed and improved for, like, decades. And you know, we think
there is something that we have learned from
these decades of experiments that we would like to share. Now, this meeting
is about how we can improve how scientists
and scientific work are evaluated,
recognized, and rewarded. Last year, we hosted
a meeting here– it’s about a year
and a half ago– a meeting co-organized
with ASAPbio and Wellcome on how we can
improve peer review. And that meeting
and this meeting we actually see as closely
linked, because both meetings had really one central element. And that is this question here. How can we diminish
the corrosive influence of journal names and metrics
in research assessment? At this peer review
meeting in 2018, we had scientific editors,
scientists, early career, senior scientists,
funders discuss how peer review can be improved. And one of the very early
outcomes of that meeting was that there was an agreement
among the participants that what we really need is that
the content of the peer reviews needs to be published. We need to have more
transparency of the evaluation process. And since then, more than
100 journals have signed up– have signed a letter
committing to making peer review reports available. We’ve also seen many
interesting experiments in peer review come on board. So there is some
progress that has happened since that meeting. But there’s a lot
more progress that we need to make in order to really
allow the kind of research assessment that we need
in the future, progress on the side of
scientific publishing. These are what we consider
the three major challenges. We already mentioned
in all the keynotes, scientists are evaluated based
on where they publish, not what they publish. And one major reason why
this is such a pervasive– the journal impact factor is
such a pervasive metric that just doesn’t go away is
because the journals just don’t give us anything else. They just tell us, this was
published in this journal. And because we don’t have access
to the evaluations, the journal impact fact that pops up
as a proxy for impact. And so what we really
need as a second point, therefore, is we
need to get– we need to have more transparency
and accountability. We need to have the results
of the peer reviews available, because then, they
can help and inform the assessment of scientists
at institutions of funders. And finally, there is a lack
of credit at the moment. Scientists do not receive
discoverable credit for all their contributions. For example, how
do you figure out who did what on a collaborative
paper when you have 30 authors? There is no way of knowing
who did the experiments of a particular figure. Who did collect the data sets? Who wrote the software? All of this– it’s important. If we are talking about
interdisciplinary collaborative science, we need to make sure
that people get actual credit for the work they have done. And that applies also to
the peer reviewers if they have done a great peer review. And you know, we
want to make sure that they can be
credited for that. Yeah. So these are the
three challenges. And here is how,
very briefly, how we think scientific
publishing needs to change in order to support
a healthy research assessment practices. Autonomy and recognition
for researchers– it should be the authors, not
editors that share their work. This already happens in
the form of preprints. Authors put preprints
up on preprint servers. We think it should happen
for all versions of a paper. And then, a credit
should be assigned based on individual contributions. Articles should be published
open access, freely available to anybody,
who wants to read them on the day of publication. And all works should
have unique identifiers so that we can track
who has used them and how often they
have been used. And finally, as I
mentioned already, transparent evaluation
of scientific work– that means the peer review
reports should be available. We also think it’s important
that the peer review reports are somewhat
distilled so that it’s easier for readers and
for academic institutions to use these distillations
and the peer reviews for their assessments
and for decisions what to read in the first place so
that we can replace the journal name with these new
kinds of, you know, transparent evaluation forms. Now, all of this will
not happen overnight. It will take years to
experiment and figure out how we can move
publishing into an area– I mean, into a process,
where authors share. Everything is open access. Peer reviews are
available and distilled. It will take years. So the question is, what
can we do in the meantime? Yeah. And that’s where I want
to shift gears and now tell you about how we evaluate
our investigators when they come up for renewal. One of our guiding principles
is people, not projects. We place big bets on
creative individuals that have been
successful in the past. But we provide them with
the level of funding and the length of funding
so that they really can follow their scientific
nose and do the research that they think they should do. Now, if our focus is on
people, not projects, we need to have a robust way
of assessing that the people we have really should be renewed. And we should fund
them in the future. And so these are the criteria
that all of our investigators know very well and
advise us that I’m talking about in a
second know, as well. Now, we want our scientists
to focus on basic science, discovery science. And this– because we believe
that this is the foundation for all benefits that
stem from science, whether it’s translation,
applications– basic discoveries science
is really the foundation for anything afterwards. And that is reflected
in our guidelines for our scientists, the criteria
that we use to evaluate them. Pursue rigorous and
deep biological studies. Lead research fields into
new areas of inquiry. That’s sort of along
the lines of what Paula talked about– novelty. That is something that we really
cherish in our scientists. Develop innovative
tools and methods. Forge links between basic
biology and human health. That’s not essential. We’re not– all our scientists– I mean, many of our scientists
are basic scientists. Well, they don’t work
on something that is related to human health. But some people do. And you know, that is
not held against them. That’s a good thing. And demonstrate promise
of future contributions. And finally, an criterion that
we added a few years ago– actively serve and train
at host institutions and in the community. Now, we do not expect that
all these criteria are satisfied by our scientists. But we want a successful
combination of these criteria for scientists that are
going to be renewed. This is the process that we use. We– I mean, I’m talking about
the reviewers, materials, and in-person meeting. So we are choosing respected
leaders and advisors. We assemble a group of 20, 25
every time we review, you know, 15, 20 investigators. And these are respected
leaders in their field. At least 30% are women. Some of them are long-serving
members of our advisory boards so that we get consistency
throughout the years of reviewing our investigators. But we add expert– ad hoc
experts on each review. These advisors–
these reviewers– they serve as advisors. We trust their advice. But the decision is
made by HHMI leadership. I think that’s an
important distinction, that these people
write critiques, but they don’t
make the decision. Decision is made based on the
advice, but by HHMI leadership. And we spent a
great deal of effort of reviewing our
reviewers to make sure that we continue to have really
the best people in the room. That can be a laborious process. The materials that we
ask from our scientists are a research
report that describes what accomplishments
the scientist has achieved in the
past appointment period and what the future plans are. They will select five of
the most important papers, include– they can be preprints. I’ll talk a little bit more
about these five papers. And then, they write a statement
about mentorship and community activity. And at an in-person meeting,
they give a presentation on their past accomplishments
and their future plans, which is followed then by a
question-and-answer session, where the advisors
ask questions. After this
question-and-answer session, the advisors deliberate. And using these
deliberations as a basis, you know, and the talk and the
questions, HHMI leaderships makes a decision whether this
scientist should be renewed or not. Now, I do want to highlight
the emphasis on the five most significant papers, because
that does two things. It tells our scientists
that we care about impact. But because we are
asking our advisors to only look at five
papers, it frees them up from looking at where
their paper is published, because they can read
the paper in detail and can make up their own mind
whether that paper is actually really a high-impact paper. And the scientist will
provide a short statement that describes what they think
the major impact of this paper is. So you see by just pointing
the advisors to just five papers of the
scientists under review, you manage to take
out the journal impact factor and the
quantity of output, because they can focus their
analysis on a few key papers of the scientist. Now, all of these papers
must be publicly available. They are either
published in a journal– if they are not published
in a journal yet, they must be on a
preprint server. And that can be, for
example, important when an advisor wants
to seek some advice out from some colleague. They wouldn’t be able to
share privileged information. But if it’s on a
preprint service, it’s of course, public. And you can get advice
on that reprint. So just in summary, what
counts in this assessment of HHMI investigators? We use expert advisors. We focus on the five
critical articles. And that means we focus
on past accomplishments. It’s really– I mean,
we follow the guideline. Past accomplishments
signal future success. It really boils down
to the question, has this investigator
made a contribution that is significant
enough that if you took those contributions out,
the field would be different? If the field would
not be different, because other people have
made similar advances, that could be an
issue for the renewal. So that is what we refer
to as the deletion tests. It doesn’t count how many
papers a person has published, where the paper is published,
and how much research funding the person has received. So just to wrap up, we think
from this HHMI process, we can consider certain elements
for the evaluation of tenure and promotion committees,
including the focus on a few research articles,
rather than on the whole work. And in fact, I
think that was also on your a little bit also on
your slide, Paula, so something that we’ve heard
now several times. We’ve been fairly secretive
about our process in the past. But it’s now available
for everybody to see, because we realize
this is something that other places that are
also thinking about research assessment could benefit from. So there is a much more
detailed description here. And I just want to finish
by coming back to what I said at the beginning. There’s a lot of
expertise in the room. I mean, we can really
learn from each other. Let’s make the most
of this meeting. Thank you. [APPLAUSE] Thank you very much, Bodo. That was a superb job,
particularly given the very late
notice that you had of your position on the program. So we do now have
about 20 minutes or so for questions and
answers and discussion. I think I might like to invite
the three keynote speakers up onto the stage, then, who
are the focus of attention and invite you, the audience,
then to give us your– [INAUDIBLE] –questions. And well, comments
are also welcome. But they would be particularly
welcome if they are very pithy. [LAUGHTER] If I detect a certain
grandiloquence, I will maybe ask you to
quickly get to the point. And as you’ve seen, HHMI
have these wonderful cubic microphones, of which
there are a number around. So Gary, I see you as
the first to grab the– Oh. –grab the– Sorry. –cube. So please, Gary– [INAUDIBLE] –why don’t you to
come to the middle and if you don’t mind being
a little bit more exposed? [INAUDIBLE] Oh, we are mic’ed up, I think. Are you? Oh, you’re– Oh, OK. Oh, OK. You stay here then. That’s fine. I have one. I know. [LAUGHS] So you have one, too. Yes, OK. We’re– The question is for Bodo. So let’s say that I apply
for a grant in the same cycle as Dr. Yamamoto
applied for a grant. How do you differentiate between
my five publications and Dr. Yamamoto’s five publications? So we will– well, we– you wouldn’t get a grant. You would be employed by HHMI. But we have a– [LAUGHS] –so we have an
eligibility window, where people are considered
for an HHMI investigatorship. And so that is
from 5 to 15 years. So five years is long enough
that somebody can have and you know, somebody,
who will be successful should have evidence of
independent contributions from their lab in 15
years, because it was 10 years more opportunities. But you would– both would
only be evaluated for– you would focus on
the five papers. So I think it balances out
a little bit, the difference between five and 15 years. But you wouldn’t
compete with Keith. We have a question at the back. And then one here. Hi, thank you very much for
the very enlightening talks. Leslie Henderson, I’m at the
Geisel School of Medicine at Dartmouth. And I want to thank both Stephen
and Paula for the emphasis on increasing the ranks
of research scientists. We have recently
begun to do that. And I had a question
for you, as well as some of the funders in the room. I know NCI has a
mechanism by which you can get institutional funding
for research scientists that aren’t staff
scientists, that aren’t tied to a given individual
investigator on a lab. That seems essential,
really, for this sort of cadre of people to
have gainful employment with any sort of security. Do you know of other institutes
or other funding mechanisms that support research scientists
in this institutional sort of way? Our scientists– so our
scientists have very generous research budgets that actually
allow them to have research scientists in our– I mean, HHMI labs have actually
quite a number of research scientists employed. And that is, of course, thanks
to fairly generous budgets. But I was more hopeful of more
generic funding mechanisms, outside of the HHMI cohort. Well, you see– but
that’s– you know, that would be for a
granting organization. Right. I mean, we employ our
investigators and many of the scientists in their
labs, and that includes a lot of staff scientists. So yeah. Well I have to say that, I
think, both for the next Gen committee and for other
groups I’ve been on, we’ve looked for
examples of how to get a funding for staff scientists. And it’s hard to
find many places. And I think NCI is a great– is great what they’re doing. But I also think if
we check they’re not many of those positions yet. So I really think this is– I mean, for a long
time I’ve advocated increasing post-doc salaries. And I’ve advocated
it for two reasons. For one is equity. And two is for
helping universities to understand that post-docs
are more expensive, and to begin to substitute
staff scientist positions for post-doc positions. And so I think there also
has to be economic pressure, that the way we’ve been staffing
our labs aren’t the cheapest. That cheap– Yeah, labor. –is creating bad
incentives here. Thank you. We have one here
and then Jessica. A couple of questions related
to suggestions raised. So one– and I’ll
just ask my questions, and then you can talk. One is related to the peer
review and transparent peer review. And we have a problem now
that people don’t have time to read the papers. And to think that they have
time to read the peer reviews and the papers I think
is not realistic. The second issue that was raised
is about staff scientists. But the modular budget
of $250,000 per grant has not changed for 25
years, post-doc salaries, rightfully so, have increased. But with that modular
budget, we’re– and NIGMS is absolutely
a disaster in this way. It’s forcing young investigators
to staff their labs with cheaper and cheaper
graduate students. So it’s really working
against the system. You talked about citations
times journal impact factor. One way to change that
is to divide citations by journal impact factor. So the hot– which
is more realistic, because the higher
the journal impact factor is, if your
citations are higher than that, that means
it’s a good paper. And so– That’s a good idea. –putting the journal impact
factor in the denominator, as opposed to the numerator,
would make a difference. So I’d like your
comments on those. I can start on the transparency. So I have two comments. I totally agree
that if we already have problems reading all of
the paper, then we won’t read. But I don’t think it’s
a matter of that we need many people to read them. It’s important that
the people who do want to read them have them. Yeah. So I think access to the peer
review reports is critical, even if only a small
fraction actually reads them. And the second
point that may have gotten a little bit of
misspeak is it’s not– we don’t think it’s sufficient
to publish the peer review reports. It’s also important
that we figure out ways to distill those reports. Yeah. So that we attach
something to the paper that is a fairly quick
note to a potential reader. This is, you know,
let’s say there’s agreement of the peer review
as if this is a rigorous paper. I mean, the scientific
community needs to decide what exactly those
distillations look like. But that will help
people fairly easily to spot what papers to read
and what papers to rely on. So it’s not just
publishing the reports that can be just as
long as the paper. It’s also distilling the
reports in useful ways for the community
and for the readers. So let me just make
one kind of comment about the staff scientists. I mean it is a resource issue. And it’s how we’ve chosen
to use our resources. And I also think it
goes back to rankings. I mean university
presidents, deans of medical schools
in the United States excel at getting money
for buildings, right, and for things like that. And I think it’s
time that we really begin to put pressure on people
to understand that it’s what’s going on in those
buildings that’s going to build reputation,
and not just a new building. We drew this huge overbuilding,
a biomedical research space in the United
States with the doubling, and it’s still going
on to some effect. And it has a lot of
financial consequences on a lot of campuses. So I think there
has to be leadership here among some key
institutions to begin to devote some of
their fundraising for staffing these
labs in different ways. Now maybe the– maybe I
ate something at dinner and I’m just
dreaming, but I really think that it’s going
to take some leadership within universities. It’s not going happen
through NIH, I don’t think. And it’s got– although
we have examples, but I don’t think we’re
going to see a big change. And I don’t think any
one faculty member is going to be able to do this. I think it’s going to have
to take some leadership. Jessica. So I’d like to extend
Gary’s question. So let’s say that Gary publishes
one paper every three years. It’s a really beautiful
complete story. But I prefer to
publish papers that are sort of more intermediate
and that come out really quickly with the results. Yeah. Yeah, so how can we reconcile
this need to tell a story– So I think– –with limitation? I would say, you know, when– if we move into an era where
people publish their data in smaller units,
then we should, you know, maybe
that is time then to revisit the
number five, or that even to have a certain
limited number, then you may have
to go to something else that still
captures impact rather than the number of papers. Yeah? You know, right now
this is something that has worked because
people are trying to publish complete stories. But if that changes,
I agree that then we need to revisit this approach. Thank you. But I think the approach of
focusing on, you know, impact rather than on
numbers is can remain. OK. At the back here. So this is a question for Bodo. So in your talk you kind of
laid out the ideal HHMI review process. But I was disappointing– That’s how it– no, I’m joking. I was disappointed that you
didn’t have any data to back up the process that you outlined. So for instance,
what evidence do you have that publication venue does
not matter during HHMI review? Well, I mean, I can certainly– I mean, well, I mean right– we don’t have that data. What I can say– what I can say is that
we have investigators that are being renewed
without, you know, publications in prestigious journals. Yeah. But of course, many
of our scientists do publish in these journals. This is an effort that
will take many years. I mean you cannot– you don’t want to
penalize people if they decide to publish
their papers in Cell, Science, and Nature. So that, I think, that
would be the wrong approach at this point. But I think– but I
think, you know, it is– people who are participating
in this process are surprised how little journal
names are being mentioned or contribute. So just within
one short comment, I would suggest that it is
misleading to have a slide that says HHMI does not consider
publication venue if you don’t have the data to back
up that assertion. Well, you know, I– Can I– yeah, I
mean, what I think the problem is that we are
trying to do a cultural shift, a cultural change here. Yeah. And that will take time. The way our scientists
publish may still, you know, may still be affected
by ways of publishing. I mean and maybe it’s
trainees that would like to publish in those journals. You know, yes, many of our
scientists’ publication are in those top journals. But that isn’t
necessarily the same as saying that it
really matters. Yeah. So I think the numbers– I’m pretty sure what the
numbers that you are referring to, it’s– you’re right, I
mean many of the papers will be in Cell, Science, and Nature. But I would argue
that doesn’t tell you that that really is critical. It may, you know,
in the process. I think it’s an advance
that you’ve published your evaluation procedure. Because people will
make assumptions, I think, as you indicate. Question here. Who’s next? Brooks Hanson, American
Geophysical Union. First, a question for– a
very quick question for Erika, which is on your survey data. Was there any generational
difference– or Erika and Anna, whoever’s doing it– in the response rate on what the
criteria were for assessment? And then secondly,
my pithy comment is if open peer review is good,
is open research assessment good? Obviously, that’s
pithy, because there’s legal and other problems
that get in the way. But the challenge,
or the criticism, of both opening them up is
it kind of inhibits editorial discretion. Which is like, we’re
going to take a paper or take a person who’s actually
got low reviews, because we think that’s the
right thing to do. So does either one of
those inhibit that? And then the larger question
for all three of the panelists is the whole focus of
the discussion really. And I agree with
this last comment as to whether it’s five
papers or impact factor. We’re dancing on
the head of a pin. It’s still, you know,
I suspect there’s a huge correlation between those
five papers and the journal impact factor. I just suspect. And yes, it’s a change. But how do we
reward things like– important things that are
increasingly important, community service,
communication, engagement, solving community
problems, other things that we really want researchers
to do that are really good for science, for
communicating science, for public engagement in
science, for broad support for the scientific enterprise. None of you mentioned
that, either. I mean I think
you did mention it as one little point at the
bottom of the HHMI slide, but would be interesting to see
if there was anything in the 10 year promotion guidelines that
really are quantifiable there. So I’d be interested in thoughts
on how do we elevate that. Sorry, that was a long dialogue. Yeah. So maybe I’ll touch
on that last part. So you’re kind of, in that
last part of the question, referring to kind of the public
dimensions of faculty work. And we did do another
paper on that, independent of the impact
factor analysis, where we were going
through the documents and looking at how
often do they mention the public, or the community, or
outreach, or things like that. And unfortunately what
we saw was those terms were mentioned a lot. But when they referred
to public or community, they were usually referring
to the academic community. And they were referring
to service activities within the university. And, as we saw, those
are the least valued. They’re all the way down at the
bottom of your list, together with open science, which
is hugely depressing and any kind of
community outreach. So we saw that in the current
promotion in tenure documents, those activities are
hugely undervalued. And even when the
university is kind of talking about the importance
of public and community outreach, they don’t
provide an infrastructure for quantifying
those contributions or evaluating those
contributions in any way. So I think that’s a huge
problem in the RPT documents. And how do you kind
of assess that? It’s hard, because there
aren’t necessarily raw numbers that you can put on
that type of engagement. I think one of the
simplest things you can do is provide a lot more
flexibility in the forms that faculty are
using, provide them a space to simply describe
those outreach– it’s not going to solve everything. But it is at least is giving
the faculty the opportunity to describe those activities. Where in many of
these forms, you don’t even have a space
where you can put that in. And I think there’s also
other ways these narrative statements, you know, what
is your actual– what is the actual impact of this work? And I don’t mean journal impact
factor or academic impact even necessarily. Give them an
opportunity to describe, and it’s harder to evaluate. Because it’s not a number,
but let’s put that in there at least as one aspect
that we’re considering. All right. We’re down to our last
five minutes of Q&A. We have one question here. Diane– Diane O’Dowd Vice
Provost Academic Personnel at UC Irvine. So I really agree with the idea
that impact factor as a proxy for quality doesn’t work. But I think that
we have to think about broader aspects of
this, which is really thinking about the breadth
of the audience that you reach with
particular journals. And that is both, you
know, Science or Nature, that you reach
lots of scientists. Then you have your society,
very good society journals, where you reach your peer group. And then you might have
very small specialty group. And so that actually really
is thinking about impact. And at least in
our review system, I think that’s how we
think about impact factor a lot is sort of the
breadth of audience and also you’re very– way more
likely to get a New York Times commentary or a Scientific
American commentary or a commentary in Science
about a Nature article or vice versa, something that might
reach a broader audience, if it’s in one of
these other journals. So I think that that’s
something that, again, it can’t be just a proxy
measure, but that I really think we have to think about
the impact in that fashion. Yeah. I just want to make one
comment related to that. I understand the point, but also
taking advantage of the fact that this is international
open access week. Those journals that you’re
putting forward as examples, Cell, Nature,
Science that do tend to reach a large community,
that’s assuming you have access to those journals. And there is a huge part of the
world that does not have access to those journals. So and then, you know, the theme
of this week is equity, right. Open for whom? So who are we reaching, when
we publish in those journals. We are reaching one
part of the world that happens to work in institutions
that has enough money to subscribe to those journals. I’ve worked in journals–
in institutions in Mexico that did not have money
for those journals. And so you have to ask,
who are you reaching? What percent of the
world’s population? What percent of the
world’s scientists? And I would actually
argue that some of those smaller community
journals, but that are fully open
access, are spreading that knowledge in a
much more equitable way so that’s what I would. [APPLAUSE] [INAUDIBLE] I wanted to comment on
Paula’s point about– or actually call to separate
research or at least bigger elements of research from
academic and from academia. She probably remembers
that we don’t actually agree on whether there are
too many graduate students. I don’t– I don’t think there
are too many graduate students, but I do think there
are too many post-docs. And it’s very clear that there
is a conflict of interest in the faculty, in
terms of wanting workforce hands on at the lab
bench as opposed to trainees. So that point is
all very well taken. I actually think from our
own experience in our career exploration courses at UCSF,
what we’ve discovered is that if academia steps up to its
responsibility to familiarize students with the range of
career options that are out there, that we’re finding
that they’re leaving happier, going off to other non-academic
positions where they can use their training to contribute
in many ways across a great diversity of career– And perhaps– –possibilities that are needed. And perhaps leaving quicker. And they are leaving quicker. I mean I think that that’s
a very important thing. And UCSF has been
great in doing that. And faster, and they’re
happier with being able to make these choices. The comment I wanted to make
was about the staff scientists. So while I think there
are too many post-docs and that post-docs, the
position for post-docs, should be really
reserved for people who declare that their
aspiration really is to go into academic,
become an academic PI, then the number of
post-docs would change, would drop dramatically. The graduate
students would go off to this great diversity
of other things. That’s what we’re
seeing actually. But the number of post-docs
would drop dramatically. Lab demographics would
change dramatically. And the need for
professionals, what are now called staff scientists,
would actually increase, as would the funding
available to support them. Because there would be
dramatically fewer post-docs. So I think that what we
should aspire to in academia is defining a new
career track that has high esteem value
associated with it, stability associated with it. You’ve pointed out that academia
should take responsibility for paying these people. I think funding agencies
should do it in a shared way, as they do with faculty,
with regular faculty. So but I think the
funds would be. There the demographics
of labs would change. There would be stable people,
researchers, PhD-trained. Who we all know of PhDs who
are spectacular scientists but end up failing in
academia, because they don’t do the other stuff either. Because they don’t want to,
or they don’t do it well. And so we lose them from their
contribution from the research community. Being able to define a career
track that’s respected, probably, including changing
the name of staff scientists which has a certain
pejorative ring to it, would allow that
group now to prosper. They would have
stable positions. Their partners and
spouses wouldn’t be asking when
they’re finally going to get a job, because
they would have them. And I think that if we
can move in that way, so it’d be a dramatic change
in the response to a needed change in the demographics
of laboratories. And I think it would
really just take one or a small number
of institutions to step up and say, we
recognize this as a real need. And we’re going to define
a new stable career track. We’re going to pay these
people, give them benefits and so forth. And because we really need them
in our research environment. And I think we can begin that
by a small number, the coalition of the willing that Ron defined,
to move in that direction. Thank you very much. I’m determined to
wrap up by nine. We have time for
one quick question– I’m sorry. This lady has been
waiting very patiently. Just here. Should I go then? Please, yeah. Very quickly, if you don’t mind. My name is Maria Elena Bottazzi. I’m from Baylor
College of Medicine . And I am just in awe of
all these discussions. But I just want to just make
one comment that maybe is just to remind everyone that research
is also morphing, right. And academia now has
even different roles in the area of moving what we
call discoveries, all the way to hopefully eventually
develop new technologies, interventions, all the
way to the public good and the public use. So I just want to
make sure maybe we open up our minds a
little bit, for example, in defining when we do
research assessment, that some people do very
good in basic science. Some people do very
good or are more interested in the translational
process of translating science. And now researchers
do more and more collaborations with
clinicians, and hopefully even beyond health and
medicine, engineers, economists, social
scientists, right. So I think that
academia now is becoming a little bit of a hybrid of
traditional training of PhD students, post-docs and
morphing into biotechnology type of engines, that,
you know, really can maybe support some of
these discussions about staff scientists. So for example, just
briefly, in my case, we run a Center for
Vaccine Development. Where, even though you may
have an academic title, then we also give
them the alternative of using a appropriate
career professional title as a director of quality
assurance or director of quality control. And I think that also adds
value to the discussion. Thank you very much. I think we have a question
from outside the room. Is that correct? Yes, we do. For the sake of time, I
think I might pose it just as a quick comment to share
what our online audience is thinking, while our AV team
sets up your slide presentation Steven. But I did want to share that
someone online was wondering, for Bodo, if you think
that the subject — the experts that HHMI
chooses as reviewers, if they are used to publishing
in Cell, Nature, and Science, if they might see
publications in Cell, Nature, and Science by applicants
as more impactful than publications in other journals. And then another submission
from our online audience was whether or not
HHMI is considered that if HHMI investigators
or HHMI professors are acting as
reviewers, if there are any data on whether they
review applications differently than non-HHMI reviewers. So I’m just going to put that
out there as food for thought, so that we can move
to the wrap-up. All right. Do you want to very quickly
respond to that or uh? Yeah, I mean, I’m pretty
sure that we all– our habits and past habits
inform how we judge today. And so I think that,
even if we say, you know, the journal
venue should not matter in the evaluation, but we can– I mean we cannot know whether
scientists will still favor those. That is probably, that’s
a possibility, sure. Yeah. OK. Thank you. OK. Thank you all for a
really fascinating and big [INAUDIBLE]. And thanks particularly
to Erin, Paula, and Bodo for a really stimulating
keynote session. I’m aware my timekeeping is
not as good as I promised. It’s five to nine. I only have half a dozen
slides that I will get through, rather rapidly. But I just wanted to offer some
sort of concluding reflections as Chair of DORA about why this
meeting is so important to us as an organization,
and to sort of give you some sort of personal
reflections on my hopes and expectations
for the meeting. How do I advance the slides? Well, there we go. OK. I’ll use the mice. So I hope nobody in
this room is unfamiliar with the San
Francisco Declaration on Research Assessment. We are sort of a
growing organization. We have a growing
group of supporters. We sort of had a revamp
about a couple of years ago, refresh the steering group,
and convened very much a global advisory board. Because I think, as all
of us must appreciate, it’s not enough to change
research culture in North America, or United
States and Canada. It’s really got to happen
all over the world, because research,
academia, scholarship is a quintessentially
international action. And although DORA,
as Erika mentioned, is probably best known for
being somewhat critical of the abuse of the journal
impact factor and research assessment, we are very
much an organization that recognizes that it’s
not enough to simply lay down the criticism and then move on. It’s really– and
one of the focuses that we have for our roadmap
for the next couple of years is to be much more proactive
about seeking out, and finding, and promoting good
solutions to really what is a really difficult problem. And of course, this
meeting is very much one part of that activity. But you know research assessment
is something that none of us can avoid. We live in a world
of finite resources. There is finite money. There are finite
positions in university. And so we are going to have to
look at measures of success. And so we are having to grapple
with rather deep questions about what it is that
we mean by success. And I quite like the operational
example of Victor Frankl, although it does pose some
philosophical difficulties. Don’t aim at success, he writes. For success, like happiness,
cannot be pursued. It must ensue. And it only does so
as the unintended side effect of one’s dedication to
a cause greater than oneself. I hope, I think I
believe that that’s a statement that
probably resonates with everybody in this room. But if you are the president
of a prestigious university, you’re probably
asking yourself how the hell do I make that
work for my university? How is that going to help me
be successful as a university? And so operationally,
it’s difficult. But I think it also taps
into the high aspirations that I think bring many
people, those signing up for PhDs and post-docs,
into a career in research in the first place. They want to make the
world a better place. They want to satisfy
their curiosity and transform the world with
the insights and understanding that they bring. And that brings us then
to another problem of, how do we balance the
curiosity that people bring to research with the
aspirations that they have of changing the world? And there are
different philosophies about how best to
manage science. And this is a really
excellent article, I think somewhat
controversial, very deliberately polemic
from Dan Sarewitz, who writes very well in this area. It made me rather angry
when I first read it. But I– and I was one
of the various people that responded to it. And there’s a really
good discussion that was published in the
New Atlantis following the original proposal. But Sarewitz is actually
very good at diagnosing the problem, which I think
we all know too well. Advancing according to its
own logic, much of science has lost sight of
the better world that it was supposed
to help create. And I think we recognize that
our mechanisms of research evaluation grind
people into the ground. And we end up chasing
citations, impact factors, publications in journals. And the irony here is
the stunted imagination of mainstream science
is a consequence of the very autonomy that
scientists’ insist is key to their success. Now I’m not suggesting
that we throw out autonomy. Although, actually, Sarewitz
takes a rather extreme view, is very much that science has
to be managed if it is going to deliver that better world. And that scientists, in
the free play of intellect, have, to some degree,
got a little bit lost in the chase for
the academic prestige that we know that
drives the system. Now I think he goes
too far in that. And I think we have to find
a balance between discovery science as HHMI wants, and
the solutions to the problems that as a world, as a
species, we know we face. His vision is quite
hopeful in a way. And there’s a lot there
that I would subscribe to. So the most valuable science
will be closely linked to the people and places
whose urgent problems need to be solved. They’ll cultivate strong
lines of accountability to those whose
solution’s important. They will, crucially,
incentivize scientists to care about the problems
more than the production of knowledge. That’s a somewhat
controversial statement, one that we can perhaps debate
a bit more in the coming days. But I think there’s a
worthwhile point in there. The links, agendas, to the
quest for improved solutions, often technological ones,
and so Sarewitz’s thesis is that technology
keeps science honest. Because if your science doesn’t
produce technology that works, then there’s something
wrong with it. And the science they produce
will be of higher quality, because it will have to be. And I think we have to think
through when we are evaluation, and this comes to
the point that was made towards the end of the
Q&A session, about linking. How do we balance the
importance of a really new fundamental discovery in
cellular molecular biology, for example, and
somebody who comes up with a treatment
for a disease that is neglected
because, to be frank, only poor people suffer from it? And so there are values
here that we really need to tap into and think about. So we have to think
about our values. And this came up actually
in several of the keynotes. And I think as a community,
we are very values driven. Research in academia
is not really a career. It’s more of a vocation. And yet, you know, we have
this mechanized system of research evaluation that
we are going to discuss. I don’t pretend for a minute
that in the next two or three days we are going to
solve all those issues. But I very much
hope that our focus will be on trying to find
practical steps forward. I very much believe that those
steps are there to be found. Some of them are
already being developed among universities and
funders, for example, and around the world. But I think there
are questions that we would want to keep at the
forefront of our minds as we go to our deliberations. So how best to
work with academics to articulate our shared values
and co-create feasible systems, I think that process
of co-creation. I think Erin and Paula have
both talked about the need to talk to the participants,
not just researchers, but also provosts and
presidents of universities. So that they really know
and understand the problem, and that we know and
understand one another. I think we have to really listen
to the concerns of academics. Many of them have
high aspirations. But they worry about
their career advancement. And they know that there is a
misalignment between the values and the aspirations
that brought them in, and the things that they
need to do in order to keep within the academic system. And similarly, I think,
as Paula mentioned, provosts don’t really
understand the negative impacts of bibliometrics. And so there are
lots of conversations that still need to happen in
order to equip and acquaint the academic community
with many problems that all the people in this room
are extremely familiar with. But many of our colleagues
back home see it– see it less clearly, I think. And it’s too easy for us,
perhaps, to be in a bubble sometimes and not recognize
the need that we have to– or the issues there
that we have to address. And this, and again, we
talk about academic freedom. But I also like to talk about
academic responsibility. And I think academics,
not in general, but there is a tendency to focus
just on the academic prestige side of things and to forget
about our duties to society. And I think we have to be very
cognizant of those duties, when many of us are in
fact publicly funded. And if we are to speak and
address the problems of society as surrogates would have us
do, then are we as a community, as an Academy, asking
the right questions, in order to address the
problems of the world? And that raises
the question then, are we actually representative
of the societies who fund us and who we purport to represent? And I think when you look at
the representation of women in the Academy, or
people of color, or other
underrepresented groups, then you’d have to
say that actually we do not have yet a healthy
ecosystem within academia. And so we have to
think much more about who outside the
Academy needs to get in, and who else in
the various publics do we need to be engaging
in order to think through our research agenda. And we have to try and
incentivize those things. Because ultimately, those are
the things that we want to do. And then there’s a first mover
problem, which is, you know, everybody knows the problem. But if even the Harvards
of this world, or the HHMIs of this world are
struggling to do it because they’re worried about
losing out in some competition, then perhaps there are
alliances that we can build. And I would very much like to
see some of those universities that perform well in the
awful, awful things that are league tables actually
getting together as a group and perhaps deciding to
secede from league tables and deciding as a
community of institutions that actually they are going
to cling to their values and use their values to
forge their path to success. So I wish us good cheer. And I wish us great courage for
the deliberations the next day. So let me leave
you this quotation, I wish I’d had the courage to
live a life true to myself, not the life of
others expected to me. It’s attributed
to no one person. But that is the
number one regret of people who are
on their deathbeds. So let us seize the day. You’ll notice it’s the top
five regret of the dying. It’s not a regret of the living. OK. And that’s because
the living forget about their values
and the things that they think are
ultimately important, because we get
distracted by the system. And so let us have the
courage and let us not have this as a regret
on our deathbeds. And with that, I
would like to close this evening’s proceedings. And thank all the pres– thank all the
speakers once again. And thank you for your
enthusiastic participation. Thank you. [APPLAUSE]