Oxford. Source: Wikimedia Commons

Scholarly metrics can be used in discovery tools, as business intelligence for funders, research organizations or publishers, and for research assessment. For all these scenarios – and in particular for research assessment – it is important to not only collect metrics for a particular journal publication, dataset or other research output, but to also link these metrics to the creators of that research output.  That is why unique identifiers for researchers, and ORCID in particular, are so important for scholarly metrics, and this is also reflected in the ORCID membership of organizations such as Thomson Reuters, Elsevier/Scopus, Altmetric or F1000Prime that provide metrics in a variety of ways.

DORA

A good starting point for any discussion on metrics for research assessment is the San Francisco Declaration on Research Assessment (DORA) that was published, together with a set of editorials in several journals, last week. The first three recommendations are a good starting point for the panel discussion:

1. Do not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions.
2. Be explicit about the criteria used in evaluating the scientific productivity of grant applicants and clearly highlight, especially for early-stage investigators, that the scientific content of a paper is much more important than publication metrics or the identity of the journal in which it was published.
3. For the purposes of research assessment, consider the value and impact of all research outputs (including datasets and software) in addition to research publications, and consider a broad range of impact measures including qualitative indicators of research impact, such as influence on policy and practice. 

Persistent Identifiers

Before we can collect any metrics, we need persistent identifiers for research outputs. Most journal articles now come with a DOI, but we should make it easier for smaller publishers to use DOIs, as cost unfortunately is still an issue.

Persistent identifiers for data are a much more complex issue, but here DataCite DOIs are probably the first choice, as DataCite focusses on attribution. There are many other use cases for identifiers for data (e.g. to identify temporary datasets in an ongoing experiment), and is of course possible to use several identifiers for the same dataset. Particular challenges with persistent identifiers for research data include different versions of a dataset, and aggregation of datasets (e.g. whether we want to cite the aggregate dataset, or a particular subset).

In addition to persistent identifiers for research outputs, we also need persistent identifiers for researchers. ORCID is obviously a good candidate, but it is clear that there are also other persistent identifiers for researchers, and that we need to link these identifiers, e.g. ORCID and ISNI.

Depending on the metrics we are interested in, we might also need persistent identifiers for institutions, for funding agencies and their grant IDs, and for resources such as particle accelerators or research vessels. Unfortunately much more work is needed in these areas.

Attribution

Attribution is then the next step, linking persistent identifiers for research outputs to their creators. The Amsterdam Manifesto on Data Citation Principles that came out of the Beyond the PDF 2 workshop in March are an excellent document, but are unfortunately missing the important step of linking persistent identifiers for data to the persistent identifiers of their creators.

One important issue related to attribution is the provenance of the claims. Has a researcher claimed authorship for a particular paper, is a data center linking creators to research data, or is a funder doing this? The ORCID registry is built around the concept of self-claims by authors, but will allow the other stakeholders to confirm these claims.

Metrics

Metrics for scholarly content fall into one of three categories:

• Citations
• Usage stats
• Altmetrics

Altmetrics is a mixed bag of many different things, from sharing on social media such as Twitter or Facebook to more scholarly activities such as Mendeley bookmarks or F1000Prime reviews. I therefore expect the altmetrics category to over time further evolve into 2-3 sub-categories.

We are all familiar with citation-based metrics for journal articles. We currently see the long-overdue shift from journal-based citation metrics to article-level metrics (see #1 from the DORA statement above for the reasoning), and as the technical lead for the PLOS Article-Level Metrics project I of course welcome this shift in focus. We also see a trend towards opening up reference lists that will make citation-based metrics much more accessible, and the JISC Open Citations project by David Shotton and others is an important driver in this, as is the Open Bibliographic Data project by OKFN. Until open bibliographic data become the norm, we have to deal with different citation counts from different sources. PLOS is collecting citations from Web of Science, Scopus, CrossRef and PubMed Central, and the citation counts are highly correlated overall (e.g. R2= 0.87 for CrossRef and Scopus citations for 2009 PLOS Biology papers), but for some papers differ substantially.

The data citation community has adopted the journal article citation model, and we are starting to see more citations to datasets. Even though data citations look similar to citations of journal articles, many essential tools and services still don’t properly handle datasets. The Web of Knowledge Data Citation Index is an important step in the right direction, as is the new DataCite import tool for ORCID. Something that we should pay closer attention to is the citation counts of the paper(s) associated with a dataset. Maybe the major scientific impact is in the data, but scientific practice still dictates to the cite the corresponding paper and not the dataset itself. The DataCite metadata can contain the persistent identifier of the corresponding journal article, thus making it possible to associate the citation count of the corresponding paper with the dataset. This approach is particularly important for datasets that are always part of a paper, as is the case for Dryad.

Another problem with data citation is that citation counts might not be the best way to reflect the scientific impact of a dataset. We are increasingly seeing usage stats for datasets, and DataCite for example has started in January to publish monthly stats for the most popular datasets by number DOI resolutions. The #1 dataset in March was the raw data to a figure in a F1000Research article, hosted on figshare.

Similar to citations we see a strong trend for usage stats to move from aggregate numbers for journals to article-level metrics. COUNTER has released a draft code of practice for their PIRUS (Publisher and Institutional Repository Usage Statistics) standard in February, and increasing numbers of publishers and repository infrastructure providers such as IRUS-UK and OA-Statistics are providing usage stats for individual articles.

One challenge with usage stats, in particular with Open Access content, is that an article or other research output might be available in more than one place, e.g. publisher (or data center), disciplinary repository and institutional repository. For PLOS articles we don’t know the aggregated usage stats from institutional repositories, but we know that 17% of HTML pageviews and 33% of PDF downloads happen not at the PLOS website, but at PubMed Central.

Altmetrics provide new challenges, but they are also a more recent development compared to usage stats and citations. Similar to usage stats they are fairly easy to game, and for some altmetrics sources (e.g. Twitter) standardization is still difficult. Altmetrics not necessarily measure impact, but sometimes rather reflect attention or self-promotion. We have just started to look into altmetrics beyond the numbers, as it makes a huge difference who is tweeting, bookmarking or discussing a paper or dataset. Altmetrics provide the opportunity to show the broader social impact (as Mike Taylor from Elsevier explains it) of research, e.g. changing clinical practice or policies.

Contributions

Related to attribution is contribution, i.e. what is the specific contribution by a researcher to a paper or other research output. An International Workshop on Contributorship and Scholarly Attribution was held together with the May 2012 ORCID Outreach Meeting to discuss this topic. Authorship position (e.g. first author, last author) is used in some metrics, but overall the contributor role is still poorly appreciated in most metrics. David Shotton has proposed a Scholarly Contributions and Roles Ontology (ScoRO), and is suggesting to split authorship credit in percentage points based on relative contributions, but I haven’t seen these numbers used in the context of metrics.

Conclusions

Persistent identifiers for people, attribution and metrics are closely interrelated and we have seen a lot of exciting developments in this area in the last two years. But with all the excitement we should never forget that a) there will never be a single metric that can be used for research assessment, and b) that scientific content will always be more important than any metric. I look forward to a great panel discussions on Thursday, and welcome any feedback via comments, Twitter or email.

There are many things I like about this new DataCite/ORCID integration tool:

• it makes it easier for researchers to get credit for their research outputs.
• it shows the value of persistent identifiers for data, publications and people, and linking them together
• it shows the Creative Commons licenses for DataCite content where this info is available, facilitating reuse of content
• it demonstrates the power of open source (thanks CrossRef!), open collaboration, standard REST APIs, and lightweight programming (Sinatra/Ruby) and deployment (Vagrant, Amazon EC2, Rackspace) tools
• it shows that we don’t need a single – often closed – system, but open services that build on top of each other using accepted community standards. Tools using the ORCID API can immediately reuse the new DataCite content, altmetrics provided by ImpactStory are a good example

I want to explore some of these ideas in the panel Attribution: Managing Provenance, Ethics, and Metrics at the combined ORCID/Dryad Meeting in Oxford next Thursday.

Markdown is a simple markup language for text, and is primarily used for HTML content on the web, but can also be converted to PDF, LaTeX and others. One challenge with markdown is that there are a number of slightly different “flavors” out there, from the original markdown to multimarkdown, github-flavored markdown and pandoc. Some of the advanced formatting of scientific documents – tables, citations, math – is still a challenge for markdown.

Will markdown become our next authoring format for scientific content? Will there be yet another flavor, scholarly markdown? How will markdown writing tools be different from LaTeX tools or Microsoft Word? If you care about any of these questions and are in or near San Francisco, join us on for all full day on June 8th. Free registration is open at http://mdsci13.eventbrite.com. We are collecting workshop ideas at https://github.com/karthikram/markdown_science/wiki/workshop, the Twitter hashtag is #mdsci13.

This event is organized by Stian Haklev and myself, with generous support by a 1K Challenge prize from Force11, and hosting provided by PLOS.

Article-Level Metrics is also doing something else: instead of tracking impact by year, it looks at usage, altmetrics and citations in real-time. There might have been technical reasons to do so 20 years ago, but there really is no longer any reason why scholarly impact should be tracked on a yearly basis in 2013. Unfortunately there is one big stumbling block:

The publication date of a scholarly article is often difficult or impossible to obtain. Publication year may be the only available information.

A good example is CrossRef. They provide a lot of interesting metadata about an article and make this information available in a very nice search interface. But they only require the publisher to provide the publication year, information about the publication month and day is optional. There are many other examples of journals and services that just can’t tell you when exactly an article was published. This might have made sense when periodicals were printed on paper, but doesn’t work for digital content.

DevOps is an important trend that brings software development and administration of IT infrastructure closer together. Agile software development, server virtualization, cloud infrastructure and software automation tools such as Chef, Puppet or CFEngine are an important pars of DevOps, but it is really the collaborative aspect of IT administrators working much closer with software developers what defines DevOps. The end result is often faster and more stable software releases, and that is what is users and customers care about.

This makes DevOps particularly relevant for all areas where innovation is important, and that of course includes tools and services for Open Science. We not only need infrastructure that facilitates software development (with services like Github, among many others), but we also have to streamline IT administration. The question is not whether you do your development in Java, Python, Ruby, PHP or Javascript, but how well you integrate your software development and IT administration. The shift towards web-based tools has centralized software installation and updates, but these web-based services are becoming increasingly complex and difficult to set up and administer. Running an institutional respository, research information system or a journal is a complex task. The software may be freely available as open source (e.g. Dspace, VIVO or Open Journal Systems), but the resources required to run such a service still make this a big investment.

Two solutions to this dilemma are to pay either a vendor for installation and maintenance, or to use the software as a service (SaaS) that is hosted somewhere else. Why these two options are popular, they may not always be the best choices because they mean that you are locked in to a particular vendor or service provider, and that you may give expertise and direct access to your data away. I believe that these are helpful approaches for auxillary services, but that ideally the core services of a library, publisher or other provider of scientific infrastructure should not be outsourced. Developing software for scientific infrastructure that you want organizations to install locally should therefore always include work on integration with IT infrastructure, and just providing manual installation instructions isn’t good enough anymore.

Article-Level Metrics (ALM) and the related altmetrics are becoming increasingly popular. The collection and display of this information is a complex process, as it requires the integration of information from several upstream APIs which may be temporarily unavailable, have changed their data format, or put up restrictions on how you can use the data. In turn this information has to be processed and aggregated, and then reliably be provided to downstream users. This kind of information gathering fits perfectly with a service provider model, and organizations such as Altmetric, ImpactStory and Plum Analytics. PLOS is collecting and displaying this information with its own tool. The simple reason is that PLOS started doing this several years before the services above became available, and none of them currently provide the same comprehensive set of information about citations, usage stats and altmetrics (although there are of course a lot of things they do better than the PLOS ALM application).

But there is also the question of whether Article-Level Metrics are a core service for every publisher and are best collected in-house. This not only makes it easier to collect information from some sources (e.g. usage stats or CrossRef citations), but also gives unrestricted access to the data in real-time. When I took over as technical lead for the PLOS Article-Level Metrics project last May, I therefore not only worked on improving the ALM application for PLOS, but we are also working hard on making it easier for other publishers to install and use the application. We want to provide an attractive alternative for organizations for which the service provider model is not the best option.

To that end I want to announce the latest feature which allows the automated installation of the PLOS ALM application on an Amazon Web Services (AWS) EC2 instance. This option is great not only for setting up an ALM production service, but because of the EC2 pricing model by hour (about $1 a day for a small EC2 instance) without setup costs is a great way to test-drive the application for a publisher, to analyze a particular set of papers from different publishers for a research project, or to set up a PLOS ALM server for a hackathon or workshop.

There are of course many options to automate software deployment on a production server, including the PaaS (platform as a service) providers Heroku, CloudFoundry and OpenShift, and the recently announced Amazon OpsWorks. I am a big fan of the Vagrant software development tool in combination with Chef for automation, and in March Vagrant added support for Amazon AWS. This makes deployment of the PLOS ALM application to AWS really simple:

1. Install Vagrant and the vagrant-aws plugin
2. Setup an Amazon Web Services Account
3. Check out the PLOS ALM source code from Github
4. run the command vagrant up –provider aws

Step #4 took 898 sec or about 15 min on my computer (see screenshot), and at the end I had a PLOS ALM server where I could access the admin dashboard via the web interface. If you are familiar with Amazon Web Services – you have to think about  the right size for the EC2 instance, an appropriate AMI, security groups, elastic IPs, and DNS service – then the whole process should be done in well under an hour. I will use this instance to load and analyze some articles from a publisher for a presentation next week. When I’m done, another command (vagrant destroy) will destroy this server and Amazon will stop billing me. During testing I have created and destroyed many servers, and the vagrant-aws video shows you how easy this process is.

At this stage the installation process is working (and has been working for a local Virtualbox install for many months), but needs testing and documentation. I therefore invite everyone interested in testing this out to contact me so that we can make this well-documented and working reliably.

I first met Victor and Jan from Mendeley in 2008 and did an interview with Victor in September 2008. We worked together in the organization of two Science Online London conferences (2009 and 2010, together with Nature.com and others), and my current job started with an entry for an API programming contest co-organized by PLOS and Mendeley, with the first lines of code written in the Mendeley offices during the Science Online London 2011 hackathon. I wish Mendeley all the best with their new parent.

What this acquisition signals to me is that commercial publishers are now moving into the software tools for scientists business at full speed. They have always done this, but with ReadCube by Digital Science (a Nature Publishing Group sister company) in 2011, the acquisition of Papers by Springer last year and now Mendeley, reference management now often means using a tool owned by a publisher – this market used to be dominated academic software such as Zotero and commercial software vendors such as Thomson Reuters (Endnote) or ProQuest (RefWorks).

For me this trend signals that publishers have realized that we are moving into an Open Access publishing model, which in contrast to subscription publishing is not about owning the content, but about providing valuable services around content that is free to read and reuse.

E-BIOMED

What is now PubMed Central started out as E-BIOMED in 1999 and initially was envisioned to include a repository for preprints. It is important to look back at what happended then, and why the preprint repository was dropped from what then became PubMed Central. Harold Varmus talks a bit about this in this interview from 2006.

Nature Precedings

The article talks about why biologists have not developed a culture of sharing preprints. It would be good to mention Nature Precedings, a preprint server for the life sciences started in 2007 that stopped taking new submissions in 2012. This blog post on RetractionWatch cites the announcement by Nature Publishing Group (which doesn’t explain why the service was shut down), and there are a good number of interesting comments.

SSRN

Preprints in other disciplines are mentioned in the text, in particular ArXiv, but also RePEc. I would also include SSRN (Social Science Research Network), which uses a different model, but is as important for the working paper and preprint culture in the social sciences as ArXiV is in physics/mathematics.

Google Scholar Metrics

In April 2012 Google launched Google Scholar Metrics, listing the top 100 publications (according to their h5-index) in several disciplines. Six out of the top 10 publications in physics/mathematics are ArXiV sections (arXiv Astrophysics (astro-ph) is #2), the IZA Discussion Papers are #1 in Social Sciences, and the NBER Working Papers are #1 in Economics, and arXiv Astrophysics (astro-ph) is #12 on the top 100 list for all disciplines (#1-5 are journals in biology and medicine: Nature, New England Journal of Medicine, Science, Lancet, Cell). All these metrics are a strong indicator that preprints can be highly cited.

Citation Advantage of Preprints

Jason makes a strong case for this vision, and takes his time to also discuss the concerns and challenges. He doesn’t have the space to discuss in more detail how we get to that future, and in particular what the role of researchers, publishers, libraries and funders be in that transition.

Jason’s vision will probably be overwhelming for many researchers, and might not directly address what is probably the biggest issue for most researchers: funding for grants and jobs is limited, and the processes we use to select for good science and good scientists are inefficient and often arbitrary. Most students entering graduate school will not be able to have a career in academia, and most academics will say that they spend far too much time with evaluations – of their own work and the work of others. It is unclear to me how we can get from the current system – where one misstep such as denied grant or submission to the wrong journal can mean the end of a career – to the system that Jason envisions. The current climate doesn’t really foster experimentation by researchers and I am interested to understand how researchers can take part in this process of change.

The vision of the decoupled journal is very threatening for some of the stakeholders of the current scholarly communication ecosystem, in particular publishers and libraries. Every journal publisher and library knows that it has to reinvent itself to survive the digital transformation, but a vision that is build around a new ecosystem of service providers needs to be clear how publishers and libraries can be part of the transformation process.

Lastly, I disagree with the notion that today’s publication silos will be replaced by a set of decentralized, interoperable services that are built on a core infrastructure of open data and evolving standards — like the Web itself. I would argue that both scholarly communication and the web in general have a tendency for centralization, and that scientific infrastructure needs to be interoperable first and decentralized second. Without a focus on interoperability the future of scholarly communication will not be open and in the hands of many, but will be a race to become one of the dominant players in this new ecosystem, and we might end up with not 1000s of libraries and publishers but just a handful of technology companies holding the keys to our scientific infrastructure.

Priem, J. (2013). Scholarship: Beyond the paper Nature, 495 (7442), 437-440 DOI: 10.1038/495437a

I like simple visualizations without too many labels or axes, and wanted to do a visualization inspired by sparklines ever since we discussed this idea in our altviz breakout group at the ALM workshop hackathon in November 2012 (kudos in particular to Juan Alperin, Karthik Ram and Carl Boettiger). In the chart below every column represents the numbers for a given month, with alternating colors for the years (the article was published November 2009).

CiteULike bookmarks, usage stats from PLOS website and blog posts for article Article-Level Metrics and the Evolution of Scientific Impact by month, available at http://dx.doi.org/10.1371/journal.pbio.1000242.

You can see a pattern that is probably typical for many articles independent of the absolute numbers: most pageviews and downloads happen in the weeks after publication, as does academic bookmarking and science blogging.

The second example shows a very different pattern. This is not only the most-downloaded PLOS article, but  the distribution of downloads over time is very different, with the number of monthly downloads actually higher the last two years (this article was published in August 2005). We also see a few spikes in the usage stats, probably indicating events that triggered usage. Academic bookmarking was most active from 2009 to 2011 and not right after publication, although that might also have to do with the relative popularity of CiteULike over time.

CiteULike bookmarks, usage stats from PLOS website and blog posts for article Why Most Published Research Findings Are False by month, available at http://dx.doi.org/10.1371/journal.pmed.0020124.

Citation data are unfortunately more difficult to get with exact publication dates (why is that so difficult?), but we can at least look at CrossRef numbers by year for the same article.

CiteULike bookmarks, usage stats from PLOS website and blog posts for article Why Most Published Research Findings Are False by year, available at http://dx.doi.org/10.1371/journal.pmed.0020124.

The citation numbers by year are still increasing (the last bar is for 2013), indicating that this article is still of general interest 8 years after publication. This would probably be unusual for a life sciences research article, but the article is an essay looking at common pitfalls in the statistical analysis of research data.