Public Library of Science - Technology

Details on Outage and Recovery of PLoS Journal Websites

Richard Cave — Wed, 20 Jan 2010 10:52:39 -0800

UnitedLayer, our collocation facility for the production servers, experienced an outage yesterday. From UnitedLayer: "A series of power brownouts occurred today at 2:56 PM PST due to PG&E instability related to the recent storms. Our 300KVA UPS system is not working as designed, the temporary repairs from last week did not hold. We anticipate a faulty motherboard."

A number of our servers (all powered by the 300KVA UPS) lost power at that time. Our large disk array (2TB of storage) that is the file server for both Fedora and Mulgara had a boot failure and refused to power up. Russ went to the colo and restarted the disk array which went into an automatic rebuild of the disks. This took about three hours to complete. Russ then started a program that checks for disk consistency and repairs any problems in the drives. This program was still running at 8pm – any recovery would have to wait until the program ended (many more hours). We made a decision to stop the program, format the drives and restore Fedora and Mulgara from a previous backup to speed up recovery.

We estimated that it would take ~3.5 hours to restore Fedora from backup. It took ~5.5 hours. Once complete, Russ brought the systems back online at ~2:50AM PST. Big thanks to Russ for babysitting the file server the whole day/night and for bringing up the system after the backups completed.

This is the first time that we had a major hardware malfunction to the large disk array and the first time we had to restore from a backup. While the disaster recovery plan worked, it took much longer than expected due to the size of the Fedora storage. We will look into solutions that enable a quicker disaster recovery. We are also meeting with UnitedLayer to discuss mitigation options.

Trackback URL for this post:

http://www.plos.org/cms/trackback/509

PLoS Journals Outage - January 19, 2010

Richard Cave — Tue, 19 Jan 2010 16:57:47 -0800

The PLoS journal websites are experiencing an outage due to a hardware malfunction after our colocation (United Layer) experienced a brown out due to lightning. We're working to resolve the problem and hope to have the journal websites back online soon.

Trackback URL for this post:

http://www.plos.org/cms/trackback/508

Update of the PLoS Journals to Ambra 0.9.5

Richard Cave — Wed, 16 Dec 2009 16:45:54 -0800

We will update the PLoS journals with the Ambra 0.9.5 release tonight. This release focused on development of the long-awaited PLoS Queue syndication module. Starting in January, PLoS will send article packages directly to PubMed Central at publication time. Article packages are currently sent to PubMed Central the same time as they arrive at PLoS so if any errors are found by the PLoS editorial or production teams, we need to send the fixed article packages back for revision. Sending these article packages at publication time will save a lot of time/effort if any errors are found. This also allows us to automatically send article packages, XML and PDFs to other external repositories in the future. The admin panel and publication workflow were updated for this new syndication service.

New features and updates:

Added a new syndication service to the Admin panel. This will allow administrators to export article files and metadata to external sources. PLoS is using this syndication service with an internal application built on Apache Camel to send published files to PubMed Central and other external repositories. This is not an out-of-the-box solution as only the service hooks are available in Ambra.
Added RDFa for the Dublin Core elements in the article HTML. Look for a blog posting on this feature soon!
Added a new action to support a Google Scholar XML feed for published articles.
Update to the article RHC to display articles cross-published in other journals.
Improvements to how XSL sheets are configured.
Upgraded the Dojo library to 1.3.2.

Bug fixes:

Promote a note to a formal correction and the annotation body was removed.
Templates from other journals were displayed if freemarker.properties template_update_delay was set.
Templates changes to fix a number of formatting errors encountered in IE6, IE7 and IE8.
Minor fixes to annotation displays, formal corrections, formatting and messaging.

For a complete list of 0.9.5 features and bug fixes, please see the Ambra 0.9.5 RC1 Tickets and the Ambra 0.9.5 RC2 Tickets.

Trackback URL for this post:

http://www.plos.org/cms/trackback/504

Network Outage at UnitedLayer

Richard Cave — Tue, 13 Oct 2009 13:54:38 -0700

It seems that the storm slamming the Bay Area also affected the co-location facility UnitedLayer. A power "glitch" in their building brought down one of their big UPS units. This caused power outage to all of their network equipment. When their routers restarted, they were not serving external traffic. And their backup routers also failed.

We were notified of the PLoS website outages at 7:22am. We could not connect to any PLoS servers which meant a network problem occurred at the co-location facility UnitedLayer. A phone call confirmed the worst and we set about trying to get backup environment running on Amazon EC2 ("the cloud"). We redirected PLoS ONE to an instance that was already running but the site was too slow to be useful. We redirected website traffic to the everyONE blog. UnitedLayer fixed their network issues at 10:30am. We switched the redirects back and the journals websites are back online.

Trackback URL for this post:

http://www.plos.org/cms/trackback/489

Update to the PLoS Journals

Richard Cave — Wed, 16 Sep 2009 11:17:17 -0700

Last night (September 15), we updated the PLoS Journals to Ambra 0.9.4. This release culminates a many-sprint development effort and huge data migration to provide per-article usage statistics. The article usage statistics join the other article usage data (citations, bookmarks, blog posts, etc.) to allow users new ways to evaluate the value of articles. Mark Patterson has posted a blog entry about the Article-level Metrics at PLoS. More information about our article-level metrics program can be found in our PLoS FAQ and the Article-level Metrics website.

To get the article usage statistics, we ran Apache log files from the last four years through a massive data migration pipeline to provide per-article usage data for the number of HTML page views, PDF downloads and XML downloads. For this data, we conformed to the COUNTER 3 standards (industry standard guidelines used to report the usage of online journals to subscribing libraries). But we found that COUNTER's list of robots was extremely limited, so we exceeded the COUNTER standards by excluding even more robots from the usage data. For detailed information about the usage data, see the Usage Data Help section of PLoS ONE.

Every article now has a new tab called Metrics which displayed a graph of the cumulative usage data. You can see a great example of the article usage data on the Ten Simple Rules for Getting Published article. Since we have all this data, we also created journal summary usage data showing the average lifetime usage per PLoS journal. And we provide a summary Excel file containing the full data set for every PLoS article up to July 31, 2009.

But that's not all! Other features that have been implemented include:

We now display the citations to articles by CrossRef. You can see an example of the citations to this article as recorded by CrossRef.
Users can now search across multiple journals.
All administrators to override annotation citation information. Correction citations used to be dynamically generated from the article information. If this article information was part of the correction (e.g. title, author name), then the error is propagated in the Correction citation. Now, admins can update the citation information for any annotation to display the correct citation.
A number of outstanding bugs were fixed

For a complete list of features implemented, see the release notes.

During the maintenance window last night, we also restarted all of the production servers. We run all of our servers on Linux CentOS so we don't restart the servers very often. Most of the servers had been running non-stop for 450 days but one of the web servers had been up for 890 days! Unfortunately, one of the production servers rebooted with a bad drive and caused the maintenance window to be a bit longer than expected. We're off to the colo today to fix the drive but this won't affect the PLoS web sites while we replace the drive.

Congrats to everybody involved in this enormous effort to provide the article usage statistics!

Trackback URL for this post:

http://www.plos.org/cms/trackback/486

PLoS Journals – measuring impact where it matters

Mark Patterson — Mon, 13 Jul 2009 05:22:00 -0700

In 2009, in this online world, how do most scientists and medics find the articles they need to read? The answer for the content published by PLoS (and no doubt by many other publishers) is via one of the now ubiquitous search engines, be it one that only searches the scientific literature, or more likely, one that searches the entire web. Given that readers tend to navigate directly to the articles that are relevant to them, regardless of the journal they were published in, why then do researchers and their paymasters remain wedded to assessing individual articles by using a metric (the impact factor) that attempts to measure the average citations to a whole journal? We’d argue that it’s primarily because there has been no strong alternative. But now alternatives are beginning to emerge.

A few months ago, PLoS initiated a program to provide a series of metrics on the individual articles published in all the PLoS Journals. You can see some examples here, here, here and here. There are two complementary benefits to the new approach.

First, we are focusing on articles rather than journals. The dominant paradigm for judging the worth of an article is to rely on the name and the impact factor of the journal in which the work is published. But it’s well known that there is a strong skew in the distribution of citations within a journal – typically, around 80% of the citations accrue to 20% of the articles. So the impact factor is a very poor predictor of how many citations any individual article will obtain, and in any case, journal editors and peer reviewers don’t always make the right decision. Indicators at the article level circumvent these limitations, allowing articles to be judged on their own scientific merits.

Second, we are not confining article-level metrics to a single indicator. As summarized by Michael Jensen, and discussed by many others including recently over at the Scholarly Kitchen, there’s a lot more to scientific impact than citations in the selection of journals covered by the Web of Science – the proprietary source of data that provides the impact factor calculation. Citations can be counted more broadly, along with web usage, blog and media coverage, social bookmarks, expert/community comments and ratings, and so on. Our own efforts are so far confined to citations (as measured by Scopus and PubMed Central), social bookmarks (as made by users of Connotea and CiteULike), and blog coverage (as recorded by Bloglines, Postgenomic and Nature Blogs), and these metrics will be improved and expanded over the coming months. The good news is that many of these indicators can be collated automatically, using openly available web tools that constantly update information on the article itself.

The presentation of a comprehensive array of this data is an enticing prospect. When an article has been published, we have tended to regard that as the end of the story (barring corrections or the occasional retraction). But if, as frequently happens, a very good article has been published in a specialist journal after being rejected from a highly selective one, it would be great to indicate to a user that this article is actually looking pretty significant, and show how its influence develops over the months and years.

Rather than basing judgments on the importance of research on the opinions of two or three reviewers and editors, article-level metrics will attempt to capture the actions and opinions of entire communities of readers to give a rich and sophisticated picture of research impact that will be helpful to authors and readers alike. Readers may then frame that picture in the context of their particular field and their own work.

To realize the vision for article-level metrics there are still some significant hurdles to clear: it won’t be enough simply to provide indicators without some context or guidance on how to interpret them; some indicators (particularly citations) take months to build up limiting their value as early indicators of impact; and standards will need to be developed so that the indicators are reliable and as free as possible from gaming and manipulation.

A clear editorial selection process will always have a place before publication in a scholarly journal. But a reduction in the reliance on the impact factor for so many aspects of research assessment could be massively liberating. PLoS Medicine, to cite an example close to home, has recently restated its mission – focusing on the diseases and risk factors that have the most profound impacts on global health. By carefully selecting articles that are likely to have the biggest influence on global health and using innovative and diverse approaches to assess and indicate that influence, PLoS Medicine will be a greater force, regardless of how many citations an average article accrues

Looking towards other modes of publishing, PLoS ONE is predicated on the notion that judgements about impact and relevance can be left almost entirely to the period after publication. By peer-reviewing submissions purely for scientific rigour, ethical conduct and proper reporting before publication, articles can be assessed and published rapidly. Once articles have joined the published literature, the impact and relevance of the article can then be determined on the basis of the activity of the research community as a whole. Article-level metrics and indicators, along with other post-publication features are part and parcel of the PLoS ONE approach, and could help readers to filter and sort literature after it is published. Ultimately, the aim of adding value to articles after publication is to improve the whole process of scientific communication and accelerate research progress itself. You can read more about article-level metrics in the context of PLoS ONE, and a talk is also available online from Pete Binfield (Managing Editor of PLoS ONE).

Article-level metrics and indicators will become powerful additions to the tools for the assessment and filtering of research outputs, and we look forward to working with the research community, publishers, funders and institutions to develop and hone these ideas. As for the impact factor, the 2008 numbers were released last month. But rather than updating the PLoS Journal sites with the new numbers, we’ve decided to stop promoting journal impact factors on our sites all together. It’s time to move on, and focus efforts on more sophisticated, flexible and meaningful measures.

Trackback URL for this post:

http://www.plos.org/cms/trackback/478

Inaugural Meeting of the Concept Web Alliance

Richard Cave — Mon, 18 May 2009 10:54:53 -0700

On May 7 - 8th, I attended the inaugural meeting of the Concept Web Alliance. CWA wants to enable interoperability between large triple stores like the Large Knowledge Collider (LarKC) and provide an Open Access mechanism for accessing the triple stores. This is great for the projects in life sciences as the semantic triple stores are becoming the de facto way to store data for gene expression and sequencing, biobanks, etc.

The CWA mission statement is:

”To enable an open collaborative environment to jointly address the challenges associated with high volume scholarly and professional data production, storage, interoperability and analyses for knowledge discovery.”

You can read the entire CWA declaration.

There were a number of representatives from the STM publishing world (Abel Packer from Bireme is a founding CWA member, Nature, Springer, SEED, The Scientist, Thomson Reuters) and I had some good conversations about the vision of the CWA in relation to STM publishers. All agree that the CWA is a much needed initiative but there are questions on how it can feed back into a revenue model. Most STM publishers don’t have triple stores that can be offered to the CWA endeavor. They publish the final result - research articles based on the triple stores.

But I see a few ways that publishers can benefit from working with the CWA:

1. The CWA can provide tools that link the data stores directly to the content of the research article. Search, data mining, and visualization tools can be created for publishers which would give their users new ways of interacting with the research article. Users can find the research articles that they really want and can dig into the underlying data even if the data is a massive data store. For the publisher, this can increase revenue by bringing more traffic, focused advertising campaigns, etc. CWA can provide these tools for a fee which would be used to sustain and further the CWA mission.

2. The CWA can provide tools to automate semantic encoding of the research article. As an example, David Shotten has shown how this can be used by publishers by encoding a PLoS NTD article - see Adventures in Semantic Publishing: Exemplar Semantic Enhancements of a Research Article and has another paper titled ““Semantic Publishing: the coming revolution in scientific journal publishing” - preprint available here. Knewco also has some great technology in this space – check out their Concept Web.

3. Publishers can provide Open Data back to the CWA. Publishers could provide access to content tagged with RDFa for easier auto-machine discovery, allow access to the data from the supplemental information in the research article or provide triples generated from the content of the research article itself. PLoS is a bit ahead of the publishing curve as all of the PLoS journals run on the Ambra/Topaz platform which stores the content of the research articles as triples. We’re looking at ways to provide access to a subset of these triples (we would need to remove user information) through a SPARQL endpoint or other means of access. This would allow for direct access to the triples that could then be given back to the CWA.

What other ways can publishers interact with the CWA? The CWA wants to know how your organization can participate.

Trackback URL for this post:

http://www.plos.org/cms/trackback/469

PLoS Biology Migration to Ambra/Topaz

Richard Cave — Wed, 13 May 2009 12:38:35 -0700

Yesterday, we migrated PLoS Biology to the Ambra/Topaz platform. This completed a two year, 5 journal, ~9000 article migration involving many of the PLoS staff. Now all of the PLoS journals have the same feature set including notes, comments, ratings, article impact metrics, etc. Migrating all of the PLoS journals to a single platform is a major milestone for PLoS and will allow us to finally create cross-journal features such as cross-journal search.

We released a snapshot version of Ambra to production just before the PLoS Biology migration that fixed a number of bugs that were uncovered with the PLoS Medicine migration in March. These bug fixes will likely be unnoticed by most users.

We also standardized our environments for performance testing of the Ambra/Topaz platform in Amazon EC2. The development team created scripts to automate the launch of the Ambra/Topaz platform in the Amazon “cloud” with a snapshot of our production data. We will continue working on these scripts so that others to easily launch Ambra/Topaz instances and test the platform (stay tuned for more info).

Liz has provided a bit more information on migration at the everyONE blog.

Trackback URL for this post:

http://www.plos.org/cms/trackback/468

PLoS Journal Outage on April 9

Richard Cave — Thu, 9 Apr 2009 14:55:28 -0700

A number of factors contributed to the long outage today. The outage was caused by the sabotaged fiber-optic cable lines San Jose. This affected the network traffic going to United Layer, our co-location facility. United Layer is supposed to have a redundant network line for failover in case something like this happens. I don’t know the details, but this redundant network line wasn’t working. Their engineers finally rerouted their customer’s traffic around the San Jose disruption at 1:43pm PST.

During the outage, we were able to redirect journal traffic to the everyONE Blog which Liz updated throughout the morning. We also launched an Amazon EC2 instance and were (literally) minutes away from having the sites running on EC2 albeit with a snapshot of production data from March 17.

United Layer will be held accountable for their part in the outage. We’ll also look to improve our disaster recovery plans to try and limit the downtime caused by future "catastrophic" outages.

A comment about the outage today left at sfgate.com: "The more complicated they make the plumbing, the easier it is to plug up the pipes." - Lt. Com. Montgomery Scott

Trackback URL for this post:

http://www.plos.org/cms/trackback/461

PLoS Journals upgrade to Topaz 0.9.2

Richard Cave — Thu, 26 Mar 2009 22:00:16 -0700

Tonight, we upgraded the PLoS journal websites to Topaz 0.9.2. This release is chock full of user interface changes and enhancements. We've completely redesigned the article page to accommodate new features and give a better visual experience to the user. Since this is a significant design change for the article layout, we'd like to hear from our users. Email us or reply to this blog post and let us know what you think about the changes.

The article page now has three tabs:

Article: Much of the content in the right hand column has been moved into the other tabs to make way for new features. Links to the appropriate issue or collection appear in the right hand column of the article page. In PLoS ONE related subject categories have been added to the right hand column to allow easy access to other related articles. We've also designed the right hand column to allow for some new feature growth in the future (e.g. user tags).
Related Content: Data from external sources is provided on the this tab. Sources include the number of citations from PubMed Central and Scopus; the number of bookmarks from CiteULike and Connotea; and the number of blog posts linking to the article from Postgenomic, Nature Blogs and Bloglines. More sources will be added in the future.
Comments: All of the comments, minor corrections and formal corrections are easily viewed in one location.

Many other features were added to this release:

Competing interest statements were added to all notes, comments and ratings.
The creation of retraction annotation types.
The Most Recently Published homepage block can be configured to display a set number of articles, articles from within a published date range and an article white list (e.g. display only research articles).
Articles can be ordered within an issue and in the table of contents.
Formal corrections and retractions are now displayed in the table of contents next to the appropriate article.
Support for NLM DTD 2.3.
The administration portal was overhauled to provide a better workflow for creating volumes and issues.
The administration portal was updated to allow manual ordering of articles within an issue.

This is the first release that the PLoS development team has used SCRUM for software development. We're believers! We have a lot of lessons to learn from our first sprint, but the overall consensus is that SCRUM allowed us to quickly (in just six weeks!) developer a healthy amount of new features for the Ambra platform. As we continue to improve on the process, we'll be able to push out new features in rapid succession. The hopes are for a new release every four weeks.

Trackback URL for this post:

http://www.plos.org/cms/trackback/459