Article-Level Metrics Information
This page contains information about each of the article-level metrics that we track. Summary tables of 'average usage' are also available, as well as an FAQ page, a page containing a technical description of our usage data in particular; and a summary Excel file containing the full data set.
Background
At PLoS, we believe that research articles should primarily be judged on their individual merits, rather than on the basis of the journal in which they happen to be published. As a result, in March 2009, we started a program to provide "article-level metrics" on every article in all of our titles. Then, in September 2009, we enhanced the program to also include usage data. At that time we created a new tab on our articles labeled "Metrics", and all of the following article-level metrics can be found from within this tab.
Article-level metrics place relevant data on each article to help users determine the value of that article to them and to the scientific community in general. Importantly, they provide additional and regularly updated context to the article, which currently includes data on citations, online usage, social bookmarks, comments, notes, blog posts about the article, and ratings of the article. Each of these items is described further in the sections below (together with a brief discussion on the limitations of each metric) as well as in our general FAQ.
It is important to note that with the exception of online usage, these metrics tend to take time to accrue. Therefore, newly published articles will typically show lower levels of activity (for any given metric) for the initial weeks or months after publication than older articles.
There are a number of known issues with some of the metrics, and details on these issues are provided in the section below.
Finally, PLoS is committed to the open provision of these metrics, and we hope that by making such data available, researchers will investigate and analyze them in new and interesting ways. Therefore, we are making the entire dataset, for all article-level metrics that we are tracking, available as a summary Excel file. This file will be updated periodically, and the current version contains data accurate as of January 31st, 2010. Future releases of our platform may well automate this download and/or provide the data via an API interface.
Citation Information
We are providing citations to each article as measured by a number of third-party citation measuring services: Scopus; Web of Science; PubMed Central and CrossRef. The data are displayed as a single number for each service (representing the number of citations to that article, as recorded by each service) with a link to a landing page containing information relating to the citing articles.
The citation data reported for each service are different, as each service draws upon a different database of articles to analyze. In order to get the most complete picture of how many (and which) articles cite the article in question, you should consult all three lists and "de-duplicate" the results.
We appreciate that Google Scholar is a powerful tool with which to discover the citations to a specific article, and is also able to identify citations beyond the formal scholarly literature. At this time API services are not provided by Google Scholar (which would allow our automatic capture of citation numbers per article), but we have added a link that will run a search for article citations at Google Scholar.
Online Usage Data
We provide our articles in three different formats: the HTML page (suitable for onscreen viewing), a PDF file (which many people prefer when printing an article), and the original XML (the raw data used to generate the HTML and PDF files). We record the number of times that each format is accessed and we provide this information as a number of "article views" for each article. Article views (split into the three types of file format) are provided as an aggregate metric, as well as in a month-by-month breakdown in graphical format. Detailed, technical information relating to our usage data can be found at our Usage Help page.
Clearly, it is useful to have data on the amount of online usage that an article receives. Although we are not the first to provide usage data of this type, they have rarely been provided by journal publishers, and therefore such data should be interpreted with caution. In general, usage is dependent on the age of the article and its subject area. To assist in the interpretation of usage data, we have provided summary tables showing useful average figures. In addition, interested researchers can download our entire dataset as an Excel file (updated periodically).
We have been careful to report our online usage data according to certain industry standard definitions of usage (so-called COUNTER 3 compliant statistics, which primarily refer to the usage of an entire journal in the context of a subscribing library). Although we have calculated the data "in the spirit" of the COUNTER 3 requirements, COUNTER has not yet ratified a protocol for article-level usage, so we may need to adjust these figures in light of any future standards.
One example of the way in which usage data should be treated with caution is with respect to robot activity. COUNTER 3 requires that certain robots be excluded from any reporting, and we have complied with this. However, COUNTER 3 does not define an exhaustive list of all robots, and so we have actually excluded more robot activity than was required, hence exceeding the standards in that respect. As COUNTER releases new standards, which may include standards devoted to article-level usage or expanded numbers of robot services to exclude, we may therefore need to restate some of this data to adjust for these new definitions.
We have also provided detailed technical information about our online usage data.
Social Bookmarks
Just as members of the public use social bookmarking services such as Delicious to bookmark Web pages of interest to them, academic researchers use similar services to bookmark papers and collate references for use in their research. Two of the primary providers are CiteULike and Connotea, and we are providing data from both services to indicate how many times their users have bookmarked the article in question. As with citations, the data are shown as a number for each service, with a link to a landing page at that service. Once at the landing page, you can navigate through other articles bookmarked by the users in question, as well as see the subject tags that they have assigned to the article. At present, it appears that CiteULike is a more heavily used service than Connotea.
We believe that appropriate use of social bookmarking data will aid the discovery of related papers, as well as informing readers how "popular" an article is. In collaboration with Cameron Neylon, we have produced an informational video explaining the power of this metric as a research and discovery tool.
Comments and Notes
Our publishing platform allows users to leave Comments (about an entire article) or Notes (about specific parts of the article). Users may not be anonymous, their comments must adhere to our guidelines for commenting, commentators must declare competing interests (when they exist), and PLoS staff monitor all comments.
We are providing information on the number of Comment/Note threads that have been created, but please be aware that any given Comment/Note thread may contain multiple replies which are not (currently) counted separately by us in the Metrics tab (although they are identified in the summary Excel file that we are providing). Under the tab "Comments" you will find the full text of these Comments/Notes along with all replies. Please also be aware that users may leave text comments when making a 'Star' rating of an article - these comments can only be accessed by clicking on the Rating itself (although, again, we are indicating which articles have Comments of this type in our summary Excel file).
It should also be noted that not all comments about an article are made using the functionality on our own site. People may also choose to comment about an article in a blog post, in the news media, in other discussion forums, etc., and so we are attempting to locate these external commentaries, compile them, and present links to them on the paper. Linking to blog coverage is the first step in this process.
Blog Coverage
As mentioned above, many blog articles are written about articles published in PLoS journals. To identify and link to them from each article, we are making use of third parties who already aggregate blog articles from across the internet. These services are Researchblogging.org, Nature Blogs, and Bloglines. For each service (as with citations and social bookmarks), we provide a numerical indicator of how many blog articles they have identified along with a link to a landing page listing the actual blog posts in question.
As with citations, the blog activity reported for each service is different, as each service aggregates blogs in different ways. In order to get the most complete picture of how many (and which) blogs cite the article in question, you should consult all these lists, bearing in mind that there will be duplicate entries.
Linking to blog coverage is not yet comprehensive: we rely on the ability of third parties to find blog postings and match them to PLoS articles. In many cases, blog authors do not reference the article in a way that allows for automated aggregation, and the aggregating services we link to cover only a selection of all possible blogs. Therefore, there will potentially be many more blogs about an article than these aggregators are able to identify. In recognition of this fact, we also provide a link to a generic search at Google Blogs that may provide additional links to unlisted blogs. We are also planning to expand the number of aggregators with which we are working.
Finally, our platform supports "trackback" functionality (which provides another way for bloggers to link to an article, and for us to automatically show that link on our site). If you are a blogger, we encourage you to use trackbacks and to make sure you reference the article using the Digital Object Identifier (DOI).
Star Ratings
Our platform supports the ability of users to leave "Star" ratings on articles. As with Comments and Notes, users may not be anonymous. We provide information on the number of ratings as well as a breakdown of the ratings in each of our three categories, and the overall average rating. By clicking on the links, you can view the actual rating information as well as see which user made it (this page may also include text notes made by that user).
Known Issues with Article-Level Metrics
As alluded to above, there are limitations to the data that we are supplying. A list of the known limitations is as follows:
Robot activity: We have excluded a large list of robots from our online usage data - however, no robot list can ever be exhaustive and some level of robot usage will undoubtedly remain in the data.
Differences in usage data for article published prior to July 1st 2005: Due to problems with our early log files, for those articles published prior to July 2005 it was not possible to separate usage between HTML, PDF and XML view types. Therefore, the usage data reported for those articles, for all months prior to July 2005, is shown as an HTML view but actually represents a 'combined' figure made up of the 3 view types. This primarily affects articles published in PLoS Biology and PLoS Medicine.
Scopus Citations: We are aware that Scopus sometimes significantly undercounts the number of citations to a specific article. This is due to Scopus having two records in their database for many of our articles (and hence the citations are spread across the two records). We are working with Scopus to improve their database in this respect, so Scopus citation counts may increase in the future.
Web of Science® Times Cited count: This figure is supplied by Web of Science® - the count for an article is calculated across all years and all five citations indexes in the Web of Science® database and will reflect the sum total of citations. When individual users access Web of Science to verify these figures they may obtain results which are lower than the quoted Times Cited count. This is because they may not necessarily have full access to the complete suite of Web of Science® databases. To read more about how the Web of Science® counts citing articles, please visit their help file
Although we originally provided blog coverage data from Postgenomic, that service was closed down by Nature Publishing Group in 2010 and so that data source has been removed.
"Go-live" Dates for Different Functionalities: Commenting, Note making, Star rating and Trackback functionality was only made available on each journal at a certain point in time, and this date differs between journals. Therefore, articles published before this functionality was made live will typically show fewer comments/notes/ratings than articles published after this date. In addition, PLoS Biology, PLoS Medicine, PLoS Pathogens, PLoS Computational Biology, and PLoS Genetics were all migrated from a prior publishing platform (Allen Press) to the current platform (Ambra). When this migration occurred, it was not possible to migrate the "posting" dates of the comments that had been made up until that point. Therefore, all commenting data that was migrated shows a "posting date" of the date of migration (however, the original posting date can still be found by clicking into each comment).