Why a NISO effort to standardise AltMetrics?

June 22, 2013

As has now been widely reported, NISO have a $200K grant from the Alfred P Sloan Foundation to develop standards for AltMetrics.

Why?

If there’s one consistent lesson from standardisation processes, it’s that standards which codify existing practice do well, while those that try to invent new practice in the form of a standard do badly. The various new facilities introduced into C++ by well-meaning standards people are a classic example of non-standard standards that no-one uses.

So I fear that trying to build standards around AltMetrics may well be premature. The thing that would be worth doing is codifying a standard XML format in which to express AltMetrics. But that’s something for a few practitioners to do in an couple of days. not something to spend $200k on.

I’ve been involved with enough standards to have a pretty good idea what goes into them. I’ve also been involved with drafting a fair few informal specifications that are used in the same way standards are. At this point, the latter (much, much more lightweight) process seems far more appropriate than the full lumbering machinery of a formal standards committee.

The bottom line is that there is nothing to standardise yet. We need several years of actual experience before we’re at the point where formal standardisation is anything more than an expensive waste of valuable people’s time.

Finally: you can just bet that the working groups will be dominated by the kinds of corporations that (A) can afford to fund staff to invest significant time in such efforts, and (B) have no vested interest in any kind of change. So what I see happening here is a committee full of people from IEEE, Elsevier, Emerald and Springer, achieving nothing or actively impeding progress — while a few valiant people who are actually doing altmetrics are effectively distracted from getting on with useful work by banging their heads against the standards-committee wall instead.

Much better to just do the work — for people actually involved in AltMetrics to pool their own experience and insight outside of any controlling formal arrangement — and put together whatever specification documents they find useful, free of the retarding influence of The Usual Suspects.

Once that’s been done, then there will be something to standardise.

 

Advertisements

7 Responses to “Why a NISO effort to standardise AltMetrics?”


  1. Mike, Your post brings forth a number of comments which are worthy of some considered thought. I’ll take the opportunity here to respond to a few of them.

    One of the most frequently referenced comments regarding the effort to standardize altmetrics at this point is that it is “too early” in the adoption curve of alternative metrics to standardize anything. The field of alternative metrics has been growing at a rapid pace for a few years now with a variety of established players now providing services in this space. However, each provider is gathering different data from different sources and calculating metrics in their own unique way. If the community is going to adopt these new metrics, administrators, granting organizations and others who rely on performance assessment need accepted, trustworthy and verifiable information. At this point, there are no community practices for even counting the most basic of elements to these equations. Even the notion of counting downloads is not a simple process, as anyone involved in Project COUNTER since its onset in 2001 will attest.

    While that is true is some regards that innovation is still taking place, realistically this project is scheduled to take about two and half years to complete (with phase 1 lasting 10+/- months and phase two lasting about 22+/- months. If alternative metrics hasn’t coalesced around some core infrastructure, definitions and collections methods by the point this project is completed, the long-term adoption of altmetrics will be significantly compromised, because people will have tested them across multiple vendors and be unable to verify them.

    In addition, we don’t see this project as inhibiting innovation, simply providing a community framework for comparing base-level apples to apples. Additional development will grow and continue, which we don’t expect to hold up, just like the existence of COUNTER didn’t stop the altmetrics community from advancing their own work.

    While your experience at IEEE or other standards bodies might be informative of your expectations, this is generally not the case within NISO. First of all, NISO does not require membership to participate in a working group, although a majority of member representatives is required on the group. Also, in this age, more than 90% of the activities of working groups is conducted through teleconferences, web documents and webinars. The only constraint on participation by anyone in the community is the time and interest to participate. If one could invest in structures to advance alternative metrics in a broad international effort, why not engage in a community consensus activity versus one’s own independent activity?

    Finally, it is not yet clear what the outcome of this initiative will be, whether they be a formal standards, or community best practices or commonly understood definitions — identifying what those needs are is Phase 1 of this two stage process. The first half of this project will be an effort to bring together very diverse stakeholders including a far broader set of stakeholders than publishers (i.e., administrators, assessment specialists, systems developers, and grant funding agencies, et al), to explore community needs and expectations, then identify a limited set of core community elements necessary for verifiable and trusted altmetrics. When the working group organizes to work toward the identified goals, every effort will be undertaken to engage a diverse set of stakeholders — not just publishers, who only represent 40% of NISO’s membership. Given the importance and interest among the variety of players in this initiative, I will be astounded if the only people to step forward to engage are large commercial publishers.

    I would be happy to discuss this further with you, or anyone else, to be clear about what NISO is actually trying to do and how we are going to go about it. Feel free to email me or send me a note on twitter @TAC_NISO

  2. Mike Taylor Says:

    Thanks, Todd, for this thoughtful, detailed comment.

    Of course you’re right that there’s a need for us all to mean the same thing when we say that an article has been viewed 4,665 times.

    The problem is, it’s very far from clear that at this stage, that’s best established by a formal standards process. Such processes are inevitably dominated by the better resourced organisations — it would be difficult to avoid this even if you were actively trying to — and that is not something that this space needs. In the field of academic publishing, many of the larger organisations have repeatedly shown themselves to be bad players, and to be perfectly frank they are doing much better under old metrics than they could expect to do under new ones (e.g. the very fact that Impact Factor scores their established journals at all while newer journals are simply not listed). So I’m struggling to see how we can trust those organisations to display good judgement in their inevitable role reading a standardisation effort.

    What I see happening here is the active players saying that they’re doing A, B and E — pretty similar things that need to converge somewhere around D — but the standardisation process, dominated by those who have least to offer, ends up recommending X instead, leaving the active players looking they’re the ones marginalised by being far away from the standard. That’s not a pretty prospect.


  3. Every community effort is, by definition, pursued by those who have the resources (skill, energy, time, money, etc.) to participate. In this case, it would be wrong to presume that the only institutions interested in this effort are publishers who aim to skew the outcome to their benefit. A great deal of interest has been expressed by the institutions who are the end-users of these statistics, including those involved in appointments, promotion and tenure, and grant funding decisions. Each of these communities definitely have the resources to participate and shape the end result to suit their needs. Furthermore, getting the buy-in and the trust of those communities who will rely on altmetrics (if they can be proven valuable) will be critical to their eventual adoption. To this point, development efforts haven’t at all included these diverse stakeholders, which is one element of what we’re seeking to do.

    In addition, noting your comment about why this costs $200K, a large portion of the funding is to support engaging those under-resourced community participants in the process through travel support to the two in-person planning meetings we will hold in Phase 1.

    I don’t disagree that there is potential that this effort could be hijacked by communities on either side to suit their social-political-economic interests. The benefit of pursuing an open consensus process is that no one interest can dominate the process, unless the other interests choose not to engage. So this is the strongest case I can make for engagement: If you represent a community that is concerned about altmetrics being used in opposition to your interests, please reach out to participate. We will do as much as we can to get your voice heard.

  4. Mike Taylor Says:

    Every community effort is, by definition, pursued by those who have the resources (skill, energy, time, money, etc.) to participate. In this case, it would be wrong to presume that the only institutions interested in this effort are publishers who aim to skew the outcome to their benefit.

    Of course no-one thinks the big publishers are the only ones interested in this. But they are ones with — by far — the most resources to plough into it. We can be sure of this: that the outcome of any committee-based effort will more fully favour the big corporations that have the resources to dominate it than the small organisations that are busy trying to establish themselves. As since we’ve seen over and over again that the big publishers want the opposite to what everyone else wants, that’s not going to be a happy ending.

    Still, against that depressing backdrop, it is encouraging to be told that NISO is actively trying to mitigate the inevitable regressive tendency of such a process.

    So this is the strongest case I can make for engagement: If you represent a community that is concerned about altmetrics being used in opposition to your interests, please reach out to participate.

    Thanks for the invitation. I’ll do that in my spare time, shall I?

  5. Mike Taylor Says:

    … what I’ve not said, but should have led with, is this: thank you for engaging with this blog. However curmudgeonly my responses might appear, the truth is that I greatly appreciate it (and I’m sure the readers do, too.)


  6. Mike, your thoughtful comments, suggestions, and warnings are well-meaning and I do appreciate them. It is this vigilant criticism that when aknowledged by the organization along with whatever group is organized should lead to better outcomes overall. Hopefully, it will motivate others to contribute. Whatever time one can spare would be most welcome and appreciated, be it formally through serving on the group, participating in our in-person or virtual meetings, or simply responding (curmudgeonly or not!) to drafts and calls for public comment.


  7. […] Taylor has written a good post about it, expressing some skepticism. David Smith has a Google+ post up asking what people think. […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: