In spring 2017, I briefly discussed the issues with scholarship impact factor in law as a response to a recommendation by a law professor to create a rankings methodology based on Google Scholar citation.

Now US News is trying to get in the game of creating a ranking of law faculty by scholarship impact factor using Hein publication metrics. US News is asking each law school for the names and other details of its fall 2018 full-time tenured and tenure-track faculty. US News plans to link the names of each individual law school’s faculty to citations and publications that were published in the previous five years and are available in HeinOnline.

Using this data, HeinOnline will compile faculty scholarly impact indicators for each law school. This will include such measures as mean citations per faculty member, median citations per faculty member, and total number of publications.

Those measures will then be provided to US News for use in eventually creating a comprehensive scholarly impact ranking.

Thanks to all of the folks who have opined on this issue. After reading and listening to many varying opinions on topic, here is a list of aggregated pros and cons*:

Pros:

  • USNews Quality Assessment scores may become more objective than the opaque peer assessment scores. 
  • Assist regional law schools that do not currently have a great peer assessment score. 
  • Incentive for faculty to publish. 
  • HeinOnline is the most robust collection of law reviews. 

Cons: 

  • Authors of books/book chapters are not represented in HeinOnline. 
  • Authors of articles in journals other than law reviews are not in HeinOnline (ex. most interdisciplinary work). 
  • Issues with journals that are embargoed for 3-5 years in HeinOnline (ex. Journal of Empirical Legal Studies). 
  • Update on 2/25/19: A rep from HOL has indicated that embargoes will not affect citation metrics (see comment below). 
  • HeinOnline could potentially skew results by including minor pieces (forwards, introductions).
    • Update on 2/25/19: A rep from HOL has indicated that these minor pieces are not included the citation analysis (see comment below). 
  • Citation counts are reductionist and bias:
    • Current research being done that shows 80% of the publications in top 10 law reviews for 2018 are written by authors whose alma mater is one of those schools.
    • Embeds gender, class, and race issues. 
    • Strategic self-citing or not citing “rivals” could be used to up impact measure.
  • Could result in a change in publication topics, selected formats, and selected publication. 
  • High-profile emphasis on faculty scholarship could be detriment to focus on teaching. 
  • Citation counts will be a detriment to institutions with a high number of more junior faculty members because their scholarship hasn’t had as much time to produce citations. 
  • USNews is using the bright line of tenure/tenure-track regardless of tenure classification or scholarly requirements.
    • How will it affect tenure lines that do not have as much of an emphasis on publishing?
    • Update on 2/25/19: USNews will still ask schools to list all tenure-stream faculty, but will also ask for their primary role to be identified (e.g., “doctrinal” or “clinical” or “legal research and writing”). USNews has not yet decided what to do with this information. 

    Recommendations regarding authority control: 

    • Multiple forms of author names – issues with typos, misspellings, and errors in author names that make count inaccurate should be remedied. 
  • Authors should use the same email address across all platforms, create a Google Scholar profile, and an ORCiD (personalized ID number that is used for all scholarship) to ensure proper scholarship metrics. 
  • Explore the use of PlumX to aggregate scholarly profiles and altmetrics. 

    • Ultimately, the success of this project depends on law schools providing the data.

    As has been said by others, it seems prudent to only provide the requested data on the condition that USNews creates an internal, unpublished sample report demonstrating its methodology.

    This will allow the various stakeholders to provide much needed input.

    Update: Withholding the data is not an option as US News is compiling faculty data if a law school did not provide it.

    *Individual ideas not attributed for sake of confidentiality