Goodhart's Law
Goodhart's Law states that when a measure becomes a target it ceases to be a good measure. Keep in mind that his can be a concern in electronic discovery. Maura Grossman and Gordon Cormack (authors of the famous TAR glossary, see the night of June 3, 2015) published a paper in The Federal Courts Law Review, entitled, "Comments on “The Implications of Rule 26(g) on the Use of Technology-Assisted Review”. This paper criticizes a study on TAR by Karl Schieneman and Thomas Gricks that specifies that enough random sampling must be conducted in order estimate precision and recall within a margin of error of plus or minus 5 per cent. Grossman and Cormack submit that the goal of TAR is always to identify as much responsive ESI as possible for a proportionate cost and that statistics about precision and recall are just measures of success in getting to that goal. They assert that TAR directed by sampling specifications reflect the problem stated in Goodhart's Law. 2014 Fed. Cts. L. Rev. 285, 287.
Comments