Metametrics

About metametrics

“Metametrics” are to metrics what “metadata” are to data.
In other words, metametrics are information-about-metrics.

Metametrics include various characteristics of metrics. For example, metrics lists and catalogs typically describe each metric in terms of its scope, purpose, parameters, sources and calculations: these are all metametrics.

Who decides that these particular characteristics are important? How do they decide? Which are the most/least important factors? On what basis are certain metametrics taken into account or ignored? Unfortunately, such important considerations are seldom discussed. Generally, all we see is the end result without the opportunity to understand the thinking process behind it.

Most of all metametrics provide an objective basis for designing and selecting metrics that are actually worth measuring, analyzing, reporting and using. Without metametrics, we’re saddled with hocus-pocus, guesswork and blind faith.

In the information security context, metametrics are valuable:

To assess different security metrics on a meaningful, rational and comparable basis, helping us decide objectively which (if any!) might be worth adopting, developing further or parking;
When reviewing and reassessing the security metrics currently in use, for example if management is unsatisfied with the current crop or needs factual information in order to manage other security risks;
To consider the merits of security metrics recommended by others, and to explain the basis for recommending specific security metrics to anyone, not least our own management;
For benchmarking security metrics used by different organizations, or by business units within a large organization, particularly to identify and share more creative and productive ways of measuring information security;
In support of a more rigorous, professional approach to information security measurement, which is itself essential for the governance (direction, management and control) of information security.

Just as security metrics are used to measure, manage and improve our information security management system, metametrics help us measure, manage and improve an information security measurement system. Applying systems design to measurement is an example of the innvative thinking behind the PRAGMATIC approach.

Information-about-metrics includes metrics-about-metrics. For example the number of security metrics used by the organization is a relatively crude metametric, while the quality of the organization’s security metrics is potentially much more informative and useful. Metametrics also includes books-about-metrics, such as Information Security Management Metrics by Krag Brotby and How to Measure Anything by Douglas Hubbard.