What is the Actual Value of The Yearly Ritual Certification Dance?

Summary

ISO and many other certifications are an increasingly expensive ritual dance. Every year, the auditor and client company go through the same motions with the sole outcome a stamp on a piece of paper. Even worse, with every new version, the ISO standard expands in scope and required paper trail. The only benefactor is this trend are the consultants, trainers, auditors and standardization bodies.

In real life, it is the attitude and skills of the individuals responsible for a product or service that drive quality, security and performance.

So why do companies continue to use these wasteful practices?

Why we use certifications

Ask an academic why we invented certifications and she/he will answer: to reduce information asymmetry. Due to the distance between the buyer and supplier of a product or service, it is difficult for the buyer to observe the qualifications of the supplier. Certifications allow the supplier to signal for the buyer unobservable attributes like quality and security. Similarly, employers use academic qualifications to differentiate between job applicants, “independent of whether or not students learn anything in the process of attending college” (I). But more than that quote later.

Last but not least, certifications are used to demonstrate regulatory compliance. IT service providers use standards like ISO 27001 and others to signal a certain level of control to their clients and regulators.

In short, certifications act as a ‘market signal‘, reducing the search stress and agency cost between buyer and supplier.

When certifications are useful

As mentioned, certifications like ISO 9001 (quality), ISO 27001 (security), ISO 20000 (service management), ISO 31000 (risk management) and ISO 22301 (Business Continuity) allow an IT department or IT service provider to communicate certain attributes to other stakeholders. They signal that, at least on paper, the IT department or IT service provider acknowledges the importance of the topic covered by the ISO or other auditable standard.

According to Ter Laak and King, certifications may even provide a competitive advantage in markets where buyers can choose from numerous suppliers. They observed that suppliers with an ISO 9001 certification tended to grow harder than supplier without a certification (II). In other words, early adopters may enjoy a competitive advantage when they are able to convince their buyers of a certificates’ added value. Until the competition catches up that is.

In mature and especially risk-averse markets (e.g. healthcare, banking, insurance, government), certifications are a precondition to do business. Buyers include them as a mandatory requirement when inviting suppliers to tender, disqualifying any bid that fails to comply. Here, certifications are used to reduce liability risk and accompanying lawsuits.

Another advantage of certifications is the body of knowledge embedded in the underpinning standards. All define - at a relatively high abstraction level - a set of desired outcomes, and the activities to achieve and control those outcomes. The quality of the standard itself is ensured through a combination of committees, a centralized governing body and strictly enforced development and update processes. In principle, everybody can participate in the committees responsible for the 21,378 published ISO standards or 4,938 ISO standards under development (note: data retrieved on May 2017). Numbers which provide a natural point to move on to the next topic.

In short,

• Some standards and certifications enjoy a, temporarily, first mover advantage.
• Certifications prevent opportunistic companies from entering certain markets.
• Why invent the wheel when it already has been invented?

When certifications lose their effectiveness

Your lunch can be ISO 22000, BRC, SQF, IFS, USDA Organic, AHA and ISTA & Hygiene modified approval scheme -certified. Does this information in any way influence your decision to buy?

The ISO catalogue dedicated to Information Technology includes 71 published standard and standards under development (note: data retrieved June 2017). This is only the tip of the iceberg however as the ISO 27000 standard alone consist of 45 underpinning standards, including:

1. ISO/IEC 27005 — Information security risk management
2. ISO/IEC 27010 — Information security management for inter-sector and inter-organizational communications
3. ISO/IEC TR 27016 — information security economics
4. ISO/IEC TR 27019 — Information security for process control in the energy industry
5. ISO/IEC 27042 — Analyzing digital evidence

This Wikipedia page  points out that 45 ISO 27,000 related standards are still not enough: “Further ISO27k standards are in preparation covering aspects such as digital forensics and cybersecurity, while the released ISO27k standards are routinely reviewed and updated on a ~5 year cycle.”  The last part of the latter sentence means that soon your current quality management system becomes obsolete and your team is faced with a mandatory update to the new version. A new version which, in my experience, only grows in scope and consequently paperwork and cost (see below).

More importantly, the unchecked growth of standards causes confusion among both buyers and suppliers.

Confronted with a relentless and unchecked growth of standards and certifications, both B2C and B2B buyers lose track and either ignore them or stick to what they know from the past. Another side effect of the proliferation of standards is misinterpretation. Recently, I red a tender requiring the supplier to be ISO 25,000 certified. ISO 25000 is a family of standards (again, one was not enough), focusing on the quality of software in terms of:

• functionality (e.g. suitability, accuracy),
• reliability (e.g. maturity, fault tolerance),
• usability (e.g. understandability, learnability),
• efficiency (e.g. time behavior, resource utilization),
• maintainability (analyzability, changeability), and
• portability (e.g. adaptability and installability).

Hence, the standard is a useful reference guide for architects and software developers, but it is not a standard one can certify against. Yet. Similar to the Agile Manifesto, consultants, trainers and auditors have identified standards and certifications as an easy source of revenue. ISO 25,000 may well be their next victim.

Misinterpretations are part of a broader issue: buyers considering certifications a quick fix. These buyers think along the following lines: if you are ISO 9001 certified, I get high quality products and services. If you are ISO 27001 certified, my data and applications are safe. If my payment processor is Payment Card Industry Data Security Standard (PCI DSS) certified, my credit card information is secure.

Sorry to burst your bubble, but retailer Target lost credit card data of 40 million people and Meiman Marcus exposed 1.1 million payment card cards despite being PCI certified. The link between the actual security level and ISO 27000 is even far weaker than achieved through PCI. PCI actually provides a solid defense against hackers as long as the security specialists of the company regularly assess their readiness against new threats. Attack vectors and threats evolve as stealing data and ransomware can be very lucrative. Companies solely focusing on the piece of paper tied to PCI compliance will therefore inevitably become vulnerable for attacks somewhere down the line.

An equally dangerous example of misinterpretation is assuming ISO 27000 safeguards against security weaknesses in the application.

The most valuable commodity hackers are after is stored in the databases and applications: personal data and commercial data (e.g. Game of Thrones scripts). To safeguard both types of data, the client company must look beyond ISO 27,000 as the latter focuses on the support and operations phase of the IT life cycle, while the security level of the application and database is shaped during the design and development phases. Not ISO 27000, but Secure Software Design, Security Development Lifecycle (SDL),  OWASP top 10, OWASP SAMM, NIST SP-800, and NIST SP 1800 should be the terms to look for.

Besides regulatory compliance and reducing information asymmetry between buyer and supplier, some standards are also promoted as a means to improve performance (e.g. financial, market share). What most scholars agree on is a difference in performance between companies without any ISO 9000 or other quality management system and companies with an ISO 9000 or other quality system. One example is Heras et al (III): “Using the return on assets employed (ROA), the average level of profitability was calculated for the 400 certified firms and the 400 non-certified firms for each of the years 1994, 1995, 1996, 1997, and 1998. […] In all five years, it can be observed that the average profitability of the certified firms is superior.” However, with 1 138 155 certifications in 2014, one can hardly call it a differentiating capability. All but the smallest niche players have an ISO 9000 certification like all but the smallest hosting providers are ISO 27000 certified.

Even worse, the emphasis on bureaucratic hard controls in combination with the ever-expanding scope of every new version may tip the scales in the wrong direction. More in general, the main criticisms levelled at ISO 9000 are (IV):

• It is bureaucratic (e.g. if it is not on paper or in a tool it does not exist for an auditor)
• It is costly to implement (e.g. writing and maintaining procedures, hiring quality manager, tooling auditor fees).
• It does guarantee the quality of the product (e.g. it does not safeguard against a garbage-in garbage-out scenario).
• It is not suitable for small organisations (e.g. due to the high cost)

Tsiotras and Gotsamani observed that many companies indeed certify “just for the sake of it”, listing the following issues with ISO 9000 (V):

• Low flexibility and slow response to change.
• Lack of correlation between certification and high quality or increased customer satisfaction.
• An excessive obedience to documented procedures, which may discourage critical thinking.
• A lack of focus on continuous improvement beyond the achievement of certification.

Hence, the following quote to wrap things up on the impact of ISO 9000 on firm performance from Cagnazzo et al (VI): “Despite substantial literature on the ISO 9000 standard, there is still much debate concerning the standard’s impact on firm performance, competitiveness and operations management. […] Although the number of firms that want to implement ISO 9000 quality management system is increasing day by day, many of them increasingly started questioning the link between ISO 9000 and firm performance.”

In short,

• The unchecked proliferation of standards and certifications reduces their effectiveness
• Lazy and/or badly informed buyers misuse them as a quick fix.
• There is a weak correlation between the desired result (e.g. secure data, quality) and certifications.
• The never-ending scope increase of standards will increase their cost to a point whereby the business case turns negative.

But the auditor saves the day, right?

Unfortunately, the auditor is of little help. Auditors only check whether you performed the mandatory quality or risk assessment, they don’t (and are rarely knowledgeable enough) to determine the quality of the assessment itself. They tick boxes. Take the following objective and controls from ISO 27,000-2005 for example.

” Objective: To provide management direction and support for information security in accordance with business requirements and relevant laws and regulations.

A.5.1.1 Information security policy document.

Control: An information security policy document shall be approved by management, and published and communicated to all employees and relevant external parties.

A.5.1.2 Review of the information security policy.

Control: The information security policy shall be reviewed at planned intervals or if significant changes occur to ensure its continuing suitability, adequacy, and effectiveness”

Simply stated:

• A.5.1.1 means the auditor wants to see a document with the title ‘security policy’, a signature of a manager somewhere in the document and a location on the intranet where the employees can find it.
• A.5.1.2 means the auditor looks for evidence that somebody has reviewed the document (e.g. new version number), but leaves it up to the company to determine where significant changes occurred regarding the suitability, adequacy and effectiveness of the policy.

The logic behind this approach is the impossibility of auditors to thoroughly understand the specific risk profile of every individual company. However, it also limits the actual value of the standard and accompanying certificate as they demonstrate their inability to protect against a garbage-in garbage-out scenario and certifications for the sake of it.

The actual level of security depends on the professionalism, skills, culture and leadership style of the IT team. Aspects which are difficult to catch with the abstract ‘hard controls‘ most standard are based on.

In short,

• The added value of an auditor is very limited.

The winners

In 2014, ISO.org reported 1.609.294 valid certificates world-wide, an increase of 3 percent compared to 2013. According to this survey, the three year cost charged by a certification body for ISO 9001 varies between $5,400 and $7,425. That translates into $2.9 billion and 3.9 billions out of pocket costs per year related to for ISO-related certifications (VII). These amounts exclude the cost that companies incur for hiring and retaining an internal quality manager, the additional administrative burden, internal audits, tooling, training and so on. Depending on the size of the company, think of at least $100,000 for a small company and up to a million for a large corporation.

Other certifications are equally lucrative of consultants, trainers and auditors. The previously mentioned ISO 22000, BRC, SQF, IFS, USDA Organic, Kosher, Halal, AHA and ISTA & Hygiene certifications translate into a global food certification market that is expected to reach a value of $14.5 billion by 2019, growing at a CAGR of 5.2%.

Certifications are a money printing machine for thousands of consultants, auditors and standard organizations.

In short,

• The primary beneficiaries of certifications are consultants, auditors, trainers and organizations creating and publishing standards.

Turning the tide

As with building airplanes, the solution starts with a CEO who understands that qualified engineers and other professionals directly involved in the day-to-day-day business determine - in the long run - the financial performance of the company. Professionals that posses an intrinsic drive and the expertise to create and deliver products with an adequate level of quality, security and privacy.

The term 'adequate' is crucial here, as the flipside of CEOs/CFOs solely focusing on short-term financial results is development teams and CISOs which set quality and security objectives which prevent the company from launching any product at all or turning the company into an internally oriented bureaucratic control-framework-nightmare.

Quality, security and other important properties should be fit-for-purpose, requiring a dialogue between the key stakeholders to determine the optimal equilibrium. Without a healthy profit, the company will go bankrupt. However, cutting too many corners (e.g. relying too much on certifications, failing to address technical debt & legacy) to increase the share price quarter after quarter will come back to bite you.

There are no short cuts, delivering quality products securely is hard work. There is no other way around it. 

Notes and references

(I) Spence, M.. Job Market Signaling, Quarterly Journal of Economics 87, pages 355-374, 1973.

(II) Ter Laak, A., King, A., The effect of certification with the ISO9000 quality management standard: a signaling approach, 2006.

(III) Heras, I., Casadesus, M., Dick, G., ISO 9000 certification and the bottom line: a comparative study of the profitability of Basque region companies, Managerial Auditing Journal, 2002.

(IV) Barnes, D., Operations Management: An International Perspective, 2007.

(V) Tsiotras, G.,  Gotzamani, K., ISO 9000 as an entry key to TQM : The case of Greek industry, International Journal of Quality and Reliability Management, 1996.

(VI) Cagnazzo, L., Taticchi, P., Fuiano, F., Benefits, barriers and pitfalls coming from the ISO 9000 implementation: the impact on business performances, WSEAS Transactions or Business and Economics, Volume 7, 2010.

(VII) Assuming average yearly cost for ISO 9001 certification is on average equal to others.