• What is a Benchmark? It’s 4 Things
• Cynicism about Benchmarks
• Benchmarking to Remain Competitive
• Benchmarking & Competitor Rresearch – Comparison Chart
• Three Types of Benchmarking
• Why We Use Passmark

What is a Benchmark? It’s 4 things

A benchmark is basically a guidepost with which you can gauge something. When surveyors use the term bench mark, they are referring to a permanent mark that has a verified elevation and can therefore be used to determine elevation of any other point.

In cloud and general IT, the meaning of benchmark is actually fourfold:

Benchmark definition #1: A scenario of established parameters that serves as a level playing field with which you can test hardware or software.

“PC magazine laboratories frequently test and compare several new computers or computer devices against the same set of application programs, user interactions, and contextual situations,” explained technologist Margaret Rouse.

In this case, the benchmark is all of the tools, processes, stipulations, and constraints of the standardized test environment.

Benchmark definition #2: An application that has been built to supply metrics on specific software.

Benchmark definition #3: A technology or approach that has high awareness among users and can be used as a reference point for comparison purposes.

Benchmark definition #4: A system of guidelines that establish criteria through which performance can be determined.

That last definition is the one that’s of the most interest to cloud hosting companies and to their customers. For instance, our cloud plans have a specified total Passmark performance rating per core ranging from 156.25 to 625.

Cynicism about Benchmarks

The one thing to keep in mind about benchmarks that are used in test environments is that they don’t always accurately represent true day-to-day operation. In some crowds, the failure of some benchmark programs to have any serious validity is a reason to forget the whole thing.

Technologist Eric Raymond is completely dismissive of benchmarks, calling them “an inaccurate measure of computer performance” and arguing that “[i]n the computer industry, there are three kinds of lies: lies, damn lies, and benchmarks.”

Although we agree with Raymond that some benchmarks are nonsense, we believe strongly that the PassMark rating provides an objective standard, allowing us to create cloud virtual machines that fit your needs and your budget.

Benchmarking to Remain Competitive

Before we get into the specifics of what makes PassMark the highest-quality system available, let’s see how the ratings fit within benchmarking as a broad business concept.

In that context, benchmarking is the delineation of ideal performance, as exhibited internally or by any organization that can then be used for comparison. Once the benchmark is compared to a system or process, the company can identify weaknesses and make improvements.

In this sense (more elaborate than a CPU performance test), benchmarking is incredibly complex, and it typically requires permissions. Usually organizations have a data policy that limits the type of information you can access. If you are within a large enterprise, other groups have probably started this process even if yours hasn’t, allowing you to branch off from their efforts.

“Benchmarking is not just a matter of making inquiries to other companies or touring and documenting another company’s facilities or processes,” commented DeLeeuw Associates consultant J. DeLayne Stroud. “[A] company should not limit the scope to its own industry, nor should benchmarking be a one-time event.”

Benchmarking & Competitor Research – Comparison Chart

Although this general benchmarking effort can definitely give you a competitive advantage, it should not be misconstrued as competitor research. Here is a basic comparison chart of the two approaches:

Benchmarking	Competitor Research
Sustainable changes leading to continuous results	Symptom-targeting, “Band-Aid” solution to quickly handle competitive threat
Collaboration to share insights	May fall under the umbrella of corporate espionage
Necessary to remain a leader in the marketplace	Can be positive but is nonessential
Changing to meet expectations once you have studied leading approach	Mimicking the efforts of another organization

Three Types of Benchmarking

Roughly speaking, benchmarking can be categorized as internal, competitive, or strategic.

Internal benchmarking – This approach is employed when an organization wants to share its best practices between different departments. You also may have to use this type of benchmarking if you want to avoid competitors and can’t find a similar scenario in another industry.

Competitive benchmarking – With this style, you take a look at the top of your industry so that you can develop a better sense of ideal performance.

Strategic benchmarking – This category “is used when identifying and analyzing world-class performance,” said Stroud. “This form of benchmarking is used most when a company needs to go outside of its own industry.” Many businesses use the Hoshin process for strategic planning, which matches the company’s objectives to benchmarks of top companies. Those companies are frequently not within the same industry.

Why We Use Passmark

Although benchmarking as general analysis is more complex than CPU comparison, standardized approaches essentially allow for competitive benchmarking of IaaS cloud servers. In this context, Raymond’s cynicism is understandable: much of it is worthless. However, we believe that one system is incredibly useful for us to describe our plans and for you to know what you’re getting.

Determining the amount of gigahertz and other parameters across different CPU generations is meaningless, and that’s the basis of some benchmarks. PassMark is the only practical and objective comparison of the actual performance of different CPU. See the “PassMark per core” row in each of our cloud plans.

NOTE: To read Part Two of this article, please click HERE.

By Kent Roberts