The Art of Benchmarking

Executive Summary

Many outsourcing agreements provide clients with the right to benchmark price and service levels. Some clients struggle to successfully gain the benefit of benchmarking. Contrary to popular belief, the primary obstacle to a successful benchmark is not the service provider, but the benchmarking method itself. Service providers often support benchmarking as a way to “prove” to their clients they are competitive. Successful benchmarking requires clients to understand the different types of benchmarks available, how they work, and how benchmark results can best be applied.

The Basics of Benchmarking

When an outsourcing provider assumes responsibility for a client’s operating environment, the provider often adopts the existing, in-house solution. This, coupled with client-specific requirements, makes each outsourcing agreement unique. The specific scope of work, service levels, operating environments, and technology solutions vary between IT organizations. It is important to take these differences into account when conducting a benchmark. Benchmarking is more than just entering numbers into a spreadsheet: it requires knowledgeable experts who understand contract pricing, terms, and technology solutions, and how each impacts market prices and delivery.

The “best practice” for benchmarking is for the client and the service provider to jointly select and compensate a third-party benchmarking firm so that the firm is not motivated to favor either organization during the review. Through the process (usually 6-10 weeks depending on scope), the benchmarking firm should maintain objectivity (both real and perceived) between the two parties.

Among the various providers and methodologies for benchmarking, two distinct methodologies exist: price and cost. Price benchmarks seek to answer the question: “How much would these services go for in the open market?” while cost benchmarks answer the question: “How much would it cost for a well managed organization to deliver these services?” Firms that provide price benchmarks claim the “moral” high ground by benchmarking against actual prices within the industry. Cost benchmarking is more widely performed (many internal shops have historically used cost benchmarking to determine their own competitiveness) and therefore offers larger peer groups and more robust industry statistics. The problem with cost benchmarking is that cost is rarely equivalent to price. Cost benchmarkers often get by this by applying “margin” to the cost benchmark result to arrive at price. But this can introduce estimation errors, as benchmarkers rarely have access to specific service offering margins.

Cost Benchmarking

Cost benchmarking is typically used by internal shops to understand their competitiveness as it relates to the cost and efficiency of other IT organizations. Cost benchmarks are most accurate when compared to similarly sized operations in similar industries. Similarity is established using cost and staffing ratios. Metrics are created with a statistical review of the client performance and cost data gathered by the benchmarker over time. For example, a cost benchmark might reveal that staffing levels for a 2,000 midrange server environment might be higher than the “industry average” of 40 FTE (50 servers per support FTE).

While cost benchmarking may seem to have a smaller margin of error, the addition of a margin to arrive at a price benchmark can often skew results. In addition, cost benchmarks do not take into account market dynamics or contractual requirements (for example, internal shops rarely have to pay penalties for poor service and therefore don’t need to build financial risk into their delivery cost). In the eyes of most vendors, cost benchmarks have limited usefulness.

Price Benchmarking

Benchmarking service agreements with a price benchmark is similar to conducting a real estate appraisal on a house (see A Price Benchmark Metaphor). Real estate appraisers understand current market conditions that impact housing prices.

Similarly, for outsourcing agreements, benchmarking firms understand current market prices as well as price drivers (including efficiency levels, service levels, and geography) and use this information to determine a fair market price for a particular outsourcing transaction.

In a price benchmark, the benchmarker relies on its database of pricing data from industry contracts. Each time a benchmark is conducted; data regarding the client’s contract are captured and used for future benchmarks. Since price benchmarking is about predicting current market prices, the data typically have a short life-span (often less than 18 months) before they are considered obsolete. A price benchmark will compare contracts of similar size and scope of work, usually with 5-6 other contracts (i.e., “peers”). The peer group prices are “normalized” to account for differences between the target environment and each peer, and the client’s target price for the services is computed.

The primary obstacle in price benchmarking is access to data. Research firms gather pricing data when reviewing or evaluating contracts for clients. Outsourcing vendors (e.g., IBM, EDS, or CSC) also gather pricing data and reverse engineer it into models against which they “benchmark the benchmarkers” for accuracy and consistency (and to gain insight into competitor pricing).

 Quartiles, Deciles, & Other Statistical Illusions

A contract clause periodically adopted by IT organizations requires that vendor pricing be within the lowest quartile (or sometimes even lowest decile) of the market. While such language sounds scientific, such clauses work against the basic structure of benchmarking data and are unenforceable when negotiating price changes. The following reasons articulate the scientific fallacy of using quartiles.

1. Using quartiles implies statistical rigor and validity that does not exist because:

A. The peer group typically includes a limited number of data points, usually less than ten;

B. The benchmark peers are not chosen randomly but are selected by the benchmarking firm using qualitative and quantitative criteria; and,

C. The data from the peer group are often interpreted and therefore have inherent subjectivity. This margin of error (from subjectivity) often increases at the extremes (top or bottom 15%).

2. The “quartile clause” has been implemented widely by several leading consulting firms and legal advisors (e.g., TPI and Shaw Pittman). We believe that greater than 40% of industry contracts include either a quartiles or deciles benchmarking clause. Interestingly, this creates a logical dilemma as it is impossible for more than 40% of contracts to be in the bottom 25% of industry pricing.

This incongruity creates a downward spiral in pricing without regard to the technology savings or other real benefits achieved within the industry.

3. Using the “quartile clause” is an attempt to impose a statistical or measurable results on benchmarking, making it less subjective for both the client and service provider. In reality, focusing on quartiles centers the benchmark and subsequent negotiations primarily on price. A thorough benchmark (and benchmark clause) should include a qualitative analysis regarding service levels, process alignment, and an overall outsourcing relationship “health check.”

Optimizing Negotiations

A common expectation by user organizations is that benchmarking will yield precise pricing and prescribe contract adjustments. When the benchmark results return, users show the vendor price disparities and are surprised to see the results viewed as more of a suggestion to change than as a mandate. Service providers will consider benchmark results as an invitation to begin negotiating again. If the benchmark shows a dramatic gap and the benchmarking firm was not jointly contracted, the provider will likely hire a different benchmarking firm to conduct another study. The discussion usually then digresses into “my benchmark” vs. “your benchmark”, and the client should not expect productive results from the dispute.

The most difficult part of a benchmark is accurate interpretation and implementation of the results. While service providers recognize the need to remain competitive, they are also unwilling to make unnecessary or excessive compromises. Outsourcing providers view negotiations over the long term; if major compromises are made in the current cycle of negotiations, then the client will expect more concessions in the future.

The first step toward successful negotiations is to jointly select and pay the benchmarking firm. Benchmarking should be embarked upon as a mutual journey to validate that the service levels and pricing remain competitive. The goal of benchmarking is to ensure that service agreements are competitive, and it should not be used in a vindictive manner to negotiate unreasonable objectives.

The Bottom Line

Benchmark results will always be subject to interpretations rather than offering crisp, clear courses of action. Clients who expect the ensuing negotiations prepare properly and are ready to discuss changes in subjective behavior, measuring and monitoring, and service levels, and to work with their vendor to bridge gaps in the overall relationship. The negotiation around benchmark results opens the door for other elements of the relationship to be renegotiated as well.

[!] Report this post

Processing your request, Please wait....

Report as:	Invalid Contents Other
Write in Words: (Optional)