Principles of building web studios rating

Hi, Habr. I would like to talk about the principles of building company ratings in the market for custom web development in Russia. We published a detailed methodology for constructing the 2010 rating , but I would like to make out the topic a little wider. Therefore, I propose to talk about how metrics in our market can be.

What factors in general it makes sense to take into account in the rating:

1. The real business performance of the company

• Finance (turnover, profit, etc.)
• Staff
• Price segment
• Number of projects released

2. Brand, marketing, PR

• Brand awareness among customers
• Brand awareness in the industry
• Market positioning and weight
• Number and format of mentions in the media

3. Quality and customer service

• Customer reviews about working with the company
• Satisfaction in various aspects of customer service
• Quality of the projects produced
• Organizational model, management techniques and production

What are the features of the web development market in Russia?
')
First of all, it is necessary to clarify why metrics adopted in “adult markets” are not very suitable for building a rating of studios, and why we are trying to “invent a bicycle”:

• The market is spread over several thousands of companies.
• Even the leaders, in fact, have a very small share of the entire market (with a turnover of 10 million dollars, and this is 1-2 companies, the share will be only about 3%).
• Companies on the market appear quickly and burn out quickly (low entry threshold)
• Most companies are very small.
• Most companies are working on "gray" schemes, and are not ready to provide reports
• Even if reporting is provided, it is impossible to separate the development budget from the budgets for advertising and promotion and transit (CMS, hosting) (and they cannot be put together).
• In-depth interviewing and checking “in their territory” data submitted by thousands of players is insane

Description of the methodology proposed by Tagline in 2010

I will describe our methodology and formulas in simple words to make it clearer. With the expanded text (and all sorts of bonuses, like "what is considered cheating") can be found here .

The project in 2010 involved about 1,000 studios, 100 clients and 50 experts.

1) Studios provide their own version of the 10 leaders of the market (indicator - turnover for 2009), placing the rating points from 1 to 5. For each company is considered a score. Mutual votes are counted with a factor of ½. Further, the votes of the first 300 companies (“electors”) are counted as they are, and the rest of the votes - from the 1/5 coefficients. So each company gets its Rball_internal, which shows the opinion of the market itself.

2) Customers and experts call their top 10 of the most recognizable brands on the market. Rating points are not placed, a dozen is a general list. For each company is considered the number of its references in this vote - RBall_external.

3) For each company, its overall rating is considered to be RBall = RBall_internal + RBall_external * 3 .

4) For each company, 7 correction factors are considered, according to business performance indicators (which the company itself submits during the survey). They are multiplied and the resulting correction coefficient K is obtained.

5) For each company is considered a client ratio. Customers indicate with which companies they have a positive experience of cooperation and negative. The difference in the number of these references is the coefficient C.

6) The final score is calculated by multiplying the rating score by a single correction factor K and client coefficient C: Ball = RBall * K * C.

Short FAQ on our methodology:

Q: Studios vote for each other, is this nonsense? Ratings auto-manufacturers do not build, you can imagine that Audi asked to vote for BMW?

A: There is a completely different market - completely transparent finance, a clear pool of leaders (about 50 brands), and not several thousand companies in 3-10 people.

Q: This rating can be only for the real turnover of companies.

A: Rating can be for anything - at least for the total number of freckles among company employees. But, indeed, many ratings in adult markets are based on turnover. In our market, it is impossible to make such a rating with the required quality. There are two reasons. Firstly, the activities of companies are non-transparent, and only a few are ready to provide their financial statements. Secondly, even if reporting is provided, it is not possible to separate the budgets for development and advertising (and most companies are engaged in development and advertising). In any case, within the framework of our project, one of the sub-ratings will be a cut according to the turnovers declared by the players themselves (but I would recommend using this information only as a reference). In addition, the representativeness of the rating can be seriously violated, as it is possible to get into the lists of leaders of companies that either do not specialize in web development (system integrators, etc.), or are created under 1-2-3 large projects.

Q: All places are bought, you are crooks!

A: Exactly. You owe us a thousand dollars for answering this question =)

Thank you for your attention, we will try to answer questions and constructive (!) Criticism in the comments :)