Qualimetric approach to rating startups
In the 20th century, experts from various countries faced the most acute problem of finding a reliable methodology for assessing quality, including the results of the activities of creative teams, their employees, products, projects and services. As a result, in 1968, the scientific discipline qualimetry appeared in the world [1], the demand for the use of which has grown significantly recently due to the rapid development of the Internet and the expansion of areas of human activity that are characterized by quantification of methods, tools and methods of research. The qualimetric method was developed by G. G. Azgaldov [2] and developed in numerous works of Russian and foreign researchers.
One of the successful examples of using a qualimetric approach in rating is the National Brand in the field of franchising “Golden Brand” [3, pp.121-143], the summing up in three nominations of which has been held since 2006 using the qualimetric method (in the main nomination the complex indicator - "Integral quality").
Since the methodological approaches used in qualimetry can be effectively implemented in the evaluation and selection of startups, especially at an early stage of development, it is proposed to use qualimetric analysis for rating startups.
Among the many different approaches to evaluating start-ups [4-6, 13], the use of qualimetry is preferred. And that's why. If you turn to the dictionaries, you can find that "rating" comes from the English term 1 , which means the subjective assessment of any poorly measurable phenomenon on a given scale. In a situation where the main criterion for a ranked list of startups is “quality” (or “integral quality”), and the given scale is determined by the world level, then the synonym for “rating” is “qualimetric analysis”.
')
In [5], a brief review of some expert methods for evaluating start-ups is given, among which at the early stages can be used:
Figure 1 presents the methodology for rating the startups of the PRUFFI Agency in the form of a property tree, which reflects the professional prospects and degree of professionalism (both operational and organizational) of start-up project teams [13].
Fig.1 Rating "30 most promising start-up teams of the Russian Internet and IT" in the form of a tree of properties
This rating is based on the scoring (R = S + P + K) given by the experts for the stability and cohesion of the team (complex property 14 in Fig. 1), professional competencies (Property 15 in Fig. 1) and team activity (Property 16 in Figure 1). According to the authors, this rating demonstrates typical errors inherent in the majority of ratings, which consist in the arbitrary selection of rating criteria and the weights of these criteria (in this case, points).
The Ave Maria model is useful when comparing startups and choosing the preferred direction of investment, but, unlike the qualimetric approach, it does not provide numerical estimates. A significant disadvantage of the Bill Payne method is the arbitrary choice of criteria for comparison, the arbitrary choice of weights of these criteria and the uncertainty of the object of measurement. Unlike Bill Payne's method, qualimetry allows you to logically justify the choice of criteria for comparing the objects being evaluated, based on the results obtained in the process of determining the assessment situation, determine the weights of the assessment criteria, relying on information on the global quality level of similar objects of assessment, reduce uncertainty due to scientifically based rules for the construction of trees of evaluation criteria. Some methodological problems of quality assessment and their solutions are considered in [7].
When rating startups, for example, the “project cost” can be used as one of the criteria, and from the valuation approaches - the cost estimation methods. Such methods are successfully used, for example, by professional appraisers and investment consultants (but most often at later stages of development — Early stage, Expansion stage, Later stage). At the early stage of development (Seed stage), the use of valuation methods for rating purposes is not effective for two main reasons: 1) due to the high costs of using methods; 2) due to the high degree of uncertainty of the model's initial data (in forecasts of market development, cash flows, in the evaluation of supply / demand, risks). Compared to the valuation methods, the use of qualimetric analysis allows reducing the total costs of the rating process and uncertainty due to scientifically based rules for constructing property trees (evaluation criteria).
When conducting national ratings of startups (as well as selecting the appropriate experts), it is proposed to use qualimetric techniques that allow ranking of startups by quality (or integral quality), while ensuring transparency of measurement procedures, validity of selection criteria for comparison and comparability of startups' performance [8, 9 ]. The use of qualimetry in evaluating (ranking) startups not only minimizes the costs of rating organizers when summing up, but also allows them to be comprehensively assessed, since qualimetry has an apparatus by which any economic and non-economic objects and processes can be quantified and thus, both quality assessment criteria and criteria for evaluating the effectiveness of 2 startups can be considered at the same time. At the same time, assessment methods based on qualimetry can be customizable taking into account the strategic goals of the stakeholders with a given accuracy and allow not only measuring individual quantitative and qualitative parameters of a startup, but also comparing their performance indicators, for example, on the integral quality.
All start-ups differ from each other in areas of activity, development stages, results and costs to achieve their goals. Qualimetric analysis (especially at the Seed stage) allows you to take into account all criteria and performance indicators of startups (for which no restrictions are imposed), bring all indicators together, convert heterogeneous indicators into homogeneous, ensure accuracy, necessary accuracy, convolution of indicators and, most importantly comparability of results.
In order to answer the question which of the methods is most preferable when rating startups in terms of quality, we turn to the works [10, 11]. Based on a study of the literature on the theory of economic efficiency, the logic of estimates, operations research and decision theory 3 , 14 requirements were formulated for which 16 different methods of quality assessment were analyzed. In [10] it was shown that the qualimetric method of quality assessment is the most acceptable of all currently used methods for this purpose. Of course, the superiority of the qualimetry method over the others cannot be understood as superiority over all methods in all requirements. The qualimetric method is better than others only in the aggregate of all 14 requirements, and in terms of some requirements taken separately, it may be inferior to other methods. For example, on demand for “non-labor”, it is, of course, inferior to the method “evaluation by one expert” [11, Table 1, “B”]. However, due to automation, even the laboriousness of the qualimetric quality assessment method can be significantly reduced up to its comparability with other methods.
Thus, for rating startups, it is proposed to develop a qualimetric method in which the indicator “integral quality” will be used as a criterion for ranking startups at an early stage of development (Seed stage), and “quality” when selecting experts (or taking their expert ratings into account).
The qualitative approach for the quantitative assessment of quality has been published in open sources, for example, in [2, 3, 7–10, 12] and is used in various fields of human activity.
- 1 "rating" - rating, order, classification. Rating is a numerical or ordinal indicator reflecting the importance or significance of a particular object or phenomenon.
2 In qualimetry, the term “efficiency” is similar to “integral quality”.
3 took into account only those requirements, the validity of which is confirmed by a sufficiently large frequency of their mention in the literature. Therefore, the requirements that are mentioned in the literature very rarely or ill-founded are not used.
The main terms used in qualimetry in assessing the quality of startups:
Qualimetry is a scientific discipline that studies the methodology and issues of comprehensive quantitative assessment of the quality of any objects (processes, products or services).
Quality is a set of properties of an object that characterizes the results it generates (both positive and negative), but which do not include the costs of its creation and consumption.
Efficiency is a property of an object that characterizes the cumulative (total) costs of the estimated object, i.e. include the cost of its production, consumption and disposal.
Integral quality is a property of an object that characterizes the totality of its quality and efficiency.
The quality of a startup is a complex characteristic of the processes of creating and launching a new product on the market (device, work, service) that meets the needs of interested parties and is a combination of the results obtained, but which do not include indicators of resources (for example, cash) used in these processes.
If the resources spent are additionally taken into account, then the characteristic investigated in this way is called “the integral quality of a startup”, which is synonymous with “efficiency of the startup”.
Qualimetric analysis is a research method that allows you to compare and evaluate the quality of objects in relation to the world level.
Startup rankings - a comprehensive assessment and subsequent ranking of young Russian technology companies by degree of development, in order to identify their investment attractiveness, as well as to describe and structure the market for such companies. Source - www.rg.ru/2012/12/09/reyting-site.html
Authors :
Kostin A.V., Ph.D., Head of Innovative Projects Evaluation Department, CJSC "Technology Transfer Center", kostin@rusttc.ru
Azgaldov GG, Doctor of Economics, Chief Researcher, Central Economics and Mathematics Institute, Russian Academy of Sciences, gazgaldov@mail.ru
Literature
1. Kostin A.V. The birth of qualimetry // Standards and quality. - M., 2010, â„–2. - C.90-91. URL: www.labrate.ru/kostin/2010-02_nucleation_qualimetry.pdf (access date: 06.06.2013).
2. Azgaldov G.G. Development of the theoretical foundations of qualimetry: Dis. on the competition studied Degree / Military Ing. Acad. them. V.V. Kuibyshev. - M., 1981. URL: www.labrate.ru/azgaldov/azgaldov-qualimetry-referat-1981.doc (appeal date: 06.06.2013).
3. Qualimetry: initial information. Reference book with an example for the Agency for Strategic Initiatives to Promote New Projects, an autonomous non-profit organization: Textbook. manual / G.G. Azgaldov, A.V. Kostin, V.V. Sadov. - M .: Higher. Sc., 2011. - 143 p. URL: www.labrate.ru/kostin/051326.pdf (appeal date: 06.06.2013).
4. Dmitry Chernyak. Methods for evaluating a startup [Electronic resource]. URL: habrahabr.ru/company/findstartup/blog/144521 (access date: 06.06.2013).
5. Dmitry Chernyak. Methods for evaluating a startup. Continued [Electronic resource]. URL: habrahabr.ru/company/findstartup/blog/144221 (access date: 06.06.2013).
6. Startups received rating / Rossiyskaya Gazeta, 09.12.2012 [Electronic resource]. URL: www.rg.ru/2012/12/09/reyting-site.html (appeal date: 06.06.2013).
7. Garry G. Azgaldov, Alexander V. Kostin. Applied Qualimetry: Its origins, errors and misconceptions // Benchmarking: An International Journal, Volume 18, Number 3, 2011, pp. 428-444. - www.labrate.ru/articles/428-444_applied_qulimetry_azgaldov-kostin.pdf (appeal date: 06.06.2013).
8. Azgaldov GG, Kostin AV (2012) Increasing the number of participants' meeting, Barcelona (Spain), 26-28 January 2012. - www.labrate. ru / kostin / 054548.pdf (appeal date: 06.06.2013).
9. Qualimetry for all: Proc. manual / G.G. Azgaldov, A.V. Kostin, V.V. Sadov. - M .: InformZnaniya, 2012. - 165 p. - www.labrate.ru/kostin/064571_qualimetry_azgaldov-kostin-sadovov-2012.pdf (appeal date: 06.06.2013).
10. Azgaldov G.G. Qualimetry in architectural and construction design - M .: stroiizdat, 1989. - 264. www.labrate.ru/azgaldov/qualimetry_in_architectural_and_structural_design.pdf (access date: 06.06.2013).
11. Abstracts Kostina A.V. to the meeting of the round table “How and why to measure the quality of education in the university?” (November 20, 2012, CEMI RAS) // Library LABRATE.RU [Electronic resource]. URL: www.labrate.ru/20121120/063804.doc (access date: 06.06.2013).
12. Azgaldov G.G. Theory and practice of assessing the quality of goods. Fundamentals of qualimetry .. - M .: Economy, 1982. - 256s. - www.labrate.ru/azgaldov/azgaldov_theory_and_practice_of_quality-assessment.pdf (access date: 06.06.2013).
13. About the rating of “30 most promising start-up teams of Runet and IT” // Electronic resource - pruffi.ru/30startups/about (circulation date: 06.06.2013)
One of the successful examples of using a qualimetric approach in rating is the National Brand in the field of franchising “Golden Brand” [3, pp.121-143], the summing up in three nominations of which has been held since 2006 using the qualimetric method (in the main nomination the complex indicator - "Integral quality").
Since the methodological approaches used in qualimetry can be effectively implemented in the evaluation and selection of startups, especially at an early stage of development, it is proposed to use qualimetric analysis for rating startups.
Among the many different approaches to evaluating start-ups [4-6, 13], the use of qualimetry is preferred. And that's why. If you turn to the dictionaries, you can find that "rating" comes from the English term 1 , which means the subjective assessment of any poorly measurable phenomenon on a given scale. In a situation where the main criterion for a ranked list of startups is “quality” (or “integral quality”), and the given scale is determined by the world level, then the synonym for “rating” is “qualimetric analysis”.
')
In [5], a brief review of some expert methods for evaluating start-ups is given, among which at the early stages can be used:
- Ave Maria model (short for Acquisition, Value, Engagement, Monetization, Retention, Intellectual Property). Model author - Maxim Krainov
- The scoring method (Scorecard Method, also known as the benchmarking method - Bill Payne Method, Benchmark Method). The author - investment "angel" Bill Payne.
Figure 1 presents the methodology for rating the startups of the PRUFFI Agency in the form of a property tree, which reflects the professional prospects and degree of professionalism (both operational and organizational) of start-up project teams [13].
Fig.1 Rating "30 most promising start-up teams of the Russian Internet and IT" in the form of a tree of properties
This rating is based on the scoring (R = S + P + K) given by the experts for the stability and cohesion of the team (complex property 14 in Fig. 1), professional competencies (Property 15 in Fig. 1) and team activity (Property 16 in Figure 1). According to the authors, this rating demonstrates typical errors inherent in the majority of ratings, which consist in the arbitrary selection of rating criteria and the weights of these criteria (in this case, points).
The Ave Maria model is useful when comparing startups and choosing the preferred direction of investment, but, unlike the qualimetric approach, it does not provide numerical estimates. A significant disadvantage of the Bill Payne method is the arbitrary choice of criteria for comparison, the arbitrary choice of weights of these criteria and the uncertainty of the object of measurement. Unlike Bill Payne's method, qualimetry allows you to logically justify the choice of criteria for comparing the objects being evaluated, based on the results obtained in the process of determining the assessment situation, determine the weights of the assessment criteria, relying on information on the global quality level of similar objects of assessment, reduce uncertainty due to scientifically based rules for the construction of trees of evaluation criteria. Some methodological problems of quality assessment and their solutions are considered in [7].
When rating startups, for example, the “project cost” can be used as one of the criteria, and from the valuation approaches - the cost estimation methods. Such methods are successfully used, for example, by professional appraisers and investment consultants (but most often at later stages of development — Early stage, Expansion stage, Later stage). At the early stage of development (Seed stage), the use of valuation methods for rating purposes is not effective for two main reasons: 1) due to the high costs of using methods; 2) due to the high degree of uncertainty of the model's initial data (in forecasts of market development, cash flows, in the evaluation of supply / demand, risks). Compared to the valuation methods, the use of qualimetric analysis allows reducing the total costs of the rating process and uncertainty due to scientifically based rules for constructing property trees (evaluation criteria).
When conducting national ratings of startups (as well as selecting the appropriate experts), it is proposed to use qualimetric techniques that allow ranking of startups by quality (or integral quality), while ensuring transparency of measurement procedures, validity of selection criteria for comparison and comparability of startups' performance [8, 9 ]. The use of qualimetry in evaluating (ranking) startups not only minimizes the costs of rating organizers when summing up, but also allows them to be comprehensively assessed, since qualimetry has an apparatus by which any economic and non-economic objects and processes can be quantified and thus, both quality assessment criteria and criteria for evaluating the effectiveness of 2 startups can be considered at the same time. At the same time, assessment methods based on qualimetry can be customizable taking into account the strategic goals of the stakeholders with a given accuracy and allow not only measuring individual quantitative and qualitative parameters of a startup, but also comparing their performance indicators, for example, on the integral quality.
All start-ups differ from each other in areas of activity, development stages, results and costs to achieve their goals. Qualimetric analysis (especially at the Seed stage) allows you to take into account all criteria and performance indicators of startups (for which no restrictions are imposed), bring all indicators together, convert heterogeneous indicators into homogeneous, ensure accuracy, necessary accuracy, convolution of indicators and, most importantly comparability of results.
In order to answer the question which of the methods is most preferable when rating startups in terms of quality, we turn to the works [10, 11]. Based on a study of the literature on the theory of economic efficiency, the logic of estimates, operations research and decision theory 3 , 14 requirements were formulated for which 16 different methods of quality assessment were analyzed. In [10] it was shown that the qualimetric method of quality assessment is the most acceptable of all currently used methods for this purpose. Of course, the superiority of the qualimetry method over the others cannot be understood as superiority over all methods in all requirements. The qualimetric method is better than others only in the aggregate of all 14 requirements, and in terms of some requirements taken separately, it may be inferior to other methods. For example, on demand for “non-labor”, it is, of course, inferior to the method “evaluation by one expert” [11, Table 1, “B”]. However, due to automation, even the laboriousness of the qualimetric quality assessment method can be significantly reduced up to its comparability with other methods.
Thus, for rating startups, it is proposed to develop a qualimetric method in which the indicator “integral quality” will be used as a criterion for ranking startups at an early stage of development (Seed stage), and “quality” when selecting experts (or taking their expert ratings into account).
The qualitative approach for the quantitative assessment of quality has been published in open sources, for example, in [2, 3, 7–10, 12] and is used in various fields of human activity.
- 1 "rating" - rating, order, classification. Rating is a numerical or ordinal indicator reflecting the importance or significance of a particular object or phenomenon.
2 In qualimetry, the term “efficiency” is similar to “integral quality”.
3 took into account only those requirements, the validity of which is confirmed by a sufficiently large frequency of their mention in the literature. Therefore, the requirements that are mentioned in the literature very rarely or ill-founded are not used.
The main terms used in qualimetry in assessing the quality of startups:
Qualimetry is a scientific discipline that studies the methodology and issues of comprehensive quantitative assessment of the quality of any objects (processes, products or services).
Quality is a set of properties of an object that characterizes the results it generates (both positive and negative), but which do not include the costs of its creation and consumption.
Efficiency is a property of an object that characterizes the cumulative (total) costs of the estimated object, i.e. include the cost of its production, consumption and disposal.
Integral quality is a property of an object that characterizes the totality of its quality and efficiency.
The quality of a startup is a complex characteristic of the processes of creating and launching a new product on the market (device, work, service) that meets the needs of interested parties and is a combination of the results obtained, but which do not include indicators of resources (for example, cash) used in these processes.
If the resources spent are additionally taken into account, then the characteristic investigated in this way is called “the integral quality of a startup”, which is synonymous with “efficiency of the startup”.
Qualimetric analysis is a research method that allows you to compare and evaluate the quality of objects in relation to the world level.
Startup rankings - a comprehensive assessment and subsequent ranking of young Russian technology companies by degree of development, in order to identify their investment attractiveness, as well as to describe and structure the market for such companies. Source - www.rg.ru/2012/12/09/reyting-site.html
Authors :
Kostin A.V., Ph.D., Head of Innovative Projects Evaluation Department, CJSC "Technology Transfer Center", kostin@rusttc.ru
Azgaldov GG, Doctor of Economics, Chief Researcher, Central Economics and Mathematics Institute, Russian Academy of Sciences, gazgaldov@mail.ru
Literature
1. Kostin A.V. The birth of qualimetry // Standards and quality. - M., 2010, â„–2. - C.90-91. URL: www.labrate.ru/kostin/2010-02_nucleation_qualimetry.pdf (access date: 06.06.2013).
2. Azgaldov G.G. Development of the theoretical foundations of qualimetry: Dis. on the competition studied Degree / Military Ing. Acad. them. V.V. Kuibyshev. - M., 1981. URL: www.labrate.ru/azgaldov/azgaldov-qualimetry-referat-1981.doc (appeal date: 06.06.2013).
3. Qualimetry: initial information. Reference book with an example for the Agency for Strategic Initiatives to Promote New Projects, an autonomous non-profit organization: Textbook. manual / G.G. Azgaldov, A.V. Kostin, V.V. Sadov. - M .: Higher. Sc., 2011. - 143 p. URL: www.labrate.ru/kostin/051326.pdf (appeal date: 06.06.2013).
4. Dmitry Chernyak. Methods for evaluating a startup [Electronic resource]. URL: habrahabr.ru/company/findstartup/blog/144521 (access date: 06.06.2013).
5. Dmitry Chernyak. Methods for evaluating a startup. Continued [Electronic resource]. URL: habrahabr.ru/company/findstartup/blog/144221 (access date: 06.06.2013).
6. Startups received rating / Rossiyskaya Gazeta, 09.12.2012 [Electronic resource]. URL: www.rg.ru/2012/12/09/reyting-site.html (appeal date: 06.06.2013).
7. Garry G. Azgaldov, Alexander V. Kostin. Applied Qualimetry: Its origins, errors and misconceptions // Benchmarking: An International Journal, Volume 18, Number 3, 2011, pp. 428-444. - www.labrate.ru/articles/428-444_applied_qulimetry_azgaldov-kostin.pdf (appeal date: 06.06.2013).
8. Azgaldov GG, Kostin AV (2012) Increasing the number of participants' meeting, Barcelona (Spain), 26-28 January 2012. - www.labrate. ru / kostin / 054548.pdf (appeal date: 06.06.2013).
9. Qualimetry for all: Proc. manual / G.G. Azgaldov, A.V. Kostin, V.V. Sadov. - M .: InformZnaniya, 2012. - 165 p. - www.labrate.ru/kostin/064571_qualimetry_azgaldov-kostin-sadovov-2012.pdf (appeal date: 06.06.2013).
10. Azgaldov G.G. Qualimetry in architectural and construction design - M .: stroiizdat, 1989. - 264. www.labrate.ru/azgaldov/qualimetry_in_architectural_and_structural_design.pdf (access date: 06.06.2013).
11. Abstracts Kostina A.V. to the meeting of the round table “How and why to measure the quality of education in the university?” (November 20, 2012, CEMI RAS) // Library LABRATE.RU [Electronic resource]. URL: www.labrate.ru/20121120/063804.doc (access date: 06.06.2013).
12. Azgaldov G.G. Theory and practice of assessing the quality of goods. Fundamentals of qualimetry .. - M .: Economy, 1982. - 256s. - www.labrate.ru/azgaldov/azgaldov_theory_and_practice_of_quality-assessment.pdf (access date: 06.06.2013).
13. About the rating of “30 most promising start-up teams of Runet and IT” // Electronic resource - pruffi.ru/30startups/about (circulation date: 06.06.2013)
Source: https://habr.com/ru/post/182806/
All Articles