How-to: tools for competitive analysis of software products

Image: Stephen Bowler, Flickr

To get an idea about the product being developed, you need to learn as much as possible about its properties and qualities. For this, there is testing - this process just allows you to gather the necessary information. But you can get food for thought and action in another way - for example, by comparing your own product with a competing solution.
')
Competitive analysis (KA) of software products allows you to identify such properties and qualities that it would not be possible to learn with the help of “ordinary” tests.

We at Positive Technologies began the process of immersion in the spacecraft a few years ago - this is our article on the development of analysis methods. In the future, it received its development in the form of an internal tool for competitive analysis - we'll talk about it today.

How it works

Below is a simple example of applying such a method. There are two products - "our" and competing. In the first one, we see four colors corresponding to some properties, and in the second solution there are only three. Also our product has more blue color, and our competitor has more gray.



The practical application of this knowledge in real work can be so. Suppose we drove ten tests and nine of them led to a crash. It would seem that this indicates the presence of serious bugs in the product, everything is bad. If the competing product in the same way does not pass the same tests, then the situation becomes not so critical. Or vice versa - out of ten tests, only one leads to failure, and it seems that everything is not bad. But if the competing product passes all ten tests, then everything is not so rosy.

Our experience

We are engaged in a competitive analysis of security scanners and firewalls, so our methods are focused on products from the field of web security, but they are also suitable for other niches. Below is a story about how we conduct a competitive analysis.

In our work we have passed two important stages of development. At the first one, it was important for us to conduct as many different tests as possible, and we compared our product with only one competitor. The report was the answer to one question - which product found more vulnerabilities.

Then we decided that we would reduce the number of tests to 10-15 of the most important ones and increase the number of competitors to 2-3, and in the report we will include a comparison of more parameters. The format of the answer to the question “who found more” is preserved, but a deeper analysis of the data allows us to find the answer to the question of why a competitor has bypassed us in a specific parameter?

Subsequently, we plan to further increase the number of tests, and the number of competitors for analysis will exceed 3 products. In the future, we want to move to the next stage of development of competitive analysis in the company and learn to give an answer to the question of what features should be implemented in order for our products to become head and shoulders above competing solutions.

Now our competitive analysis process looks like this: we run tests, analyze results, prepare reports, deal with automation and refinement of tests, and then everything repeats again.

Not so simple

On paper, everything looks smooth, but in real life there are also difficulties. Very often we find ourselves in a situation as in the picture below, when the products differ, something like an apple and a bird cherry:



If the task is to compare two different vulnerability scanners, then we usually encounter such difficulties:

1. Competing products have different formats of reports and logs - this means that you need to come up with a single format and convert everything into it.
2. At the exit, products provide different information - for example, even when performing the same tasks, products may use different descriptions in their reports. So you need to develop automation tools to understand when we are dealing with the same concepts.
3. Competitors are able to search for different types of vulnerabilities - one product searches for vulnerabilities A and B, and the second defines B and C. A simple comparison of the number of vulnerabilities in two sets will be ineffective, it is necessary to analyze their intersection, only then the picture will be complete.
4. Competitors call the same vulnerability differently - here the situation repeats the second paragraph.
5. A competing product on some target shows a large number of false positives - as a result, our scanner can find 1000 vulnerabilities, and a competitor is still 10,000, which means we need to figure out how to automatically detect such false positives in order to be able to correctly compare the results.
6. The scanner cannot completely scan any CMS - even if we have found an excellent target for scanning, all this will not help if the competing product cannot do the same (for example, it cannot pass authorization in the CMS). We have to adapt the goal in order to still be able to make a comparison.

Competitive Analysis Tools

Let's talk about what tools can be used for competitive analysis of software. To manage the analysis process, we use our own tool called InAC (Intellectual Analysis of Competitors). Without the OpenStack cloud, the process would also be impossible, TFS and a network drive are used to save the results, in addition we use the neural network analysis (this will be a separate article), but it’s impossible to give the machines everything, so the analysis of the results is done manually using browsers, IDE and a burp, finally, a report is created on the web. Initially, we used Excel, but it is not able to display a large number of graphs in a user-friendly form.

Conclusion: what you need to be able to conduct spacecraft

Competitive analysis allows us to obtain information about the properties and qualities of software products that cannot be recognized by other means. However, this is a specific tool, and in order to achieve high-quality conclusions, the specialists who carry out the analysis should have a deep understanding of the subject area.

In our case, testing specialists have the skills of IB researchers and pentest, besides, they need to know PHP, Java EE, ASP.NET (C #), Python (preferably), to be able to work with Docker, TFS, Salt, git

PS The story about our experience in the competitive analysis of software products was presented in the framework of DevOps-mitap, which took place in the fall of 2016 in Moscow.

Video:



Slides



The link presents presentations of 16 reports presented during the event. All presentations and video presentations are added to the table at the end of this topic-announcement .

Author : Vladimir Sofin