Forrester study: a comparison of the top ten vendors of Software Composition Analysis

In April 2019, Forrester Research, an analytical agency, released a quarterly report of The Forrester Wave ™: Software Composition Analysis, Q2 2019, with the evaluation and comparison of ten software solution class providers (hereinafter referred to as SCA). This article is an adapted translation of the key points of the report.

Key findings

• WhiteSource and Synopsys topped the list of leaders (Leaders) of the market;
• Snyk and Sonatype are named Strong Performers;
• WhiteHat Security , Flexera and Veracode are called Contenders;
• GitLab , FOSSA and JFrog are Challengers.

Translator's Note

What is Software Composition Analysis? The Software Composition Analysis process is a sub-aspect of the broader Component Analysis process of the Cyber ​​Supply Chain Risk Management framework ( see the NIST website ) and is designed to determine the potential risks of using third-party and open source software.
')
Software solutions such as Software Composition Analysis (SCA), respectively, are intended for automated risk detection and elimination of vulnerabilities in code, control over the use of external (ready-made) libraries.

For a general overview of Component Analysis and Software Composition Analysis in particular, we recommend that you read the material on the OWASP project website .

The importance of decisions of the Software Composition Analysis (SCA) class in the implementation of modern approaches to creating a safe application development environment

In conditions when it is necessary to quickly add new functionality to applications while preserving the quality of the code, developers are increasingly resorting to using already created software modules located in public repositories, and also customize and automate their code relying on common open source components. One of the latest studies of the ^source showed that from year to year the number of open source components in the code base analyzed in the study increases by 21%.
It is important to understand that these third-party open source components may contain critical vulnerabilities, which in itself already represents a risk to the business. A recent study found that every eighth open source component contains a known ^source security vulnerability. All this is aggravated by the fact that recently information security specialists have less and less time to identify and eliminate newly discovered vulnerabilities, since the time between the publication of a vulnerability and its operation has decreased from 45 to 3 days.

Considering these trends, consumers of SCA class solutions should take a closer look at vendor solutions that:

• Recommend developers how to fix vulnerabilities.
  To significantly reduce the risk posed by vulnerabilities and licensing conditions, developers need to receive notifications about security risks and the use of someone else's code in terms of licensing purity, as well as receive recommendations on their security risks as part of the SDLC (software delivery life cycle) processes. to eliminate. SCA solutions should not only provide high-quality recommendations for eliminating risks, some products also offer solutions for correcting the code, providing information about the presence of a secure version of the open source component or creating patches if there are no corrected versions.
• Allows you to create policies that are consistent with different business units and application types.
  To increase the speed of release of releases, information security specialists should switch from manual testing using SCA tools to the development and implementation of secure development standards in organizations using the SCA functionality. Information security specialists approve secure development policies that all applications in a company must satisfy (for example, none of the known critical vulnerabilities should go into release) and control the minimum security requirements for more critical applications. To be effective, the information security specialist requires flexible management of adopted policies, including the use of tools provided by SCA.
• Provide strategic risk reports for information security and CISO specialists.
  CISOs should organize the process in such a way as to reduce the amount of time spent on addressing the identified vulnerabilities. In the past, information security specialists collected information about risks and vulnerabilities from different sources or simply did not do anything. Today, an information security specialist can receive out-of-the-box reports (“out of the box”) for CISO and development teams that describe the identified risks.

Criteria for the inclusion of vendors in the review

Forrester has included 10 vendors in this benchmarking analysis: Flexera, FOSSA, GitLab, JFrog, Snyk, Sonatype, Synopsys, Veracode, WhiteHat Security and WhiteSource.

Each of these vendors has:

1. A full-fledged enterprise-class SCA tool. All of these vendors offer a number of SCA capabilities that are suitable for information security professionals. Participating manufacturers were required to offer most of the following out-of-the-box functionality:
   
   
   • the ability to provide recommendations to address licensing risks and vulnerabilities in open source software (open source),
   • ability to integrate into SDLC automation tools,
   • provide proactive vulnerability management,
   • ability to edit and create policies,
   • the ability to provide clear risk reports in open source software.
2. Annual revenue in excess of $ 10 million just from SCA offers.
3. The presence or interest of Forrester customers. In other cases, by decision of Forrester, the participating vendor could guarantee its inclusion in the list due to the technical characteristics of its products and significant presence in the market.

The Forrester Wave ™ report highlights Leaders, Strong Performers, Contenders, and Challengers. The report is an assessment of the leading vendors in the market and does not constitute a complete landscape of vendors. More market information can be obtained from other Forrester reports on the SCA ^link .

Method Forrester Wave

Forrester Wave reports are a guide for customers looking at software acquisition options in the technology market. To offer an equal process for all participants in the comparison, Forrester follows the Forrester Wave ™ Methodology Guide methodology when evaluating participating vendors.

The report conducts a primary research to form a list of estimated manufacturers. The initial pool of vendors is narrowed down based on the selection criteria and is included in the final list. Then, detailed information about the products and strategies is collected through questionnaires, demonstrations / briefings, customer reviews / interviews. These inputs, along with expert research and market knowledge, are used to evaluate vendors in a rating system in which each participant is compared with others.
Forrester clearly indicates the publication date (quarter and year) of Forrester Wave in the title of each report. Evaluation of the vendors participating in this study Forrester Wave, made on the basis of materials provided by Forrester to 01/28/2019, additional materials after this time were not accepted.

Before publishing the report, to assess the accuracy of the study, Forrester asks vendors to study the results of the study. The manufacturers listed as non-participating in the Forrester Wave chart met the criteria and could have been included in the report but refused to participate or only partially participated in the study. They are evaluated in accordance with the Forrester Wave ™ policy and the Forrester New Wave ™ Nonparticipating and Incomplete Participation Vendor Policy, their position is published along with the position of the participating vendors.

Comparison Overview

1. Current vendor offer
   
   The position of each vendor on the vertical axis of a Forrester Wave chart indicates the attractiveness of the current offer. Key criteria for such solutions are license risk management, vulnerability assessment actions, proactive vulnerability management, policy management, SDLC integration, container scanning and serverless scanning, as well as built-in strategic reporting capabilities.
2. Strategy
   
   The location on the horizontal axis indicates the stability of the vendor strategy. Forrester evaluated the product strategy, the approach to market promotion, the roadmap and the availability of training for users and distributors.
3. Presence in the market
   
   Estimation of the presence of vendors in the market is represented by the size of a circle on the chart and reflects the volume of product installations for each vendor, the growth rate and profitability of the company.

Translator’s note: The following table contains Forrester’s expert estimates and weights, which are the basis for which the compared SCA providers are located on Forrester “waves”. We were unable to find a detailed transcript of the estimated indicators, however, it is important to note the following features of the methodology:

1. Forrester weights were set according to their importance for the subscribers (clients) of this analytical agency, so for those companies that are primarily concerned with the specific functionality of the SCA solution, the vendor rating may differ significantly from the one presented.
2. The indicator of market presence has a coefficient of zero, and is only informative in nature, which means that the total figures of this indicator affect only the size of the vendor’s point (circle), and not its vertical and horizontal position, i.e. Do not affect the rating and hit the leaders.

Vendor profiles

The analysis revealed the following strengths and weaknesses of each vendor.

Leaders

WhiteSource reduces the time it takes to eliminate risks with prioritization. WhiteSource recently introduced the ability to prioritize vulnerabilities through static scanning to see if the vulnerable part of a component is being invoked by an application. If not, the priority of the vulnerability is reduced. Another relatively recent possibility is the automatic elimination of vulnerabilities through the creation of requests (pull requests) to update a component to a version that complies with company policy.

Users appreciate the support of a large number of languages ​​and the quality of customer support, but note that the product could better visualize transitive dependencies. The WhiteSource solution has very few flaws, but the functionality for drawing up the list of components used in the project (bill of materials or BOM) does not meet expectations. WhiteSource is well suited for companies that need to scan in the very early stages of the SDLC and who are looking for a solution to prioritize and automatically eliminate threats.

Synopsys capitalizes on acquiring Black Duck by scanning a binary code and generating reports. Synopsys combined the functionality of its solution and the functionality of Black Duck (Black Duck Hub and Protecode SC). However, for a more comprehensive analysis of compliance with licensing requirements, users can also use Black Duck Protex and Black Duck. For example, you can analyze the difference between the declared licenses found, but you will need to use two different tools for this. The Synopsys solution also has another functionality for interactive and static scanning - the company recently released the Polaris platform, designed to combine all data from pre-release scanning tools.

Users rely on Synopsys code scanning products, which are fast and reliable, provide detailed recommendations for eliminating threats, but consider false positives to be high. Synopsys offers opportunities for good policy management, integration with SDLC solutions, robust proactive vulnerability management, including the project specification comparison (BOM) feature, which highlights component changes. However, Synopsys does not meet expectations when it comes to the ability to automatically fix vulnerabilities that are offered by other leading vendors. Synopsys is well suited for companies whose application development teams have clear SDLC integration requirements and need different policies for different types of applications.

Strong Performers

Snyk focuses on developer scripts for version upgrades and patch fixes. The company's goal is to allow developers to fix vulnerabilities and, as a result, not only offers the possibility of fixing them through creating update requests, but also allows you to work with custom fixes when there is no acceptable component version in the repository. Snyk also provides developers with a visualization of queries in the form of a graph that displays links and associated vulnerabilities, which helps developers understand why these or other corrections are necessary.

Users appreciated Snyk’s focus on developer needs, including simple integration into the SDLC, automatic fixing of vulnerabilities, including custom patching tools without the ability to manually correct the code, and visualization of dependencies. However, in order for the solution to fully satisfy the needs of developers, it is necessary that Snyk allows not only to detect software code vulnerabilities, but also to provide information about all versions of the used software program code components. Snyk pays great attention to the needs of developers and less to the needs of information security specialists, so as a recommendation it can be noted that it is necessary to expand the functionality of the built-in audit and report generation in Snyk. Snyk is perfect for companies that have the task to implement automatic elimination of software code vulnerabilities during its development.

Sonatype continues to develop the Nexus platform for performance improvement. Sonatype uses the Nexus repository to enrich its vulnerability base. The Sonatype research team further complements the data related to the discovered new vulnerabilities, information on how to fix them and recommendations on changing settings, updating components, as well as recommendations on changing the program code. The Nexus platform has several licenses for various functionalities — DepShield, IQ Server, Nexus Auditor, Nexus Firewall, Nexus Lifecycle, Nexus Repository, and Nexus Vulnerability Scanner; in order to get maximum efficiency from using the Nexus platform products, you need to find the right combination of these components.

Users of this solution note a low level of false positives, integration with the Nexus repository and a high level of user support. However, it also stresses that Sonatype solutions require more attention to refine the functionality for tracking transitive dependencies, as well as providing reporting forms on the results of scanning. In addition, the complexity of product users is caused by pricing and licensing policies when choosing the right solution from a wide variety of company products.

This solution will be of interest to companies that already use Nexus products, or who require a very low level of false positive scan results.

Contenders (Rivals)

WhiteHat Security offers an SCA solution without manual intervention for maximum speed. WhiteHat Security is known for their specialists, in order to reduce the level of false positive scan results, analyze the scan results before sending the verdict to the user. WhiteHat is now able to offer a fully automated solution using Sentinel SCA Essentials in addition to WhiteHat Sentinel SCA Standard, which is still verified by the security team.

The WhiteHat Sentinel Essentials solution is new, so users may experience minor flaws. The shortcomings of the role-based access system, difficulties in identifying vulnerable dependencies are noted, but at the same time the detailing of information about vulnerabilities is highly appreciated. WhiteHat Security continues to actively work on improving its product, which is positioned as a universal scanner (SCA, SAST, DAST).

WhiteHat Security is best suited to companies whose developers differ in maturity level: some require speed and have enough experience to independently use the capabilities offered by the product, others need additional help through manual viewing of the results of the operation by the vendor’s security team.

Flexera is different from its competitors in security research. Secunia, the Flexera Research Group, conducts an early exploration of new vulnerabilities, providing Flexera users with information about vulnerabilities and how to deal with them before they are officially recorded in the National Vulnerability Database (NVD) vulnerabilities database. The success of this group is measured on the basis of accuracy and response time, all vulnerability reports were received in NVD and published in it. Vulnerabilities that Secunia detects are displayed in Flexera FlexNet Code Insight and are identified as Secunia found.
Although users of Flexera confirm the flexibility of the user interface and useful automation functionality. At the same time, they also report that the documentation and training in working with the system are not very informative, and they need to be supplemented with the level of technical support service, as well as system maintenance during implementation, and the implemented API has limited functionality.

Veracode provides separate use of its two products. In 2018, Veracode acquired SourceClear to complement its Veracode Software Composition Analysis solution. SourceClear Software Composition Analysis is an agent-based scanning tool, and Veracode Software Composition Analysis is a cloud-based SaaS solution. The functionality of these products has not yet been merged - for example, SCA is possible using the SaaS solution, but not using agent scanning. Veracode also offers static and dynamic pre-release scans as an addition to its SCA products.

Until Veracode completely combines SourceClear and Software Composition Analysis, users will experience inconveniences when purchasing and renewing product licenses, with delays in updating the functionality and different levels of API support. Nevertheless, the vendor continues to work on the integration of its solutions, and there is an expectation that Veracode will work to correct the remaining differences between the products, especially over the joint support of languages ​​and the common engine of policies.

Veracode is best suited for companies that already use Veracode products and want to limit the number of security vendors. Current Veracode users will appreciate the prospect of this platform for secure application development, in which SCA data complements other scan data from Veracode products.

Challengers (Challengers)

GitLab quickly launched its solution to the market, but focused on solving the problems of developers, paying less attention to the problems of information security specialists. GitLab has been offering security products since 2017, and now static and dynamic code analysis has been added to the functionality, in addition to binary SCA. However, some of the SCA features targeted at developers will be inconvenient for information security specialists. For example, the ignore feature allows developers to ignore vulnerabilities of any severity level. This forces the information security specialists to carefully monitor which vulnerabilities the developers chose to ignore. In addition, GitLab is prone to not slow down the assembly with a quality check at the release stage. Instead, he recommends using the browsing feature, which forces security specialists to manually view the status of each assembly.

GitLab actively and quickly implemented and continues to develop security functionality. However, many of these features are still in their infancy or at the planning stage. In evaluating the solution, users emphasized the lack of support for a large number of languages, the unsustainable detection of vulnerabilities, and basic policy management capabilities.

Paying attention to GitLab is worth it when most, if not the entire development team use it as a GitLab open source repository.

FOSSA improves the product with open source identification and proprietary licenses. FOSSA Compliance can automatically detect a raw copyright title, as well as distinguish between a private code, a third party code, and a copyrighted external code. In addition to this discovery technology, FOSSA claims to work with legal advisors with expertise in open source licensing. FOSSA is an open source solution that allows users to improve product performance and add support for new languages ​​and platforms.

Professionals working with this tool love the vulnerability assessment offered by FOSSA during assembly, and they feel that the results of the vulnerability scan can be trusted. Some FOSSA users publish scan results and cite FOSSA as the source. FOSSA itself is an open source project, so it’s best to treat it as a toolkit, in which, and this is confirmed by users, poor documentation and advanced functionality, such as container scanning. A detailed long-term roadmap is difficult to implement for an open source project, as FOSSA cannot anticipate when external project participants will create new functionality.

FOSSA should be considered if you have the desire and the ability to invest resources in the refinement and customization of the SCA product to the specific requirements of the company.

JFrog is limited to scanning a binary code located in its Artifactory repository. Using JFrog XRay, you can determine with a high degree of detail what is scanned inside a binary code, using the ability to select objects for observation (watches), and then correlate policies with what needs to be scanned. Policies can be applied to the entire binary code and to assemblies from various JFrog repositories.

Users rated JFrog as average and noted lack of functional flexibility, non-intuitive interface and limitations in reporting. The advantages were noted: good integration with JFrog Artifactory, the rapid speed of developing new features and fixing the problems found.

JFrog XRay should be considered when the JFrog Artifactory is already in use and automatic threat elimination is required.

Translator's note.
In conclusion, I would like to note the difference between the views of Forrester and Gartner on the source code security analysis tools market: if Forrester considers SAST and DAST functionality a useful extension of SCA products, then Gartner in its research of Magic Quadrant for Application Security Testing considers solutions in this market, on the other hand, it considers the SCA functionality important, but optional for SAST and DAST.
What do you think about this? Do you believe that the SCA market will remain a separate direction (and, for example, competition will evolve towards an increase in the number of supported integrations with third-party systems) or do you consider buying niche SCA-market players more affluent * AST solutions as a more likely scenario?

Translation authors:
Vyacheslav Vovkogon, Cross Technologies Inc.
Daria Oreshkina, Web Control