Congratulations to the programmers on their professional day! In connection with this holiday, our company Positive Technologies decided to talk about its activities directly related to the development, namely, open source and GitHub.

Recently, more and more companies, such as Google, Microsoft, Facebook, JetBrains, spread the source code of both small and large projects in open access. Positive Technologies is famous not only for its high-class information security specialists, but also for its large number of professional developers. This allows it to also contribute to the development of the Open Source movement.

PT has the following GitHub organizations supporting the company's open projects:

• Positive Technologies
• Open DevOps Community
• Positive Research
• Positive JS
• Libprotection

We described in detail the first organization with its projects and briefly all the rest.

Content

• Organizations
  
  • Positive Technologies
  • Open DevOps Community
  • Positive Research
  • Positive JS
  • Libprotection
• Projects
  
  • PT.PM
  • Grammar ANTLR
  • PT.SourceStats
  • Aspxparser
  • FP Community Rules
  • Demonstration projects
• Conclusion

Organizations

Positive Technologies

The main community in which the development of both initially open projects and those that were previously developed exclusively within the company is underway. It also hosts training and demonstration projects.

Open DevOps Community

The goal of the community is to create open, ready-made solutions for managing the complete cycle of the development, testing and related processes, as well as the delivery, deployment and licensing of products.

At the moment, the community is in its initial stages of development, but now you can find some useful tools written in Python in it. Yes, we love him.

Active projects:

1. crosspm is a universal package manager that allows you to download packages for multi-component product assemblies using the rules specified in the manifest.
2. vspheretools is a tool that allows you to manage virtual machines on vSphere right from the console. It is also possible to connect it as an API library in your Python scripts.
3. YouTrack Python 3 Client Library - A Python client for working with the YouTrack API.
4. TFS API Python client - Python client for working with Team Foundation Server API from Microsoft.
5. A Python client for Artifactory is a Python client for working with the Artifactory binary data storage API.
6. FuzzyClassificator is a universal neuro - fuzzy classifier of arbitrary objects, whose properties can be evaluated on a fuzzy measuring scale.

Each tool has an automatic assembly in Travis CI with a display in the PyPI repository, where they can be found and installed via a standard pip install.

Several more tools are being prepared for publication:

1. CrossBuilder is a system for organizing cross-platform builds as a code, a la Travis CI, but independent of the CI system used (TeamCity, Jenkins, GitLab-CI, etc.).
2. ChangelogBuilder is a release notes generator with a description of changes in a product that receives and aggregates data from various trackers (TFS, YouTrack, GitLab, etc.).
3. polyglot.sketchplugin is a plugin for the Sketch system, which designers use to simplify working with the multilingual layout.

Everyone is welcome as contributors to any instrument. We have a sample project ExampleProject , which contains the general structure and detailed instructions for creating your own project in the community. In fact, it is enough to copy it and make your project by analogy. If you have ideas or tools to automate anything, let's share them with the community under a MIT license! It is fashionable, honorable, prestigious :)

Positive Research

The repository, which publishes some research findings, presentations, utilities (for example, for determining vulnerabilities), signatures and methods for determining attacks.

• Pentest-Detections is a utility that allows you to quickly scan the network (supports IPv4 and IPv6) and identify vulnerabilities that exploit WannaCry, NotPetya.
• unME11 - tools to decode the latest versions of Intel ME 11.x.
• Bad_Tuesday_Cryptor_SIEM - MaxPatrol SIEM package for dealing with NotPetya.
• me-disablement - methods to disable Intel ME. The repository is the old method. About the new method with the High Assurance Platform (HAP) can be read on the habr in the article Turn off Intel ME 11, using undocumented mode .

AttackDetection

In this repository, the attack detection team lays out the rules for determining the exploitation of vulnerabilities using intrusion detection systems Snort and Suricata IDS . The main goal of the project is to create rules for vulnerabilities that are widespread and high level of danger (high impact). The repository contains files for integration with oinkmaster - a script for updating and deploying rules in the specified IDS. And for the test of the rules themselves are attached pcap-files with traffic. It is worth noting that the repository has already scored over 100 additions to favorites, and over the year added about 40 new vulnerabilities, including BadTunnel, ETERNALBLUE, ImageTragick, EPICBANANA, SambaCry. All announcements about new threats are published on Twitter .

Positive JS

The tooling development community (primarily the web) used in PT products.

Libprotection

An organization that brings together members of the Positive Development User Group community,
working on LibProtection library implementations for various languages ​​and platforms.
This library provides developers with a secure string handling facility.
and takes over the entire routine of sanitizing input data and automated protection
applications from injection attacks. In more detail the principles of work and functional
LibProtection opportunities were described in Vladimir Kochetkov's report “Winning injections” .

Projects

PT.PM

PT Pattern Matching Engine is a universal signature code analyzer that accepts custom patterns that are described in a special language as input. This engine is used in a free tool for checking web applications for the presence of vulnerable components of Approof , as well as in the source analyzer PT Application Inspector .

The analysis process consists of several stages:

1. Parsing source code to parse tree.
2. Convert a tree into a unified format.
3. Map the tree with custom patterns.

The approach implemented in the project makes it possible to unify the task of developing templates for different languages.

PT.PM introduces continuous integration, supports building and testing project modules for both Windows and Linux (Mono). The development process is organized using tagged tasks (Issues) and pull requests. Along with the development, the project documentation is maintained, and the results of all significant assemblies are published in the format of both NuGet packages and “raw” artifacts . The organization of PT.PM can probably be considered exemplary, to which I would like to strive for in all other projects.

For the first stage, namely, source code parsing, ANTLR based parsers are used. This tool generates them for various languages ​​(runtimes) based on formal grammars for which there is a repository. Our company is actively developing it. Currently, generation under Java, C #, Python 2 and 3, JavaScript, C ++, Go, and Swift is supported, with support for the last three added recently.

It is worth noting that ANTLR is used not only in the PT projects of the Application Security direction, but also in Max Patrol SIEM : there it is used to process the DSL (Domain Specific Language) language, which is used to describe dynamic asset groups. The exchange of experience in this area made it possible not to waste time on tasks that have already been solved earlier.

Grammar ANTLR

With the participation of Positive Technologies, grammars for PL / SQL, T-SQL, MySQL, PHP, Java 8 and C # have been developed and improved.

PL / SQL

SQL grammar have extensive syntax with lots of keywords. Fortunately, the PL / SQL grammar existed under ANTLR 3 and porting it under ANTLR 4 was not very difficult.

T-sql

No decent parsers were found for T-SQL, not to mention open ones, and we have long and painstakingly restored the grammar from the MSDN documentation. However, the result was decent: it already covers many common syntactic structures, looks neat, runtime-independent and covered with tests (examples of SQL queries from the same MSDN). Since 2015, more than 15 third-party users have contributed to it. Moreover, this grammar is now already used in DBFW , a firewall prototype at the level of database management systems, the PT Application Firewall subproject. Denis Kolegov with Arseny Reutov told about him on PHDays VII: “How to develop DBFW from scratch” .

Mysql

The grammar developed by the above team, primarily Ivan Khudyashov and Denis Kolegov, based on T-SQL. It is also used in DBFW.

Php

This grammar was translated from Bison grammar to ANTLR. It is interesting because it supports parsing immediately PHP, JavaScript and HTML. More precisely, sections of JavaScript and HTML code are parsed into text that is later processed by parsers specifically for these languages.

Java

Grammar with Java 8 support was developed quite recently. The grammar of the previous seventh version was taken as a basis. However, the new test cases with different syntax (AllInOne7.java, AllInOne8.java) have been significantly expanded and improved, and the results of performance tests for popular Java projects (jdk8, Spring Framework, Elasticsearch, etc.) have been added.

Javascript

Built on the old ECMAScript grammar without exacting standard compliance. When developing grammars, we are primarily guided by practical application and comprehensibility, not formal compliance. Another significant difference is the almost complete support for ECMAScript 6, as well as outdated constructions (Html ​​comments, CDATA sections).

Not all syntactic constructions can be described only with the help of grammar rules. In some situations it is convenient and necessary to use code inserts in the target runtime language. For example, in JavaScript the get token in some cases is a regular identifier, and in others it is a keyword describing the property getter . Those. This token can be parsed as a common identifier, and in the parser you can check the value of the token when processing the property:

 getter : Identifier{p("get")}? propertyName ; 

This grammar is interesting in that these code inserts are universal , at least for C # and Java runtimes, thanks to the
superClass .

Those. In C # code, the p("get") function is described in the parent class of JavaScriptBaseParser.cs:

 protected bool p(string str) => _input.Lt(-1).Text.Equals(str); 

In Java, this function looks like this (JavaScriptBaseLexer.java):

 protected boolean p(String str) { return _input.LT(-1).getText().equals(str); } 

C #

This is mostly an experimental grammar created for comparing the speeds of parsers based on ANTLR and Roslyn parser.

Development and prospects

Details of the development of grammars can be found in our last year’s article “Theory and practice of source code parsing using ANTLR and Roslyn” .

As can be seen from the change history and a large number of accepted Pull Request ( tsql , plsql , mysql ), these grammars are refined not only by the efforts of Positive Technologies, but also by a large number of third-party developers. During this cooperation, the repository has grown not only quantitatively but also qualitatively.

PT.SourceStats

Allows you to collect statistics for projects in various programming languages ​​and is used in the free product Approof.

Aspxparser

As part of this project, an ASPX page parser is being developed, which is used not only in the open PT.PM engine, but also in the internal analyzer of .NET applications ( AI.Net ) based on abstract code interpretation.

FP Community Rules

The repository is developing rule sets in the YARA format, which are used in the signature analysis module of projects in Approof. In August last year, within the framework of PDUG (user group for safe development), Alexei Goncharov made a report on the FingerPrint module used in PT AI and Approof.

The FingerPrint engine runs on the set of source codes for the site (backend, frontend) and, in accordance with the described rules, YARA searches for known versions of third-party components (for example, the bla-bla library version 3). The rules are designed so that they contain signatures of vulnerable versions of libraries with a textual description of the problem.

A rule consists of several conditions for checking a file. For example, the condition of the presence of certain lines in the file. If the file satisfies them, then the Approof in the final report provides information about the detected vulnerabilities in a particular component with version N, as well as descriptions of CVEs related to them.

Read more about this in the article by Denis Efremov (ISP RAS) “Developing rules for Approof” . See also his report “Automating the construction of rules for Approof” on the PDUG section of PHDays.

Demonstration projects

A master class “Appsec Outback” was held at PHDays VII as part of PDUG. Educational-demo versions of the static code analyzer Mantaray and the firewall Schockfish were developed for it. These projects have all the basic mechanisms that are used in real remedies. But, unlike the latter, their main goal is to demonstrate the algorithms and methods of protection, to help understand the process of analyzing and protecting applications, as well as to illustrate the fundamental theoretical possibilities and limitations of technologies.

Also in the repository there are examples of the implementation of protection mechanisms:

• DOMSanitizer is a module for detecting XSS attacks on the side of a web browser.
• DOMParanoid is a security linter module for checking HTML security.

License

In our projects, we use both licensing licenses (MIT, Apache) and our own, which implies free use exclusively for non-commercial purposes.

Conclusion

The process of moving to GitHub turned out to be useful and gave us experience in various areas - in configuring DevOps under Windows and Linux, writing documentation, in development.

Positive Technologies is developing Open Source projects and plans to expand this activity.