Source Ripper and AST Spring Boot Tree
Recently I came across an unusual task of accessing code comments in runtime.
This will help catch 3 hares immediately - in addition to the documentation in the project code, it will be easy to generate a sequence diagram from the project tests that analysts can read, and QA will be able to compare their test plans with project autotests and supplement them if necessary. There is a common language in the team between those who write the code and those who can not read it. As a result, a better understanding of the project by everyone in the development process, and from the point of view of the developer, there is no need to draw anything manually - the code and tests are primary. There are more chances that such documentation will be the most relevant on the project, since it is generated from working code. At the same time disciplines the developer to document the classes and methods that participate in the charts.
')
In this publication I will tell how to extract javadoc from the project source code.
Of course, before writing my code, I remembered that the best code was already written and tested by the community. I started looking for what exists to work with javadoc at runtime and how convenient it will be to use it for my task. Searches led to the project
therapi-runtime-javadoc . By the way, the project is alive and developing and allows you to work in runtime with the comments of the source classes. The library works as AnnotationProcessor when compiling and it is quite convenient. But there is one feature that does not allow using it without fear with a real code that will go into actual operation in the future - this is what it modifies the source bytecode of the classes, adding to it the meta information from the comments. You also need to recompile the code and add the @RetainJavadoc annotation, which does not work for project dependencies. Sorry, the solution at first glance seemed perfect.
It was also important for me to hear the opinion from the side. After talking with a fairly vigorous developer and listening to his thoughts, as if solving this problem, he suggested parsing HTML javadoc. It would be nice to work as in the central maven repository there are javadoc archives for artifacts, but for me it’s not a particularly elegant decision to gut the generated documentation when I have the source code. Although a matter of taste ...
It seems to me more appropriate a way to extract documentation from the source code, moreover, much more information is available in AST than in HTML documentation based on the same source code. There was experience and preparations for this approach, about which I once told in the publication “Analysis of Java programs using java programs”
So the extract-javadoc project was born, which is available as a ready-made assembly in maven central com.github.igor-suhorukov: extract-javadoc: 1.0 .
If we discard unremarkable parts of the program for working with the file system, saving javadoc as a JSON file, parallelizing parsing and working with the contents of jar and zip archives, the very stuffing of the project begins in the parseFile method of the com.github.igorsuhorukov.javadoc.ExtractJavadocModel class.
Initializing the ECJ parser of java files and extracting javadoc look like this:
The main work on parsing javadoc and mapping it on the internal model, which later becomes serialized in JSON, occurs in JavadocVisitor:
In the com.github.igorsuhorukov.javadoc.parser.JavadocVisitor # visit (PackageDeclaration) method, javadoc is currently processed only for types and their methods. I need this information to build a sequence of diagrams with comments.
Working with the AST program for the task of extracting documentation from source turned out to be not as difficult as it seemed at first. And I was able to develop a more or less universal solution while I had a break between work and I relax, coding 3-4 hours at a time for a couple of days.
For a maven project, it's easy to add javadoc extraction to all project modules, adding
So an additional artifact appears in the project, which contains javadoc in json format and it gets into the repository during install / deploy.
Also, there should not be a problem to integrate this solution into the Gradle assembly, since this is a normal console application that passes two parameters to the input - the path to the sources and the file where the javadoc is written in JSON format or compressed json, if the path ends with ".xz"
Guinea pig now will be the Spring Boot project as a fairly large project with excellent javadoc documentation.
Run the command:
And add to the file spring-boot-parent / pom.xml in the profiles tag,
After that, we will build the project, in the process for all java files from Spring Boot, the AST tree is built and the javadoc types and methods are extracted . The javadoc.json file appears in the target directories of the modules containing java sources. But the more processor cores in your system, the more memory will be required for parsing, so you may have to increase the max heap size in the .mvn / jvm.config file
As an example, the file spring-boot-tools / spring-boot-antlib / target / javadoc.json is created.
You can turn the javadoc model from JSON back into the object model and work with it in the program by calling com.github.igorsuhorukov.javadoc.ReadJavaDocModel # readJavaDoc to the method you need to pass the path to the JSON file with javadoc (or json compressed in .xz format) .
How to work with the model, I will describe in the following publications about the generation of sequence diagram from tests
This will help catch 3 hares immediately - in addition to the documentation in the project code, it will be easy to generate a sequence diagram from the project tests that analysts can read, and QA will be able to compare their test plans with project autotests and supplement them if necessary. There is a common language in the team between those who write the code and those who can not read it. As a result, a better understanding of the project by everyone in the development process, and from the point of view of the developer, there is no need to draw anything manually - the code and tests are primary. There are more chances that such documentation will be the most relevant on the project, since it is generated from working code. At the same time disciplines the developer to document the classes and methods that participate in the charts.
')
In this publication I will tell how to extract javadoc from the project source code.
Of course, before writing my code, I remembered that the best code was already written and tested by the community. I started looking for what exists to work with javadoc at runtime and how convenient it will be to use it for my task. Searches led to the project
therapi-runtime-javadoc . By the way, the project is alive and developing and allows you to work in runtime with the comments of the source classes. The library works as AnnotationProcessor when compiling and it is quite convenient. But there is one feature that does not allow using it without fear with a real code that will go into actual operation in the future - this is what it modifies the source bytecode of the classes, adding to it the meta information from the comments. You also need to recompile the code and add the @RetainJavadoc annotation, which does not work for project dependencies. Sorry, the solution at first glance seemed perfect.
It was also important for me to hear the opinion from the side. After talking with a fairly vigorous developer and listening to his thoughts, as if solving this problem, he suggested parsing HTML javadoc. It would be nice to work as in the central maven repository there are javadoc archives for artifacts, but for me it’s not a particularly elegant decision to gut the generated documentation when I have the source code. Although a matter of taste ...
It seems to me more appropriate a way to extract documentation from the source code, moreover, much more information is available in AST than in HTML documentation based on the same source code. There was experience and preparations for this approach, about which I once told in the publication “Analysis of Java programs using java programs”
So the extract-javadoc project was born, which is available as a ready-made assembly in maven central com.github.igor-suhorukov: extract-javadoc: 1.0 .
Ripper javadoc "under the hood"
If we discard unremarkable parts of the program for working with the file system, saving javadoc as a JSON file, parallelizing parsing and working with the contents of jar and zip archives, the very stuffing of the project begins in the parseFile method of the com.github.igorsuhorukov.javadoc.ExtractJavadocModel class.
Initializing the ECJ parser of java files and extracting javadoc look like this:
public static List<JavaDoc> parseFile(String javaSourceText, String fileName, String relativePath) { ASTParser parser = parserCache.get(); parser.setSource(javaSourceText.toCharArray()); parser.setResolveBindings(true); parser.setEnvironment(new String[]{}, SOURCE_PATH, SOURCE_ENCODING, true); parser.setKind(ASTParser.K_COMPILATION_UNIT); parser.setCompilerOptions(JavaCore.getOptions()); parser.setUnitName(fileName); CompilationUnit cu = (CompilationUnit) parser.createAST(null); JavadocVisitor visitor = new JavadocVisitor(fileName, relativePath, javaSourceText); cu.accept(visitor); return visitor.getJavaDocs(); } The main work on parsing javadoc and mapping it on the internal model, which later becomes serialized in JSON, occurs in JavadocVisitor:
package com.github.igorsuhorukov.javadoc.parser; import com.github.igorsuhorukov.javadoc.model.*; import com.github.igorsuhorukov.javadoc.model.Type; import org.eclipse.jdt.core.dom.*; import java.util.ArrayList; import java.util.List; import java.util.Objects; import java.util.Optional; import java.util.stream.Collectors; public class JavadocVisitor extends ASTVisitor { private String file; private String relativePath; private String sourceText; private CompilationUnit compilationUnit; private String packageName; private List<? extends Comment> commentList; private List<JavaDoc> javaDocs = new ArrayList<>(); public JavadocVisitor(String file, String relativePath, String sourceText) { this.file = file; this.relativePath = relativePath; this.sourceText = sourceText; } @Override public boolean visit(PackageDeclaration node) { packageName = node.getName().getFullyQualifiedName(); javaDocs.addAll(getTypes().stream().map(astTypeNode -> { JavaDoc javaDoc = getJavaDoc(astTypeNode); Type type = getType(astTypeNode); type.setUnitInfo(getUnitInfo()); javaDoc.setSourcePoint(type); return javaDoc; }).collect(Collectors.toList())); javaDocs.addAll(getMethods().stream().map(astMethodNode -> { JavaDoc javaDoc = getJavaDoc(astMethodNode); Method method = new Method(); method.setUnitInfo(getUnitInfo()); method.setName(astMethodNode.getName().getFullyQualifiedName()); method.setConstructor(astMethodNode.isConstructor()); fillMethodDeclaration(astMethodNode, method); Type type = getType((AbstractTypeDeclaration) astMethodNode.getParent()); method.setType(type); javaDoc.setSourcePoint(method); return javaDoc; }).collect(Collectors.toList())); return super.visit(node); } private CompilationUnitInfo getUnitInfo() { return new CompilationUnitInfo(packageName, relativePath, file); } @SuppressWarnings("unchecked") private void fillMethodDeclaration(MethodDeclaration methodAstNode, Method method) { List<SingleVariableDeclaration> parameters = methodAstNode.parameters(); org.eclipse.jdt.core.dom.Type returnType2 = methodAstNode.getReturnType2(); method.setParams(parameters.stream().map(param -> param.getType().toString()).collect(Collectors.toList())); if(returnType2!=null) { method.setReturnType(returnType2.toString()); } } private Type getType(AbstractTypeDeclaration astNode) { String binaryName = astNode.resolveBinding().getBinaryName(); Type type = new Type(); type.setName(binaryName); return type; } @SuppressWarnings("unchecked") private JavaDoc getJavaDoc(BodyDeclaration astNode) { JavaDoc javaDoc = new JavaDoc(); Javadoc javadoc = astNode.getJavadoc(); List<TagElement> tags = javadoc.tags(); Optional<TagElement> comment = tags.stream().filter(tag -> tag.getTagName() == null).findFirst(); comment.ifPresent(tagElement -> javaDoc.setComment(tagElement.toString().replace("\n *","").trim())); List<Tag> fragments = tags.stream().filter(tag -> tag.getTagName() != null).map(tag-> { Tag tagResult = new Tag(); tagResult.setName(tag.getTagName()); tagResult.setFragments(getTags(tag.fragments())); return tagResult; }).collect(Collectors.toList()); javaDoc.setTags(fragments); return javaDoc; } @SuppressWarnings("unchecked") private List<String> getTags(List fragments){ return ((List<IDocElement>)fragments).stream().map(Objects::toString).collect(Collectors.toList()); } private List<AbstractTypeDeclaration> getTypes() { return commentList.stream().map(ASTNode::getParent).filter(Objects::nonNull).filter(AbstractTypeDeclaration.class::isInstance).map(astNode -> (AbstractTypeDeclaration) astNode).collect(Collectors.toList()); } private List<MethodDeclaration> getMethods() { return commentList.stream().map(ASTNode::getParent).filter(Objects::nonNull).filter(MethodDeclaration.class::isInstance).map(astNode -> (MethodDeclaration) astNode).collect(Collectors.toList()); } @Override @SuppressWarnings("unchecked") public boolean visit(CompilationUnit node) { commentList = node.getCommentList(); this.compilationUnit = node; return super.visit(node); } public List<JavaDoc> getJavaDocs() { return javaDocs; } } In the com.github.igorsuhorukov.javadoc.parser.JavadocVisitor # visit (PackageDeclaration) method, javadoc is currently processed only for types and their methods. I need this information to build a sequence of diagrams with comments.
Working with the AST program for the task of extracting documentation from source turned out to be not as difficult as it seemed at first. And I was able to develop a more or less universal solution while I had a break between work and I relax, coding 3-4 hours at a time for a couple of days.
How to extract javadoc in a real project
For a maven project, it's easy to add javadoc extraction to all project modules, adding
in the parent pom.xml project next profile
<profile> <id>extract-javadoc</id> <activation> <file> <exists>${basedir}/src/main/java</exists> </file> </activation> <build> <plugins> <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>exec-maven-plugin</artifactId> <version>1.6.0</version> <executions> <execution> <id>extract-javadoc</id> <phase>package</phase> <goals> <goal>java</goal> </goals> </execution> </executions> <configuration> <includeProjectDependencies>true</includeProjectDependencies> <includePluginDependencies>true</includePluginDependencies> <mainClass>com.github.igorsuhorukov.javadoc.ExtractJavadocModel</mainClass> <arguments> <argument>${project.basedir}/src</argument> <argument>${project.build.directory}/javadoc.json.xz</argument> </arguments> </configuration> <dependencies> <dependency> <groupId>com.github.igor-suhorukov</groupId> <artifactId>extract-javadoc</artifactId> <version>1.0</version> <type>jar</type> <scope>compile</scope> </dependency> </dependencies> </plugin> <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>build-helper-maven-plugin</artifactId> <version>3.0.0</version> <executions> <execution> <id>attach-extracted-javadoc</id> <phase>package</phase> <goals> <goal>attach-artifact</goal> </goals> <configuration> <artifacts> <artifact> <file>${project.build.directory}/javadoc.json.xz</file> <type>xz</type> <classifier>javadoc</classifier> </artifact> </artifacts> </configuration> </execution> </executions> </plugin> </plugins> </build> </profile> So an additional artifact appears in the project, which contains javadoc in json format and it gets into the repository during install / deploy.
Also, there should not be a problem to integrate this solution into the Gradle assembly, since this is a normal console application that passes two parameters to the input - the path to the sources and the file where the javadoc is written in JSON format or compressed json, if the path ends with ".xz"
Guinea pig now will be the Spring Boot project as a fairly large project with excellent javadoc documentation.
Run the command:
git clone https://github.com/spring-projects/spring-boot.git And add to the file spring-boot-parent / pom.xml in the profiles tag,
our profile tag
<profile> <id>extract-javadoc</id> <activation> <file> <exists>${basedir}/src/main/java</exists> </file> </activation> <build> <plugins> <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>exec-maven-plugin</artifactId> <version>1.6.0</version> <executions> <execution> <id>extract-javadoc</id> <phase>package</phase> <goals> <goal>java</goal> </goals> </execution> </executions> <configuration> <includeProjectDependencies>true</includeProjectDependencies> <includePluginDependencies>true</includePluginDependencies> <mainClass>com.github.igorsuhorukov.javadoc.ExtractJavadocModel</mainClass> <arguments> <argument>${project.basedir}/src</argument> <argument>${project.build.directory}/javadoc.json</argument> </arguments> </configuration> <dependencies> <dependency> <groupId>com.github.igor-suhorukov</groupId> <artifactId>extract-javadoc</artifactId> <version>1.0</version> <type>jar</type> <scope>compile</scope> </dependency> </dependencies> </plugin> <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>build-helper-maven-plugin</artifactId> <version>3.0.0</version> <executions> <execution> <id>attach-extracted-javadoc</id> <phase>package</phase> <goals> <goal>attach-artifact</goal> </goals> <configuration> <artifacts> <artifact> <file>${project.build.directory}/javadoc.json</file> <type>json</type> <classifier>javadoc</classifier> </artifact> </artifacts> </configuration> </execution> </executions> </plugin> </plugins> </build> </profile> After that, we will build the project, in the process for all java files from Spring Boot, the AST tree is built and the javadoc types and methods are extracted . The javadoc.json file appears in the target directories of the modules containing java sources. But the more processor cores in your system, the more memory will be required for parsing, so you may have to increase the max heap size in the .mvn / jvm.config file
As an example, the file spring-boot-tools / spring-boot-antlib / target / javadoc.json is created.
[ { "comment" : "Ant task to find a main class.", "tags" : [ { "name" : "@author", "fragments" : [ " Matt Benson" ] }, { "name" : "@since", "fragments" : [ " 1.3.0" ] } ], "sourcePoint" : { "@type" : "Type", "unitInfo" : { "packageName" : "org.springframework.boot.ant", "relativePath" : "main/java/org/springframework/boot/ant", "file" : "FindMainClass.java" }, "name" : "org.springframework.boot.ant.FindMainClass" } }, { "comment" : "Set the main class, which will cause the search to be bypassed.", "tags" : [ { "name" : "@param", "fragments" : [ "mainClass", " the main class name" ] } ], "sourcePoint" : { "@type" : "Method", "unitInfo" : { "packageName" : "org.springframework.boot.ant", "relativePath" : "main/java/org/springframework/boot/ant", "file" : "FindMainClass.java" }, "type" : { "@type" : "Type", "unitInfo" : null, "name" : "org.springframework.boot.ant.FindMainClass" }, "name" : "setMainClass", "constructor" : false, "params" : [ "String" ], "returnType" : "void" } }, { "comment" : "Set the root location of classes to be searched.", "tags" : [ { "name" : "@param", "fragments" : [ "classesRoot", " the root location" ] } ], "sourcePoint" : { "@type" : "Method", "unitInfo" : { "packageName" : "org.springframework.boot.ant", "relativePath" : "main/java/org/springframework/boot/ant", "file" : "FindMainClass.java" }, "type" : { "@type" : "Type", "unitInfo" : null, "name" : "org.springframework.boot.ant.FindMainClass" }, "name" : "setClassesRoot", "constructor" : false, "params" : [ "File" ], "returnType" : "void" } }, { "comment" : "Set the ANT property to set (if left unset, result will be printed to the log).", "tags" : [ { "name" : "@param", "fragments" : [ "property", " the ANT property to set" ] } ], "sourcePoint" : { "@type" : "Method", "unitInfo" : { "packageName" : "org.springframework.boot.ant", "relativePath" : "main/java/org/springframework/boot/ant", "file" : "FindMainClass.java" }, "type" : { "@type" : "Type", "unitInfo" : null, "name" : "org.springframework.boot.ant.FindMainClass" }, "name" : "setProperty", "constructor" : false, "params" : [ "String" ], "returnType" : "void" } }, { "comment" : "Quiet task that establishes a reference to its loader.", "tags" : [ { "name" : "@author", "fragments" : [ " Matt Benson" ] }, { "name" : "@since", "fragments" : [ " 1.3.0" ] } ], "sourcePoint" : { "@type" : "Type", "unitInfo" : { "packageName" : "org.springframework.boot.ant", "relativePath" : "main/java/org/springframework/boot/ant", "file" : "ShareAntlibLoader.java" }, "name" : "org.springframework.boot.ant.ShareAntlibLoader" } } ] Reading javadoc metadata in runtime
You can turn the javadoc model from JSON back into the object model and work with it in the program by calling com.github.igorsuhorukov.javadoc.ReadJavaDocModel # readJavaDoc to the method you need to pass the path to the JSON file with javadoc (or json compressed in .xz format) .
How to work with the model, I will describe in the following publications about the generation of sequence diagram from tests
Source: https://habr.com/ru/post/336732/
All Articles