Custom refactoring tool: Swift
Any engineer seeks to make their work process as optimized as possible. We, as iOS mobile developers, very often have to work with uniform language structures. Apple is improving the tools of developers, making a lot of effort to make it convenient for us to program: highlighting the language, autocompleting methods, and many other IDE features allow our fingers to keep up with the ideas in their head.
What does an engineer do when the required tool is missing? True, he will do everything himself! Earlier we already talked about creating custom tools, now let's talk about how to modify Xcode and make it work according to your rules.
We took a tuska from JIRA Swift 'and made a tool that transforms if let into an equivalent guard let construction.
')
With version 9, Xcode provides a new refactoring mechanism that can translate code locally, within the same Swift source file, or globally, when you rename a method or property that occurs in several files, even if they are in different languages.
Local refactoring is fully implemented in the SourceKit compiler and framework, the feature is in Swift's open source repository and written in C ++. The global refactoring modification is currently inaccessible to the public, because the Xcode codebase is closed. Therefore, we dwell on the local history and tell you how to repeat our experience.
What you need to create your tool for local refactoring:
Some theoretical basics before diving into practice. Let's take a look at the work of the Swift compiler architecture. The compiler is primarily responsible for transforming the code into executable machine code.
Of the transformation stages presented, the most interesting for us is the generation of an abstract syntax tree (AST) - a graph in which the vertices are operators, and the leaves are their operands.
Syntax trees are used in the parser. AST is used as an internal representation in the compiler / interpreter of a computer program for optimizing and generating code.
After generating the AST, a parsing is performed to create an AST with a type check, which has been translated into the Swift Intermediate Language. SIL is converted, optimized, reduced to LLVM IR, which is ultimately compiled into native code.
To create a refactoring tool, we need to understand AST and be able to work with it. So the tool will be able to correctly operate with parts of the code that we want to process.
To generate an AST of a file, run the command: swiftc -dump-ast MyFile.swift
Below is the output to the AST console of the if let function, which was mentioned earlier.
There are three basic node types in Swift AST:
They correspond to three entities that are used in the Swift language itself. Names of functions, structures, parameters are declarations. Expressions are entities that return a value; for example, calling functions. Operators are parts of the language that define the control flow of code execution, but do not return a value (for example, if or do-catch).
This is a sufficient minimum that you need to know about AST for the upcoming work.
To implement refactoring, you will need specific information about the area of code that you are going to change. Developers are provided with auxiliary entities that accumulate data. The first, ResolvedCursorInfo (cursor-based refactoring), will tell you whether we are at the beginning of an expression. If so, then the corresponding compiler object of this expression is returned. The second entity, RangeInfo (range-based refactoring), encapsulates data about the source range (for example, how many entry and exit points it has).
Cursor-based refactoring is initiated by placing the cursor in the source file. Refactoring actions implement the methods that the refactoring mechanism uses to display the available actions in the IDE and to perform transformations. Examples of cursor-based actions: Jump to definition, quick help, etc.
Consider the usual actions from the technical side:
Range-based refactoring is initiated by selecting a continuous code range in the source file.
In this case, the refactoring tool will go through the same described call chain. The difference is that when implemented, the input is RangeInfo instead of ResolvedCursorInfo. Interested readers can refer to Refactoring.cpp for more details on Apple’s tool implementation examples.
And now to the practice of creating a tool.
The first step is to download and compile the Swift compiler. Detailed instructions are in the official repository ( readme.md ). Here are the key commands for cloning a code:
To describe the structure and dependencies of the project is used cmake . With it, you can generate a project for Xcode (more convenient) or under ninja (faster) by one of the commands:
or
Successful compilation requires the latest version of Xcode beta (10.2.1 at the time of this writing) - available on the official Apple website . To use a new Xcode to build a project, you need to set the path using the xcode-select utility:
If we used the --xcode flag to build the project under Xcode, then after several hours of compilation (we had a little more than two) in the build folder, we will find the Swift.xcodeproj file. Having opened the project, we will see the usual Xcode with indexing, breakpoints.
To create a new tool, we need to add the code with the tool logic to the file: lib / IDE / Refactoring.cpp and define two methods isApplicable and performChange. In the first method, we decide whether to display the refactoring option for the selected code. And in the second - how to convert the selected code to apply refactoring.
After the done preparation, it remains to implement the following steps:
This method of development will take a lot of time from you due to the long assembly of components, but the result is immediately visible in Xcode - the manual check path.
To begin with, we will assemble the toolchain Swift with the command:
Compiling the toolchain will take even more time than compiling the compiler and dependencies. At the output we get the file swift-LOCAL-yyyy-mm-dd.xctoolchain in the swift-nightly-install folder, which needs to be transferred to Xcode: / Library / Developer / Toolchains /. Next, in the IDE settings, select the new toolchain, restart Xcode.
Select the piece of code that should process the tool, and look for the tool in the context menu.
If the project was compiled under Ninja and you chose the TDD path, then developing through tests with Ninja is one of the options that suits you. Cons - you can not set breakpoints, as in the development through Xcode.
So, we need to check that the new tool is displayed in Xcode when the user selects the guard construction in the source code. We write the test in the existing file test / refactoring / RefactoringKind / basic.swift:
We indicate that when selecting a code between 266 row 3 columns and 268 row 4 columns, we expect the menu item to appear with a new tool.
Using the lit.py script can provide faster feedback to your development cycle. You can specify the test suit. In our case, this suite will be RefactoringKind:
As a result, only tests of this file will be launched. Their implementation will take a couple of tens of seconds. More about lit.py will be discussed below in the Diagnostics and Testing block.
The test fails, which is normal for the TDD paradigm. After all, so far we have not written a single line of code with the logic of the tool.
And finally, the last development method when the project was compiled under Xcode. The main advantage is the ability to set breakpoints and control debugging.
When building a project under Xcode, the file Swift.xcodeproj is created in the build / Xcode-DebugAssert / swift-macosx-x86_64 / folder. When you first open this file, it is better to choose the creation of schemas manually in order to generate ALL_BUILD and swift-refactor yourself:
Next, we build the project with ALL_BUILD once, after that we use the swift-refactor scheme.
Refactoring tool is compiled into a separate executable file - swift-refactor. Help about this file can be displayed using the –help flag. The most interesting parameters for us are:
They can be set in the schema as arguments. Now you can set breakpoints to stop in places of interest when you start the tool. In a familiar way, using the p and po commands in the Xcode console will display the values of the corresponding variables.
The isApplicable method takes as input ResolvedRangeInfo with information about the AST nodes of the selected code fragment. At the output of the method, it is decided whether or not to show the tool in the Xcode context menu. The full ResolvedRangeInfo interface can be found in the include / swift / IDE / Utils.h file .
Consider the most useful in our case, the fields of the class ResolvedRangeInfo:
To test isApplicable, add sample code to the file test / refactoring / RefactoringKind / basic.swift .
In order for the test to simulate a call to our tool, you need to add a line in the tools / swift-refactor / swift-refactor.cpp file.
This method is called when selecting a refactoring tool in the context menu. The method has access to ResolvedRangeInfo, as in isApplicable. Use ResolvedRangeInfo and write the logic of the code conversion tool.
When generating code for static tokens (regulated by the syntax of the language), entities from the namespace tok can be used. For example, for the guard keyword, we use tok :: kw_guard. For dynamic tokens (modified by the developer, for example, the name of the function) you need to select them from the array of AST elements.
To determine where the inserted code is inserted, we use the full selected range using the RangeInfo.ContentRange construction.
Before you finish work on the tool, you need to check the correctness of his work again. This will help us again tests. Tests can be run one at a time or with all available options. The easiest way to run the entire Swift test suite is with the --test command on utils / build-script, which will run the main test suite. Using utils / build-script rebuilds all the target, which can significantly increase the debug cycle time.
Be sure to run the utils / build-script --validation-test verification tests before making major changes to the compiler or API.
There is another way to run all compiler unit tests through ninja, ninja check-swift from build / preset / swift-macosx-x86_64. It will take about 15 minutes.
And finally, an option when you need to run the tests separately. To directly invoke the lit.py script from LLVM, it must be configured to use the local build directory. For example:
This will run the tests in the 'test / Parse /' directory for 64-bit macOS. The -sv option provides a test execution indicator and shows the results of only unsuccessful tests.
Lit.py has several other useful features, such as timing tests and latency testing. See these and other features with lit.py -h. The most useful can be found here .
To run one test we will prescribe:
If we need to tighten up the latest changes to the compiler, then we need to update all the dependencies and rebase. To upgrade, run ./utils/update-checkout.
We managed to achieve our goal - to make a tool that was not previously in the IDE to optimize work. If you also have ideas on how to improve Apple products and make life easier for the entire iOS community, feel free to tackle the content-feeding, because it’s simpler than it seems at first glance!
In 2015, Apple posted the open source Swift source codes, which made it possible to dive into the implementation details of its compiler. In addition, local refactoring tools can be added with Xcode 9. Basic C ++ knowledge and compiler devices are enough to make your favorite IDE a little more convenient.
The described experience was useful for us - in addition to creating a tool that simplifies the development process, we got a really hardcore knowledge of the language. Pandora's open box with low-level processes allows you to look at daily tasks from a new angle.
We hope that the knowledge gained will also enrich your understanding of the development!
The material was prepared in collaboration with @victoriaqb - Victoria Kashlina, iOS developer.
What does an engineer do when the required tool is missing? True, he will do everything himself! Earlier we already talked about creating custom tools, now let's talk about how to modify Xcode and make it work according to your rules.
We took a tuska from JIRA Swift 'and made a tool that transforms if let into an equivalent guard let construction.
')
With version 9, Xcode provides a new refactoring mechanism that can translate code locally, within the same Swift source file, or globally, when you rename a method or property that occurs in several files, even if they are in different languages.
Local refactoring is fully implemented in the SourceKit compiler and framework, the feature is in Swift's open source repository and written in C ++. The global refactoring modification is currently inaccessible to the public, because the Xcode codebase is closed. Therefore, we dwell on the local history and tell you how to repeat our experience.
What you need to create your tool for local refactoring:
- Understanding C ++
- Basic compiler knowledge
- Understanding what AST is and how to work with it
- Swift source code
- Hyde swift / docs / refactoring / SwiftLocalRefactoring.md
- A lot of patience
Little about AST
Some theoretical basics before diving into practice. Let's take a look at the work of the Swift compiler architecture. The compiler is primarily responsible for transforming the code into executable machine code.
Of the transformation stages presented, the most interesting for us is the generation of an abstract syntax tree (AST) - a graph in which the vertices are operators, and the leaves are their operands.
Syntax trees are used in the parser. AST is used as an internal representation in the compiler / interpreter of a computer program for optimizing and generating code.
After generating the AST, a parsing is performed to create an AST with a type check, which has been translated into the Swift Intermediate Language. SIL is converted, optimized, reduced to LLVM IR, which is ultimately compiled into native code.
To create a refactoring tool, we need to understand AST and be able to work with it. So the tool will be able to correctly operate with parts of the code that we want to process.
To generate an AST of a file, run the command: swiftc -dump-ast MyFile.swift
Below is the output to the AST console of the if let function, which was mentioned earlier.
There are three basic node types in Swift AST:
- declarations (Decl subclasses),
- expressions (subclasses of type Expr),
- operators (subclasses of type Stmt).
They correspond to three entities that are used in the Swift language itself. Names of functions, structures, parameters are declarations. Expressions are entities that return a value; for example, calling functions. Operators are parts of the language that define the control flow of code execution, but do not return a value (for example, if or do-catch).
This is a sufficient minimum that you need to know about AST for the upcoming work.
How the theory refactoring tool works
To implement refactoring, you will need specific information about the area of code that you are going to change. Developers are provided with auxiliary entities that accumulate data. The first, ResolvedCursorInfo (cursor-based refactoring), will tell you whether we are at the beginning of an expression. If so, then the corresponding compiler object of this expression is returned. The second entity, RangeInfo (range-based refactoring), encapsulates data about the source range (for example, how many entry and exit points it has).
Cursor-based refactoring is initiated by placing the cursor in the source file. Refactoring actions implement the methods that the refactoring mechanism uses to display the available actions in the IDE and to perform transformations. Examples of cursor-based actions: Jump to definition, quick help, etc.
Consider the usual actions from the technical side:
- When you select a location from the Xcode editor, a request is made to the sourcekitd (the framework responsible for highlighting, code completion, etc.) to display the available refactoring actions.
- Each available action is requested by a ResolvedCursorInfo object to check if this action is applicable to the selected code.
- A list of applicable actions is returned as a response from sourcekitd and displayed in Xcode.
- Xcode then applies the refactoring tool changes.
Range-based refactoring is initiated by selecting a continuous code range in the source file.
In this case, the refactoring tool will go through the same described call chain. The difference is that when implemented, the input is RangeInfo instead of ResolvedCursorInfo. Interested readers can refer to Refactoring.cpp for more details on Apple’s tool implementation examples.
And now to the practice of creating a tool.
Training
The first step is to download and compile the Swift compiler. Detailed instructions are in the official repository ( readme.md ). Here are the key commands for cloning a code:
mkdir swift-source cd swift-source git clone https://github.com/apple/swift.git ./swift/utils/update-checkout --clone To describe the structure and dependencies of the project is used cmake . With it, you can generate a project for Xcode (more convenient) or under ninja (faster) by one of the commands:
./utils/build-script --debug --xcode or
swift/utils/build-script --debug-debuginfo Successful compilation requires the latest version of Xcode beta (10.2.1 at the time of this writing) - available on the official Apple website . To use a new Xcode to build a project, you need to set the path using the xcode-select utility:
sudo xcode-select -s /Users/username/Xcode.app If we used the --xcode flag to build the project under Xcode, then after several hours of compilation (we had a little more than two) in the build folder, we will find the Swift.xcodeproj file. Having opened the project, we will see the usual Xcode with indexing, breakpoints.
To create a new tool, we need to add the code with the tool logic to the file: lib / IDE / Refactoring.cpp and define two methods isApplicable and performChange. In the first method, we decide whether to display the refactoring option for the selected code. And in the second - how to convert the selected code to apply refactoring.
After the done preparation, it remains to implement the following steps:
- Develop tool logic (development can be conducted in several ways - through the toolchain, through Ninja, through Xcode; all options will be described below)
- Implement two methods: isApplicable and performChange (they are responsible for accessing the tool and its work)
- Diagnose and test the finished tool before sending PR to Swift's official repository.
Check tool operation through toolchain
This method of development will take a lot of time from you due to the long assembly of components, but the result is immediately visible in Xcode - the manual check path.
To begin with, we will assemble the toolchain Swift with the command:
./utils/build-toolchain some_bundle_id Compiling the toolchain will take even more time than compiling the compiler and dependencies. At the output we get the file swift-LOCAL-yyyy-mm-dd.xctoolchain in the swift-nightly-install folder, which needs to be transferred to Xcode: / Library / Developer / Toolchains /. Next, in the IDE settings, select the new toolchain, restart Xcode.
Select the piece of code that should process the tool, and look for the tool in the context menu.
Development through tests with Ninja
If the project was compiled under Ninja and you chose the TDD path, then developing through tests with Ninja is one of the options that suits you. Cons - you can not set breakpoints, as in the development through Xcode.
So, we need to check that the new tool is displayed in Xcode when the user selects the guard construction in the source code. We write the test in the existing file test / refactoring / RefactoringKind / basic.swift:
func testConvertToGuardExpr(idxOpt: Int?) { if let idx = idxOpt { print(idx) } } // . // RUN: %refactor -source-filename %s -pos=266:3 -end-pos=268:4 | %FileCheck %s -check-prefix=CHECK-CONVERT-TO-GUARD-EXPRESSION // CHECK-CONVERT-TO-GUARD-EXPRESSION: Convert To Guard Expression We indicate that when selecting a code between 266 row 3 columns and 268 row 4 columns, we expect the menu item to appear with a new tool.
Using the lit.py script can provide faster feedback to your development cycle. You can specify the test suit. In our case, this suite will be RefactoringKind:
./llvm/utils/lit/lit.py -sv ./build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/test-macosx-x86_64/refactoring/RefactoringKind/As a result, only tests of this file will be launched. Their implementation will take a couple of tens of seconds. More about lit.py will be discussed below in the Diagnostics and Testing block.
The test fails, which is normal for the TDD paradigm. After all, so far we have not written a single line of code with the logic of the tool.
Debugging and Xcode Development
And finally, the last development method when the project was compiled under Xcode. The main advantage is the ability to set breakpoints and control debugging.
When building a project under Xcode, the file Swift.xcodeproj is created in the build / Xcode-DebugAssert / swift-macosx-x86_64 / folder. When you first open this file, it is better to choose the creation of schemas manually in order to generate ALL_BUILD and swift-refactor yourself:
Next, we build the project with ALL_BUILD once, after that we use the swift-refactor scheme.
Refactoring tool is compiled into a separate executable file - swift-refactor. Help about this file can be displayed using the –help flag. The most interesting parameters for us are:
-source-filename=<string> // -pos=<string> // -end-pos=<string> // -kind // They can be set in the schema as arguments. Now you can set breakpoints to stop in places of interest when you start the tool. In a familiar way, using the p and po commands in the Xcode console will display the values of the corresponding variables.
Implementation isApplicable
The isApplicable method takes as input ResolvedRangeInfo with information about the AST nodes of the selected code fragment. At the output of the method, it is decided whether or not to show the tool in the Xcode context menu. The full ResolvedRangeInfo interface can be found in the include / swift / IDE / Utils.h file .
Consider the most useful in our case, the fields of the class ResolvedRangeInfo:
- RangeKind - the first step is to check the type of the selected area. If the region is invalid (Invalid), you can return false. If the type suits us, for example, SingleStatement or MultiStatement, then go ahead;
- ContainedNodes - an array of AST elements that belong to the selected range. We want to make sure that the user selects the range that includes the if let construct. To do this, take the first element of the array and check that this element corresponds to IfStmt (the class that defines the AST node of the statement of the if subtype). Next, look at the condition. To simplify the implementation, we will display the tool only for expressions with one condition. Based on the type of the condition (CK_PatternBinding), we determine that this is let.
To test isApplicable, add sample code to the file test / refactoring / RefactoringKind / basic.swift .
In order for the test to simulate a call to our tool, you need to add a line in the tools / swift-refactor / swift-refactor.cpp file.
Implement performChange
This method is called when selecting a refactoring tool in the context menu. The method has access to ResolvedRangeInfo, as in isApplicable. Use ResolvedRangeInfo and write the logic of the code conversion tool.
When generating code for static tokens (regulated by the syntax of the language), entities from the namespace tok can be used. For example, for the guard keyword, we use tok :: kw_guard. For dynamic tokens (modified by the developer, for example, the name of the function) you need to select them from the array of AST elements.
To determine where the inserted code is inserted, we use the full selected range using the RangeInfo.ContentRange construction.
Diagnostics and Testing
Before you finish work on the tool, you need to check the correctness of his work again. This will help us again tests. Tests can be run one at a time or with all available options. The easiest way to run the entire Swift test suite is with the --test command on utils / build-script, which will run the main test suite. Using utils / build-script rebuilds all the target, which can significantly increase the debug cycle time.
Be sure to run the utils / build-script --validation-test verification tests before making major changes to the compiler or API.
There is another way to run all compiler unit tests through ninja, ninja check-swift from build / preset / swift-macosx-x86_64. It will take about 15 minutes.
And finally, an option when you need to run the tests separately. To directly invoke the lit.py script from LLVM, it must be configured to use the local build directory. For example:
% $ {LLVM_SOURCE_ROOT} /utils/lit/lit.py -sv $ {SWIFT_BUILD_DIR} / test-macosx-x86_64 / Parse / This will run the tests in the 'test / Parse /' directory for 64-bit macOS. The -sv option provides a test execution indicator and shows the results of only unsuccessful tests.
Lit.py has several other useful features, such as timing tests and latency testing. See these and other features with lit.py -h. The most useful can be found here .
To run one test we will prescribe:
./llvm/utils/lit/lit.py -sv ./build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/test-macosx-x86_64/refactoring/RefactoringKind/basic.swift If we need to tighten up the latest changes to the compiler, then we need to update all the dependencies and rebase. To upgrade, run ./utils/update-checkout.
findings
We managed to achieve our goal - to make a tool that was not previously in the IDE to optimize work. If you also have ideas on how to improve Apple products and make life easier for the entire iOS community, feel free to tackle the content-feeding, because it’s simpler than it seems at first glance!
In 2015, Apple posted the open source Swift source codes, which made it possible to dive into the implementation details of its compiler. In addition, local refactoring tools can be added with Xcode 9. Basic C ++ knowledge and compiler devices are enough to make your favorite IDE a little more convenient.
The described experience was useful for us - in addition to creating a tool that simplifies the development process, we got a really hardcore knowledge of the language. Pandora's open box with low-level processes allows you to look at daily tasks from a new angle.
We hope that the knowledge gained will also enrich your understanding of the development!
The material was prepared in collaboration with @victoriaqb - Victoria Kashlina, iOS developer.
Sources
Source: https://habr.com/ru/post/460227/
All Articles