How the AST Finder and Builder tools can help developers build codemods
In this article, we will look at two tools called AST Finder and AST Builder, which will significantly improve the developer experience for writing codemods.
Codemod is a tool/library for large-scale codebase refactors that can be partially automated but require human oversight and occasional intervention. Codemod was developed at Facebook and released as open-source.
If you want to know more about codemods, their building blocks, and how they work, please check out this detailed post.
Using codemods: What AST Builder is all about
AST Builder is a playground for building abstract syntax tree (AST) nodes using source code. Because AST plays a significant role in writing codemods, this tool will assist the developers to a great extent in writing codemods. One of the primary reasons codemods are more resilient in making effective code transformations is because codemods do AST-to-AST transformation on your source code.
It supports Javascript (ES5, ES6, and some ES7 constructs), JSX, and Glimmer.js handlebars syntax. For coverage information, see Cover core API andCover ES6 API. And I am planning to include more language syntax and semantics.
Using codemods: Why we leverage the AST Tooling
We already have a well-established and battle-tested tool called AST Explorer for visualizing ASTs. So why do we need a new tool? Because AST Explorer is only for exploring your ASTs—it doesn’t tell you how to create AST nodes. Even though AST Explorer offers IntelliSense in its editor for the jscodeshift APIs, it doesn’t work for all the parsers. For instance, you can only use the autocomplete API for the recast parser. If you choose any other parser other than recast, you won’t get IntelliSense in the codemod editor.
Most of the time, you will be creating nodes for transforming code using codemods, so we definitely need a tool that makes it easy to create nodes. The problem is that there is no proper documentation on creating AST nodes using the jscodeshift API. All you have to do is learn from other codemods out there, sift through the code, and find out how you can create new nodes.
For that, you need to understand the parser internals, the node schema, and the types of node builders available for the language you are using.
If you are still not convinced why this tool will make a difference in developer experience for building codemods, read what others have to say here.
For example, for Javascript, you need to know the ESTree spec or the node builder definition in ast-types. This module provides an efficient, modular, Esprima-compatible implementation of the abstract syntax tree type hierarchy pioneered by the Mozilla Parser API.
Let’s say you want to replace a CallExpression, foo(), with a new one like foo.bar(). The AST representation for the above two expressions will be:
I have omitted a lot of information in the above code for clarity and readability purposes. It only contains the relevant information for the actual CallExpression AST node. If you want to explore the full tree structure of the AST, you can check it in AST Explorer.
As you can see from the above two AST nodes, the only difference between the two is the callee object, which is a simple Identifier in foo() and a MemberExpression in foo.bar(). Usually with codemods, we will be replacing the original expression with the new one. Hence here, we will be replacing the original CallExpression with a new one like this:
In order to replace the old CallExpression with a new one, first we need to find the existing CallExpression. From the above codemod, you can see we are querying the AST using jscodeshift API like this:
If you try to build the above CallExpression within the AST Explorer transform editor the first time, you will have a tough time because you are not familiar with the builder API in the first place and you don’t know the correct order and type of parameters to supply to correctly build the AST node. And don’t forget the typos and punctuation errors you make while typing the code.
This is where AST Finder comes into the picture. It acts as a reference guide for finding APIs to easily query your AST nodes. Just input the code in the input editor (see the image above to identify the input editor, which is always in the top-left pane) and you will get the find API automatically generated for you without any mistakes. So if you input foo.bar() into the AST Finder, it will give you something like:
Now you can simply copy the query from AST Finder and use it in your codemods. How cool is that?
To replace the old CallExpression with a new one, we need to build the new one. From the above codemod, you can see we are building the new one using jscodeshift API like this:
If you try to build the above CallExpression within the AST Explorer transform editor the first time, you will have a tough time because you are not familiar with the builder API in the first place and you don’t know the correct order and type of parameters to supply to correctly build the AST node. And don’t forget the typos and punctuation errors you make while typing the code.
There are also some subtle nuances with the jscodeshift API that beginners won’t know. For example, the API j.callExpression is a constructor for building CallExpression nodes, whereas j.CallExpression is an instance of the type CallExpression, which is basically used to find nodes of the type CallExpression.
This is where AST Builder comes into the picture. It acts as a reference guide for builder APIs to easily build your AST nodes. Just input the expected code in the input editor (see the image above to identify the input editor, which is always in the top-left pane) and you will get the builder API automatically generated without any mistakes. So if you input foo.bar() into the AST Builder, it will give you something like:
You can safely omit the ExpressionStatement wrapper if you are just replacing the nodes.
Now you can simply copy the builder API from AST Builder and use it in your codemods. How easy is that?
Using codemods: How AST tooling makes a difference
AST Builder uses ast-node-builder with an npm package underneath, which provides the APIs for building AST nodes through jscodeshift. All the APIs take an object as a parameter and return the builder API in string format, which you can use with jscodeshift to create new nodes. The object that is passed to the API as a parameter is actually a node in the AST generated by the respective parser. When you feed the node to the ast-node-builder API, you get back the jscodeshift API to build that node.
This allows developers to easily and effectively create AST nodes from source code instead of tinkering with the autocomplete API in AST Explorer. All you have to do is enter or paste the source code into the input editor and you can see the jscodeshift API automatically generated in the output editor.
You can also use AST Builder to visualize your AST in the top-right pane without all the noise and clutter of meta information. We deliberately filter out the loc nodes and tokens from the AST, since we feel they are not of much use for working with codemods. To dig deeper into the builder, you can take a look at the source code here. It is built in Ember.js.
Community and adoption
The Ember.js community has been pioneering the use of codemods for upgrades and deprecations for a long time. Since the community is actively building codemods for their use cases, AST Builder proves to be a valuable tool for the task. It was even featured in an EmberConf 2021 talk called “Cross-File Codemodding” by Joshua Lawrence.
EmberConf 2021 – Cross-File Codemodding by Joshua Lawrence
In this talk, Joshua talks about how easy it is to use AST Builder for automatically generating jscodeshift APIs for building handlebars nodes to transform AST with codemods.
Stay tuned for more about the exciting tools we are building around ASTs and codemods.
Using codemods: References for developers
Here’s a list of resources that can empower developers to use codemods effectively.
