Migrating from Webpacker to esbuild

Why?

With the retirement of Webpacker we needed to find an alternative.

So what is Webpacker and why do we need it? Webpacker is a gem that makes it easy for Rails developers to bundle JavaScript assets. Basically it is a thin layer over Webpack (Or that was the original intention). Which is how the README describes it.

Webpacker makes it easy to use the JavaScript pre-processor and bundler webpack 4.x.x+ to manage application-like JavaScript in Rails. It coexists with the asset pipeline, as the primary purpose for webpack is app-like JavaScript, not images, CSS, or even JavaScript Sprinkles (that all continues to live in app/assets).

However, it is possible to use Webpacker for CSS, images and fonts assets as well, in which case you may not even need the asset pipeline. This is mostly relevant when exclusively using component-based JavaScript frameworks.

Given that Webpacker has been retired and the fact that there are few developers who know how to work with it, it felt like an opportune time to revisit how to package our JavaScript assets.

When considering a replacement for Webpacker we had the following non-functional requirements:

Reduce the complexity of our current JavaScript Packaging
Packaging Javascript Assets should be easy for Developers to configure and understand
Potentially allow the JavaScript packaging to be extracted from the rails app
Address the slowness in our current build process
Should have a similar dev experience to Webpacker (e.g. recompilation and reloading in the browser on file changes)
The changeover should be relatively seamless - with as little code changes as possible
Ease our eventual transition of our JavaScript assets to TypeScript

What do we replace Webpacker with?

So with the non-functional requirements in mind, we considered the following options

1. Continue using Webpacker

Pros

Nothing needs to be done we can keep calm and carry on

Cons

We are delaying the fact we need to migrate to another solution. Eventually with Rails 7 we will need to revisit this again
The assets packaging will still be slow
Does nothing to address the complexity - nor the siloed knowledge on JavaScript packaging

2. Migrate to Shakapacker (a fork of Webpacker)

Pros

Relatively easy transition process
Developer experience is similar to the way everything works now

Cons

Developers will need to understand Webpack
Does little to address the speed issues - from the README

Check out 6.1.1+ for SWC and esbuild-loader support! They are faster than Babel!
Doesn't allow us to extract the JavaScript packaging from the Rails application
Does nothing to address the complexity - nor the siloed knowledge on JavaScript packaging

3. Move to jsbundling-rails and cssbundling-rails

As per the README in the jsbundling-rails repo:

Use esbuild, rollup.js, or Webpack to bundle your JavaScript, then deliver it via the asset pipeline in Rails. This gem provides installers to get you going with the bundler of your choice in a new Rails application, and a convention to use app/assets/builds to hold your bundled output as artifacts that are not checked into source control (the installer adds this directory to .gitignore by default).

It does not dictate what tool to use for the bundling.

Pros

Allows us to use TypeScript easily
Building assets will be faster
Configuration is more approachable to non frontend developers - (Don't need to understand Webpack)

Cons

Packaging scripts would need to written
Sprockets gem needs to be added and configured
Webpacker helpers need to migrated to standard sprocket asset helpers
Webpack aliases in our css using ~ are not supported in standard sass
esbuild is still relatively earlier in its development not version 1 yet

The Decision

After considering the options listed above, options 1 and 2 do little in reducing the complexity that webpacker introduced. We felt that option 3 would be a more future proof and maintainable solution.

We chose Sass to bundle the CSS, and for our JavaScript bundling tool, we chose to use esbuild for the following reasons:

It's fast
Relatively small build API
Wide community usage

How did we migrate?

We have answered the Why? and What?, now on to the trickiest question How?

Whilst jsbundling-rails does provide instructions on how to switch from Webpacker, the document only covers the migration to webpack based asset compilation. This document did provide us with some insight on how to proceed with the migration.

Our migration process

Note: some of this process was taken from the original migration instructions linked above.

Setup jsbundling-rails
- Install jsbundling-rails
- Run ./bin/rails javascript:install:esbuild
This is where we hit our first issue - the install script expects a sprockets manifest to exist.
```
if (sprockets_manifest_path = Rails.root.join("app/assets/config/manifest.js")).exist?
  append_to_file sprockets_manifest_path, %(//= link_tree ../builds\n)
end
```
Well this is going to be fun - our rails app doesn't even have sprockets.
Setup sprockets
- Install sprockets-rails
- Add configuration for production test and development
- Create the manifest file for sprockets app/assets/config/manifest.js
  - include the jsbundling-rails build output by adding //= link_tree ../builds

Setup cssbundling-rails

Install cssbundling-rails
Run ./bin/rails css:install:sass

running yarn run build:css

This is where we hit our second problem: our Sass files were using ~ prefixed imports e.g. @forward ~@bugcrowd/bc-flair/. In sass-loader < 11.0.0, this was the way to tell the plugin to resolve sass files in the node_modules folder. This is still supported but is deprecated. The original usecase for the ~ prefix was to simplify sass imports without having to write long relative paths.

So how did resolve this? We wrote a script using the Sass JavaScript API that would process all the Sass files and call out to a custom importer to process the ~ imports.

For the importer, we reverse engineered an esbuild Sass plugin that supported ~ prefixed imports.

/**
 * The Sass syntax for a given file
 * 
 * https://sass-lang.com/documentation/syntax
 */ 
function fileSyntax(filename) {
   if (filename.endsWith('.scss')) {
     return 'scss'
   } else if (filename.endsWith('.css')) {
     return 'css'
   } else {
     return 'indented'
   }
 }

/**
 * Custom SASS importer to process @import statements with "~" prefix
 *
 */
 const importer = {
   load(canonicalUrl) {
     const pathname = fileURLToPath(canonicalUrl)
     const contents = fs.readFileSync(pathname, 'utf8')
     return {
       contents: contents.replace(/(url\(['"]?)(~)/g, `$1`),
       syntax: fileSyntax(pathname),
     }
   },
   canonicalize(url) {
     const baseDir = dirname('.')

     if (resolved.hasOwnProperty(url)) {
       return pathToFileURL(resolved[url])
     }

     let filename = null
     if (url.startsWith('~@')) {
       // @forward '~@bugcrowd/bc-flair/dist/main';
       filename = decodeURI(url.slice(1))
       try {
         requireOptions.paths[0] = baseDir
         filename = require.resolve(filename, requireOptions)
       } catch (ignored) {
         filename = resolveRelativeImport('./node_modules', filename)
       }
     } else if (url.startsWith('~')) {
       // @use '~css/modules/organization-roster'
       const { 0: alias } = url.split(sep)
       filename = resolveRelativeImport(aliases[alias], url.replace(alias, '.'))
     } else if (url.startsWith('file://')) {
       filename = resolveImport(fileURLToPath(url))
     }

     return filename == null ? filename : pathToFileURL(filename)
   },
 }

And usage of importer

const result = sass.compile(source, {
    loadPaths,
    importers: [importer],
    style: minify ? 'compressed' : 'expanded',
  })

Replace Webpacker pack tags

# Webpacker tag       # Sprockets tag
javascript_pack_tag = javascript_include_tag
stylesheet_pack_tag = stylesheet_link_tag

Deploy and test

Issues

The migration to jsbundling-rails and cssbundling-rails was relatively straightforward. However, we did run across some minor issues during the migration:

Assets weren't compiled when running tests. Reading through the documentation for jsbundling-rails provides an explanation of why they aren't being be generated:

If your testing library of choice does not define a test:prepare Rake task, ensure that your test suite runs javascript:build to bundle JavaScript before testing commences.

RSpec-rails does not provide a test:prepare Rake task. After little bit more googling we found a relevant post on how to ensure assets are precompiled with RSpec Stack Overflow link. Once we added the script, it just needed to be called before all RSpec tests
```
RSpec.configure do |config|
  config.before(:suite) do
    MaintainTestAssets.maintain!
  end
end
```
Some images were not being displayed in React components. This was due to the way esbuild processes external images. Roughly the process is:
- Create a digest for the image
- Copy the file with the digest appended e.g. app/assets/builds/slack-CJBP27SE.svg
- Reference the image by the new name - so the compiled JavaScript would have reference like this
```
x(()=>{o0n='/assets/slack-CJBP27SE.digested.svg'})
```
After esbuild has completed compilation - sprockets will process the app/assets/builds folder. During processing of this folder, Sprockets will fingerprint each file and copy it to the assets folder with the fingerprint appended. So we would end up with a filename of slack-CJBP27SE-ed7f62b217d969163106eb3bdd9ce7fbcab95813affe1274145398edf6850c23.svg. Once again the internet had a solution, we just had to get esbuild to append the file with digested to prevent Sprockets from adding its own digest. Which was as easy as adding --asset-names=[name]-[hash].digested.
No HMR(Hot module Replace) - Webpacker provided us with a nice tool webpack-dev-server to auto reload the page as you update the source. Sadly, esbuild is not planning to add support for HMR see. This article provided an alterative.

Conclusion

To give you an some idea on the complexity and scale of the migration from webpacker, our codebase has:

1727 JavaScript files
107 scss files
106 usages of asset_pack_path
11 usages of javascript_pack_tag
20 usages of stylesheet_pack_tag

Was the migration worth the effort?

Here are some benefits we can see:

The build process is easier for us to maintain
Our eventual transition to TypeScript will be an easier endeavour
No longer need babel plugins for compilation
No longer ship es5 code
Potentially, we could bundle the JavaScript and CSS outside of the Rails application

I think the best way to sum it up is our builds are now Blazingly™ fast

Comparison