> ## Content Index > Fetch the complete content index at: https://www.jonathancreamer.com/llms.txt > Use this file to discover other available public pages before exploring further. # How to successfully manage a large scale JavaScript monorepo aka megarepo - URL: https://www.jonathancreamer.com/how-to-successfully-manage-a-monorepo-aka-megarepo/ - Published: 2019-11-20T18:31:55.000Z - Updated: 2019-11-21T21:27:48.000Z - Description: There are many tools already in the wild for managing a monorepo. This post will discuss using Bolt to manage a monorepo aka megarepo. - Author: Jonathan Creamer - Tags: monorepo, megarepo, bolt, jest, typescript, divops First off, this is a follow up to a post I did recently! [Say bye to monorepos say hello to megareposBreaking up a monolith can be a daunting task. Oftentimes a monolith gets brokenup into many repos, and sometimes into a monorepo. But, here’s the problem I have with the term monorepoo. > Monorepos are NOT monoliths!I recently heard a talk by Marcel Cutts called... MonoRepos for the Masses\[https://www.youtube.com/watch?v=rdeBtjBNcDI&=&feature=youtu.be…![](https://www.jonathancreamer.com/favicon.ico)Jonathan CreamerJonathan Creamer![](https://storage.ghost.io/c/42/63/4263df8f-a33f-403d-aa31-ba113a54d1b7/content/images/2019/11/IMG_2348.JPG)](https://www.jonathancreamer.com/say-bye-to-monorepos-say-hello-to-megarepos/) tldr; of the previous post is, when switching from a monolith to a monorepo, call it a megarepo instead... ![](https://storage.ghost.io/c/42/63/4263df8f-a33f-403d-aa31-ba113a54d1b7/content/images/2019/11/image.png) Rather than a Death Star sized monolith which is prone to weakness with a single proton torpedo, prefer a **megarepo**, a force of specialized ships and pilots, which when working in concert together, can turn the tides of the galaxy. ## Tools for managing a megarepo There are many tools already in the wild for managing a megarepo. They handle things like running scripts against each package, deploying packages to NPM, determining which packages have changed, etc. - [Lerna](https://github.com/lerna/lerna?ref=jonathancreamer.com) - [Bolt](https://github.com/boltpkg/bolt?ref=jonathancreamer.com) - [Rush](https://rushjs.io/?ref=jonathancreamer.com) - [Yarn Workspaces](https://yarnpkg.com/lang/en/docs/workspaces/?ref=jonathancreamer.com) Each has its own pros and cons as all tools do. The particular tool we'll discuss going forward in this post is Bolt. ## Bolt Bolt and lerna were both created by the same folks. Bolt was just an iteration on a lot of the ideas lerna had, but with a few deviations. Bolt is installed as a global module as well ```shell npm i -g bolt # or yarn add global bolt ``` Bolt is meant to be even more of a "project" based, and is actually kind of a wrapper around Yarn and Yarn Workspaces. It also locks the dependencies of the shared packages. Meaning, a single version of React, or React Router, etc is installed across all of the packages in the megarepo. There are a few reasons why this is done. In a bolt megarepo, unlike lerna, when you add a project to a package, it is also automatically added to the top level package.json. This makes installs a bit quicker and easier as you won't ever have forked dependencies like you can in Lerna. ![](https://storage.ghost.io/c/42/63/4263df8f-a33f-403d-aa31-ba113a54d1b7/content/images/2019/11/Lerna-vs-bolt.png) The above diagram illustrates how allowing packages to have differing versions of pacakges can cause the nested node\_modules directories to fork in many different places. This bloats the repo size as well as complexity of maintenance, and overall yarn install times. This also makes running the megarepo with Docker a bit less trivial as well ```Dockerfile FROM node:12 WORKDIR /code COPY package.json yarn.lock ./ RUN yarn RUN npm i -g bolt COPY . . RUN bolt ``` With lerna and other monorepo tools, you have to constantly add new lines like... ```Dockerfile COPY ./packages/alpha/package.json /code/packages/alpha/package.json COPY ./packages/beta/package.json /code/packages/beta/package.json COPY ./packages/charlie/package.json /code/packages/charlie/package.json ``` But, since `bolt` has all the dependencies of every package listed at the top level, it's much simpler and faster to install all the dependencies. It also makes testing upgrades easier because the internal dependency graph of bolt makes it to where testing upgrades across packages is very simple. You can bump a package, run the tests and see how you're upgrade affects other parts of the megraepo. ## Structure of a megarepo Here is an example project... [jcreamer898/bolt-demoRepo for demonstrating how to build a bolt megarepo - jcreamer898/bolt-demo![](https://github.githubassets.com/favicon.ico)GitHubjcreamer898![](https://avatars3.githubusercontent.com/u/472487?s=400&v=4)](https://github.com/jcreamer898/bolt-demo?ref=jonathancreamer.com) Most of the megarepo tools have similar ideas about how to structure themselves. In general there's a top level `package.json` folder, and a `packages` directory. At least in the case of working with `bolt` and `lerna`. For lerna, a `lerna.json` file is created which contains a tiny bit of config for setting which packages are a part of the And with `bolt` the configuration is done in the `package.json`. ```JSON { "name": "rebel-alliance", "private": true, "bolt": { "workspaces": [ "./packages/*/*" ] }, "dependencies": { "react": "^16.12.0", "react-dom": "^16.12.0" } } ``` ![](https://storage.ghost.io/c/42/63/4263df8f-a33f-403d-aa31-ba113a54d1b7/content/images/2019/11/Screen-Shot-on-2019-11-20-at-11-13-44.png) See in the above how react in each package is just a symlink to the top level node\_modules directory. ## Creating shared tooling One of the major benefits of having a megarepo is the ability to have one set of tooling which builds, tests, and deploys every package. It takes the worry out of having to configure all of our mountains of front end code across multiple repos. To get started... Create a `build` directory and add it to the `bolt` section of the `package.json` ``` "bolt": { "workspaces": [ "./packages/*/*", "./build/*" ] }, ``` Let's create a builder project called hoth in `./build/hoth`. We'll also use a library called scritch to help us create a super easy CLI... [scritchA small CLI to help you write sharable scripts for your team![](https://static.npmjs.com/1996fcfdf7ca81ea795f67f093d7f449.png)npm![](https://static.npmjs.com/338e4905a2684ca96e08c7780fc68412.png)](https://www.npmjs.com/package/scritch?ref=jonathancreamer.com) ## Transpiling with Babel Babel is undoubtedly going to be the first thing we need in our project. So, in hoth, add a few things... First, a `package.json`... ```JSON { "name": "@rebels/hoth", "version": "1.0.0", "dependencies": { "@babel/cli": "^7.7.0", "@babel/core": "^7.7.2", "@babel/plugin-proposal-class-properties": "^7.7.0", "@babel/plugin-syntax-dynamic-import": "^7.2.0", "@babel/preset-env": "^7.7.1", "@babel/preset-react": "^7.7.0", "@babel/preset-typescript": "^7.7.2", "scritch": "^1.3.1" } } ``` Add a `cli.js` file... ```shell #!/usr/bin/env node require("scritch")(__dirname); ``` Scritch will scan the `scripts` directory and auto generate a CLI. Then a `scripts` directory with a `babel.sh` script in it... ```shell #!/usr/bin/env bash set -e bolt workspaces exec \ --parallel-nodes \ -- \ babel \ --extensions .ts,.tsx,.js,.jsx \ --root-mode upward \ --source-maps true \ -Dd dist \ src ``` Don't forget to `chmod +x ./build/hoth/scripts/babel.sh` as well as the `cli.js` too. This is to make these scripts executable. Make sure and always do that for any new script added. This will run `babel` in every workspace in parallel. Then let's create a `src/configs` directory in hoth, and add `babel.js` ```javascript module.exports = { presets: [ [ '@babel/env', { targets: { browsers: ['last 2 versions'], }, }, ], '@babel/react', '@babel/typescript', ], plugins: [ '@babel/proposal-class-properties', '@babel/plugin-syntax-dynamic-import', ], }; ``` Next, since we are placing these config files in a package, we have to add some top level files that point to these... ```js // babel.config.js module.exports = require('./build/hoth/src/configs/babel'); ``` One more thing to make life easy is, you can hop up to the top level package.json and add... ```json "scripts": { "hoth": "./build/hoth/cli.js", "build": "yarn hoth babel" }, ``` Now, what you can do is... ```shell > bolt hoth > bolt build ``` `bolt hoth` will run scritch to tell you what all scripts you have available, and `bolt build` will run the `babel.js` script to run babel in every package. ![](https://storage.ghost.io/c/42/63/4263df8f-a33f-403d-aa31-ba113a54d1b7/content/images/2019/11/Screen-Shot-on-2019-11-20-at-12-14-34.png) ![](https://storage.ghost.io/c/42/63/4263df8f-a33f-403d-aa31-ba113a54d1b7/content/images/2019/11/Screen-Shot-on-2019-11-20-at-12-14-47.png) We're well on our way now! Since we're using the `@rebels` scope, we'll need to make sure that babel knows how to properly resolve that scope. Let's say we add a new package called `@rebels/endor` for shared utils, ```js const resolver = { root: ['.'], alias: { '@rebels/alpha': './packages/theme-two/endor/src', '@rebels/beta': './packages/theme-three/endor/src', '@rebels/charlie': './packages/theme-one/charlie/src', '@rebels/endor': './packages/theme-one/endor/src', }, }; module.exports = { presets: [ [ '@babel/env', { targets: { browsers: ['last 2 versions'], }, }, ], '@babel/react', '@babel/typescript', ], plugins: [ '@babel/proposal-class-properties', '@babel/plugin-syntax-dynamic-import', ['module-resolver', resolver], ], }; ``` The `alias` piece could be configured to be dynamic as well by reading in all the packages and setting them up with so packages aren't manually added. The premise here though is the resolver will at compile time swap `@rebels/endor` or any of the packages for its actual location on the file system at transpile time. This is mostly used for local development as ideally during a production build you'll run the `bolt build` step anyways. So, you can consider wrapping the resolver stuff up in a check for local dev or not. ## Prettier Let's add a format script which will run prettier on everything because who wants to format their code anyways. Add a `prettier.js` to the configs folder. ```js module.exports = { tabWidth: 4, arrowParens: 'always', trailingComma: 'all', proseWrap: 'always', singleQuote: true, overrides: [ { files: '**/package.json', options: { tabWidth: 2, }, }, ], }; ``` And add the prettier.js at the root of the project... ```javascript // prettier.config.js module.exports = require('./build/hoth/src/configs/prettier'); ``` You can add dependencies just like you would with yarn. ```shell bolt workspace @rebels/hoth add prettier ``` Now we'll add another script to `./build/hoth/scripts`. ```shell #!/usr/bin/env bash set -e prettier --write '**/src/**/*.{ts,tsx,js}' ``` Now you can run... ```shell bolt hoth format ``` ![](https://storage.ghost.io/c/42/63/4263df8f-a33f-403d-aa31-ba113a54d1b7/content/images/2019/11/Screen-Shot-on-2019-11-20-at-12-22-00.png) Which will format all the code in the repo. You can continue adding more and more scripts to the CLI as needed for you and your team. ## Typescript We're relying on the `@babel/preset-typescript` to remove the types from the `.ts` files as shown in the babel config previously. We'll use `typescript` itself with `noEmit: true` to only do type checking. We can setup typescript type checking by adding a `tsconfig.json` in our configs directory, and linking it at the top level like with babel and prettier. For TypeScript we first run `bolt w @rebels/hoth add typescript`, and then add a `./build/hoth/src/configs/tsconfig.json`... ```json { "exclude": ["**/node_modules/**", "**/test/**"], "compilerOptions": { "target": "es5", "module": "commonjs", "lib": ["dom", "es2015"], "jsx": "react", "outDir": "./dist", "noEmit": true, "downlevelIteration": true, "strict": true, "moduleResolution": "node", "esModuleInterop": true } } ``` And at the top level add a tsconfig... ```json { "compilerOptions": { "resolveJsonModule": true, "paths": { "@rebels/alpha": ["packages/theme-one/alpha/src/index"], "@rebels/beta": ["packages/theme-one/beta/src/index"], "@rebels/charlie": ["packages/theme-one/charlie/src/index"] }, "baseUrl": ".", "rootDir": "." }, "extends": "./build/hoth/src/tsconfig.json", "include": ["./types/**/*", "./packages/**/*", "build/**/*"] } ``` In here you can see we've also specified some paths. This will help TypeScript know what the `@rebels` prefix points to. Now we can add a `typecheck.sh` script into the cli... ```bash #!/usr/bin/env bash set -e tsc ``` The final thing to make sure we have installed is the `@types/react` package. ```bash bolt w @rebels/alpha add @types/react bolt w @rebels/beta add @types/react bolt w @rebels/charlie add @types/react ``` Now running `bolt hoth typecheck` will perform ONLY typechecking. ## Jest Setting up jest requires a few additional config files, but nothing we can't handle. Add a `./build/hoth/src/configs/jest.js` file... ```js module.exports = { resolver: require.resolve('./jest/resolver.js'), testPathIgnorePatterns: ['dist'], moduleFileExtensions: ['ts', 'tsx', 'js', 'jsx', 'json'], transform: { '^.+\\.(js|jsx|ts|tsx)$': require.resolve('./jest/transformer.js'), }, }; ``` And a `./build/hoth/src/configs/jest/transformer.js` ```js const babelJest = require('babel-jest'); module.exports = babelJest.createTransformer({ configFile: './babel.config.js', }); ``` The resolver is where things get interesting... Let's say in the future you have a package which imports another package. ```ts import { ewoks } from "@rebels/endor"; ``` Well, jest will look in the `dist` folder of `@rebels/endor` . That's not good... So, we can use `jest-enhanced-resolve` to update the mainFields for where to find source code during imports. ```js 'use strict'; let createResolve = require('jest-enhanced-resolve').default; let resolve = createResolve({ mainFields: ['rebels:source', 'main'], extensions: ['.js', '.jsx', '.ts', '.tsx'], }); function resolver(modulePath, opts) { return resolve(modulePath, opts); } module.exports = resolver; ``` Then just make sure to add `"rebels:source": "./src/index.ts"` in all the package.json files. The final thing is to add your `test.sh` script.. ```bash #!/usr/bin/env bash set -e jest ``` Now you can run `bolt hoth test` or add it to the `"scripts"` block to simply do `bolt test`. ## Wrapping up This is all just the start of working with a megarepo! There's so many more things you can add. Webpack, a development server, versioning the packages, etc. All things to keep an eye out for in future posts. [Building a megarepo with webpack · Issue #2 · jcreamer898/blog-post-ideasGotta do it.![](https://github.githubassets.com/favicon.ico)GitHubjcreamer898![](https://avatars3.githubusercontent.com/u/472487?s=400&v=4)](https://github.com/jcreamer898/blog-post-ideas/issues/2?ref=jonathancreamer.com) [Creating a common local dev server for a megarepo · Issue #3 · jcreamer898/blog-post-ideasGotta talk about how to serve apps locally![](https://github.githubassets.com/favicon.ico)GitHubjcreamer898![](https://avatars3.githubusercontent.com/u/472487?s=400&v=4)](https://github.com/jcreamer898/blog-post-ideas/issues/3?ref=jonathancreamer.com) [Using changesets/cli to version packages in a megarepo · Issue #4 · jcreamer898/blog-post-ideasVersioning is hard, but not with changesets![](https://github.githubassets.com/favicon.ico)GitHubjcreamer898![](https://avatars3.githubusercontent.com/u/472487?s=400&v=4)](https://github.com/jcreamer898/blog-post-ideas/issues/4?ref=jonathancreamer.com)