> ## Content Index > Fetch the complete content index at: https://www.jonathancreamer.com/llms.txt > Use this file to discover other available public pages before exploring further. # Inside the pain of monorepos and hoisting - URL: https://www.jonathancreamer.com/inside-the-pain-of-monorepos-and-hoisting/ - Published: 2021-12-10T23:01:51.000Z - Updated: 2023-11-29T01:22:44.000Z - Description: Working in a monorepo comes with a long list of pros, and a few cons. One of the most painful cons, when it comes to working in a specifically JavaScript based monorepo, is the pain that comes from hoisting. - Author: Jonathan Creamer - Tags: node, monorepo, javascript Working in a monorepo comes with a long list of pros, and a few cons. One of the most painful cons, when it comes to working in a specifically JavaScript based monorepo, is the pain that comes from hoisting. Here's a video that talks about it, and you can also keep reading to learn more! ## What's this hoisting thing? ![](https://storage.ghost.io/c/42/63/4263df8f-a33f-403d-aa31-ba113a54d1b7/content/images/2021/12/image-5.png) As we all know, node\_modules is a deep dark place with lots and lots of stuff. That problem is even more massive in a large monorepo. Let's take a step back for a sec and take a look at what happens when you `require` something. ### node module resolution If you take a read through [the docs](https://nodejs.org/api/modules.html?ref=jonathancreamer.com#all-together), you'll find this... ``` require(X) from module at path Y 1. If X is a core module, a. return the core module b. STOP 2. If X begins with '/' a. set Y to be the filesystem root 3. If X begins with './' or '/' or '../' a. LOAD_AS_FILE(Y + X) b. LOAD_AS_DIRECTORY(Y + X) c. THROW "not found" 4. If X begins with '#' a. LOAD_PACKAGE_IMPORTS(X, dirname(Y)) 5. LOAD_PACKAGE_SELF(X, dirname(Y)) 6. LOAD_NODE_MODULES(X, dirname(Y)) 7. THROW "not found" ``` What this is basically saying is... If you require `X`, see if it exists in node, things like `fs`, `child_process`, etc. If you start `require('/X');` you're at the file system root. If you `require('./X');` then see if `.X` is file, then a directory. *...and this is the interesting one...* If you do `require('X');` and it's not a node thing, then keep traversing the file system looking in every `node_modules` along the way. ### Hoisting Package managers such as `yarn` and `npm` implemented a hoisting algorithm as a part of their different `workspaces` implementations. What this hoisting does is, it scans your `package.json` files across your workspaces and figures out what the most common versions of dependencies are. If you have 50 packages, and 47 of them are using react 16.9.0, but 3 are using 16.13.0, it'll "hoist" the common version of react, 16.9.0 to the top level `./node_modules` directory. That way you don't have to have 50 different versions of react in your project, the `import React` or `require('react')` calls will simply pull from the root `node_modules/react` one in the case of 16.9.0, or `./packages/*/node_modules/react` for the 3 cases of 16.13.0. Voila, space saved. However, the plot thickens... ### Monorepo structure So, *most* monorepo structures have a root `package.json`, and a `packages` folder. ``` ./packages/foo/ ./packages/foo/node_modules ./packages/bar/ ./packages/bar/node_modules ./package.json ./node_modules ``` Let's say we're working with the `./packages/foo/` and it does an `import React from 'react';`. Following the node module resolution from above, it'll eventually look in the `./packages/foo/node_modules` directory. If it's *not* there, it'll look at `../packages/node_modules/`, which definitely should not be there, and then it'll look at the `./node_modules` directory and see if it's there. Ok, so that seems fine, where can we go wrong here? Well, a couple of different bad things can happen here. Let's say the `foo/package.json` has `"react": "^16.9.0"`. Then, let's say over in `bar`, they forgot to add react in the package.json, but in `bar/src/index.ts` someone does an `import React...`, and it also happens that `bar` used a feature that only exists in the `16.13.0` version of React. What's gonna happen? Welp, because of the fact that `bar` doesn't have a `react` in its `package.json`, the node module resolution algorithm will kick in and look up to the `../../node_modules/` directory and grab react from there. But OH NOES, that code won't work at runtime because the code written in `bar` needs that fancy new feature in React 16.13! This is a problem that is also nicely summed up in the Rush docs as a "[Phantom Dependency](https://rushjs.io/pages/advanced/phantom%5Fdeps/?ref=jonathancreamer.com)" and a "[Doppleganger](https://rushjs.io/pages/advanced/npm%5Fdoppelgangers/?ref=jonathancreamer.com)". ### Real world ouchies Here's a real world example of how this came into play in a recent release pipeline failure. In the monorepo supported by my team, the most common version of `jest-environment-jsdom` that was hoisted to the root `node_modules` folder in the repo was `24.9.0`. A pull request when in which added `jest-environment-jsdom` 26.10.0 to one of the packages in the `./packages` folder. Well, what happened was, there were a couple of other places using that same version across the repo so, `yarn` in its attempt to de-duplicate by hoisting decided to switch the hoisted version to `26.10.0`! Here's where things got bad. Let's say the package which added `jest-environment-jsdom` was called, `cool-button`. The pull request for adding the new dependency will get pushed to CI, and the CI server does a check for all the places where `cool-button` is used, and start running tests, builds, etc on the dependencies in the repo to make sure that the changes `cool-button` doesn't break any of it's downstream dependencies. Ok, PR is green, all looks great! Well, let's say there's another package called `cool-color-picker`. And `cool-color-picker` had some tests which was more or less like... ```JavaScript it('should pull from memory storage when localStorage isnt there', () => { expect(storageUtilThing.get('item')).toBeTruthy(); }) ``` Well, in this crazy insane case... `cool-color-picker` *was* relying on the hoisted `jest-environment-jsdom` version, 24.9.0\. Well, that particular version used jsdom `11.11.0`. In that particular version of `jsdom` there was NO local storage support in the `jsdom` environment, so the test would pass. However! In the new version of 26.10.0 of `jest-environment-jsdom`, the version of `jsdom` gets bumped to `16.x.x` and it just so happens every so ironically that version `11.12.0` of `jsdom` implemented localStorage. See [Changelog](https://github.com/jsdom/jsdom/blob/master/Changelog.md?ref=jonathancreamer.com). ![](https://storage.ghost.io/c/42/63/4263df8f-a33f-403d-aa31-ba113a54d1b7/content/images/2021/12/image-6.png) All the sudden, the test is now failing because the test had previously assumed that there was no localStorage support in jsdom without some kind of 3rd party mocking like `jest-localstorage-mock`. What's worse here is, because `cool-color-picker` isn't in the dependency graph of `cool-button`, it never got tested as a part of the pull request. Therefore, the change landed and attempted to go through the release pipeline. THANKFULLY, although also painfully, the release pipeline currently does a "build the world" strategy regardless of what changed. It's here where the release pipeline which always builds all the packages that the failure happened and broke releases for a few hours until we figured out what happened which involved LOTS of code spelunking. ![](https://storage.ghost.io/c/42/63/4263df8f-a33f-403d-aa31-ba113a54d1b7/content/images/2021/12/cave-dive.gif) ## So what next? Well, in an ideal world we'll soon be switching to a "strict" package manager like the one we're currently building called [midgard-yarn-strict](https://www.npmjs.com/package/midgard-yarn-strict?ref=jonathancreamer.com), which will eventually migrate much of it's feature set into the implementation of [Isolated Mode](https://github.com/npm/rfcs/pull/436?ref=jonathancreamer.com) in NPM. Yarn v2 or later can also solve this problem with plug and play, or `pnpm` as well, but we're currently focusing on the former of the solutions with NPM and midgard yarn strict. A `strict` package manager such as these options will do a couple of things. First of all, the packages are forced to correctly declare their dependencies in the `package.json` so as to eliminate the phantom dependency / doppleganger problem. Secondly, the root `node_modules` folder can be moved to a different location on disk, and pointed to with symlinks. ``` ./packages/foo/ ./packages/foo/node_modules/react -> node_modules/.store/react-16.9.0 ./packages/bar/ ./packages/bar/node_modules/react -> node_modules/.store/react-16.13.0 ./package.json # This breaks the node module resoution ./node_modules/.store ./node_modules/.store/react-16.13.0 ./node_modules/.store/react-16.9.0 ``` By moving the `node_modules` folder somewhere else, and simply symlinking all of the package's dependencies to the new location, the hoisting algorithm breaks. So, in the previous case where the `bar` package had forgotten to declare react in its package, `bar` can no longer rely on hoisting! Other things you can do to help are implementing a couple of solutions like [DepCheck](https://www.npmjs.com/package/depcheck?ref=jonathancreamer.com) to make based on scanning your code that your dependencies are all correctly declared. There's also a tool called [TypeSync](https://github.com/jeffijoe/typesync?ref=jonathancreamer.com) which does something similar for the `@types` packages to make sure they're present. The hoisting problem with`@types` packages presents a whole different set of complexities because their usage isn't in code necessarily, but only in the TypeScript setup. ## Conclusion Hoisting helped a lot in solving *some* problems for *many* repos out there. However, the moment your monorepo starts to scale a bit, you'll inevitably run into these problems.