Harmony UIThursday, October 17, 2024
Harmony Technical Hurdles
If there is one about Harmony we have learned, it is that the problem we are solving is complex. A SaaS app that transforms any existing React app into a no-code editor and allows you to update the underlying code without having access to it is tough stuff. When we first started tackling this problem, we quickly learned why this sort of thing does not really exist yet.
This blog looks to go through some of the big technical hurdles we faced and the solutions in code.
And what is the biggest thing we have learned from these hurdles? Simplify your MVP. Most of these hurdles were tackled before we even launched, which we realized was not needed. Regardless, here are those hurdles.
Hurdle 1: Turning an Existing Application into a No-Code Editor
Hurdle 1 Commits
The first hurdle for us was how do we turn an application into a no-code suite? This has a lot of moving parts and hurdles to it, but the first was simply making an application selectable--like the chrome inspect tool!
One tool that already exists that does this is the React Developer Tools chrome extension. What we need is this functionality but more complex. So how did we solve this hurdle? We forked the repo and used this code as a base. We used the inspecting and highlighting parts of this extension to be able to hover over and select DOM elements on a page.
What was left after this was just expanding no-code functionality. Not too bad.
Hurdle 2: Update Code from UI Edits
First Implementation of Hurdle 2
With Github
Now that we can turn our application into a no-code editor, how do we actually make the edits? What if an edit comes from a database, we should tell the user that they can't make that edit right? What if the component is highly dynamic, being reused in a lot of places? Which component do we edit, just one or many?
These are all questions that took a lot of pacing around to find the answers to. The system we came up with: index the codebase. Indexing the codebase means going through each file in the codebase using a babel AST transformer and understanding it. Then we create data structures of the code contents with useful information to be able to make updates. This all happens ahead of time so that we can recognize where we can and cannot make updates and notify the user.
Hurdle 3: Connect to Github
Making changes to the local file system is great. But Harmony is for designers who do not have access to the code! That means we need to be able to create pull requests for the developer to review.
Even though this seems straight forward, this hurdle took a lot longer than anticipated. Sifting through Github documentation and making sense of it proved to be a pretty hard task! But we were able to get our Github App with octokit up and working, allowing you to connect your repository and create pull requests.
Hurdle 4: Deploy the Editor
First Bundle
Ok we can have an editor that overlays on an application, but how do we get people to use it? There are a few different ways we thought of: npm script, cdn script, chrome extension. We eventually went with all three, but started with a npm package, which comes with a cdn script.
Bundling React code for an npm script is also not an easy task. We opted for webpack, but that meant getting all of the configuration settings correct. We first went with an approach similar to Heap where you install a script tag, then we moved to an npm package (see Hurdle 6 below), then finally a Chrome extension.
Hurdle 5: Identifying Elements + Speedy Web Compiler
The next hurdle is mapping the code from the DOM to the code base. Up to this point we were just creating a unique hash using the element's className and its DOM position and matching it up when we do the indexing. But this is not sustainable. The solution: at compile time, add a data-harmony-id tag to each element that has information about the file and location where it is defined.
We are already using babel for the indexing, can we just use babel again for this? In NextJS apps: no. NextJS uses the Speedy Web Compiler instead of babel, which meant we would not only need to teach our selves Rust, but also figure out how to make a SWC plugin!
This proved to be a not easy task because there is not a lot of documentation on how to make a SWC plugin, and ChatGPT was not that helpful. Luckily, we finally figured it out. Now we can more easily uniquely identify and find where elements are defined!
Hurdle 6: Put Application into Zoomable Editor
Progression of Putting Harmony into an Editor
A functionality that we wanted was for Harmony to behave like Figma or Canva where you have your edit tools on the side and in the middle, you can zoom in and out of the application. Doing this inside of an existing application was very difficult. Here is what we tried:
- Using an iframe. This would be ideal but it does not work well for authenticated applications. Also, we had to setup a proxy for all the api requests, so no bueno.
- Manually taking everything inside the body of an application and putting it inside of our editor. This is the approach we opted into for the longest time until we got rid of the zoomable functionality altogether.
Moving everything inside of the body tag posed a few problems.
1. What if there is a modal that spawns? This modal needs to spawn inside of our editor, not on top of our editor. The solution is to setup a mutation observer that will listen for any added elements to the body and then move them to be inside of our editor
2. When moving an element into the body tag manually, the React fiber for that element is now out of date. So navigating across the application would result in a bunch of React errors. The solution is to manually update the React fiber references so that the parent element is no longer the body tag, but instead our editor root tag.
3. Whenever a user's application used React.createPortal and we used our mutation observer to move this element inside our editor root tag, the React fiber was all messed up in a different way than problem 2 above. The solution is to overwrite the React.createPortal function with our own implementation that fixed this problem. This, however, led to problem 4
4. In order to overwrite a user's React.createPortal function, we have to have a reference to this function. This meant that a simple cdn script would not do--it had to be an npm package. This removed our ability to have a simple script tag that would work for React, Angular, and other frameworks. We were too locked into React at this point...
Another problem we faced with the zoomable application was getting the zoom right. On Figma, it zooms in exactly where the cursor is. This turned into a math problem where we had to figure out the correct formula to make sure that it behaved like Figma.
Hurdle 7: Automated Drag and Drop
A Full Month of Working Mainly on This
Another functionality we wanted was, instead of using a complex attribute panel like Figma, have the ability to drag and resize elements on the screen and have the underlying padding, margin, and gap be automated. We also wanted it to snap to certain css friendly properties like flex justify and align-items properties. This, of course, is very complex and has proved to be one of our biggest hurdles.
We went through many iterations of systems that would track the edges of an element and its parent in order to calculate the margin, padding, and snap points that correlated to the correct justify between, around, etc. properties. After we would get one application to work, we would try another application and see a wave of new bugs/scenarios to consider. After a month of iterating, we decided that we would push off this hurdle for a later date. But we were able to make a lot of good progress and showed that this thing is definitely doable!
Hurdle 8: CSS Break Points
At this point we can handle text and css updates pretty well. But only for desktop screen sizes. Tailwind has a specific way that it handles break points for screen sizes. The problem we were running into is that maybe we were editing an application that has 16px of margin for a large screen size and 8px for a small screen size. Updating from 16px to 12px meant updating the tailwind class lg:m-4 to lg:m-3 NOT the class that shows just m-2.
The way we solved this was to try and merge tailwind classes at different break points, starting from the desktop size one and going to mobile, and see which one is merged. As soon as we can merge one, we stop. If we never are able to merge, we just put on the tailwind class without the break point. So in the previous example, it would try to merge 2xl:m-3, then xl:m-3, then lg:m-3. Here, lg:m-3 would merge with the existing lg:m-4, and we would stop.
To be continued (we have a lot more hurdles to unveil)...