I orignally wrote this on RemoteBase. You can now read it here because RemoteBase is not live.

RemoteBase was originally written with Meteor, a fullstack JavaScript framework. I recently rewrote it without Meteor. In this post, I would like to share why and how I migrated the whole application from Meteor to React + Redux + Express.

About two weeks ago, I pulled an all-nighter because my server kept failing due to an unexpected level of traffic RemoteBase was suddenly getting.

I wasn’t sure if I should be happy for the high traffic or frustrated due to the server fault. I was sort of feeling an odd mixture of both emotion. One thing was sure; I was pretty tired and the current setup wasn’t scalable.

What happened?

It was the end of a long weekend in Sydney, Australia. I thought I’d spend the day working on RemoteBase’s the user interface. So I got to work early in the morning.

a whiteboard showing UI drawings
I was working on UI on that day

In the morning, I was sort of bored. So I put RemoteBase on Hacker News in the Show HN section. While I had been meaning to do so, I had been putting it off , looking for the best moment. But I knew that perfect timing would never come, and I just went ahead.

Later that day, it got to the first page of Hacker News and was also hunted on Product Hunt. The number of concurrent users rose to 200.

And the problem began to rise. First of all, it was taking forever to fetch the data from DB, even though the DB was up. Secondly, RemoteBase was dying periodically and giving internal server error to visitors.

What did I do wrong that made the whole app crumble with meager 200 concurrent visitors?

Let’s have a look at what was wrong with the setup.

Original setup

As mentioned, RemoteBase was using Meteor. The database was MongoDB, and the frontend was written in React. If you are curious, the original version is open sourced on this repo for the community.

The original version was running on a virtual server from Digital Ocean with 2GB of RAM, and a dual core CPU. MongoDB was running on the same server locally.

Here is the post-mortem of why the app broke:

Not enabling oplog tailing in a Meteor app

Meteor observes oplog on MongoDB so that it can publish the latest data to all connected clients in real time. Basically, whenever a data is mutated in the DB, that operation is pushed to the tail of the oplog. Meteor detects the new operation and notifies the clients.

While such real time feature is very novel, RemoteBase doesn’t really need it. All in all, the core functionality of RemoteBase is just a filter and a list of companies. So I simply ignored setting up oplog tailing; I did not provide OPLOG_URL environment variable required by Meteor.

What I overlooked was that data publications from the server were still real time. I forgot to turn off reactive updates for publications. And the Meteor was spawning processes to poll the DB and calculate diffs every 10 seconds or so. Such task is very CPU intensive especially at scale.

Here is how server was publishing the list of companies to the client (here is the actual file).

Meteor.publish('companies', function (query, limit) {
  check(query, Object);
  check(limit, Match.Optional(Number));

  query['hidden'] = {$ne: true};

  let option = {sort: {official: -1, name: 1}};

  if (limit) {
    option.limit = limit;
  }

  Counts.publish(this, 'companies-counter', Companies.find(query), {noReady: true});
  return Companies.find(query, option);
});

option should have included {reactive: false} and the server would have not tried to poll/diff the DB. See this documentation.

Inadequately setting up MongoDB

I did not have indices in my database. There were only about ~190 companies in at the moment, so I did not think much about indexing. But indexing the database would not have hurt.

Also, putting the MongoDB instance on the same server as the app was not wise. The server supported two threads because it had a dual core CPU. The MongoDB process uses a thread for each socket.

On top of all the threads the MongoDB was using, Node.js event loop, and nginx also uses new threads. So at scale, the server’s resources could not keep up with the requirement.

Only having a single instance of an app

Although the server had two CPU cores, the app was not making a full use of them because there was only a single instance running. There should have been multiple instances of app and a load balancer.

Luckily, there was a package called cluster that allowed me to load balance multiple Meteor apps simply by installing it, without trying to set up nginx or HAProxy.

Simply installing the package didn’t help much. I had to buy more CPU cores before I could see the benefit of load balancing. The reason was probably that Meteor was already spawning so many processes for polling, and that MongoDB was already consuming a large amount of resources.

Rewriting without Meteor

After wrestling with the server all night, I concluded that there needed to be a more scalable and hassle free setup. So I decided to move to a simpler stack, without Meteor.

Certainly I could have addressed the problems identified above and kept using Meteor. Yet the main reason those issues substantiated in the first place was because I was using Meteor, a framework that RemoteBase does not really depend on.

Meteor helped me build and test my idea really quickly and it was time to move on. I immediately started rewriting the app. After a week, I shipped the new version.

New technology stack

The new app uses React and Redux in the front-end. The server side written in node.js with Express. The database is still MongoDB.

The app no longer subscribes to a data source from the server when fetching a list of companies. It simpley makes an API call and saves the result in the Redux store. MongoDB is now hosted externally.

How I rewrote the app

1. Setting up development environment

The frontend development is kind of overwhelming because there are so many things to set up before one can get an app up and running.

There are a lot of boilerplate repos for setting up the dev environment. I decided not to use them because I knew that technologies in those repos will come and go, and a day will come when I have to fix or customize the setup.

Without being familiar with the technologies, it’s not wise to base a production app on a boilerplates like webpack-react-redux-babel-autoprefixer-hmr-postcss-css-modules-rucksack-boilerplate By the way, that repo does exist and it shows how complex front end development is.

So I spent a two days learning and setting up my own development environment. You can call it webpack-react-redux-babel-autoprefixer-hmr-scss-modules-universal-auth-boilerplate. Now that I know every nook and cranny of the setup, I feel confident keeping stuff up-to-date or replacing parts of the setup.

2. Copying over React components

I copied over all my React components from the old repo to the new repo. Their functionalities were decoupled from Meteor because the business logic was not hard-coded, but was injected.

In Mantra, which is the application architecture RemoteBase is based on, all dependencies and behaviors are injected into the components by ‘containers’. So I simply discarded the containers, and stubbed out the dependencies in the components. After that, the new app was able to render the components.

The decoupled architecture of the app made it very easy to remove things I didn’t need. Our lives as developers can be made easier if we constantly ask ourselves: “When, not if, I have to rewrite this part, how painful would it be?”

3. Making the components work with real data

The components were using fake data at this point. To fetch the real data from the DB, I wrote a simple API with Express and Mongoose which is a node.js MongoDB driver. It did not take long to render the components with real data.

But I wasn’t sure where to store the data on the client side. In Meteor, I had MiniMongo, which was a MongoDB-like storage on the client side, but it had been long gone now.

Well, I heard about Redux before, so I learned Redux one morning and started using it. There’s really no sense in saying I learned Redux because it’s just a simple library. Anyway, the data had a place to call home on the client side, and I was pretty happy with the implementation.

By this time, the app’s core functionality was working without Meteor.

4. Server side rendering

For the original version, RemoteBase was using a third party service to render HTML to serve to search engines. Doing so was necessary because Meteor apps are single page apps. The server does not generate static HTML, but rather puts data on the wire.

But relying on the third party service for such an important feature means that I don’t have control over the problem, and that I have to continually pay for someone else’s service.

Luckily, server side rendering was trivial when I moved out of Meteor.

I mounted an Express middleware to match the incoming request’s URL to the routes defined with React Router, and serve a raw HTML using React’s built-in renderToString API. And the server side rendering was working.

But to be honest, the process wasn’t all a breeze. I spent a full day trying to figure out how to ensure that the server fetches the necessary data before rendering the components.

For example, when rendering the main page shown below, how is the server supposed to know that it needs to first fetch the company list? On the client side, I can simply use a component lifecycle method componentDidMount to fetch the data, but the method never fires on the server side.

a screenshot of the main page of RemoteBase
Main page - the list component needs company data before it can be rendered

In other words, I was faced with a problem of binding data to React components without relying on component lifecycle method. redux-connect library solves that problem excellently and RemoteBase uses it in production.

What changed after rewriting?

Although the current version looks and functions similarly to the old version, so many changed under the hood.

1. Deployment is faster

Deployment was kind of slow in the previous version of the app, because building the Meteor app alone used to take me 2~3 minutes. Then I needed to upload the bundle to the server.

Although the process was automated by meteor-up and a continuous deployment, the wait time was not pleasant. And the version of meteor-up I was using was no longer maintained and gave me error messages every now and then.

I tried to fix it, but meteor-up’s source code is not mine, and it takes a while to comprehend what goes on inside. So it was not economical to delve into the source code in a hope that I could optimize the tool just for my use case.

With the new version, deploying takes me up to 30 seconds. I ssh into my server and execute a shell script I wrote. It pulls from the remote, compiles with webpack, and runs the app. Soon, I will probably automate the whole process so that a simple git push will trigger a deployment.

2. I have a full control of my stack

Now that RemoteBase is not bound by a full stack framework, I can easily switch parts of the application. If I don’t like the database, I can write a migration script and use Postgres, for instance. I almost did so just because I could, but ended up staying with Mongo because it performed just fine for now.

If there is a performance or maintenance problem in the future, I could rewrite the API server with compiled languages like golang. In fact, I almost did so, but didn’t deploy that version because node.js allows me to iterate really fast, and works fine.

3. I miss user management system of Meteor

User authentication is one thing I kind of miss about Meteor. In Meteor, I could simply call Meteor.loginWithTwitter() to allow users to login with Twitter. Meteor would take care of everything including login tokens and securities.

But with the current setup, I actually have to write some code. I have not enough time to write it. So I dropped the Twitter login functionality for now.

Concluding thoughts

RemoteBase was able to launch within 10 days using Meteor. Meteor has served its purpose very well in allowing me to launch and test an idea so quickly.

But every problem has a right tool for fixing it. And I don’t see Meteor as the right tool for RemoteBase in the long term.

I think it’s important not to get overly attached to or be religious about technologies we use. Once in a while, we should stop to reevaluate if the current tools and architecture are a good fit for the problem. If not, we should be ready to move on.


This used to be published on my RemoteBase blog, my previous project.