Building a Data Visualization Tool in JavaScript

Rendering more than 200k data points on a web map is not an easy job. You need to pick the right tools and refine some parts of the application until you achieve the best performance.

We wanted the application to perform rendering as fast as possible. Rendering thousands of data points on the map was crucial for the product. Therefore, we decided to choose small-yet-powerful libraries.

With this simple stack, we started the development of our data visualization tool which we’ll call ORM-Viz for reference.

The rest of our architecture is very straight-forward: we use React to build the application’s components and Leaflet to draw markers on the map, each marker represents a record in the Database.

Here we’re rendering a thousand markers, each one representing the train location at some point in the selected time frame. The whole process takes in average 800 ms from the moment when the user clicks on “Apply” until the moment where the points are drawn on the map. For us, this is considered slow and this slowness is caused mainly by the fetching of data. Adding to this point we had to make a DB request every time we wanted to render the train’s location which quickly became a bottleneck in our goal to achieve smooth visualizations.

Simply projecting the data on the map is already helpful to analysts who want to verify the data they collect from the trains but we think that animations improve the level of analysis, for this reason we implemented a basic feature to animate the time slider while displaying the data corresponding to the selected time range.

Under the hood, we use setInterval inside a react hook and every second it triggers a DB query to select the records that match the time frame, then the application’s state is updated with the new markers which kicks off a re-rendering of the Map component. We got a satisfactory result in the animation considering the fact that we are still fetching the data from the SQLite DB every time the slider is updated with the next time range.

It took one week to have a working prototype based on Kepler.gl, we had to integrate the Open Railways Map which involved a significant amount of time spent on research, and because our users don’t need all the features that come with Kepler.gl we also had to simplify the User Interface. Fortunately, UI customization is well documented in the project therefore it was done quickly.

Another advantage of using kepler.gl as the foundation of our application is that we can have in-memory datasets therefore we don’t need to make DB requests every time we want to render the markers. With Kepler.gl you load a dataset from a source (file, service, etc.) keeping all the data in-memory. This improved drastically the rendering time during animations.

Developing a web-based visualization tool requires special focus on rendering times and techniques to leverage resources as much as possible. In case you need to build a web-based data visualization tool, I strongly suggest you use WebGL for rendering data and avoiding multiple requests for data-fetching as we discovered in our journey.

Related posts:

A fun perk about working in publishing is: sometimes you get free stuff. By sometimes I mean “often” and by stuff I mean “books.” Pre COVID, my job had a thrice yearly Free Book Order period for…

The greatest scientist of the 21st century, Stephen Hawking, believed that it is possible to move in time not only a person, but also individual objects. True, he thought that such a trip was like a…

It seems that these days “become a freelancer” is the end all be all solution to your problems. Is your job not fulfilling your underlying desires? Go freelance. Do you struggle with the everyday…