User avatar
AndrewC
GpsGate
Posts: 315
Joined: Tue Apr 01, 2014 12:30 pm
Location: Sweden
Contact: Website

Dispatch 5 developer blog

Mon Oct 30, 2017 6:48 pm

Dispatch 5.1 architecture

As a developer myself, whenever I read a set of patch notes or a feature blog post, I'm always interested in the details behind the note. We try to make our notes a little more specific than the infamous "Various bug fixes and improvements" when we can, but there's always a deeper story behind that one line of text. For this Dispatch beta period, we're going to release a series of developer-centric blog posts to provide some more context to those who are interested in the direction of Dispatch.


If you are light on time, our recently released patch notes will tell you about the new features we've made and the old ones we've retired. Our GpsGate blog posts will explain how our new features can help your business and fulfill your customers' needs.

These developer blog posts are designed to provide some extra insight on the creation of the plugin for those looking for more technical details or the intent behind the various features and are purely optional reading material.

In the coming weeks we will have some deep dives on the design and the implementation of our various key features touching on topics like our new native apps, our route management and optimisation, the analysis panel and our developer APIs.

This week we'll be looking at our motivations for redesigning both the server and the user interface of Dispatch.

Let's start at the very beginning.

Why?

The main motivation for this redesign of the Dispatch plugin was to address 3 key issues that we were having with Dispatch 4.

  • Performance
  • The resistance involved in creating new features
  • Usability of the UI at scale


Performance

Performance is one of the simplest concepts to understand in software for both users and developers. While using the features of the program, everything should work responsively and not stutter and stop when it's stressed. To get to this level of usability in modern software with its terabytes of data flying around between thousands of interlinked features, performance, for developers becomes a never-ending quest and a state of mind.

In a piece of software, the very slowest component immediately becomes your performance bottleneck. It's obvious when you think about it. Who cares if the server is flying at 88 miles per hour if your user interface is immediately choked by the data it's getting. If your data storage takes a long time to load your data, every subsequent component is waiting around idle.

To achieve good performance, every aspect of the system between the user's screen and the store with your data needs to be designed in a way that does not block anything else. Every time you implement a fix or feature, you need to consider how it will work when there is a lot of data and a lot of users. If you do this in every aspect of your system, then you can have good performance.

Updates

GpsGate Server is used by many different types of partner and solves so many business cases that we won't ever be able to add every feature that everyone would like. Despite this, we have a strong commitment to working with our partners and turning their feedback into features whenever we can. We typically have a much faster turnaround time on features and fixes than most software developers and we look for opportunities to design our systems in a way that can help us with this goal. It's important to us that Dispatch provides a platform that aids our ability to be responsive with updates and we saw an opportunity to make it better at this with Dispatch 5.

Usability

When a large system is working smoothly and there are no performance issues, the next thing to address is whether the data that you can serve to the user is useful. If your system can display 1 million items to the user, can they use that information in a meaningful way? There's a chance that you're just making a lot of noise that is hard to interpret. For the performance to be meaningful the system needs a way of displaying all the information that will help the user the most, while ideally having the rest of the data close at hand.

How?

The concept of usability is a very complex one and is best addressed in discussions about specific features. We will leave this topic for the posts about our features in the coming week and instead concentrate on performance and adaptability.

These two concepts are unrelated at first glance but it turns out that a shift in design can help you in solving both at once.

Traditionally the best way of gaining performance has been by optimising code to do as little work as it can to achieve the goal that you want. The fewer operations that the processor must perform, the faster it will go. If your processor is faster, it can do more operations in a certain time and the faster your software will go. Both principles still hold true and are still valuable tools to improve performance but in a modern system there are a massive amount of operations taking place at once. Very quickly you will hit a point of optimisation and server capacity that can't really be tweaked all that much better.

With the increase in complexity of systems come more moving parts in the system. This is a blessing as well as a curse. A lot of the functionality of these smaller parts don't need to be very connected to each other aside from when they start and when they finish. With this in mind, it's almost always possible to break each small part of the system down into a self-contained little black box that only needs two things outside of their own world. When they start an operation, they will need to be given instructions and data to process, and when they finish they'll need to tell some other part what they managed to produce.

The Server

In Dispatch 5 we have rewritten the server back end to use a message-driven architecture. All the system has been split into smaller parts that do their own work and spring to life when they are sent a message telling them to do something. When they are finished, they send a message to the other parts that will want to know about the result, who in turn can act on this message.

Breaking the system down into very self-contained parts like this creates a small amount of extra work compared to extremely optimised code that directly links into parts of other systems to bypass a few redundant operations. In exchange for losing a few extreme optimisations you get many benefits:

  • The code to deal with each part of the system is completely separated from other aspects of the system. This instantly solves a lot of our concerns about making sure that the system can be updated and fixed easily. You can change inside one feature without causing a lot of side effects.
  • The code to run an operation of a certain type has nothing to do with other parts of the system. They communicate only by sending a message to each other when a message is ready. This means that we get access to “asynchrony” when it makes sense; the ability to run the code at the same time as another operation without waiting for the other operation to finish. Asynchronous programming is not a new concept and GpsGate Server has taken advantage for it for years, but a message-driven architecture allows asynchrony without adding a lot of the complexity that comes with traditional asynchronous programming.
  • As we just created some asynchrony we now also have scalability. Instead of trying to supercharge our processor even more, we can just throw more processor cores at the workload and benefit from them. An operation can run on any free resource and fall asleep when it's either waiting for a message that it needs, or has no work to do, freeing up the resources for another operation.
  • As all our processes communicate with passing messages we don't need to stop at multiple cores. We can have the different parts, usually called micro-services, live on different servers that are tailored to what they're trying to achieve. A server with a lot of disk storage for the database and a lot of processor power for the route optimisation, for example. GpsGate 5 now has support for the track store running on a cluster of these servers and in the future Dispatch will also be able to be scaled to a cluster that you can tailor for your business case.
This architecture isn't a magic solution and even when you separate code into self-contained parts you will need to occasionally wait for other operations to finish so that you can use their data. You can't work on rendering a report before you have the data finished, for example, but the benefits to a server workload are clear and substantial.

The Interface

Things have subtly changed on the server now. In a message-driven architecture you should no longer start an operation and wait for a response. Waiting is a waste of time as the part of the system you are communicating with will send you a message when it's finished. If you would like, you can even ask it to stream messages to you as it finishes each part of an operation.

This concept is something that we want to exploit on the interface side. Using HTML5 server sent events we can split up our server communication logic to be two different systems. One system sends requests to the server when the user wants to do something or wants more data. One system processes any incoming data that is received from the server. This means that operations in the interface finish immediately and release control back to the user. It also means that the instant the server is finished processing some data, it can inform all the connected clients - web or app - that the new information is available. This messaging happens without leaving any processes sitting waiting for responses on the client side. As there are no processes waiting, it also means that on the server side you don't need to have an operation that is sitting and waiting for the response data for each request that is sent in. Now you don't have a server that is being choked by lots of longer running requests.

As well as the communication with the server being rewritten, our web interface refresh and new apps are designed to work with large amounts of data. A lot of effort has been put into making sure that with this increase in server responsiveness, the user interface can keep up.

We will share more details about the interface in the upcoming blogs.



What is next?

Throughout the beta period for our live Dispatch 5 release we will be concentrating on fixing bugs and tuning the performance even more based on our own testing and feedback from all our beta testers. We are proud of our new system but we're not ready to call it finished. There will be glitches and bugs and we'll be working to get them smoothed out.

Alongside bug fixing, the development team will be making sure that all the features listed in our Known Issues patch notes section are included in the plugin and working before we go live.

Our design team is working on constant improvements to both the user interface and the wider user experience for GpsGate Server and their efforts will be debuted in Dispatch 5 as soon as they are ready. Other plugins might take some time to be updated but Dispatch 5 will be at the front of the line.

While the beta period is ongoing we have some of the team working on our first major feature release. Our current plan is to have a beta build ready very soon after Dispatch 5 is released to the update channel. We will share more about our first feature patch nearer to the end of our beta, but it will introduce some very requested features such as the ability to add media to Dispatch custom fields and a more powerful search functionality in the Dispatch panel.

Feedback group

We will be releasing blog posts during the beta phase highlighting new and upcoming features for Dispatch 5, but we are also running a Dispatch 5 beta feedback group. Members of our beta feedback group will get emails with the developer deep dive posts, be assigned specific feature feedback assignments to communicate with the developers and be a part of shaping the development of Dispatch 5 for our first feature patch and beyond!

Our first round of specific feedback gathering will be targeted at our Mobile apps so sign up now if you haven't already!

Sign up to our beta feedback group here.


Andrew is a developer at GpsGate in Stockholm. He works mostly on designing systems for GpsGate 5 and in his spare time enjoys trail hiking to the fridge.
Last edited by AndrewC on Tue Oct 31, 2017 12:27 pm, edited 1 time in total.
Reason: Sticky during Dispatch 5 beta period only.
Regards,
Andrew
GpsGate Support

slester
Posts: 2
Joined: Thu Nov 16, 2017 4:32 pm

Dispatch 5 developer blog 2

Mon Nov 20, 2017 5:40 pm

Dispatch Mobile Apps: designing mobile apps for end users

This time we are going to talk a bit about which technologies we have used in our mobile app for the Dispatch feature of GpsGate Server.
If you are light on time, our recently released patch notes will tell you about the new features we've made and the old ones we've retired. Our GpsGate blog posts will explain how our new features can help your business and fulfill your customers' needs.
These developer blog posts are designed to provide some extra insight on the creation of the plugin for those looking for more technical details or the intent behind the various features and are purely optional reading material.

Goals
For Dispatch 5’s worker UI, our goal was to have a smoother, higher performing, native mobile-feeling experience for users. We now have a native mobile app for Android and iOS that offers many advantages over our previous solution. The previous worker UI was in the GpsGate Fleet app, a hybrid web/mobile app.
In order to reach our goal, we figured we needed a true native mobile app, which does all of the above better than a web solution. We needed both Android and iOS compatibility, with the platforms having the same features and functions. However, we did not want to maintain two different apps (one for Android and one for iOS). Instead, we looked at solutions that let you write code once and reuse it on both platforms.

After doing some research, we decided on React Native, which is a JavaScript framework for building native mobile apps made by Facebook. We use a lot of JavaScript in our VehicleTracker web app, so our developers already know it well.

Development
Now you might be thinking that JavaScript and native mobile apps don't sound right together. Let me quote Facebook: "With React Native, you don't build a “mobile web app”, an “HTML5 app”, or a “hybrid app”. You build a real mobile app that's indistinguishable from an app built using Objective-C or Java. React Native uses the same fundamental UI building blocks as regular iOS and Android apps. You just put those building blocks together using JavaScript and React."
So React Native lets us write code in JavaScript. We can also write and use true native Android/iOS components, so we are in no way restricted to JavaScript. For example, we use the phone's camera to scan QR codes for fast connection to your server (see more below).
While developing with React Native, you don't have to recompile the app every time you make a change. Instead, the app reloads automatically on your phone (or simulator) whenever you make a change. Less time waiting on compilers is always welcome. This makes development much faster.
We liked React so much that after doing the Dispatch mobile app, we have also started using React in our VehicleTracker web app. We will talk more about that in a future blog post.
Read more about React and React Native here: https://facebook.github.io/react-native/

Architecture
Server connection
We have integrated our new GpsGate Server registration system (pairing service) which makes it easier to connect the app to your GpsGate Server instance.
Operators can generate a QR code for the server in the Dispatch for VehicleTracker plugin. When a worker opens the mobile app, they can scan the QR code using the phone’s camera. The server address is entered automatically, so the workers are connected in seconds.
Once logged in, the app is connected real-time to the server using server-sent events (see https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events). Traditionally, you need to send a request to the server to receive new data (polling). Using server-sent events means that whenever an operator updates a job, that change is immediately received in the app.
We use the same technique in the VehicleTracker web app. For example, vehicle position data and in the new Dispatch for VehicleTracker plugin (but once again, we will talk more about this in a future blog post).

Client state
All the data received from the server must be stored in some way. Most React and React Native projects use the Flux architecture/pattern for that. Flux utilizes a unidirectional data flow (data only flows one way). The other option is two-way data binding that is also popular. See below for a comparison.
twoway.png
twoway.png (23.16 KiB) Viewed 60 times
Image 1. Two-way data binding.
flux.png
flux.png (25.52 KiB) Viewed 60 times
Image 2. Flux architecture. Notice the added Action and Dispatcher steps.

In Flux, the code is much more explicit with less "magic" involved. This should reduce the number of bugs, and allow easier debugging when we actually find bugs. However, it does require a bit more boilerplate code.
In our app, we use an implementation of the Flux ideas called Redux (see https://redux.js.org/). The data flow is similar to the second image above.
1) When you click a button OR receive data from the server, you dispatch an action
2) The client state store is updated according to the action you dispatched
3) Views are re-rendered IF the client state changed

Work offline
A major new feature in the app is the ability to work offline. To solve this, we save worker action history on the phone's storage, if a worker is offline. Once back online, all actions will run again to synchronize with the server.

This was quite simple to implement using Redux actions (see above) and optimistic updates. ‘Optimistic updates’ means that the UI is updated even if the server did not confirm the change.
An example:
1) Login
2) Wait for all the jobs to download
3) Go offline
4) Start a job (a Redux action is sent)
5) Cannot send to server. Store the Redux action in queue and optimistically update the UI
6) Go online
7) Send all Redux actions in the offline queue to the server
8) The UI is reverted in case of failure, otherwise nothing happens since the UI was already optimistically updated

Push notifications
With push notifications, workers will never miss new jobs, even when their phone is locked or the app is closed. Whenever an operator assigns a job to a worker, a notification is sent to the worker's phone, as long as he/she is logged in.
For sending push notifications, we use Google Firebase Cloud Messaging (see https://firebase.google.com/docs/cloud-messaging/), which is a cross-platform messaging solution. We just need to implement it once on the server, and once in the app, and it does not matter what type of phone we are sending to.

What's next?
Right now, everything is in place to make a great Dispatch app, but we need your help!
How can we improve the workflows within the app to better suit your business?
What features are you missing?
Join our beta feedback group to tell us! <a href="https://docs.google.com/forms/d/e/1FAIpQLSdgLAAnZyTzAbcWOh3axqDiWH-pScLzlxkQkSJGLkbbs2Ryow/viewform?usp=sf_link">here</a>.

Mattias is a developer at GpsGate in Gothenburg, Sweden.

Return to “GpsGate Server - How to setup and use”