Recovery of Failed Services

[Georgette]

Not sure where this belongs, but wanted to document it somewhere.

If any one of the services fails, for example, I had an issue with the github service failing due handling the data, and not accounting for certain scenarios. When this happens, obviously we can restart the service, but in the case of the npm updater, how do we not lose data while a service is recovering, because the updater doesn't know about the services failing?

This also comes paired with scaling the services so we have multiple copies of services for bulk loading data from npm.

It feels like we need a leader, or a puppet, that directs the message traffic for both load balance, and service recovery.

[mcdonnelldean]

It depends on the microserice. Npm, Github, and Travis are all backed by redis, search by elasticsearch.

In the case of the updater, modules downloaded via the download all function should be first placed in redis (just like in github) and then batched to the queue. The queue itself is redis based so it is hard saved.

We will still lose some data in transit but this is ok, the nature of modules is eventual consistency. We will also be offering a sync button to modules who's cache is outside a certain timeframe. This means we can override an update in case of loss.