In this morning’s inbox from the founder of roamsoftware.com.
As some of you already know - this week, Roam committed the cardinal sin for a writing app and failed to save some of our community's notes.
Even if you weren't affected or the impact on you was minimal you deserve to know what happened, so I'm emailing everyone who has signed up for an account.
As best as we can tell, here is what happened.
There were a number of windows of time this week where we saw a huge spike in users writing large amounts of data. This was predictable result of Nat Eliason's excellent intro to Roam post (see here: https://cwhitesullivan.acemlnb.com/lt.php?s=57e0e22b2822ec7764fbc27434319c09&i=14A158A1A37 hitting the top of hacker news recently, and a big influx of new users ramping up usage with Roam. What we did not predict was that during these windows, our backend servers essentially got "clogged" as we hit our write limit.
When this happened, Roam failed in a few ways
1. Users who had multiple tabs open, or who were using on multiple devices, or collaborating with others on a shared database saw their devices go out of sync and start writing over each other
2. Some changes may have been dropped completely if you closed your Roam tab or refreshed the page before the server had processed them.
What was especially bad about this failure is that we did not have any way of knowing that the server had failed to process changes, so all of this was happening silently.
We've had largely the same architecture for saving and syncing changes to Roam workspaces for the past 2.5 years, and this sort of thing has never happened before. But things change as you scale, we were wrong about how long we could keep doing things the way we've been doing them, and the responsibility for that is on the Roam team - in particular myself.
This afternoon PST we pushed up some changes that should solve the problem of clogged servers during peak use times -- and we're working on larger changes that so that silent failures to save data don't happen in the future, and for a more robust offline saving system so that even if you lose your connection you'll still be backing up the changes locally.
I don't know when we'll have these improvements finished. I can tell you that we're working on it now, and reliability is our highest priority.
We know it takes a lot of trust to put your second brain into an unproven startup, we're exceptionally grateful for that trust, and it pains me that (rightly) we lose some of that trust every time a failure on our end causes anyone in the community to lose notes.
The #RoamCult has been growing by leaps and bounds in the past few months, we went from a few dozen workspaces in August to tens of thousands today, but the team is still just myself and Josh full-time at the moment.
We really appreciate the patience and support we've gotten in the slack channel, and on intercom, and those of you who messaged us there and over email to first report these issues.
I hope we can continue to win some of this trust back.
As a final point - a few of you have asked how you can help
Right now, my personal focus is on hiring. At the end of last year (thanks in large part to all of the noise you all have been making on Twitter) we were able to raise a sizable war chest to ensure that Josh and I can keep working on Roam indefinitely, and I'm happy to say we now have the resources to grow the team and pay competitive salaries.
Our goal in the next few months is to hire 1-2 full time senior engineers (with experience in Clojure and preferably Datomic) to help us improve our backend, ship offline apps on desktop and mobile, enable more granular sharing and develop an API for all of you who want to start building apps on top of Roam and pulling in notes and data from other tools you use.
We're also on the lookout for someone with extensive React experience -- and preferably some experience with Clojurescript -- who can help us speed up performance and rendering across the site and ship a host of new features and workflow improvements that we hope will make Roam an even more powerful tool for thinking, learning, and organizing your life.
If you, or anyone you know might be a good fit for either or both of those roles, please send them our way -- ideally via this link https://cwhitesullivan.acemlnb.com/lt.php?s=57e0e22b2822ec7764fbc27434319c09&i=14A158A1A36
Thanks again for all your support, for those of you whose notes have been lost from this bug - or any other - know that we're mourning their loss with you -- even if we aren't always promptly responsive on Intercom or to your emails.