• Log In
  • Sign Up
    • Max Stoiber shared a great blog post today about the technology choices he regrets making while building Spectrum, a community-focused chat platform. It's an interesting and informative read:

      I found it particularly interesting because there are many similarities between Spectrum's tech stack and Cake's. We faced some of the same choices while developing Cake, so I thought it might be informative to compare and contrast our choices and how things have worked out so far.

      Regret: Building Cake before Next.js existed

      Like Spectrum, Cake needed to support server-side rendering in order to ensure good SEO (Search Engine Optimization). Not all search crawlers can execute JavaScript, so even though we were building Cake as an API-backed React app that could have been rendered entirely in the browser, we also needed to be able to render Cake pages as HTML on the server.

      Next.js is a framework that makes this pretty easy. Unfortunately it didn't exist yet when we started building Cake. In fact, at the time it seemed like we were blazing new trails that not many people had even written about.

      We ended up building our own universal routing layer and data fetching layer. It works well, but it took a lot of time and effort. If Next.js had existed then it could have saved us a lot of headaches.

      Regret: Trying to manage our own frontend infrastructure on AWS EC2

      I wasted tons of time early on by trying to self-manage all of Cake's web infrastructure on Amazon EC2. This included using Terraform to provision AWS resources, using Ansible and Packer to build and configure EC2 machine images, using AWS CodeDeploy to deploy code to our web servers, and more.

      I spent weeks and weeks deep-diving and learning all of these things, setting up infrastructure, coming up with plans for how to manage and scale that infrastructure, and so on. I was miserable. I constantly felt like my brain couldn't possibly absorb new information, yet there was no end to how much more I needed to learn to do things right. Even worse, I somehow needed to figure out how to efficiently share all that knowledge with other engineers. And if I did something wrong, we might regret it for years.

      Finally I realized this just wasn't a smart use of my time. I should have been building Cake, but instead I was building infrastructure and it was stressing me out. So I scrapped it all and switched to Heroku.

      We still use other cloud services, including AWS, for various parts of our infrastructure, but Heroku runs our web servers and makes deployments and frontend scaling dead simple. Now, instead of worrying about that stuff, we can focus on building Cake. I wish I had done it sooner.

      Regret: Building full-text search on Elasticsearch

      We spent a ton of time and engineering effort building full-text search functionality on top of Elasticsearch. What we built actually works really well and overall we're pretty happy with how it turned out, feature-wise, but it took way longer to build than we expected, and the knowledge and time required to keep it running well and to add new functionality is, in hindsight, more of a burden than our tiny engineering team should have taken on.

      Elasticsearch is great, but for our current needs and at our current scale, it's a little like building a car factory instead of calling a taxi.

      If I had it to do over again, I'd probably outsource our search functionality to a cloud service like Algolia. I think that could have saved us a lot of time and a lot of headaches and would have simplified our infrastructure.

      Don't Regret: Switching from RethinkDB to MySQL early on

      We originally used RethinkDB as Cake's primary data store, just like Spectrum did. It's a very promising database with some great features, and overall I really like it. But about five months into developing Cake, we realized it just wasn't going to be right for us. We weren't live yet so we didn't experience the operational issues Max wrote about, but we found that RethinkDB had several limitations that made it a bad fit for some of the things we needed to do.

      Here's a Stack Overflow question I posted back then about one of the problems we ran into:

      So we made the hard choice to switch to MySQL. It's old and boring, but we knew it well, we knew it could do what we needed, and we knew it could scale. We haven't regretted that choice.

      Don't Regret: Switching from Markdown to a WYSIWYG editor

      Originally Cake's post editor was a plain text editor that relied on Markdown for formatting. This worked well, but it made it difficult to support features we knew we wanted (like rich embeds and mentions) and we also knew that non-technical users weren't likely to be familiar with Markdown.

      So after evaluating several options including Draft.js, we switched to a WYSIWYG editor based on Slate. This allows us to have near-complete control over the editing experience, and it means users don't need to be comfortable with Markdown to use links, formatting, and other features in their posts.

      Behind the scenes we store post content in an XML format we call Cakedoc. This gives us a lot of flexibility to add new editing features that would be difficult to support well with Markdown alone, such as rich link embeds and other features that are still in the works. It also makes it easier to render posts consistently on different platforms, whereas Markdown parsers tend to vary widely in their behavior and features and are often focused on converting Markdown to HTML, which isn't the best format for every rendering environment.

      It's not without its challenges: rich text editing on the web is incredibly complicated, so we've invested a lot of time in contributing improvements and bug fixes to Slate. Slate and other rich text editors (including Draft.js) still don't work very well on Android devices at the moment due to some limitations of the Android platform, which means that right now Android users don't get the WYSIWYG editing experience on Cake. This is a big problem, but it's one we think can eventually be solved.

      What we've learned

      It's still early days for Cake so we're far from having learned all we can learn. But like Max, I've learned that flexibility and speed of iteration are incredibly important, and boring tech is often a smarter choice than something cutting edge that's likely to be full of surprises.

      Outsourcing hard or non-core stuff to cloud services and open source tools while leaving pathways open to bring it in house in the future if necessary has so far worked out well for us.

      When in doubt, make the choice that allows you and your team to spend more time focusing on improving your product and meeting the needs of your users, as opposed to scratching an engineering itch or satisfying a desire to play with cool new tech.

    • I <3 this post so much. I always love knowing about the guts.

      MySQL may be old, but you get predictability, and massive amounts of already solved problems with that. It scaled Facebook to 100MM users, so I figure it will work just fine for y'all.

      I still love PHP as a backend language, especially since the advent of Laravel (which plays very nicely with Vue for a front end) and the release of PHP 7. It often also gets sniffed at for a variety of reasons, but again, so many already solved problems. That lets me focus on product building, not framework building, which I find a lot of Devs spending a helluva lot of time on.

    • SmugMug is built on PHP and for awhile some devs would sniff at it as we tried to recruit them. But at the SmugMug/Flickr (also built on PHP) party this Christmas, the engineers I spoke to had come to be very satisfied with it, a change from the complaints I used to hear from those same engineers.

    • Kudos are due to Facebook for investing in Hack which showed how much faster and more scalable PHP architecture could be. 7.x takes a lot of that to heart, and solves a ton of issues that OOP devs complained about for years.

    • I've spent much of my career writing PHP, sometimes loving it and sometimes hating it. It's not my language of choice these days (Cake is JavaScript all the way down), but it's hard to beat PHP for getting shit done without a fuss!

      When I was younger and less experienced I spent a lot of time turning up my nose at languages or tools I didn't like. I've come to realize I was wrong about most of them, and that what matters is getting the job done and doing it well. It's less about which things are right or wrong, period, and more about which things are right or wrong for the job at hand.

      I think a perception that MySQL was old, boring, and not much fun was what initially led me to choose RethinkDB for Cake. It was new, cool, fun, and different. Those things made it seem right, but in reality it was the wrong tool for this job.

    • One of the things I like about having a dev driven team now is that I spend most of my time focused on architecture and devops, and let the developer leads take the reins on testing new approaches for how we build product. That lets them taste the flavor of the day, and keeps them excited, but also provides quick case studies, and reveals some of the flaws inherent in that approach.

      Usually that gets us a "move fast and break things" type result without actually affecting Prod, but not always. We've had a few "ElasticSearch" type rabbit holes along the way, and it's always painful to come to the realization that you did things the hard way, or worse, that you couldn't cover the last 10% no matter how hacky you got, and had to start over.

    • Great post, enjoyed reading it very much (I love when companies do this type of technical transparency).

      From my personal experience, watch that AWS-to-Heroku thing though. I agree 100% that at the current scale and complexity (well, I'm just guessing) of Cake's infrastructure, Heroku would have been an excellent choice and would have saved you time and money, no questions asked. However as you grow, there will be a point where you will have to graduate from Heroku and do your own infra. That threshold is pretty high and can wiuldly vary from business to business and specifics of the situation, but it's there. If you miss the bell, it will become harder and harder to manage and will bite you in the bottom line in multiple ways (runaway costs, developer pipelines crumbling/overcomplicating, issues with monitoring/observability, etc and so on).

    • However as you grow, there will be a point where you will have to graduate from Heroku and do your own infra. That threshold is pretty high and can wiuldly vary from business to business and specifics of the situation, but it's there. If you miss the bell, it will become harder and harder to manage and will bite you in the bottom line in multiple ways

      Definitely agree!

      The parts of Cake that run on Heroku are deployed as Docker containers that have no actual tight coupling to Heroku. We think we've got a long scaling runway on Heroku before switching to bare AWS will be more cost effective, but when the time comes we should be able to make the switch without too much pain (except for the additional operational overhead of managing our own infrastructure).

    • Awesome! Hopefully you won't get bit by the whole Kubernetes bug, too :) (I have a love/hate relationship with k8s - it's definitely a nifty thing to happen but it's a) a huge platform and IMHO 99% of the people do not need it as it is too complex and heavy and b) it's excruciatingly raw inside for something that is at versions like 1.13 and on and trumpeted around on every corner). There are much simpler orchestration solutions.