Why are we not using Cassandra?
That was a common question around 2016, right around the crescendo of the Big Data Hype Cycle. We had just come off the high of HDFS and MongoDB, and engineers were seeking the next hot thing to put on their résumés: should it be Kafka, or Cassandra? How can I steer the discussion topics in my interviews towards the CAP and FLP theorems?
I was one of them, unfortunately, which explains why I know all the buzzwords.
Now that I have crossed over to the dark side of engineering management, I wonder: how should an organization triage and respond to these questions?
- Why should we use X?
- Why are we not using X?
A knee-jerk reaction could be to use some embedded default in the organization’s culture and set expectations around who should bear the burden of proof. Counter-ask: what’s the problem you are trying to solve? Ask the Five Whys.
However, new hires bring fresh ideas, and we want to encourage their questions and suggestions. We want to learn from their past experiences and their past companies. We want them to help uncover our blind spots, so we can continuously do better. Edit the company! Throwing such obstacles in their way could hinder what is already an insecurity-inducing onboarding experience. (I used to know stuff! Now I don’t!) Sometimes people are just looking to make an impact quickly and establish street cred when they join a company.
On the other hand, there is an infinite list of “why nots”. For every observability tool, there are half a dozen competing options, all slightly different, and each slightly better in a nuanced subset of features. The same can be said of every data store. (Dare I suggest that the choice of a recommended default IDE also falls in this bucket?) It does not seem feasible to maintain a list of documents on the wiki explaining “why we are not using X” for every possible X. It is also certainly not feasible to try a Proof of Concept for every Why Not, or integrate with every possible tool out there to get the best of all worlds. (I tried that at Quixey and it was super fun for me, but a terrible waste of time for the company.)
This is exacerbated by increasingly aggressive bottom-up sales techniques. SaaS Sales is reaching out to every engineer in your company, encouraging them to start the conversation. What’s the harm in asking? Such tactics, when coordinated, can be very effective, creating an illusion that everyone is suddenly unhappy about Y and wants you to adopt X to solve it. This can be distracting for the organization, and also exhausting for decision-makers.
What can engineering leaders do to find the sweet spot between these trade-offs?
The Power of Focus
An act of leadership is to point people at specific problems or problem areas that are indeed going to make a difference for the company. It is an act of strategy: identifying the set of problems that contribute to a competitive advantage, or the areas where we want to catch up with the competition.
New hires, when they join the company, are unaware of where these areas of opportunity are. They will jump at the first thing that they can improve. Understanding the holistic view of the company requires context, which takes time. Using the context you have to share your view of the opportunity areas is a valuable act of leadership.
Why are we not using Cassandra?
Instead of asking “why?”, or “why not?”, or “what is the problem you are trying to solve?”, we could instead stem the exploration in this direction and point them in another one.
Using more specialized storage technologies is probably going to make a small improvement for this set of concerns, but it comes at a cost and do not contribute to these other concerns that we really care about. Looking at the product roadmap for the next 18 months, we think that we should focus on these other concerns that will make a significant contribution to our broader strategy.
And maybe that is all that they are looking for.
