A few months back our CTO Dan was so stressed out dealing with a scaling issue that he started to look green in the face.
He wrote a great post about that outage, and it inspired me to talk to a bunch of other CTOs and find out what is keeping them up at night. I solicited feedback through a few of our networks: Tech Stars, 500 Startups, and Heavybit, and ended up with 17 great interviews, 6 of which I’ve already posted here.
What do CTOs fear most? Turns out it’s not unexpected downtime. For the most part, CTOs are not worried about how to be great technologists. They’re thinking about how to be great leaders.
I really enjoyed these thoughtful responses. Hope you will too!
In this post:
- Allen Rohner, CTO & Co-founder of CircleCI
- Alex Haro, CTO of Life360
- Joe Ferris, CTO of thoughtbot
- Matthew Romaine, CTO & Co-founder of Gengo
- Rahf Noor, CTO of Markerly
7. Allen Rohner, CTO & Co-founder of CircleCI
What was the biggest problem you faced in the past 6 months?
When we started CircleCI it was 2 guys. Now we have employees & customers. I spend a lot of time thinking about how to scale the team and how to scale our product.
When should we take on technical debt and when not? Is a duct-tape solution OK for now? At our stage, if the technical solution is 100% stable and not being strained at all, it probably means we’ve spent too much time on it. We need to move fast to figure out what our customers need, but if we go too fast we’ll release bugs and they will have a bad experience. We’ll also build up technical debt that will bite us later. There’s always that tension between safety and agility.
On the team side, there’s a similar kind of debt you incur when you don’t invest in recruiting and hiring. Hiring is hard and takes a long time. Do we invest the time now for the long-term gain?
What is something that you’ve been worried about in the past 2 days?
My focus this week is marketing, which ended up meaning I’ve been working on our analytics backend so we can track conversions. I’ve been thinking about what things are important to track and the best ways to do that.
I think right now our most important metric is new users getting to “success” — how many of them get there and what percentage of them get there. But even once we measure that, how do we know if the number is good or bad? How do we know what is possible?
What’s different about being a CTO vs something like a lead engineer?
Understanding the business model. Making technical choices that shift the business.
My experience before was in big companies where it seemed they didn’t even want you to know the business side. Pricing is something I think about a lot now and never had to think about before.
Also, there are trade-offs between business goals and code. We can spend a day optimizing some algorithm to reduce server costs, or we can just buy more hardware.
Describe a far-out future risk or challenge you’ve been thinking about.
What the business model will look like in two years. What are the technical choices we will make that will change our market? We started in a small niche, but there are any number of features we could build that expand our addressable market. Those technical choices impact sales, marketing and hiring. What features are possible to build? Which features are good investments?
What is your biggest fear?
It feels like a juggling act. Keeping servers running. Getting new customers. Hiring. Things have to be going well in many areas or the whole things falls apart. How do you know you’re prioritizing the right things?
What’s your favorite war story (work related crisis you overcame)?
I have lots of those stories :). Thankfully they are getting less common as we’ve added more quality checks. I don’t have a particular story to share but there’s definitely a pattern. All of the interesting failures involve multiple bugs at the same time, and you find out that thing you never thought mattered actually really does matter.
One thing I’ve learned is that your success in handling a crisis is almost always more communication-related than technical. If you let your customers know you’re having issues right away, they’re very forgiving. Wait for them to find the problem on their own and their reaction will not be so nice. On the triage side, we’ve found that putting all of our team members in a chat room is crucial during an issue. Everyone knows what’s happening and you have people to double-check your thought process — -“Are you sure you want to restart that server? It’s going to impact X”. That kind of thing.
What do you like about your job?
Multiple things. I like having control of my destiny. In a big company “you are doing X this quarter”. In our company, we talk to users every day, find out new problems, and can make them happy on the spot. That’s really rewarding.
I think one reason big companies make poor decisions is that one person rarely understands the cost and benefits of a given decision. The engineer understands how to build it, the product manager sees the time it takes, the CEO sees the money it makes, and the CFO sees the costs. One of the things I love about being on a small team is that you can see all of that at once. You have a much better understanding of how your decisions will impact the business.
CircleCI gives web developers powerful Continuous Integration and Deployment with easy setup and maintenance. @circleci @arohner
8. Alex Haro, CTO & Co-founder of Life360
Alex, what was the biggest problem you faced in the past 6 months?
Maintaining a great company culture. When we were small, it was easy to make sure everyone was in sync and motivated to accomplish great things. As we’ve grown, maintaining that focus and energy has become a lot harder. It really requires dedication throughout the entire team to make sure that the culture scales.
One thing that has worked really well at Life360 is our designation of an official meeting day. By setting aside all major meetings for Tuesday, we now have a day when everyone is in the same place at the same time to plan, collaborate, celebrate and commiserate. It also allows everyone to be empowered to get their jobs done for the rest of the week.
What is something that you’ve been worried about in the past 2 days?
How do I hire the next 5 people?
Is our release schedule on time?
Are our servers going to stay up?
Where did that bug come from?
Priorities priorities priorities!
Focus focus focus!
How is everyone getting along? Are there any issues I need to address?
Is everyone clear on the vision of where we are going?
How can we make this simpler?
Is my inbox at zero? Where did all these emails come from?
What’s next?
Describe a long-term challenge you’ve been thinking about.
Managing a healthy work / life balance. It is a cliche for startups to say they are work-hard play-hard, but too often only the first half actually is true. I need to make sure everyone knows when to push hard and when to step back and decompress. Life does exist outside of work and it’s important to not lose sight of that.
What is your biggest fear?
Falling off the rocket ship. Right now, Life360 is in the very fortunate position of having great traction and massive scale. It is entirely in our hands not to mess up. I spend every day making sure we can all hang on to the rocket ship even longer.
Life360 is a family locator, messaging tool and communication app all in one. @life360
9. Joe Ferris, CTO of thoughtbot
What was the biggest problem you faced in the past 6 months?
In the past year, we grew from one office to four. When I first started as CTO, I was helping to grow our single office while maintaining and improving our processes, tools, and talent. Working hands-on with a few teams is much easier than tracking and participating in discussions spread out across the globe. It’s been a real challenge to put outlets and processes in place for this kind of communication and collaboration.
What is something that you’ve been worried about in the past 2 days?
We spend almost every Friday on “investment time.” Our developers and designers work on pet projects within the company, contribute to products, and work on open source. A lot of this effort is undirected and happens in solo, though. The past couple of days I’ve been thinking about how to push towards working in teams to improve cross-pollination and give people more of a sense of purpose with their projects.
Describe a long-term challenge you’ve been thinking about.
We’ve done a great job of developing talent and knowledge with specific tools like Rails and iOS. We’ve also streamlined a lot of our processes around those technologies. I spend a decent amount of time figuring out how to make it possible to experiment with new platforms without introducing too much risk into our current strategies.
What is your biggest fear?
I think my biggest fear is that our industry will dry up overnight because of some kind of investor crash or something. It’s always the stuff that seems totally out of your hands that’s scariest.
What’s your favorite war story (work related crisis you overcame)?
There was a period where we weren’t very focused on hiring and three of our developers left the company, mostly because they were moving away from Boston. We’re a talent company, so it was really important to replace them with other great developers. Since we hadn’t hired anyone for a while, though, our pipeline had gotten a little stale. We did a few things to get the process moving again, but my favorite was http://www.apprentice.io/, which was a new iteration of our apprenticeship program. It turned out to be an amazing way of discovering, growing, and hiring talented developers and designers. We continue to use it when growing our offices today.
What do you like about being a CTO?
It’s been a more difficult and rewarding challenge than anything else I’ve ever done. I get to work with talented people every day and discuss how we can make great things better. Being at the center of such an amazing group of developers every day is awesome. It exercises more, different areas of my brain than just doing development.
thoughtbot. Expert designers and developers who turn your idea into the right product. @thoughtbot @joeferris
10. Matthew Romaine, CTO & Co-founder of Gengo
What is something that you’ve been worried about in the past 2 days?
Not giving my team the space they need to flourish, because I enjoy building product so much 🙂 The hardest work to delegate is what you enjoy doing. I don’t code as much as I used to, but there are still parts of the system that I know best and it’s hard not to get sucked in. So my most recent worry was “Am I stepping on someone’s toes?”
Describe a long-term challenge you’ve been thinking about.
Scaling up the team; it’s one thing to scale infrastructure — lots of opinions online about that — but it’s another to scale human interaction. There’s much written about the benefits of small, modular teams. But for all pieces to work together, teams need to have consistent guidelines, processes, communications, etc. I also think it’s important to have a feedback loop with customers and users as engineering operations grow. That doesn’t have to mean engineers should be speaking directly with users. Feedback via customer support, marketing, and other touch-points needs to be fed back to the engineers. As operations grow, ensuring there are no holes in the feedback loop is a challenge.
What’s different about being a CTO vs something like a VP Eng Role?
As CTO I’m now much more visible in public. I spend a lot of time sharing the fruits of the engineering team and explaining why what we’re doing is cool, as well as obtaining new tech info to feed back into team. Blogs and prose online only give you so much perspective, where as in-person feedback from personal meetings and conferences like OSCON are important because not everyone with valuable experience is an avid writer.
At Gengo, the VPE is the true team leader, responsible for day-to-day management of the team and execution of deliverables. It’s an incredibly challenging job — dealing with inter-personal and team dynamic issues, driving deadlines, selecting and solving technical issues, etc.
Together we collaborate on the bigger-picture items too — from architecting the system for long-term growth to hiring, motivating, and rewarding the team.
What is your biggest fear?
Miscalculating opportunity cost. By that I mean not adopting new technologies that would improve our system significantly at the right time due to implementation concerns, and then having a competitor zip by because of it. Or, not adopting important technologies early enough and increasing our technical debt instead. In other words, I fear missing the tipping point of when building on one technology turns into managing it as technical debt.
What do you like about being a CTO?
I have so many opportunities to meet incredibly smart and amazing people — sometimes the CTO title itself can help open doors. I love sharing stories and challenges and learning from others’ lessons, then bringing them back to the team. I also enjoy sketching and discussing new architectures with my VPE, and making big plans for the future growth of Gengo.
Gengo is the translation service behind global companies. @gengoit @quanza
11. Rahf Noor, CTO of Markerly
What was the biggest problem you faced in the past 6 months?
One of the biggest challenges we had was scaling our infrastructure to support our rapidly growing user base. Six months ago, we were collecting data on users, scoring them and placing them based on the interest graph we’d created. We collected dozens of interaction points on each user, and were rapidly scaling. At the time, we were at around 9 million users, and growing significantly month over month. Our infrastructure needed a complete overhaul. We moved to a schema-free database, and switched all data collection to logs. Additionally, we moved all client/user loaded assets to a CDN (content distribution network).
What is something that you’ve been worried about in the past 2 days?
We’re going to be on-boarding our first major brand resulting in significant revenue for us. We need to make sure that we deliver the data/reporting on time and have no server interruptions/hiccups.
Describe a long-term challenge you’ve been thinking about.
Eventually, we’d like to make all of our reporting real-time. Currently we are generating end-of-day-of reports, and future clients may want reporting on demand.
What’s different about being a CTO vs something like a lead engineer?
A lead engineer is responsible for their team and current project. A CTO must balance the current project and team while budgeting time and considering long-term business goals.
What is your biggest fear?
Not necessarily a fear, but something that lingers in the back of my mind — I don’t have an extensive engineering background (~ 5 years), so I wonder if we will get to a point where my experience, background, and skill set are insufficient for our company.
What do you like about being a CTO?
Being able to choose how we want to attack a problem from an engineering standpoint. There are many ways to accomplish a task, and I’ve been afforded the ability by my co-founders to make the call when need be.
Markerly is a content agency that connects your brand to influencers. @markerly @NoorRahf
Checkout these other great interviews, too!
- Robert Fan, CTO & Co-founder of Sharethrough
- Charlotte Genevier, CEO and former CTO of The Cotery
- Dean Shold, CTO of Stanford Hospital & Clinics
- JJ Zhuang, CTO & Co-founder of Acompli
- Dan Kador, CTO & Co-founder of Keen IO
- Subbu Balakrishnan, CTO and Co-founder of Good.co
Coming Next:
- Tony Amoyal, CTO of GetSidewalk
- Matti Paksula, CTO of AppGyver
- Florian Motlik, CTO of Codeship
- Mahmoud Arram, CTO of TriggerMail
- Tim Jenkins, CTO of SendGrid
- Anonymous, CTO of Mystery Company
