Introduction

BigPanda provides an autonomous operations platform to empower IT operations at large, complex enterprises. BigPanda's platform turns IT noise into insights, automates incident management, and unifies fragmented IT operations. Naturally, their infrastructure is a highly critical component of the services they provide.

Challenge

BigPanda uses a microservices architecture and saw migrating to Kubernetes as the right move to help the company address several goals and challenges:

  • Simplify the development process by removing time-wasting manual tasks (and the potential for human error)
  • Prepare infrastructure that allows easy migration out of the California region
  • Unify management of all BigPanda cloud environments—from development to testing and production

With several talented DevOps engineers on staff, BigPanda initially tried to make the switch to Kubernetes on their own, starting with their development environment. However, they encountered some problems because they didn’t have prior experience with the complex Kubernetes ecosystem and its ready-made tools. BigPanda's VP of Engineering at the time, Shahar Kedar, described it like this: 

When our engineers engaged other developers to interact with the environment, a lot of integration issues became apparent. We didn't have ready solutions to these issues, so the engineers had to go back and research and debug. We needed a robust Kubernetes platform from day one. We couldn’t start with something half baked and then experiment and slowly improve over time.

Shahar Kedar, former VP of Engineering, BigPanda

After two months, the BigPanda team decided to enlist outside expertise so they could make the switch faster and with fewer issues. They sought a vendor that was familiar with the Kubernetes ecosystem and that could quickly troubleshoot integration issues.

Shahar saw Opsfleet as the right fit given our deep experience with setting up Kubernetes environments for other companies. However, BigPanda still had some concerns about outsourcing the project:

When you bring in someone external, you know they will leave at the end of the project, taking all their knowledge with them. I was looking for someone trustworthy who could build the solution and then be available to solve issues as we started to work with that solution. Most importantly, they’d need to be able to transfer their knowledge to our full-time engineers.

Shahar Kedar, former VP of Engineering, BigPanda

Solution 

We started with a review of BigPanda's old development process and reviewed their attempts to switch the dev environment to Kubernetes. They had set the environment up so that all processes had to be run locally. 

We proposed a new architecture in which most of the application would run on a remote shared Kubernetes cluster. We suggested Telepresence for creating a transparent two-way proxy so developers can work with remote applications as if they are running locally. 

Leo came with a very clear plan for how the platform would look and how Opsfleet intended to build everything. He came very prepared, knew exactly what he was going to do, and was very good at explaining it to us.

Shahar Kedar, former VP of Engineering, BigPanda

We helped the team clean up their Dockerfiles and improve their CI to automatically build and publish the Docker images for all applications. We created Helm charts for each application and created a few utility scripts so developers could operate the system autonomously. 

Within a month, we had the dev environment up and running on Kubernetes. We traveled to BigPanda's offices to train the team, and we recorded the session so it can be an ongoing asset to orient new team members. Shahar says one of the more intangible benefits of the project was a motivation boost for the engineers on the team. 

I could see that everyone on my team was hyped when Opsfleet presented the solution and showed us how to work with the new Kubernetes and the new features. The team was thrilled to have a faster way to work; they knew this speed would help the company do great things. Everyone was happy that they could stop learning the Kubernetes platform and start using it every day. 

Shahar Kedar, former VP of Engineering, BigPanda

To minimize the impact on the development team, we rolled out access to the new development environment slowly, starting with a few team members. We fixed bugs and added to the automation scripts, repeating this process until all were satisfied with the results. We also built the development environment in such a way that its building blocks can be reused for the production environment.

Results 

With the migration complete, the BigPanda engineering team has already seen great improvements. The new dev environment has replaced the previous environment—and improved on it. BigPanda can now shift from noncontainerized services to fully containerized services running on a top-of-the-line container-orchestration platform.

My experience with Opsfleet is that within a few months, we had a stable dev environment built on top of Kubernetes and it included a lot of features that the environments we'd built ourselves didn’t have. Opsfleet’s work definitely improved how we develop our service. Our new dev environment will enable us to scale much more easily and with higher velocity than before. 

Shahar Kedar, former VP of Engineering, BigPanda

Now that it’s so easy to add new services to the environment, BigPanda's developers work more efficiently. They have fewer complaints about the development workflow and have even been able to test some very urgent features on the new platform with minimal preparation and minimal operations-team involvement. 

Opsfleet helped BigPanda's development team level up their expertise in Kubernetes. The team is now operating the development environment completely on their own and plan to take the same setup to production without needing much support from Opsfleet. 

The groundwork is now set for BigPanda to achieve a major goal: migrate out of the California zone to a region with better AWS support.

We needed to easily provision environments from scratch on different data centers in different regions. Opsfleet helped provide a vital building block that will eventually allow us to migrate out of our main AWS region.

Shahar Kedar, former VP of Engineering, BigPanda

Shahar says he particularly appreciates the professionalism and flexibility Opsfleet provides as well as our ability to understand both the operational and business sides of an organization.

I think Leo offers the best of both worlds: he is strong technically and can also create plans, provide visibility to management, understand high-level business needs, and build solutions that accommodate those needs.

Shahar Kedar, former VP of Engineering, BigPanda