"No one wants to end up under a bridge! How to keep Kubernetes costs in check."
Welcome to this edition of Coffee with Calyptus! Today, we’re diving deep with Dmitry C., Senior DevOps Engineer at 1inch Labs, to explore his journey from coding experiments on Linux to pioneering DevOps strategies in the Web3 ecosystem.
What first sparked your interest in DevOps? And how did that path eventually lead you into the world of Web3 with 1inch Labs?
From a young age, I was fascinated by technology. In school, I learned programming and participated in local coding competitions. At some point, my uncle introduced me to Linux, which became a turning point in my journey. I began exploring it deeply, and one of my early experiments was attempting to write a Linux kernel module to support a display embedded in a floppy drive slot. This, along with switching my personal laptop to Gentoo Linux, further solidified my passion for digging into how systems work.
My introduction to DevOps came about rather unexpectedly. At one point, I was developing a chatbot for Twitch streams and thought it would be fascinating to create an environment where different application components, written in various languages and interacting over a network, could run together seamlessly. When I shared this idea with a friend, he introduced me to Kubernetes and Ceph. He also helped me set up a small lab environment for experimentation, and eventually encouraged me to apply for a DevOps position at his company. That’s how I landed my first role in DevOps.
Over time, I worked at several product-based companies, gradually realizing that I wanted to experience the dynamic environment of a startup. The startup culture seemed more appealing, with its fast pace and opportunities for innovation. At one point, a former colleague invited me to contribute to a grant project funded by the Graph Foundation. This marked my first step into the Web3 space. Later, when I was exploring new job opportunities, I received a message from 1inch. Since 1inch was a Web3 startup—combining two of my key interests—it felt like the perfect opportunity to dive deeper into the Web3 ecosystem.
In a field that’s always changing, you’ve worked with some cutting-edge tech and strategies. What’s a new trend or tool in DevOps that you’re most excited about, and why do you think it’s a game-changer?
The tool that truly amazed me was Kubernetes. Its ability to orchestrate applications using open-source technology across any cloud or even on-premises infrastructure is remarkable. Beyond that, Kubernetes has fostered an entire ecosystem of tools and extensions that address almost any operational need. It fundamentally changed how I think about deploying and managing applications at scale.
Today, however, I find two trends particularly promising. The first is the Zero Trust approach with Identity-Based Access, which ensures security by verifying every user and system, regardless of location. The second is FinOps, which helps teams identify cost inefficiencies and understand the true expenses of various components. By providing transparency, FinOps empowers businesses and developers to make informed decisions, including revisiting architectural choices to optimize both performance and cost.
You’ve done some impressive work with Kubernetes, managing dynamic environments and controlling costs. If you had to share your top secrets for mastering Kubernetes without breaking the bank, what would they be?
To manage Kubernetes effectively without overspending, my approach varies based on the situation:
For quick setup and minimal support effort, managed services like GKE, EKS, or AKS are great options. They take care of updates, scaling, and cluster management, saving significant time and effort. However, these services aren’t cheap, so it’s critical to set up spending alerts early—cloud costs can spiral quickly, and no one wants to end up under a bridge because of unchecked expenses.
For low-cost environments needing significant resources, bare-metal hosting providers like Hetzner or OVH offer excellent value. While they require more manual setup and maintenance, the cost savings are substantial for static or predictable workloads.
For dynamic workloads, tools like HPA, Karpenter, and Cluster Autoscaler are essential. When managing short-lived environments for end-to-end tests, I’ve found optimizing scheduling policies to be key. Using affinity rules to group pods from the same environment onto shared nodes allows you to deprovision entire nodes once the workload is done, avoiding wasteful partially used nodes.
Finally, incorporating FinOps tools and robust monitoring ensures you stay on top of costs and resource usage. With clear visibility into what’s consuming your budget, you can fine-tune workloads and clusters to balance performance and cost effectively.
Looking ahead, where do you see the biggest opportunities for DevOps in the Web3 space?
Projects in Web3 vary widely—some combine smart contracts with traditional backends and frontends, while others operate entirely on decentralized principles. One critical area of focus is enhancing security and reliability in hybrid Web3 systems, particularly where Web3 intersects with traditional components.
Web3 brings significant advantages over traditional applications, such as immutability, transparency, and decentralized trust, but it doesn’t eliminate all traditional challenges. For example, frontend vulnerabilities, like dependency compromise, can still lead to serious exploits, such as wallet drainers. At the same time, Web3 introduces its own unique challenges, such as ensuring rigorous auditing and verification of smart contracts, which are core to system integrity and user trust.
Addressing these challenges requires balancing the strengths of Web3 with careful attention to security risks inherited from traditional architectures. By leveraging best practices from DevOps and developing new solutions tailored to Web3, teams can build more secure and resilient systems that capitalize on the unique benefits of blockchain technology.
Web3 brings unique challenges to DevOps, especially when it comes to managing decentralized and often unpredictable systems. What would you say is the biggest challenge DevOps faces in Web3, and how are you or your team approaching it?
One of the most significant challenges in Web3 DevOps is managing archival blockchain nodes used for data indexing and retrieval. These nodes are resource-intensive, costly to operate under high request rates, and difficult to scale horizontally. They often require maintaining vast amounts of historical blockchain data in a consistent state, making them behave more like "pets" than "cattle." This creates substantial challenges in automation, scaling, and cost-efficiency as blockchain datasets continue to grow.
To address these issues, we utilize Nodes-as-a-Service providers and implement proxy layers with health checks and failover mechanisms. This approach ensures higher availability and reliability while reducing the operational overhead of directly managing such nodes. However, scalability and cost efficiency remain persistent concerns.
Recently, I’ve been intrigued by the idea of decoupling data storage from node computation, enabling nodes to access historical blockchain data stored in a scalable, shared storage backend. For instance, DexterLabs’ solution for Solana leverages distributed, cost-effective storage, allowing archival nodes to focus on processing and computation while minimizing the need for each node to maintain a full dataset. This approach strikes me as particularly promising—it could facilitate treating nodes as "cattle," reduce discrepancies in their state, and enable dynamic scaling to handle workload demands, especially for indexing and querying. Exploring similar innovations across other blockchains could transform how archival nodes are managed in Web3 infrastructure.
Solidity Challenge 🕵️♂️
💰 Can you hack this contract to get all the reward tokens without sending a dime to someone else?
Jobs 💼
We have over 200 roles on our platform from companies like Aave, Aragon, Balancer, Gnosis, etc. Check out a select few below:
Staff Software Engineer, Non-EVM Integrations - remote, fulltime - $100-250k
Senior DevOps Engineer - remote, US, fulltime - $150-250k
Researcher - remote, Europe/Asia, fulltime - $100-120k
SEO Manager - remote, 4hr overlap with CET, fulltime - $75-90k
Solidity Challenge Answer ✅
Answer: One can send eth to oneself and earn reward.
