Klustered & Rawkode Academy

In today’s episode we have the pleasure of Audun Fauchald Strand, Principal Software Engineer at NAV.no, Norway’s Labour & Welfare Administration. We will be talking about NAIS.io, the application platform that runs on-prem, as well as on the public cloud. Imagine hundreds of developers shipping on an average day 300 changes into a system which processes $100,000,000 worth of transactions on a quiet week. If you think this is hard, consider the context: a government institution which must comply with all laws & regulations.

15 years ago, Gerhard discovered magic in the form of Ruby on Rails. It was intuitive and it just worked. That is the context in which Gerhard fell in love with infrastructure and operations. Today, for special episode 77, we start at Seven Shipping Principles, and, in the true spirit of Ship It, we’ll see what happens next. Our guest is David Heinemeier Hansson, creator of Ruby on Rails, co-founder of Basecamp & HEY, and a lot more - check out dhh.dk.

In today’s episode, we talk about distroless, ko, apko, melange, musl and glibc. The context is Wolfi OS, a community Linux OS designed for the container and cloud-native era. If you are looking for the lightest possible container base image with 0 CVEs and both glibc and musl support, Wolfi OS & the related chainguard-images are worth checking out. Ariadne Conill is an Alpine Linux TSC member & Software Engineer at Chainguard.

Few genuinely need a multi-cloud setup. There is plenty of advice out there which mostly boils down to don’t do it, you will be worse off. Vex.dev is a startup that provides APIs for video and audio streaming. The hard part is real-time combined with massive scale - think hundreds of thousands of concurrent connections. They achieve this by using a combination of Fly.io, AWS and GCP. Jason Carter, founder of Vex Communications, is joining us today to talk about the multi-cloud setup that vex.dev runs.

I don’t think that you can imagine just how excited Gerhard was to find out that Audi, his favourite car company, has a Kubernetes competence centre. We have Sebastian Kister joining us today to tell us why people, followed by tech make the process. The right thing to focus on is the genuine smiles that people give in response to something we do or say. That is an important SLI & SLO for reducing friction between silos. How does this impact the flow of artefacts into production systems that design & build cars?

Matias Pan is a Staff Software Engineer at Lemon Cash, a crypto startup based in Argentina. Lemon infrastructure runs digital wallets & physical cards, which technically makes them a bank. How does Matias & his team think about enabling developers get code from their workstations into production? Remember, we are talking about a bank - a bad deploy is a big deal. And when a bad database migration goes out, what happens then?

Dave Farley, co-author of Continuous Delivery, is back to talk about his latest book, Modern Software Engineering, a Top 3 Software Engineering best seller on Amazon UK this September. Shipping good software starts with you giving yourself permission to do a good job. It continues with a healthy curiosity, admitting that you don’t know, and running many experiments, safely, without blowing everything up. And then there is scope creep…

In today’s Kaizen episode, we talk about shipping Adam’s Christmas present: chapter support for all Changelog episodes that we now publish. This feature was hard because there are many subtle differences in how the ID3 spec is implemented. Of course, once the PR shipped, there were other issues to solve, including an upgrade the world kind of scenario. Since Lars Wikman did all the heavy ID3 lifting, he is here with us too.

Maybe it’s the Californian sun. Or perhaps it’s the time spent at Disney Studios, the home of the best stories. One thing is for sure: Taylor Dolezal is one of the happiest cloud native people that Gerhard knows. As a former Lead SRE for Disney Studios, Taylor has significant hands-on experience running cloud native technologies in a large company. After a few years as a HashiCorp Developer Advocate, Taylor is now Head of End User Ecosystem at CNCF. In his current role, he is helping enable cloud native success for end-users like Boeing, Mercedes Benz & many others.

Most of you already know what it’s like to work in a startup or a small company. A few of you have been asking us for conversations with engineers that work for big companies, the kind that run everything from big title games to banking, and even critical national infrastructure. In today’s episode, we talk to Ganeshkumar, a Software Engineer in the Azure Kubernetes Service team, who works on Node Lifecycle and Kubernetes Versioning, and Brendan, Kubernetes project co-founder and engineering Corporate Vice President of Microsoft Azure OSS and Cloud-native Compute. We talk about what it’s like to work for Microsoft, how mentoring works in practice, and what Kubernetes, Omega, & Borg have to do with it all.

A few weeks ago, Jerod spoke with Liz Rice about the power of eBPF on The Changelog. Today, we have the pleasure of both Liz Rice, Chief Open Source Office at Isovalent & Thomas Graf, CTO & co-founder at Isovalent, the creators of Cilium. Around 2014, Facebook achieved a 10x performance improvement by replacing their traditional load balancers with eBPF. In 2017, every single packet that went to Facebook was processed by eBPF. Nowadays, every Android phone is using it. Truth be told, if it’s network-related and it matters, eBPF is most likely a part of it.

Why are the right values important for a company that changed the way the world builds software? How does pair programming help scale & maintain the company culture? What is it like to grow a company to 3000 employees over 30 years? Today we have the privilege of Rob Mee, former CEO of Pivotal, the real home of Cloud Foundry and Concourse CI. Rob is now the CEO of Geometer.io, an incubator where Elixir is behind many great ideas executed well, including the US COVID response programme.

Tammer Saleh, founder of Super Orbital, a tiny team of exceptional Kubernetes engineers and teachers, is joining us today to talk about what is cool in the Cloud Native world. Yes, it’s the same Tammer that we had the pleasure of on shipit.show/31 - Is Kubernetes a platform? In today’s episode, we also cover two great blog posts: Zero to GitOps: Terraform and the AWS EKS Blueprints project by Sean Kane Hunting Down an Intermittent Failure in Cilium by James McShane We wrap up with ✨ The Cool Wall of Cloud Native ✨

Our today’s guest spent 4 days building a feature for his side project so that we could ship it together on Ship It!, while recording. The feature is called rave mode, and the context is Bass, an interpreted functional scripting language written in Go, riffing on the ideas of Kernel & Clojure. When the local build runs, you can now press r to synchronise the beats of your currently playing Spotify track with the build output. For a demo, see bass v0.9.0 release. Please welcome Alex Suraci, a.k.a. vito, the creator of Concourse CI and Bass. This episode is dedicated to the late John Shutt, the creator of Kernel. Your ideas continue in Bass. Thank you for getting them out into the world.

We know that many of you listen to this podcast while running 🏃‍♀️ or cycling 🚴‍♂️ Hey Dan! How many of you cycled to a conference? Gerhard knows a single person that cycled 764 miles for 8 days straight from Switzerland to Spain for this year’s KubeCon EU. His name is Johann Gyger, a CNCF ambassador & a cloud consultant at Peak Scale. Johann is a cloud engineer at heart that is all in on sustainability. He is the main reason why Gerhard is super excited to talk about electric cars & Dagger at the Swiss Cloud Native Day this September.

Gerhard’s transition to a senior engineer started 10 years ago, when he embraced the vim mindset, functional core & imperative shell, and was inspired to seek simplicity in his code & infrastructure. Most of it can be traced back to one person: Gary Bernhardt, the creator of Execute Program, Destroy all Software and the now famous Wat idea. Few stick around long enough to understand the long-term impact of their decisions on production systems. Even fewer are able to talk about them as well as Gary does.

Today we talk with two lovely folks from Transistor.fm: Jason Pearl, Senior Software Developer & Jon Buda, co-founder. Gerhard was curious to find out about their setup & how did it change with the launch of the new podcast website builder. After all, you have been hearing us talk about our setup for years, so it was high-time to challenge some assumptions and learn how another team is solving similar problems. TL;DL: keeping it simple is at the root of smooth operations & stable systems.

In our 6th Kaizen, we talk with Jerod about all the things that we cleaned up after migrating changelog.com from a managed Kubernetes to Fly.io. We deleted the K8s cluster and moved wildcard cert management to Fastly & all our vanity domain certs to Fly.io. We migrated the Docker Engine that our GitHub Actions is using - PR #416 has all the details. We did a few other things in preparation for our secrets plan. Thank you Maikel Vlasman, James Harr, Adrian Mester, Omri Gabay & Owen Valentine for kicking it off in our Slack #shipit channel. Gerhard’s favourite improvement: the new shipit.show domain.

Ben Johnson, the creator of Litestream, joined Fly.io a few weeks after we migrated changelog.com - episode 50 has all the details. That was pure coincidence. What was not a coincidence, is Gerhard jumping at the opportunity to talk to Ben about Postgres vs SQLite with Litestream. The prospect of running a cluster of our app instances spread across all regions, with local SQLite & Litestream replication, is mind boggling. Let’s find out from Ben what will it take to get there. Thanks Kürt for kicking off this dream.

Rob Barnes (a.k.a. Devops Rob) and Rosemary Wang (author of Infrastructure as Code - Patterns & Practices) are joining us today to talk about infrastructure secrets. What do Rosemary and Rob think about committing encrypted secrets into a repository? How do they suggest that we improve on storing secrets in LastPass? And if we were to choose HashiCorp Vault, what do we need to know? Thank you Thomas Eckert for the intro. Thank you Nabeel Sulieman (ep. 46) & Kelsey Hightower (ep. 44) for your gentle nudges towards improving our infra secrets management.

Today we are talking with Frederic Branczyk, founder of Polar Signals & Prometheus maintainer. You may remember Frederic from episode 33 when we introduced Parca.dev. This time, we talk about a database built for observability: FrostDB, formerly known as ArcticDB. eBPF generates a lot of high cardinality data, which requires a new approach to writing, persisting & then reading back this state. TL;DR FrostDB is sub zero cool & well worthy of its name.

Today we are talking with Maikel Vlasman, technical lead for a large Dutch machine construction company, and a cloud engineer by heart. We cover self-updating GitLab & ArgoCD, Maikel’s thinking behind dev environment setup and a Kubernetes workshop that he is preparing for his team. The goal is to function as a true DevOps team with shared responsibilities. This conversation started as a thread in our community Slack - link in the show notes. Thank you Maikel for being a long-time Changelog listener and for reaching out to us - we enjoyed telling this story.

Today we are talking how to optimise sociotechnical systems with Ben Ford, founder & CEO of Mission Control. The correct order is: people, process & technology. The tools are important, and we talk about specific ones in the second half of this episode, but there are rules and principles that govern how people interact, and we need to start there.

This is the post-KubeCon CloudNativeCon EU 2022 week. Gerhard is talking to Matt Moore, founder & CTO of Chainguard about all things Knative and Sigstore. The most important topic is swag, because none has better stickers than Chainguard. The other topic is the equivalent of Let’s Encrypt for securing software.

Today we are at KubeCon CloudNativeCon EU 2022, talking to Adolfo García Veytia about securing Kubernetes releases. Adolfo is a Staff Software Engineer at Chainguard, and one of the technical leads for SIG release, meaning that he helps ship Kubernetes. You most likely know him as Puerco, and have seen first-hand his passion for securing software via SBOMs, cosign and SLSA. Puerco’s love for bikes and Chainguard are a great match 🚴‍♂️

Today we talk to Priyanka Sharma (E.D. at the Cloud Native Computing Foundation) about all things KubeCon Europe 2022. We start with Gerhard’s favourite subject - Priyanka’s Happy Hour - and then we switch focus to the conference. For many, this will be the first in-person KubeCon since 2019. As for Gerhard, he is not sure that he remember how airports work. If he succeeds, he looks forward to meeting some of you in Valencia. If not, send help.

Today we talk to Mark Ericksen about all the things that we could be doing on the new platform - this is a follow-up to episode 50. Mark specialises in Elixir, he hosts the Thinking Elixir podcast, and he also helps make Fly.io the best place to run Phoenix apps, such as changelog.com. In the interest of holding our new platform right, we thought that it would be a great idea to talk to someone that does this all day, every day, for many years now. We touch up on how to run database migrations safely, and how to upgrade our application config to the latest Phoenix version. We also talked about some of the more advanced platform features that we may want to start leveraging, like the multi-region PostgreSQL.

This is our 5th Kaizen where we talk about the next improvement to changelog.com: we are now running on Fly.io and our PostgreSQL is managed. This is a migration that many were curious about, including Simmy de Klerk, the person that requested this episode. After migrating all our media files to AWS S3 (check episode 40), we thought that this part was going to be easy. Plan met reality. Pull request 407 has all the details. We want to emphasise the type of partner relationships that we seek at Changelog & why they are important to us, as well as to our listeners. Honeycomb & Fly embody the principles that we care about, and Gerhard thinks that we are currently missing a Kubernetes partner.

Alex Sims, a Senior Software Engineer at James & James, an eCommerce fulfilment company, reached out to us about the Kaizen story of the third-party logistics (3PL) platform that he has been involved with for several years now. The system delivered 16 millions of orders in 10 years, and 4.5 million in the last year alone. All the numbers are going up, and there is only so much that a single PHP monolith deployed as VM images can handle. So how do you even start thinking about the architectural improvements, and inspire everyone involved to move towards better? We encourage you to look at the architectural diagrams in the show notes, especially the 10 year roadmap, and ask Alex for a blog post follow-up. While today’s episode was a good conversation starter, there is a lot that we did not have time to cover.

In this episode we talk about launching Dagger with all four founders: Andrea, Eric, Sam & Solomon. While you may remember Sam & Solomon from episode 23, this time we assembled all four superheroes in this story and went deeper, covering nearly three years of refinements, the launch, as well as the world-class team & community that is coming together to solve the next problem of shipping software. Container images and Kubernetes are great steps in the right direction, but now it’s time for the next leap into the future. You can use Dagger to run your CI/CD pipelines locally, without needing to commit and push. You can also use Dagger as a Makefile alternative, which resonates with Gerhard, but go further and your perspective on documentation & automation may start shifting. Gerhard believes that this is the Docker moment of CI/CD.

This episode was requested by Tyler Smith who feels that he may not need Kubernetes just yet. Tyler has a few questions about Docker & Docker Swarm, so Andrea Luzzardi, former Docker Swarm Lead, joins us today to answer them. We talk about Docker Swarm beginnings, some of the challenges that it faced, and what Andrea’s recommendation is for Tyler’s journey with Docker Swarm. After dedicating four years of his professional career to Docker Swarm, Andrea is the best person that Gerhard knows to talk about this subject. And guess what, the same thing happened now as it did at KubeCon 2015: Sam pointed to Andrea. It will all make sense in the first five minutes. This one is going to be fun!

Nabeel Sulieman, Senior Software Engineer at Vercel, talks about KCert, a simpler alternative to cert-manager that he built. Gerhard tried it out, and he thinks that Nabeel is onto something. If you want to see the video that they recorded, ping us on Twitter or Slack. We love this story, especially the long-term approach of working on something that one truly believes in, and the only reason is because it’s fun. The world needs more people like Nabeel, and we hope that this episode inspires you to go all out, and do just that.

Pia Wiedermayer, Lead QA at Zühlke, is talking with Gerhard today about software quality. If the name sounds familiar, check out episode 28. Thank you Romano for the introduction 👋🏻 Do you remember the last time that you used an app, whether it was in the browser or on your mobile, and everything just worked? What about that intuitive feel, snappiness and you achieving the task that you intended to without feeling that you are fighting tech? Experiences like those take a lot of effort across multiple disciplines. They are designed, built and maintained over long periods of time. It all starts with people like Pia that really care about quality. It’s so much more than just automated testing…

Today’s conversation with Kelsey Hightower showed Gerhard what he was missing in his quest for automation and Kubernetes. The fundamentals that Kelsey shares will most certainly help you level up your game. This is a follow-up to the last 45 seconds of the Kubernetes documentary. Oh, and we finally cleared where we should run our changelog.com PostgreSQL database 🙂

In this week’s episode Cameron Dutro, a software engineer at GitHub, Ship It listener and someone with an extraordinary attention to detail, joins us to talk about Kuby, a convention-over-configuration approach to deploying Rails apps. The question that we will be trying to answer is what happened to Rails Active Deployment. The path to that promise land is paved with good intentions, but it’s complicated.

This week we have the pleasure of Rich Burroughs, Senior Developer Advocate at Loft Labs and host of the Kube Cuddle podcast. We talk about multitenancy in Kubernetes and how to run Kubernetes in Kubernetes with vcluster. If you are using KiND, you will find this episode interesting, and maybe even helpful. We also talk about the role that Kelsey Hightower played in Rich joining the CNCF ecosystem. The key take-away is that people make all the difference. ADHD is something that Rich thinks about often. Gerhard was curious about the difference between ADHD and burnout, as well as this Twitter thread on re-reading sent emails.

In today’s episode, Gerhard is talking to Mauricio Salatino (@salaboy) about the Continuous Delivery for Kubernetes book that he is currently writing. Mauricio is a Staff Engineer at VMware where he spends most of his time contributing to Knative, an open source platform for running serverless workloads on Kubernetes. Gerhard & Mauricio spent a few months in 2021 working on Knative Eventing, and they both appreciate shipping great software continuously. Mauricio helped ship Knative 1.0. The from-monolith-to-k8s application used throughout this book has been a few years in the making. It doubles-up as a workshop-style guide for rearchitecting a Java monolith to a Cloud Native architecture running in Kubernetes.

We finally did it! All our static files are served from AWS S3. This is the most significant improvement to our app’s architecture in years, and now we have unlocked the next level: multi-cloud. We talk about that at length, and how it fits in our 2022 setup. The TL;DR is that changelog.com will fly, both literally and figuratively. We also address Steve’s comment that he left on our previous Kaizen episode - thanks Steve! Towards the end, we talk about Gerhard’s new beginnings at Dagger, where he gets to work with a world-class team and build the next-gen CI/CD. That’s right, Gerhard is now walking the Ship It talk all day, every day. If you want to watch him code live, you can do so every Thursday, in our weekly community session. Kaizen!

This week we are talking to Robin Morero, the person behind fabled.se, a DevOps consultancy from Gothenburg, Sweden. Their motto is “move faster and prosper”, which Gerhard prefers to the initial “move fast and break things”. Fabled works with startups primarily, and after 26 years, Robin has a few interesting insights to share. What do you think, are haunted codebases real? At what point do pull requests become harmful? What about k3s running on KVM as a simple starting point for production? If this reminds you of #7, and the follow-up YouTube stream with Lars, it’s no coincidence.

Gunnar Holwerda (Engineering Manager) and Tom Pansino (DevOps Team Lead) share with us a few stories about how the teams at opensesame.com manage AWS operational complexity. The first link in the episode show notes are the slides that Tom & Gunnar prepared for this conversation. Check them out as you hear us speak about the Inverse Conway Manoeuvre, and why you should always go for the bananas. If you like this episode, and have a similar story to share, please reach out to us. We all love real-world stories that we can learn from, and perhaps contribute to.

Vincent Ambo –the person behind nixery.dev, tvl.fyi, and a former Google engineer– shares his take on monorepos, Nix, and fully declarative systems without any Flux, Argo or Kubernetes. While the tooling is impressive, it’s the principles behind it that captivated Gerhard’s imagination. Vincent has a rather interesting take on the monorepository idea, including one change - one version - one deploy. There are a lot of interesting links in the show notes, including all the code that Vincent uses to manage infrastructure. As a result of this conversation, Gerhard is running Nix on one of his Macs, and also started experimenting with his first NixOS production instance.

Gerhard loves simple ideas executed well, which is why he is excited to be speaking today with Ildar Iskhakov & Matvey Kukuy about their startup Amixr, a.k.a. Grafana OnCall. Ildar & Matvey started with a simple idea and a simple stack - Django, Celery, RabbitMQ & MySQL - all running on Kubernetes. Because they kept their main thing their main thing, and kept improving it every day for a couple of years, now your on-call can be simple too. This is another Big Tent philosophy story with a Black Swan moment towards the end.

Today Gerhard shares the entire story behind his lost packets. He is talking with Drew Marshall, director at Trunk Networks and No One Internet, a Cloud Services Provider & ISP based in Sussex, UK. Gerhard’s Vodafone ISP gateway was losing packets, and recording some of the previous episodes used to be challenging as his internet connection would cut out up to 10 seconds at a time, multiple times per recording session. He was convinced that his Unifi Dream Machine Pro was not the issue. Drew helped Gerhard realise that it actually was. Not only has Gerhard’s DNS latency improved by 3x, but he can now fail-over between two WAN connections. And because nothing beats a real-world experiment, you can guess what is coming in this episode 😉 You will find latency & packet loss graphs, speed test runs, and a few other interestings in the show notes. We hope that they inspire you to setup a better home network. Most importantly, may you find your humble & brilliant Drew.

In our first 2022 episode, Alexis Richardson, co-founder and CEO of Weaveworks, is talking to Gerhard about going fully remote, what a great team looks like, and GitOps. While you may have heard of GitOps, now is a good time to check out opengitops.dev. The most interesting part of today’s conversation is the missing cloud native App Store. While Apple revolutionised the world with the App Store and the iPhone, we don’t yet have something similar for cloud native apps. You may be thinking “But what about OperatorHub?”, or all the Helm registries out there? The registry fragmentation, operator deprecations and lack of curation are not what people have in mind when they think App Store. But there is more to it, so let’s hear how Alexis thinks about this.

Merry Shipmas! This is our special Christmas episode which sums up two months of very early mornings and a few late nights. After many twists and turns, stuff which didn’t work out, as well as pleasant surprises, this is what we ended up with: 🎁 PR #395 - CI/CD Lego set with Guillaume de Rouville & Joel Longtine 🎁 PR #396 - Continuous CPU profiling with Frederic Branczyk 🎁 PR #399 - Auto-restoring Kubernetes clusters with Dan Mangum & Muvaffak Onuş While we initially intended to have five Christmas presents in total, only three got delivered in time. We planned, worked hard and eventually shipped the best we could just in time for this special Christmas episode. Our hope is that the latest additions to our changelog.com GitHub repository will help you just as much as they will help our 2022 setup. 🎄Merry Shipmas everyone! 🎄

In 2014 Gerhard joined CloudCredo, a startup co-founded by Colin Humphreys, Paula Kennedy & Chris Hedley. They stuck together through two acquisitions: Pivotal & VMware. This year, Colin, Paula & Chris co-founded Syntasso, the Platform-as-a-Product startup. Today they all get together to talk about about what it takes to build a platform team, why Team Topologies is a good conversation starter and why a curated blend of off-the-shelf, composed, and self-created services are required in any organisation operating at scale. Your hunch is right, all of them used to share the same Pivotal London office with Tammer Saleh, our guest from episode 31. Chris used to win all table tennis matches without even breaking a sweat, and today Gerhard gets his comeback. Touché!

Tammer Saleh, founder of SuperOrbital and former VP of Engineering at Pivotal, is joining Gerhard to talk about table tennis, remote work, and challenges that teams have with K8s. Some years ago, both Tammer & Gerhard used to work in the same London office on CloudFoundry, and nowadays they are both into Kubernetes. Tammer and the SuperOrbital team are deeply experienced in this topic, and they help teams at companies like Bloomberg, Shopify, and federal U.S. agencies tackle hard Kubernetes and DevOps problems through engineering and training. Why do companies need Kubernetes in the first place? Which are the right reasons for choosing it? Is Kubernetes a platform? Gerhard’s favourite: we are doing Kubernetes wrong, but it works better than when we were doing it right, so what’s up with that? This last one was a lot of fun, and we left the entire minute of laughter in at your request. Enjoy!

This is our third Kaizen episode in which Adam, Jerod & Gerhard talk about GitOps the wrong way, ask questions with Honeycomb and realise that they must be holding the CDN wrong, and the effort that has been going into moving all changelog.com static files from regular volumes to an S3-like object store. If you like a good yak shake, listening to this one is a lot more fun than doing it. Gerhard is most excited about the Ship It Christmas gifts that we have been preparing for you. While GitHub Codespaces is not going to be part of the upcoming Christmas special episode, today’s talk covers why investing in a Codespaces integration is worth it. Changelog #459 and Backstage #20 are related to this topic.

Zac Smith, managing director Equinix Metal, is sharing how Equinix Metal runs the best hardware and networking in the industry, why pairing magical software with the right hardware is the future, and what Open19 means for sustainability in the data centre. Think modular components that slot in (including CPUs), liquid cooling that converts heat into energy, and a few other solutions that minimise the impact on the environment. But first, Zac tells us about the transition from Packet to Equinix Metal, his reasons for doing what he does, as well as the things that he is really passionate about, such as the most efficient data centres in the world and building for the love of it. This is a great follow-up to episode 18 because it goes deeper into the reasons that make Gerhard excited about the work that Equinix Metal is doing. This conversation with Zac puts it all into perspective. By the way, did you know that Equinix stands for Equality in the Internet Exchange?

This week Gerhard is chatting with Romano Roth, Head of DevOps at Zühlke, a company founded by Gerhard Zühlke in 1968. Nowadays they help companies all over the world build, ship and run anything from factory robots, to AI assistants in complex regulatory environments, and even medical devices that perform autonomous robotic surgery. When Romano is not leading a team of 30 software engineers that specialise in operations, infrastructure and cloud, he is one of the organisers of DevOps Days Zürich, and also the DevOps Meetup group, which is how Gerhard and Romano met in 2019. Having started his career as a .Net developer back in 2002, Romano had his fair share of dev and ops challenges, and he always enjoys seeing real business value delivered continuously in an automated way. In recent years, Romano’s perspective broadened, and now he sees DevOps realities across many companies. If you are curious about what good DevOps looks like, and what are the real challenges, then Romano has some good insights for you.

In this episode, Gerhard is joined by Cyrille Le Clerc, Product Manager Lead on Observability at Elastic, and Oleg Nenashev, Principal Engineer at CloudBees. It all started with Oleg’s tweet back in July, in which he was promoting Akihiro Kiuchi’s work on Jenkins monitoring with OpenTelemetry. This was done in the context of Google’s Summer of Code - a link to Akihiro’s demo is in the show notes. As you may remember from episode 20, instrumenting our changelog.com pipeline is on Gerhard’s mind, and this conversation helped him clarify a few things. If you are thinking of instrumenting your CI/CD pipeline with OpenTelemetry, this episode is for you.

In the second set of interviews from KubeCon North America 2021, Gerhard and Liz Rice talk about eBPF superpowers - Cilium + Hubble - and what’s it like to work with Duffie Cooley. Jared Watts shares the story behind Crossplane reaching incubating status, and Dan Mangum tells us what it was like to be at this KubeCon in person. Dan’s new COO role (read Click Ops Officer) comes up. David Ansari from VMware speaks about his first KubeCon experience both as an attendee and as a speaker. The RabbitMQ Deep Dive talk that he gave will be a nice surprise if you watch it - link in the show notes. Dan Lorenc brings his unique perspective on supply chain security, and tells us about the new company that he co-founded, Chainguard. How to secure container images gets covered, as well as one of the easter eggs that Scott Nichols put in chainguard.dev.

This is Gerhard’s first set of interviews from KubeCon North America 2021. William Morgan shares with us some of the finer Linkerd details, such as the underlying security theme, why native Kubernetes objects are preferable to more CRDs, and the joy of meeting team members in person. Frederic Branczyk speaks about Parca, a new continuous system profiling tool that uses eBPF to help you understand what is happening on your hosts. Andrew Rynhard gives us a great Talos OS and Kubespan perspective, and shares some really good follow-up videos on these topics. The last conversation is with David Flanagan - you know him as Rawkode - about new beginnings. It’s only been less than two months since we’ve had him in episode 18, and he kept really busy. Caleb, his 3 weeks old baby boy, was the youngest attendee at this conference, and some talks made him sleepy, so good job everyone.

This week Gerhard is talking with Arnaud Porterie, founder of EchoesHQ, a new utility that measures and communicates engineering activity. They start by re-creating the 60 seconds Y Combinator pitch, and then shift focus to what it was like to get EchoesHQ off the ground. Next, they tackle something which is always on Gerhard’s mind: Why is it important to connect our daily engineering activity to intent? Before EchoesHQ, Arnaud used to run the core team and the open source project at Docker, and combined with other engineering leadership roles that he held for over a decade, he kept encountering misalignment that was preventing organisations from making meaningful progress. Let’s hear why EchoesHQ might just be a great way of addressing this.

In today’s episode, Gerhard is talking to Sam Alba, Docker’s first employee, and Solomon Hykes, the Docker co-founder. Together with Andrea Luzzardi, they are the creators of Dagger, a universal deployment engine that trades YAML for CUE, and uses Buildkit as the runtime. Why? Because we should stop rewriting the same application deployment logic in scripts, makefiles or continuous delivery configuration. That’s right, this is the YAML vaccine that we have all been waiting for. Gerhard believes that one day, Dagger will become just as meaningful for application delivery, as Docker is today for application code.

Today we have a very special episode, where Gerhard gets to share his favourite learnings from Steve Jobs. If it wasn’t for his determination to build a better personal computer, Gerhard would have most likely continued with a career in physics. We know what you’re thinking: it’s crazy and impossible to interview Steve Jobs, but on his 10th memorial anniversary, Gerhard was determined to combine the things that Steve said with his passion for computers, automation, and infrastructure. Live your life and ship your best stuff because there’s nothing like the present. Thank you, Steve.

Things go wrong all the time. We all make mistakes. And that is okay. What is not okay, is to think that it won’t happen, or that there will be someone else around when it does. In that moment, it doesn’t matter who wrote that module, package or microservice. But there is a better way to think about this, and there is an approach that makes people actually look forward to incidents. It all starts with thinking of incidents as opportunities to learn, and then share those learnings with everyone, so that you can all improve. In this episode, Gerhard is joined by Stephen Whitworth and Chris Evans, incident.io co-founders, and former Staff Engineers at Monzo. They get it, we get it, and now you can get it too.

This is our second Kaizen episode, where Adam, Jerod & Gerhard talk about changelog.com improvements since episode 10. OK, so Gerhard deleted the DNS API token. Not only did he take the time to understand how that happened, so that he could actually learn from his mistake, but now we have a system in place so that we can share learnings from incidents. By the way, these are publicly available in our #incidents Slack channel. A great & unexpected thing that happened since we recorded this episode, is Jerod fixing 99% of all the errors that were happening in prod. The top error was the broken Twitter auth - sorry Matt - which was a result of us upgrading to OTP 24 a few months back. Episode 3 show notes include a YouTube stream which captures it all. We wrap up this episode by each of us sharing the improvements that we would like to do until our next Kaizen. You heard it from Adam first: Ship It Driven Development

This week Emile Vauge, founder & CEO of Traefik, joins Gerhard to share a story that started as a solution to a 2000 microservices challenge, the real-world implications of shipping many times a day for years, and the difficulties of sustaining an inclusive and healthy open-source community while building a product company. Working every day on keeping the open-source community in sync with the core team was an important lesson. The second learning was around big changes between major versions. The journey from Travis CI to Circle CI, then to Semaphore CI and eventually GitHub Actions is an interesting one. The automation tools inspired by the Mymirca ant colony is a fascinating idea, executed well. There is more to discover in the episode.

In this episode, Gerhard talks to David and Marques from Equinix Metal about the importance of bare metal for steady workloads. Terraform, Kubernetes and Tinkerbell come up, as does Crossplane - this conversation is a partial follow-up to episode 15. David Flanagan, a.k.a. Rawkode, needs no introduction. Some of you may remember Marques Johansson from The new changelog.com setup for 2019. Marques was behind the Linode Terraforming that we used at the time, and our infrastructure was simpler because of it! This is not just a great conversation about bare metal and Kubernetes, there is also a Rawkode Live following up: Live Debugging Changelog’s Production Kubernetes 🙌🏻

I’m Gerhard Lazu, host of Ship It! A show with weekly episodes about getting your best ideas into the world and seeing what happens. We talk about code, ops, infrastructure, and the people that make it happen. Like Charity Majors from Honeycomb… clip from episode #11 And Dave Farley, one of the founders of Continuous Delivery… clip from episode #5 We even experiment on our own open source podcasting platform so that you can see how we implement specific tools and services within changelog.com. What works and what fails… clip from episode #10 Listen to an episode that seems interesting or helpful and if you like it, subscribe today. We’d love to have you with us.

On this week’s episode, Gerhard is joined by Kathy Korevec, former Senior Director of Product at GitHub, and now Vercel’s Head of Product. Docs play an essential role in GitHub Actions, and Gerhard’s experience has proven that. Building, testing, and shipping code with GitHub Actions works better because of their excellent docs. However, the docs that Kathy pictures are not what you are imagining. She explains it best in her post, Maybe it’s time we re-think docs, which is what started this whole conversation. The bottom line is, just as you wouldn’t ship untested code, shipping code without documentation is not optional. Today’s conversation with Kathy explains why.

This week Gerhard is joined by Justin Searls, Test Double co-founder and CTO. Also a 🐞 magnet. They talk about how to deal with the pressure of shipping faster, why you should optimize for smoothness not speed, and why focusing on consistency is key. Understanding the real why behind what you do is also important. There’s a lot more to it, as its a nuanced and complex discussion, and well worth your time. Expect a decade of learnings compressed into one hour, as well as disagreements on some ops and infrastructure topics — all good fun. In the show notes, you will find Gerhard’s favorite conference talks Justin gave a few years back.

In this episode, Gerhard follows up on The Changelog #375, which is the last time that he spoke Crossplane with Dan and Jared. Many things changed since then, such as abstractions and compositions, as well as using Crossplane to build platforms, which were mostly ideas. Fast forward 18 months, 2k changes, as well as a major version, and Crossplane is now an easy choice - some would say the best choice - for platform teams to declare what infrastructure means to them. You can now use Crossplane to define your infrastructure abstractions across multiple vendors, including AWS, GCP & Equinix Metal. The crazy ideas from 2019 are now bold and within reach. Gerhard also has an idea for the changelog.com 2022 setup. Listen to what Jared & Dan think, and then let us know your thoughts too.

In today’s episode, Gerhard is joined by Uma, CEO and co-founder of ChaosNative, as well as Karthik, CTO and also a ChaosNative co-founder. They talk Chaos Engineering and Litmus. Chaos Engineering is not just for super SREs. It is not meant to prevent outages. And, it is not just about hardware. Chaos Engineering is about testing how reliable your systems are. It’s meant to show you how things fail, including when other dependent systems fail - think cascading failures. This is a good way to discover inconvenient truths about that beautiful code that you wrote. Everything fails, and great insights are to be found when it does.

In this episode, Gerhard talks to his Skyhook Adventure friends: Alan Cooney, Saul Cullen & Wycliffe Maina. They are the ones that introduced Gerhard to the world of serverless in the context of Amazon Web Services. Gerhard shared his experience with remote work, how to ship software with confidence and consistency, and what to look for in infrastructure as code. At the heart of Skyhook Adventure are adventure trips, and 2020 was not a good one for this business. As you can already tell, code and infrastructure was not the biggest challenge for this team. Having said that, serverless, microservices, a monorepo and the event-based architecture played a big part in successfully navigating the challenges. This is a story about what happens when a good team allows itself to be guided by solid experience and keeps doing the right thing, long-term. It’s fun, real, and it applies to many.

Gerhard talks to Tom Wilkie, VP of Product for Grafana Labs. They talk about Loki, Tempo, and how can Grafana Cloud offer such a generous free tier. The solution is in the Cortex architecture, which was used in Loki and in Tempo too. Yes, Tom is the Cortex co-author. We recommend that you listen to this episode in combination with episodes 3 and 11. That’s the best way to get a more complete picture of the topics that we discuss today. Lastly, would you like to watch Gerhard & Tom pair-up and build Grafana dashboards like pros? Tom has this really interesting approach that Gerhard would like to learn too. We can either have a live YouTube stream, or record and then publish the video. Let us know your preference via our Changelog Slack, or just plain Twitter.

Gerhard talks with Charity Majors, ops engineer and accidental startup founder at honeycomb.io about high-performing teams, why “15 minutes or bust,” and how we should start using Honeycomb in our own monolithic Phoenix app that runs changelog.com. There is just one step, and it’s actually really simple! They also talk about how Honeycomb uses Honeycomb to learn about Honeycomb, which is one of Gerhard’s favorite questions. As for key take-aways, deploying straight into production is really important, but not as important as optimising for humans - which are not replaceable cogs, that learn and share their learnings continuously. That is the secret to making things easy and happy for everyone.

Kaizen means “change for the better”, continuous improvement in this context. Failure is essential to learning, but how do we learn as a team? The simplest thing is to regularly dedicate time for taking a step back, talking about what works & what doesn’t, maybe writing some of it down, and eventually deciding what we should improve next. I intend to make every 10th Ship It! episode a Kaizen one. This is the first one when we talk with Adam and Jerod about the things that we want to improve in our setup over the next few months. We talk about how the June Fastly outage affected changelog.com, how we responded that day, and what we could do better. We discuss multi-cloud, multi-CDN, and the next sensible and obvious improvements for our app. Let us know via Slack or Twitter what learnings are valuable to you so that we can produce the best content for you.

This week we talk with Jean-Sébastien Pedron, RabbitMQ and FreeBSD contributor, about the importance of good release engineering for core infrastructure. Both Jean-Sébastien and I have been part of the Core RabbitMQ team for many years now. We have built some of the biggest CI/CD pipelines (check the show notes for one example), wrote and shipped some great code together, while breaking and fixing many things in the process. We have been wrestling with today’s topic since 2016. Jean-Sébastien has some great FreeBSD stories to share, as well as an interesting perspective on shipping graphic card drivers. Oh, and by the way, it’s probably our fault why your remote car key stopped working that afternoon. It will all make sense after you listen to this episode.

Why Cloud Native? What are the guiding principles that you should keep in mind as you are choosing a project from the Cloud Native Landscape? How do you build & ship an app in a Cloud Native way? Katie Gamanji, Ecosystem Advocate @ CNCF and former cloud engineer for American Express, Condé Nast and Microsoft, joins Gerhard to cover these topics in the context of the Cloud Native Fundamentals course that she developed. 15,000 students have already enrolled, and the initial feedback has been great. Tune in if you want to know why you should too, how to do it and when the course will become available for free.

This week on Ship It! Gerhard talks with Lars Wikman (independent Elixir/BEAM software consultant) why sometimes a monolith running on a single host with continuous backups and a built-in self-restore capability is everything that a small team of developers needs. That’s right, no Kubernetes or microservices. After 2 years of running changelog.com, a Phoenix monolith, on Kubernetes, what do I think? Join our discuss and find out!

This week on Ship It! Gerhard talks with Ian Miell, author of Docker in Practice as well as Learn Git, Bash, and Terraform the Hard Way. They talk about being comfortable with the uncomfortable, focusing on the tech while keeping a holistic view of the business. Following the money flows is key. Ian explains this concept really well, and Gerhard feels fairly confident you will be better off if you pay attention. Let us know in the comments!

This week on Ship It! Gerhard talks with Dave Farley, co-author of Continuous Delivery and the inventor of the Deployment Pipeline. Today, most of us ship code the way we do because 25 years ago, Dave cared enough to drive the change that we now call CI/CD. He is one of the great software engineers: opinionated, perseverant & focused since the heydays of the internet. Dave continues inspiring and teaching us all via his newly launched YouTube channel, courses and recent books. The apprentice finally meets the master 🙇‍♂️🙇‍♀️

This week on Ship It! Gerhard talks with Ben Ford, former Royal Marine and founder of Commando Development, about the OODA loop (observe, orient, decide, act). Shipping is just a small part of it. The OODA loop that you know is probably the wrong one. We explore Mission & Command, Situational Awareness and a few other practices that will help you deal with complexity as you code and ship. As a former Royal Marine Commando, Ben learned these skills the hard way, and then refined them over many years as a software engineer. Check out the diagrams in the show notes - they are a work of art and precision.

This week on Ship It! Gerhard talks with Alex Koutmos about Elixir observability using PromEx. Why do we need to understand how our setup behaves? What is PromEx and where does PromEx fit in changelog.com? Bonus! Tune in to our LIVE Friday evening deploy 😱 of Erlang 24 for changelog.com. Check the show notes for a link on YouTube. 🍿

This week on Ship It! Gerhard is joined by Constance Caramanolis, Principal Engineer at Splunk and former maintainer of Envoy Proxy, and Stephen Augustus, Head of Open Source at Cisco & self-proclaimed Caesar of Systems. Constance and Stephen are the KubeCon + CloudNativeCon co-chairs. Join us to find out what happens before and after KubeCon gets shipped.

Welcome to Ship It! This is a new show from Changelog about shipping software - and all the details, challenges, and problems that surface. Changelog SRE Gerhard Lazu is taking us on a journey into the world of shipping code, infrastructure, ops, and the people making it happen. Shipping is near and dear to every developers’ heart. We do it every day. It’s the essential first step. You have to ship it to share your ideas with the world. New episodes ship weekly.