Our monitoring caught it immediately. An alert fired the moment the change happened: deployment in the production account by a role that is not part of the approved CI/CD pipeline.

Nothing broke. The engineer had confirmed each step, and detection worked exactly as designed. But detection is not prevention. If the agent had deleted the wrong stack, the alert would have fired too. And everyone would be having a really bad day.

Nothing happened but the moment scared me enough to take a step back to rethink how we deal with engineer production access.

You've read the public versions of this story. An agent wiped a founder's production database through Cursor during ordinary development work. Replit's agent deleted a production database during a code freeze, then fabricated thousands of fake records to cover it up. The pattern is always the same: a "clean up and roll back" that looks reasonable to an agent and is catastrophic in production.

We run AI agents across engineering, and we want to. It's part of how epilot operates. But our AWS access model was designed before that, and it assumed a careful human is the only thing holding production credentials.

That assumption is dead. Today we buried it: no engineer at epilot can reach production AWS directly from their laptop anymore. Not the CLI, not the console. Here's what replaced it.

The design

Production access at epilot was already technically short-lived: time-limited sessions behind MFA. On paper, that looks fine.

In practice, sessions outlive the reason you opened them. You assume prod for a quick check, switch to another task, and a live production session stays behind on your machine. Usually as the global default AWS CLI profile.

That means every process on the laptop is connected to production by default. Your test suite. Your scripts. Every AI agent you run. Nobody decided that. It's just what a global profile on a busy machine does.

So we changed who the prod roles trust. Developer identities can't assume them at all anymore. The roles trust exactly one principal: an internal access broker. And the broker only mints a session after a human walks through two separate surfaces: a web portal behind SSO (the portal and its REST API both sit behind it) and a manual approval in Slack.

The flow:

1. Request. Open the Admin Portal, pick an access level and a duration, write why you need it. Takes 30 seconds.
2. Approve. The portal posts your request to a Slack channel the whole company can see, and you approve it there. This is the trick: the Slack approval is a second factor, not a peer review. It's a separate app on a separate surface, so an agent driving your terminal or your browser session can't complete it. Approving your own request is fine. Another engineer can approve it too, but nobody has to. The Slack thread is the audit log.
3. Connect. The broker mints a session named after you and its approver, then you choose: a one-click sign-in link to the AWS Console, or a throwaway EC2 jump host. The jump host lives in a private subnet, reachable only through Session Manager. No SSH keys, no open ports. Ready in about 90 seconds.
4. Expire. Sessions die on their own. Jump hosts are destroyed with them.

There are three access levels, and the default is read-only:

• Read-only: inspect resources, read logs and metrics. No writes, no deploys. This covers most production visits.
• Developer: day-to-day production support. Deploy through the CI/CD pipeline (never around it), read logs, manage infra. Cannot delete stacks, read SSM secrets, touch customer data directly or decrypt with KMS CMKs.
• Administrator: full access. Assume only if you absolutely need it.

Notice what the developer tier can't do. Accessing customer data at epilot is intentionally difficult. No direct reads from the data stores, no decrypting with customer-managed KMS keys, not even for day-to-day production support. And even read-only goes through the broker. We treat read access as access.

Escalation is deliberate, like everything else in the flow. You don't get developer access out of habit. You ask for it, and the reason sits in Slack next to your name.

The access page in our Admin Portal states the whole contract in four principles: deliberate and isolated, second-factor approval, attributed and time-limited, least privilege. That's not a compliance poster. It's literally the UI copy engineers read while requesting access.

The jump host is the hardened path: no AI agents installed, none allowed, only the minimal tooling for production support, with full shell-session and AWS API logging. Clickops and incident investigation still work. That's what the console link is for. But a console session is still a browser, so we treat it as the weaker option and the jump host as the default.

For emergencies there's a separate break-glass admin role. A small named group can assume it directly with MFA, and every use fires a loud alert in our security channel. Emergency access stays open without reopening the everyday hole.

Self-approval? Seriously?

The first reaction to this design is always the same: what's the point if you approve yourself?

Our engineers were always trusted with production. You build it, you run it. That hasn't changed. A second human clicking "approve" would add latency and theatre, not security. Rubber-stamp approvals are how big orgs pretend to have controls.

The threats we're closing are ambient credentials and autonomous agents. Those are handled by the credential-free laptop and the out-of-band approval, not by another pair of eyes.

What this doesn't solve

An approved human can still break production. So can an agent driving an approved console session. The goal was never to make damage impossible. It's to make production access deliberate, attributed and temporary instead of ambient.

The broker itself is now a high-value target. It's the only thing that can assume prod roles, so we scoped it tightly, locked it down and watch it closely. We've concentrated the risk in one small surface we can actually defend, instead of spreading it across every laptop in the company.

Break-glass is a deliberate bypass. We contain it with a tiny group, MFA and loud alerting rather than pretending it doesn't exist.

Can't an agent just approve itself in Slack?

Yes, technically. A rogue agent running on your machine could technically take over your browser session to create the request, then access Slack to approve it.

But to get there, a lot of things already have to have gone really badly wrong:

1. You gave the agent full access to your browser with a logged in admin session.
2. You gave the agent full access to Slack.
3. You let the agent act without verifying anything. Nothing and no one is checking what it does.
4. The agent decided to go out of its way to find out how to do this and perform all the steps

But this is of course not a realistic agent setup any engineer at epilot would run. Security always comes in layers. What we prevent with this setup are the common footguns where it was too easy to give AI unfiltered prod access.

The rollout

We ran the broker for two weeks alongside the old access and asked engineers to stop using their direct prod roles. Today we flipped the trust policies and removed direct assumption entirely.

The announcement slack post filled with celebration emojis. Engineers cheering for losing production access tells you something about how everyone actually feels about agents sitting next to prod credentials.

This is how we approach cloud security in general. Security at epilot is not a separate department reviewing things after the fact. It's a core engineering competence: every engineer builds it, runs it and secures it, and external pentests keep us honest. The architecture does the enforcing: SSO in front of everything, CI/CD as the only path to deploy, monitoring that flags a rogue change within seconds, and production access that requires deliberate, named, visible intent.

Accessing customer data is supposed to be difficult. We built it that way on purpose.

We didn't build this because we fear AI agents. We built it because we run them everywhere and intend to run more. Banning agents is easy. YOLOing them into production is even easier. Building the rails that let you run them at full speed is the actual work, and it's why we'll adopt the next generation of agents faster than the companies still deciding whether to allow them.

You build it, you run it still stands. You just don't run it from the machine your agent lives on.

Vlad becomes the sole Head of Engineering. He's been running most of the role for close to a year anyway, since we announced him as my Co-Head back in July 2025. Time to make it official.

This isn't a step away from the job. I'm still leading. I'm still in our engineering leadership meetings. I'm still committed to epilot's long game. The title change is about clarifying what my work looks like in 2026 so that it stops getting in the way of Vlad owning the HOE role outright.

Quick recap of how we got here.

A brief history

I joined epilot in October 2020 as a cloud engineer and took over as Head of Engineering in May 2021. We were about 10 in-house engineers. Our product was a Java monolith built on a 3rd-party low-code platform we didn't even own the source code to.

My job: build better software, faster.

The first few years were heavy on technical and people leadership. I wrote our first Engineering Principles, introduced RFCs, established our weekly product demos, and championed the 360 plan to rewrite the entire product on serverless AWS with microfrontends.

I led the original 360 strike team and drove the new architecture from the front. We killed the legacy platform. We managed to grow MRR fast while migrating customers. Today we ship 150+ times a week to production. The in-house engineering team has grown from 10 to over 40 senior product engineers from more than 10 countries. No management layer. No platform team. No ops team.

I took pride in keeping our headcount relatively low. Raise the bar, hire fewer, expect more out of each. Product Engineer Mindset.

I introduced Salary Fairness reviews, ran our remote-first engineer hiring round after round, and built up most of what now just feels like "how we do things" at epilot.

The shift

As the foundation got solid and the team filled out with great people, I drifted back to where I enjoy my work the most: building great products and helping others do the same.

My last 3 major initiatives have all been about spinning up teams to eliminate specific business bottlenecks:

• Blueprints, Apps & Marketplace
• ERP integrations with Dinero
• AI adoption inside the company: building the tools to safely open up our product and codebase for agents and non-engineers to start contributing

I was never a great people manager

I'll just say it. I was never great at the people manager part of the job.

I've always hated 1:1s and review discussions, which is why I barely do them anymore.

Vlad has been handling most of the people side since July 2025, when we announced him as my Co-Head of Engineering. He's better at this than I am. The team has felt the difference. So much that I want him to own the title outright instead of sharing it with me.

What I'll keep doing

Same as before. Scaling Entity API & Elastic. Integrations, CI Pipelines, Docs, Performance. Figuring out how to use AI better.

ERP integrations with Dinero are still a big bet for MRR. They open up customer segments we can't really reach without them.

AI adoption is where most of my creative energy goes right now. Building the tools, guardrails and workflows that let PMs, designers, customer success folks, and in the future our partners and customers, extend epilot with AI. Engineers then take those drafts the rest of the way to production.

I haven't been this excited about a problem since 360.

I'll keep joining our engineering leadership meetings as Distinguished Engineer alongside the other engineering leaders. Same seat at the table, less fancy title, just more clearly working from the trenches.

This is not me going pure IC. I'm still leading, still running the team solving ERP integrations, still making bets that shape where epilot goes.

Why Distinguished Engineer

Because it matches what I do.

It also sets the right expectation for new joiners. New engineers often expect a Head of Engineering to play a more people-focused role than I have any interest in playing. That mismatch confuses everyone, distracts from real product work, and gets in the way of Vlad owning the HOE role.

A Distinguished Engineer at epilot is someone who:

• Has been here long enough to have built a strong internal track record
• Has their hands on the product
• Leads through example

That's the job I want to keep doing.

Leaving the door open

I'd also like Distinguished Engineer to stay open as a title for other long-tenured engineer leaders at epilot with exceptional track records to grow into.

If you've been here for years, shipped things people still rely on, and want to keep your hands on the product without taking the people-management track, this is what that path can look like.

We're going to need a few of these.

🦄 Thank God it's Friday!

— Viljami

I've been writing about product engineering for years. The core idea: engineers should understand the whole product, the customer, the business problem. I wanted to test the reverse. Can non-engineers contribute to the codebase if you give them the right tools?

Two months of real data delivered a clear answer: yes. And it may be the most powerful shortcut I've found to building capable, business-aware product teams who truly care about what they ship.

Here is what that looked like at epilot: 629 merged PRs in 60 days. 40 PRs were from 11 non-engineers. All teams have at least one person who contributed to our codebase with Claude Code.

We're not alone in seeing this shift. Stripe ships over a thousand agent-produced PRs per week. Superhuman uses Codex to let product managers contribute lightweight code changes without pulling in an engineer (except for code review). Intercom gave Claude Code to everyone in the company, over 1,000 non-engineers (PMs, designers, support, marketers, and more), and more than 300 became active weekly users.

This post is a detailed, honest look at how we set it up, what actually worked, the mistakes we made, and the deeper lessons that emerged.

The starting point

epilot is a B2B SaaS platform for energy companies. We run a fully remote team of 40+ product engineers with no engineering managers and no dedicated platform team. Everyone owns end-to-end ownership. We already ship to production 150+ times per week on AWS serverless.

Our entire engineering team was already using AI coding agents daily. Claude Code, Codex, Cursor. Engineers had adopted them fast and the productivity gains were obvious. The question I wanted to answer next: what happens when you expand codebase access via coding agents to the rest of the company? PMs, designers, customer success, support. Not as a one-off hack day, but as a fully supported, production-grade workflow.

The infrastructure that turned agents into teammates

Claude Code and Codex still only ship connectors to GitHub, and our entire codebase lives in GitLab. So we created a single GitHub repo that bootstraps a developer sandbox with a shallow copy of all 219 repositories. This allows both AI agents or engineers to spin up a fully functioning developer environment with everything cloned & tools set up within minutes.

1. Agents live in Slack

We created #dev-squad-agents alongside our regular squad channels. Anyone could join via Claude Code, Codex, or GitHub Codespaces. I ran three one-hour onboarding sessions to showcase the tooling and make sure everyone had the required access: one for design, one for support, one for go-to-market. The PMs didn't need their own session. Their engineering teams had already invited them and they were eager to start.

This single decision drove adoption more than anything else.

2. One-click preview environments

Every PR automatically generates a live preview link in CI. Click it, see the change, test it immediately. No local setup required.

For engineers this is convenient. For non-engineers it is essential. If they cannot see and interact with their change, they can't meaningfully contribute.

3. Deep institutional context

We built an indexed knowledge base called how-to-everything. It contains our API patterns, backend conventions, code style, CI/CD and testing strategy, full business context, domain terminology, and our entire RFC archive.

Stripe solved the same problem by connecting agents to 400+ internal tools. We chose comprehensive indexing. The insight is identical: agents without your company’s context produce garbage. Agents with it produce work that feels native.

4. Design system in context

Our Volt UI component library is baked into the agent prompt. Claude now generates frontend code that respects the correct components, spacing, colours, and patterns from the first try. Consistent UI without forcing a designer to review every small change.

Claude as a coworker: the Slack pattern

Claude quickly became the most active “team member.” People tag @Claude the same way they tag colleagues, and they expect results just as fast.

How engineers use it

Simple tasks look like: “@Claude fix EPI-5660.” A PR appears.

More powerful uses scale across the organization:

• “@Claude which microfrontends still use the legacy editor?” → instant audit of all 219 repos.
• “@Claude migrate 13 backend repos from file-client to lambda-adapter.” → completed in one session.
• Mass upgrades across 20+ repositories: one message triggers 6 parallel sub-agents that handle changes, tests, and PRs.

What once took a full day of tedious cross-repo work now happens in hours. Agents worked weekends. Engineers didn't have to.

How non-engineers use it

They speak in plain product language: - “@Claude disable drag and drop of entity table columns for embedded tables.”

• “@Claude when creating a new family in the Label Builder, the drawer should close automatically.”

Claude figures out the right repositories and implementation details. Non-engineers don't need to know our tech stack or repo structure. They describe the desired outcome.

Production impact

We added Claude to #alerts-production on day two. It investigates incidents, reads logs, traces errors, and proposes fixes before most engineers open the dashboard. When a bulk action hammered Elasticsearch with 100 parallel requests, Claude designed an org-level job queue with concurrency limits that cut peak load by 90%. Engineers started replying “another one bites the dust” to resolved alerts.

Within a week, Claude felt like a trusted junior-to-senior coworker: fast executor, deep historian, and reliable pair programmer. Usage spread organically to every squad.

What people actually built

Non-engineer contributions (40 merged PRs from 11 people)

• Feature work: One PM built auto-tagging for file uploads (a feature he had spec’d three months earlier). Another shipped 13 PRs on messaging (bulk email templates, sidebar toggles, drawer behaviour, i18n). A go-to-market colleague delivered entire features, like pivot tables for displaying time-series data in epilot tables, that are now live for customers.
• Daily improvements: Customer success shipped portal config changes they handle every day. Designers and support added FullStory tracking and started prototyping new UI flows.
• Translations: Native speakers cleared 116 German translation improvements with almost no engineering involvement.
• UX via vibecoding: A designer shipped a UX improvement to our Flow Hub before heading into the weekend: when creating a new Flow, users now get prompted to enter a name first. She described it as "vibe coding" and the change reduced friction in a way she'd been wanting to fix for a while.

These were not toy changes. Several were meaningful, customer-facing updates. One PM even asked Claude to create a demo video of the feature it just built, so he could share it with the team. The full cycle from spec to demo, handled through conversation with an agent.

Engineer contributions (589 merged PRs)

Engineers use agents across the full lifecycle. Writing RFCs and technical designs. Implementing multi-service features. Running tests. Shipping to production. Auditing feature flag usage across 15 repos and cleaning up stale flags. Investigating production incidents from alert channels before anyone opens a dashboard.

Some of the most valuable uses are research and diagnostics. "Which services listen to the file upload event?" "How many microfrontends still use the legacy editor?" "Audit all listSchemas calls and upgrade to the new summary API where safe." These are questions that used to take half a day of grepping across 219 repos. Claude answers them in minutes.

GitHub Codespaces turned out to be a key piece for engineers too. Running Claude Code in auto-mode in a cloud sandbox means you can kick off a multi-repo migration and let it run without risking your work laptop. No local state to corrupt, no accidental changes to your working branch. The sandbox spins up with the full codebase mirrored and Claude works through the task autonomously. Engineers now run 6 parallel sub-agents in Codespaces for bulk upgrades across 20+ repos: package bumps, migration scripts, test updates, PRs opened. What used to be a full day of copy-paste across repos is done in one session.

The new collaboration model

Non-engineer describes the problem in business terms → Claude implements → engineer reviews for architecture and quality. The person closest to the customer now helps ship the solution. Feedback loops tightened dramatically.

A designer and engineer even deployed Agentation (a prompt-structuring tool) in 2.5 hours. The designer now uses it daily and calls it a “dream collaboration.”

Guardrails: the hard lesson

A few weeks in, one non-engineer with full GitLab push access bypassed our CI pipeline and deployed code to production without review. No damage, but it was exactly the wake-up call we needed.

We immediately tightened permissions: non-engineers and agents create PRs; engineers review and merge. No direct push to protected branches.

But that incident was just one piece. When you give AI agents to 50+ non-engineers, you need a proper governance framework. We built one.

What we had from day one

• Sandboxed execution only. Agents run in Claude Code Web, Codex sandbox, or GitHub Codespaces. No production infrastructure access. No deployment outside CI.
• Dedicated bot users. Limited API scopes. Code-level access only. Zero access to customer data, session tokens, or production databases.
• Public channels only. All agent interactions happen in shared Slack channels. Engineers see everything and can intervene. No private DMs with agents.
• Human responsibility. You trigger the agent, you own the output. No exceptions.

What we added after

We published a formal Agentic AI Tool Usage Guideline. The core principle: give the agent the minimum access it needs for the task at hand, and nothing more.

The guideline includes:

• A three-tier data classification. Red (never share with AI): customer PII, session tokens, credentials, security documents. Yellow (anonymise first): user research transcripts, survey responses, usage metrics with identifiers. Green (safe): public docs, your own drafts, anonymised research, process docs. Hard line on red: no exceptions, no "just this once."
• An approved connector matrix. Every tool combination (Claude + Figma, Claude + Atlassian, Claude + Miro, etc.) classified by risk level with specific conditions and recommended settings.
• Prompt injection awareness. Use a separate browser profile for Claude browser extension. Don't paste content from untrusted sources into agent prompts. Disconnect MCP connectors when not in use.
• Explicit dealbreakers. Leaking customer personal data, unauthorized code deployment, and feeding company data into model training are not allowed regardless of how useful the outcome might be.
• GDPR and EU AI Act compliance. The guideline maps our practical rules to specific legal obligations: data minimisation (Art. 5), DPA requirements (Art. 28), AI literacy obligations (Art. 4), transparency requirements (Art. 50). We're a European company serving energy utilities. Regulatory compliance is table stakes.

Lesson: Lock down merge and push permissions before opening access. We got lucky. And build the governance framework early. Guardrails didn't slow adoption; they accelerated trust and usage.

What surprised us most

• Adoption speed: Three one-hour onboarding sessions and people were off. Every team across the company had active contributors within three weeks.
• Non-engineers shipping real features: I expected small fixes and translations. Instead, people from across the company delivered complete, production features.
• Claude as institutional memory: Support now asks Claude product questions instead of hunting down engineers. It has become the fastest way to understand how anything works across five years of code history.

Beyond code

This is just one layer. My colleague Suresh built Vigilos, an internal BI agent that lets anyone query our data warehouse in natural language. Designers check average journey name lengths for UX decisions. Customer success identifies accounts with AI features disabled. Product managers run customer segment analysis in minutes.

We'll share that story soon.

Where this goes

Two years ago I published the Product Engineer Manifesto: engineers should care about the customer, the business outcome, and the whole product.

What these two months taught me is that the same mindset now applies in reverse. Give non-engineers the right tools, deep context, and sensible guardrails, and they will deliver.

We're still hiring product-minded engineers. That hasn't changed. Engineers at epilot own the full product lifecycle: customer understanding, architecture, quality, reliability, and shipping. Non-engineer contributions don't reduce what we expect from engineers. They amplify it. The entire organization now ships alongside them.

629 merged PRs in 60 days. 40 from non-engineers.

Same agent, same tasks, different approach to context. The difference between working and not working.

The insight clicked immediately. If we could give Claude Code reliable access to this institutional knowledge without forcing it to decide when to look things up, it would fundamentally change how people work in our codebase.

So we built it.

The Context Problem

Coding agents have a token limit. A ceiling on how much information they can process at once. Think of it like working memory. You can't hand Claude Code your entire codebase and documentation library upfront. It's too much.

The traditional solution is skills: the coding agent decides when it needs information and actively looks it up. "I need to know about authentication, let me search for that." Sounds reasonable. In practice it creates three problems:

1. Decision paralysis - the agent has to decide when to look up docs, and it often guesses wrong
2. Async delay - every lookup is a round-trip, breaks flow
3. Sequencing conflicts - exploring code vs. consulting docs creates timing issues

Vercel's approach flips this: give Claude Code a compressed index of what documentation exists and where to find it, before it starts work. The index is small enough to fit in context every turn, so it always knows what's available. When it needs details, it reads the specific file directly. No decisions, no lookups, no conflicts.

Why Compression Matters

The key innovation is compression. A full documentation tree - all the folder structures, file names, categories - takes significant space. Too much to include in every conversation turn.

The compressed index uses a simple pipe-delimited format that shrinks this by ~80%:

[epilot Docs Index]|root: ./.epilot-docs|00-general:{tech-stack.md,business-context.md,ci-cd.md}|01-apis:{api-design.md,calling-apis.md}|02-epilot360-microfrontends:{env-vars.md,local-dev.md}|...

That single line tells the agent:

• Where the docs live (.epilot-docs/)
• What categories exist (00-general, 01-apis, etc.)
• What files are in each category

It's a table of contents, not the full book. But it's enough. The agent sees the map, understands what's available, and pulls specific files when needed. The decision-making load disappears.

Why This Matters

The accuracy improvement is significant (100% vs 79% task success), but it's not the real story.

The real story is what happens when you lower the barrier to contribution. Proper context enables the person closest to the problem to fix it, regardless of job title. Your PM can fix bugs. Designers can adjust component behavior. Support engineers can patch data issues. You remove bottlenecks. Context and authority live in the same person.

Coding agents are the equalizer. But they're only as good as the context you give them.

Most companies will throw Claude Code at their codebase and wonder why results are inconsistent. The agent hallucinates patterns. Makes incorrect assumptions. Writes code that doesn't match conventions.

The difference is context. Structured, compressed, always-available context about how your codebase works.

How to Build Your Own

The pattern is straightforward. Here's what we did at epilot:

1. Curate agent-friendly documentation - organize your internal knowledge: conventions, APIs, architectural patterns, framework usage, code style
2. Structure by domain - group related docs (general, backend, frontend, infrastructure, etc.)
3. Use descriptive filenames - the agent sees filenames in the compressed index before opening files. api-design.md is better than guidelines.md. error-handling.md is better than errors.md. Make filenames searchable and specific.
4. Automate updates - pull live data where possible (OpenAPI specs, schema definitions, framework docs)
5. Generate compressed index - use a simple format (pipe-delimited works well) that reduces the doc tree by ~80%
6. Embed in agent context - add the index to your CLAUDE.md or AGENTS.md file (the context files Claude Code reads)

One thing worth highlighting: we don't just include our internal documentation. We also package docs for the frameworks and libraries we heavily use - single-spa, openapi-backend, openapi-client-axios, i18next, and Volt UI (our custom design system). When Claude Code needs to know how i18next pluralization works or how to register a single-spa parcel, it already has the answer. No hallucination, no outdated Stack Overflow posts, just accurate framework documentation.

What This Enables

We're already seeing daily usage across the team. Developers context-switch between services faster. Non-engineers contribute directly instead of filing tickets.

But the real potential is broader: if compressed context improves coding agents for technical documentation, why stop there?

• Runbooks and incident response - on-call engineers with instant access to procedures
• Customer domain knowledge - support teams with context on product behavior
• Business logic - product decisions and their rationale, preserved and accessible

The pattern is the same: curate the knowledge, compress it, embed it in context, let the agent work.

The Constraints Are Disappearing

For decades, contributing to a codebase required deep technical knowledge. You needed to understand the language, the frameworks, the architectural patterns, the implicit conventions. The barrier was high.

Coding agents lower it. Claude Code, Cursor, and similar tools don't replace engineers. They make technical knowledge more accessible. With the right tooling and the right context, a PM can fix bugs. A designer can adjust styling logic. A support engineer can patch data issues.

The question isn't whether this is possible. It's how fast you can adapt.

Organizations which enable broader contribution will move faster than those that don't. The tools exist. The research is clear. What's missing is execution.

Start Simple

You don't need to document everything upfront. Start with the knowledge that causes the most friction:

• Code style conventions - how you write TypeScript, naming patterns, file structure
• Common patterns - how you handle authentication, API calls, error handling
• Framework specifics - non-obvious usage of your frameworks and libraries
• Internal APIs - if you have OpenAPI specs, even better

Create a simple doc structure:

docs/
  00-general/
    code-style.md
    tech-stack.md
  01-apis/
    api-design.md
    calling-apis.md
  02-backend/
    error-handling.md
    database-patterns.md

Generate the compressed index (pipe-delimited format, one line per directory). Add it to your CLAUDE.md or AGENTS.md file - the context files that Claude Code and other coding agents read on startup. Done.

The compressed index approach works! 🎉 Vercel's research proved it: 100% task success versus 79% without it. We've validated it internally. Now it's about whether you'll adopt it before your competitors do.

I moved from Munich to Cologne with the goal to help out a small energy startup build their SaaS on AWS.

What I found was a crazy, ambitious team, an impressive sales track record, and a product team struggling with execution; largely due to the product having been outsourced early on to a team in Vietnam, who built it on top of a proprietary product we didn't own.

I saw the opportunity, and decided to give it six months. It turned into five years. The objective:

• Rewrite the product on a scalable tech stack we own.
• Build a top engineering team focused on speed & delivery.
• Reach $100M valuation in 3 years.

Today, epilot runs entirely on serverless AWS. We ship to production more than 100 times each week and delivered over 300 new features just in the past year.

The in-house engineering team grew from 10 to more than 45 senior product engineers. We've kept the structure flat, the ownership high, and the results speak for themselves.

That ambition doesn't feel like a stretch goal anymore.

From rewrites to real speed

Back in 2020, everything was about big risky bets: rebuild the product, migrate the customers, prove that it works.

We killed the legacy product, launched 360, and reached feature parity while shipping new features for customers.

Somehow we pulled this off while sustaining healthy revenue growth throughout.

Now we're shipping faster than ever.

Our early bets are paying off. There's still no platform team, no DevOps team, and no manager roles to oversee engineers.

With AI, we now move at a pace we couldn't imagine five years ago.

A new kind of challenge

So what's next?

The next chapter isn't about just delivering more features, quicker. It's about long term thinking.

When I joined epilot, I was a 20-something chasing my next startup exit story. I moved to Cologne on a limb, prepared and happy to work weekends & late nights to get shit done.

Now I'm in my 30s. Married. Two kids. Living back in my old hometown in Finland, working remotely, surrounded by family and pine trees.

The pace hasn't changed much, but my perspective has.

Today, I value the freedom and balance this company gives me just as much as the excitement of the early years. That's why this question matters to me more than ever:

How do we make epilot a place where top talent wants to spend the next 3–5 years and beyond, to do the best work of their career?

A place to do your best work

For me, the answer is simple.

epilot is a place where you can do the best work of your career and still have a life.

We don't demand crazy hours, we never did. That was a personal choice I made to push us forward early on.

But we still demand excellence & delivery. The bar today is much higher than when I started.

With today's AI tools, I find there's just less need to spend nights coding anyway. I no longer need to stay late to finish coding a feature, or set up infrastructure, when Claude Code helps me get it done 5x faster, during business hours.

Our Salary Fairness model is a big part of that promise.

We review compensation twice a year to make sure every engineer gets paid their fair market value. Not based on negotiation skills, but on track record and market demand.

It's simple: we want to make it as hard as possible for anyone to justify leaving epilot just for better money.

At the same time, we encourage everyone to keep growing their track record. Inside and outside epilot. We support open source and side projects, because all of it builds experience, visibility, and helps strengthen epilot's brand as a serious tech company with top industry talent.

You don't have to quit to grow.

Why it matters

If we want to keep attracting and retaining the best talent, we need to make sure the promise of working here still feels worth it.

That means paying fairly, staying transparent, and giving people the freedom to focus on what matters: building great products and growing in their craft.

Salary fairness plays a big part, but so does trust. The trust that you can make decisions, ship fast, and still log off to be with your family at the end of the day.

That balance is rare in tech.

It's one of the main reasons I'm still here.

What keeps me here

I've been building software for more than 15 years. I've worked in startups, enterprises, and built my own companies.

But I've never stayed anywhere as long as I've stayed at epilot.

Here I get to live what I wrote about in the Product Engineer Manifesto: real ownership, customer impact, and ability to take pride in my work.

The Viljami who joined in 2020 wanted to move fast and build a SaaS unicorn in record time. The Viljami in 2025 wants to make it last.

Looking ahead

We'll keep finding ways to ship faster. Always.

But we probably don't need big rewrites anymore. No need to take crazy shortcuts. We've already proven we can build.

Now the focus is growth through quality, delivering what truly matters for our customers.

That's ERP integrations for me now. Not exactly the sexiest topic, but one that still excites me through the potential to make a massive difference to both the adoption of our product, and our customers.

But we're in this for the long game now.

We've hit our goals so far. Now let's build the next chapter, from where we are today all the way to unicorn. Even if that takes longer than a couple more years.

It's still early in epilot's story. We're not even close to done yet.

🦄

Before anything else: thank you.

The work you do, aligning stakeholders, maintaining a relationship with users, finding new opportunities and communicating them is rarely easy and often invisible. It's the kind of effort that can be underestimated from the outside. But we see it. And we respect it.

As product engineers, we're sharing the manifesto not as a critique of your role, but as a challenge to ourselves. We want to step up. Not just write the code, but understand the customer. Not just deliver features, but take responsibility for outcomes.

Still, we know that can come across the wrong way. The things we post: the manifestos, the bold claims, the challenges to not blindly follow roadmaps created by PMs can read like they're aimed at you. But the truth is, we're mostly talking to ourselves. Encouraging each other to break old habits and move beyond being passive implementers.

We want to be useful upstream. We want to sharpen the problem definition. Bring context from past experiments. Understand your goals and constraints. Help shape what we're building before it's locked in. Not because we think we know better—but because we care deeply about getting it right.

However, one of the realities we often see is that having a "manager" title can sometimes create an implicit hierarchy within the team. This can unintentionally lead to a dynamic where engineers are seen primarily as executors of specifications, rather than as creative contributors to the product vision.

The manifesto isn't about diminishing the importance of product management or suggesting that engineers should take over your role. Rather, it's about fostering a space where engineers can engage more deeply with the product vision, contribute ideas, and help bring even more innovative solutions to life. We see this as a way to strengthen our collaboration, not replace it.

So don't let us off the hook. Treat us as peers. Hold us accountable when we overstep or get lost in abstraction. And when we ask questions "What's the problem? For who? Why now?" see it for what it is: a desire to think with you, not just follow.

Product engineering is a team sport. We're here to play it with you.

Sincerely,

Your engineer colleague

I hope this letter finds you well. I wanted to take a moment to share some thoughts about how we work together, and more importantly, to express my deep appreciation for the incredible work you do.

Your work is often the soul of the product. The empathy you bring, the deep research and great taste you bring add up to a joyful experience. That's what makes great products feel great. As engineers, we aspire to that level of care.

As product engineers, we often find ourselves in awe of the magic that designers create. Your ability to envision user experiences, understand the needs of our customers, and translate them into beautiful, intuitive designs is truly inspiring. For many of us, designers are like the north star: something to strive towards in terms of creativity, great taste and user empathy.

You might see talk of "product engineering" and wonder if we're trying to step on your toes. And if we're being honest, sometimes our content can come across that way. Manifestos, checklists, strong opinions—written as if we have it all figured out.

But the truth is: it's mostly written for ourselves. It's a reminder to stop coding and start thinking. To look beyond the Jira ticket and ask: Does this feel right? Is it intuitive? Would we be proud to ship this?

We want to be closer to design, not to replace it. Contribute our experience and expertise to it. Because we believe great products happen when engineers develop taste. Not just coding skills.

We're learning. Trying to notice when something feels off. Trying to ask better questions. Trying to be more than just the dev who picks up tickets and needs to be told what to build.

So don't lower your standards for us. Challenge us when we miss the mark. Treat us as partners. Help us grow into engineers who can build with you. Not after you.

Because in the end, product engineering is a team sport. And we want to be teammates you can count on.

Sincerely,

Your engineer colleague

An all too familiar and frustrating situation for many of us.

You think of yourself as a proud product engineer wanting to solve meaningful problems but then get slapped with a detailed feature specification designed by a group of non-developers without your input. Now they expect you to go deliver their vision.

This is very much the reality in most product teams. Calling yourself a product engineer doesn’t automatically mean your voice will be welcomed. And that’s totally okay.

The Ideal vs. the Reality

In my past writings, I've painted the ideal: engineers who understand customer needs, question roadmaps, challenge designs, and contribute beyond code. But the reality is often much messier.

Some PMs and designers love working closely with engineers. Others are still adjusting to the idea. And that’s fair. The product engineer mindset definitely isn’t the norm.

Let’s not forget, PMs and designers are often under pressure too. It’s only natural they sometimes default to the most familiar and streamlined path. One that doesn’t always include engineers in the early stages.

And let’s be honest. Sometimes engineers haven’t yet built the trust or skills to contribute meaningfully.

The Friction Is Normal

The moment you step outside your lane, you shouldn’t be surprised when the reaction isn’t overwhelmingly supportive.

You will face skepticism.

You might be seen as overstepping.

You will encounter:

• Requests to give estimates for implementing someone else's design
• Roadmaps shared as top-down mandates
• UI prototypes handed over "ready for dev", expecting pixel-perfect implementation
• Pushback from asking too many questions

This is the price of wanting to do more than just execute. And if we’re honest, many of us haven’t always shown up in these conversations in a way that earns trust.

Is this a culture problem? Not necessarily. Most teams don’t have a rule against engineers joining product discussions. It’s just not the default. It’s less about policy and more about patterns. Changing those patterns takes trust, initiative, and persistence.

At the end of the day, it’s the product engineer’s job to show that product engineering actually works.

Earn the Trust

Calling yourself a product engineer isn’t a free pass. No one hands you a seat at the product table just because you want it. You must earn it. Show, don’t tell.

• Do the homework. Know the names of your customers. Know the business domain.
• Talk to your colleagues, not just other devs. Find opportunities to interact with customers.
• Ask helpful questions that sharpen the team's product thinking.
• Dive into data. Gather insights. Find new metrics and ways to collect useful signals.
• Bring value to the table. Demonstrate you understand the customer problem by making thoughtful proposals that move the product forward.

You need to show up in demos, RFCs, testing sessions, and reviews. Not just in code commits.

Why Bother?

Because it’s worth it.

We do this for our own professional pride. To put great products into the hands of happy customers. And into our portfolios.

We don’t challenge product decisions because we want to take over the PM’s job or undermine the work of our teammates. We do it because we care about impact. Because we want our effort to count.

Because it hurts to pour weeks of your life into something that doesn't work, only to discover it failed because no engineer was involved in the concept phase.

Our Leverage

And here’s something to remember: We, as engineers, hold real leverage. We are the only ones who can actually turn product decisions into reality. No idea ships without us.

So while we may not always get a say by default, we do get a say in how we show up and how deeply we choose to care.

Use that leverage wisely. And proudly.

My core thesis is that with the rise of AI, the expectations for software engineers are being redefined, moving away from narrow programming roles: backend, frontend, C#, Java, React, etc. fast becoming obsolete due to AI tools like Cursor lowering the barrier of entry and unlocking productivity with any technology.

The days when you could coast on your ability to churn out code are over. If you're still clinging to the idea that you're safe just because you know how to write code in a widely used language or framework you're in for a rude awakening.

I believe this trend is likely to be the biggest industry shift in my lifetime, even surpassing the agile movement of the last 20 years. And I'm not the only one talking about it. (#1, #2, #3, #4, #5)

But I got it wrong.

Not entirely wrong, but I made it more complicated than it needed to be. In my quest to define what it means to be a product engineer, I overlooked the power of simplicity and ironically did not think enough about the customer: the ambitious software engineer thinking strategically about their careers.

I overengineered it

In my earlier attempts to help engineers break out from purely technical roles, I published an extensive checklist to help engineers think and ask questions as product engineers. It covered everything from understanding the user and the market to measuring success and staying ahead of industry trends.

Here's a taste of that checklist:

• 1 Understand
  • 1.1 Who's the user?
  • 1.2 Who's the customer?
  • 1.3 What's the market?
  • 1.4 Ask Why
  • 1.5 What do we already know?
• 2 Craft
  • 2.1 Am I proud of what I'm building?
  • 2.2 Does the product feel good?
  • 2.3 How do I get there faster?
  • 2.4 Teamwork
• 3 Growth
  • 3.1 How do I measure the success of my work?
  • 3.2 How do I maximise the impact of my work?
  • 3.3 How do I stay ahead of the curve?
• 4 Product Vision
  • 4.1 What’s our North Star?
  • 4.2 How does my work impact the overall design of the product?
  • 4.3 What's the ambition level?

Great stuff, but let's be honest — no engineer is going to run through a 15-point checklist to make a product decision. Just seeing this long list of questions can be totally overwhelming, especially if you're used to being in a technical role.

If we expect engineers to actually start doing this stuff, the core idea needs to be simple and easy to remember.

The Wake-Up Call in the Sauna

As I'm writing this post, I'm enjoying a company working vacation in a nice hotel in Mallorca with my epilot colleagues. I was having a conversation in the hotel sauna with a principal engineer where something he said in passing suddenly clicked for me.

He was venting about some engineers on his team who seemed to be diving head first into coding without bothering to understand the problem. "It's so easy. Every engineer should be able to answer what problem they're solving, who the customer is, and what the impact of the work they're doing is."

That hit me like a splash of cold water in a 100°C Finnish sauna.

He was totally right.

Boiling It Down to 3 Essential Questions

To think like a product engineer, it's already enough to start with just three simple questions:

• What's the problem?
• For who?
• Why is this important?

Let's dive a bit deeper into each one.

1. What's the problem?

Stop coding for a second. Do you really know what you're trying to solve? Not the ticket description in Jira, but the real-world issue. If you can't articulate the problem in simple plain English, you have no business writing a single line of code.

2. For who?

Identify your user and customer. Sometimes they're the same person; other times, they're not. Understanding who will use your product (and who will pay for it) is crucial. It shapes the way you approach the solution and helps you tailor the experience to meet their needs.

3. Why is this important?

As engineers there's never a shortage of things we could improve or implement. If solving the problem doesn't make a meaningful difference, why are you wasting your valuable time? We're not here to build features that nobody uses or cares about. Connect your work to something that actually matters and helps build your track record.

By anchoring your work in these three questions, you immediately move from code monkey to a high value product-minded engineer. Still a rare breed. You're not just implementing features someone else decided to build; you're elevating yourself to a position to influence product decisions and build smarter.

Leverage Your Team—They're There for a Reason

The biggest mistake engineers make is thinking they have to find all the answers themselves.

Most of us are lucky to work in a product team with other disciplines like UX researchers, designers, PMs and business stakeholders whose job is to help answer these questions.

By leaning on your team, you not only find better answers but also foster a more collaborative and innovative environment.

Product engineering is a team sport.

Taking Action: Start Asking the Right Questions Today

So, the next time you tackle a new topic, pause for a moment before diving into code. Ask yourself:

• What's the problem?
• For who?
• Why is this important?

Write down your answers. If you don't know, reach out to your team and find out. This simple practice can transform your approach to work, leading to better decisions, more impactful solutions, and a greater sense of ownership.

Congratulations, you just became a Product Engineer!

Join the Conversation

I'd love to hear your thoughts on this simplified approach to product engineering. Have you tried asking these three questions in your work? What impact did it have? Share your experiences in the comments below.

Consider giving the GitHub repository a star if the product engineer role resonates with you!

Here's why embracing uncertainty isn't just smart—it's necessary:

Trial and Error Beats the Best Research

Nicholas Nassim Taleb, one of my favourite authors and thinkers argues that in complex systems you can't predict outcomes with precision. Instead, you should expose yourself to situations where small bets can lead to significant rewards, and failures are limited in their impact. Simple trial and error over time tends to beat every other strategy.

Taleb calls it antifragility—the idea that not only can you be resilient to shocks and volatility, you can benefit from it. By allowing failure, you become stronger. Each stumble is feedback, a new learning.

Consider the world of investing. High-powered hedge funds employ armies of PhDs armed with complex algorithms and predictive models. They pour millions into trying to forecast market movements. Yet, time and again, they get blindsided by events their models didn't anticipate—black swans that wipe out gains overnight.

Meanwhile, index fund investors who follow a simple, trial and error approach outperform these sophisticated funds over the long haul. Their strategy isn't flashy, in fact it's remarkably simple. It doesn't rely on predicting the unpredictable; it focuses on adapting to reality as it unfolds.

The lesson here is straightforward: adaptability beats elaborate strategy. Trial and error is flexible. It adapts on the fly. When you make small, calculated bets, you're not risking everything. You're gaining insights that no amount of theoretical modeling and research could provide.

Building Products Is Not a Science

There is no magic formula for success in building a product. If only it were that simple. The truth is messier. Sometimes it's being first to market with a groundbreaking new idea or technology. Sometimes it's great marketing. Sometimes it's more attractive pricing. Sometimes it's just better execution to disrupt an existing market.

Take Nokia, for example—the pride of my home country Finland. 🇫🇮

They dominated mobile phones but missed the smartphone revolution. Why? They treated product development like a science, shipping calculated improvements backed up by market research with each new model. But building products is more art than science. It's about gut feeling, timing, and yeah, a bit of luck.

Develop Taste

It's not enough for a product to function and have useful features; it has to feel right. That means sweating the details others overlook. It means deeply caring about the product from start to finish.

Taste isn't just about aesthetics; it's about creating something people love. Apple's obsession with design and user experience set them apart because they understood this deeply.

Taste isn't innate—it's developed. You don't just wake up one day with great taste. It's created through experience—by immersing yourself in your craft, by building. Each project, each line of code, each design tweak sharpens your sense of what works and what doesn't.

Developing good taste is the ultimate form of professional pride. It's what allows you to move faster than others and make decisions with intuition.

Yet, the idea of taste is often seen as controversial when building products. Where's the market research? Where are the numbers?

Be Data-Informed, Not Data-Driven

Data is useful, but let's not kid ourselves – it's easy to manipulate. Cherry-picked numbers and fancy charts can convince nearly anyone of anything. The strategy consulting industry are masters at this: making executives feel they're being data-driven when really the consultants are brought in to make whoever pays for them look good, and back their decisions with impressive research and presentations.

Way too frequently data is only used as a tool to back up whatever decision you would have made anyways. Not to say this is done on purpose–it's just very difficult for us humans not to be biased towards our own preferences. This is confirmation bias, and even people doing actual science aren't immune to it.

Data tells you what's happening now, not what could happen next. It lacks context and can't predict breakthroughs. Good data helps inform your decisions, sure, but don't let it shackle you. Balance it with taste, vision, and a willingness to defy the numbers when your intuition tells you to.

Steve Jobs nailed it: "People don't know what they want until you show it to them." If Apple had been data-driven, the iPhone would never have existed.

Heuristics Are Powerful

Heuristics or rules of thumb help you make decisions without drowning in analysis. The Pareto Principle is a classic: 80% of results come from 20% of efforts. Focus on what counts.

Heuristics are the survival tools we develop over time, forged through countless trials and errors. They don't pretend to predict the unpredictable; instead, they offer robust strategies for navigating chaos.

The most powerful heuristics are the ones you craft yourself. They're born from your unique experiences—your wins, your screw-ups, the patterns only you can see. Over time, these personalized rules become your competitive advantage.

Here are some of my own heuristics:

• Show Don't Tell
• Every Week is Quality Week
• Unblock others first
• The Keeper Test
• Simplicity is worth fighting for
• Imposing a process is the last resort
• Everything is a Bet

When faced with countless options, heuristics cut through the noise. They're not infallible, but they keep you moving. Be aware of their limits, though. They can introduce biases if you're not careful. Use them as guides, not strict laws.

It's Important to Do Things Fast

Speed matters. When you're moving quickly, you focus on what's essential. There's no time for unnecessary fluff.

You learn more per unit of time because you're making contact with reality more frequently. Each iteration, each release, gives you immediate feedback. You're not theorizing in a vacuum; you're engaging with real users and real-world problems.

The market doesn't reward the perfect—it rewards the first to make an impact. Google sat on their AI research while OpenAI shipped a consumer product that changed the world overnight with ChatGPT.

Shipping fast isn't about lowering standards; it's about focusing on what truly matters and getting it into users' hands. Research and data can be helpful, but ultimately leaders need to trust their own experience, heuristics and taste to make a bet and align everyone behind a vision and deliver.

Slow is fake. It's an illusion of safety that keeps you from confronting the truths you need to hear. While you're stuck researching, planning and tweaking, the world moves on without you.

I'm talking, of course, about the agile coaches and scrum masters, the architects and platform engineers; the growth hackers and data-driven researchers. These vanguards of innovation are vital to any team that wants to maintain a perpetual state of being maximally enabled, well-informed, and facilitated to perform their work.

Let's begin with the often underappreciated roles of agile coaches and scrum masters, the only people in a company actually certified™️ to set up the correct processes and tools to run an agile team. This, of course, is paramount for implementing agile—the popular software development philosophy that explicitly tells us to avoid focusing on processes and tools.

Agile coaches and scrum masters are professionals whose primary job is to ensure all work in teams is performed in sprints, with daily stand-up meetings to guarantee everyone is constantly enabled. This allows the velocity of a team to be meticulously measured in story points, which no one quite understands, but hey, at least we're definitely data-driven.

Surely, without their meticulous facilitation, teams would be left wandering aimlessly, unsure of how to break down their work into JIRA tickets. Their presence ensures that everyone stays aligned and that every team member is reminded of their blockers and sprint goals every single morning. Because nothing screams productivity like a daily stand-up where everyone takes turns saying, "no updates from my side."

Next, we have the architects. These exceptionally gifted, usually very senior engineers haven't touched any production code in the current decade. They are often found in their natural habitat, PowerPoint, where they sketch out their grand visions of future systems and envision platform rewrites for actual software engineers to implement.

Their role is to think so far ahead that their ideas become completely theoretical, creating software designs that are so advanced, they may never actually be implemented. But that's not the point. The point is to have a shared vision, to inspire, to draw boxes and lines that will one day guide the hands of actual engineers who might one day understand what they were trying to convey.

Then, there are the platform engineers. A demanding role with zero responsibility to deliver anything of value to customers. These unsung heroes dedicate their time to building and maintaining the internal tools and infrastructure that supposedly makes everyone else's job easier. Yet, their true skill lies in making things so complicated that only they can understand and manage them.

They manage infrastructure, design intricate CI/CD pipelines, common libraries and tools that add layers upon layers of necessary abstraction, which they insist must be adopted and standardized across all teams. The result? All teams having to learn and depend on custom in-house tools so complex that even the slightest changes require a detailed consultation with the platform team, ensuring their perpetual job security.

And then we have the growth hackers, the avant-garde marketers of the tech world. Their job is to come up with innovative ways to "hack" company growth, often resorting to questionable methods that straddle the line between clever marketing and outright deceit. They dive into data, running A/B tests, tweaking landing pages, and optimizing user funnels to squeeze out the tiniest incremental gains.

Of course, the real growth usually comes from the core product being genuinely useful, but why let that overshadow the need for an entire team devoted to marginal tweaks and vanity metrics?

Data-driven researchers are the prophets of the modern tech age. Armed with vast amounts of data and research, they uncover profound insights such as "users prefer faster load times" or "clearer buttons improve user engagement." Their work truly brings scientific rigor, managing to bring impressive-sounding numbers and great data visualizations to argue for any side of a decision, usually the one they already went with before doing any research.

These bullshit roles are the pillars upon which modern tech companies stand. They enable a culture where productivity is constantly measured, documented, and discussed, albeit often at the expense of actual productivity.

Without them, who would ensure that calendars get filled with back-to-back meetings so that everyone is too busy to notice no work is getting done? Who would write all the documents and Slack messages, ensuring no detail is left unshared or undiscussed? In doing so, these roles truly create a seamless, endless flow of communication, although one might argue that less talk and more doing might yield better results.

But where's the fun in that?

How to Think Like a Product Engineer

The following is a checklist of questions for product engineers:

• 1 Understand
  • 1.1 Who's the user?
  • 1.2 Who's the customer?
  • 1.3 What's the market?
  • 1.4 Ask Why
  • 1.5 What do we already know?
• 2 Craft
  • 2.1 Am I proud of what I'm building?
  • 2.2 Does the product feel good?
  • 2.3 How do I get there faster?
  • 2.4 Teamwork
• 3 Growth
  • 3.1 How do I measure the success of my work?
  • 3.2 How do I maximise the impact of my work?
  • 3.3 How do I stay ahead of the curve?
• 4 Product Vision
  • 4.1 What’s our North Star?
  • 4.2 How does my work impact the overall design of the product?
  • 4.3 What's the ambition level?

1 Understand

1.1 Who's the user?

As a product engineer, your #1 goal is to create happy users. These are your fans! Always start with the user!

Your user is the person that primarily interacts with your product and whose experience your work will directly impact.

You may have multiple user groups. People who may have varying reasons to use your product or might interact with it in different ways from different angles, maybe on different kinds of devices.

You should understand how to help these users. What drives them to use your product? What delights them? What are their pains?

As a product engineer you should demand and support your team find answers to these questions before jumping into writing code.

1.2 Who's the customer?

Yes, this is a different question to "Who’s the user?". The customer is whoever pays for your product, not always who uses it.

You should know what makes your product valuable to your customers to make better decisions on what to invest your time.

Hint: If you're in B2B the answer always has to do with helping your customers save money or make more money. Understanding your customers' core business is key to understanding why they would pay for your product.

Most importantly, you want to make whoever is paying for your product look good. After all, they're the ones taking a risk by picking your product. You ALWAYS want to reward them for that.

1.3 What's the market?

Zooming out, let's take a look at the wider market landscape. Who are the potential customers we haven’t captured yet? Why would a customer pick a competitor’s product vs. mine? Are we leaving opportunities on the table?

What can I learn from similar products in the market? What are our USPs? How do I create a competitive advantage against competition?

Are there rules to the market? Are there regulations or industry standards I need to know about? Does my product have to look or feel a certain way to be taken seriously?

Knowing the market and regularly bechmarking yourself against other players helps become aware of your strengths and weaknesses and give ideas for where to invest strategically in your own product.

1.4 Ask Why

This is a bit of a product thinking cliche but still holds true: always pays to ask “why” a few times to uncover root causes of problems and underlying motivations.

Asking why can be helpful in almost any situation to build understanding in your team. Some examples:

• Why should we invest into building this feature?
• Why are customers asking for this feature?
• Why is this a pain for our users?
• Why do users give us that feedback?
• Why now?

1.5 What do we already know?

It’s smart to build on what’s already known rather than always starting from scratch.

Always leverage the existing knowledge and experience of peers and leaders: founders, product leadership, designers, other product engineers, etc.

Examine the status quo to see how a problem is currently solved. Look at ideas, feedback, metrics, KPIs already collected in the past.

Do we have users already doing something like this? What are their existing workflows? How could we improve their experience?

Am I duplicating or potentially deprecating some functionality that already exists? Could this be achieved by extending or leveraging existing features? How are competitors solving this?

2 Craft

2.1 Am I proud of what I'm building?

Your track record as a product engineer is the products and features you've delivered. Your last feature represents your professional competence level. No excuses.

Ask yourself what quality standard do I want to set for the work I put out there?

Can I be proud of the product I worked on? Is my work well tested and polished? Did I cut corners where I shouldn't have?

As a highly paid professional engineer your craft is to produce high quality software which includes avoiding the creation of technical debt. Never ask for permission to improve quality!

What makes a great product engineer stand out from the average software engineer is an intense sense of professional pride in their work and product.

2.2 Does the product feel good?

This may be a slightly controversial take, but I believe a surprisingly large part of building great products is about developing a good taste for it.

Simply knowing the difference between great vs. average and not settling for just ”ok” helps tremendously in making good decisions as a product engineer.

Does the product feel smooth and consistent? Is it intuitive, simple, and familiar to the user? Or does it feel cheap and janky?

Note that this doesn’t only concern the visual aspects of your product. Just slapping a fancy UI design on a shaky foundation doesn’t create a great experience.

Simple. Elegant. Clean. This is what we're after as product engineers.

2.3 How do I get there faster?

The pace of innovation especially in software is so rapid that in order to be competitive you must deliver fast, early and often.

Too slow and your customers will lose trust in you while your competitors overtake you.

What many get wrong about agile and building products is optimizing for predictability with estimates and roadmaps. Rather, what you really should care about is visible and continuous progress towards product goals.

The goal of estimates should not be to try to be as accurate as possible but rather to set ambitious and yet achievable goals for yourself.

The question is not how long you think it will take to build, but how long should it take? What’s an acceptable amount of time and effort I should invest on this?

What’s the rollout strategy? How do I get this into customers' hands as soon as possible? What’s the MVP?

2.4 Teamwork

Building products is a team sport.

You are absolutely not expected to work alone and do everything yourself from writing code to doing user research.

Am I effectively communicating with my team? Am I leveraging my team members' strengths? Are we celebrating our successes?

Great teamwork results in great products. Invest in your team, and you will see the dividends in your product's success.

3 Growth

3.1 How do I measure the success of my work?

Our work as product engineers is not just about building and shipping feature after feature.

We make educated guesses about the most valuable thing to work on, so we should also be able to answer the question: What’s the impact of my work?

How many customers did we talk to to validate our progress? Are they coming back? How much money does it generate?

Use analytics tools and user feedback to help you understand what’s working and what’s not. Talk to real users to get qualitative insights about the product.

Most importantly, make sure to share your outcomes openly and transparently. What did I ship in the last few months? What were the results?

3.2 How do I maximise the impact of my work?

The most important question you should regularly ask yourself is: am I focused on the right thing?

Being a good engineer is finding the biggest bottlenecks that hold us back and figuring out how to solve them. You should prioritise your efforts ruthlessly.

Building products is a team sport. You should actively communicate what you’re working on and why, so that others can help you and keep you accountable. Demo progress frequently and actively seek feedback.

Don’t fear taking risks. Being bold and taking the lead on delivering new and innovative features you believe in can lead to big wins.

Ask yourself: How can I align my work with our company goals? How does my work directly benefit our users? Is there a way I can communicate my work better to others to get better feedback?

3.3 How do I stay ahead of the curve?

Stay updated on market trends. Attend conferences, read up, follow experts.

Make sure to research and benchmark competitor products, or other similar products whenever possible.

Hold frequent retrospectives and brainstorming sessions. Experiment with new ideas. Encourage creativity in your team.

What are the latest trends in my industry? How can I encourage innovation within my team? Are we taking enough time to learn from our successes & failures?

4 Product Vision

4.1 What’s our North Star?

To align your work in the context of the broader product vision, you should deeply care about what others in the company are doing and saying, making sure you fully understand and are committed to the overall product strategy. Asking “why” is crucial here.

What are our current product goals? What’s our growth strategy? Where do we want to be in the next 2-5 years?

When presenting your work it always helps to put it in the context of how it pushes us forward in the big picture. How does my work help us reach our North Star?

4.2 How does my work impact the overall design of the product?

Understanding the existing software’s design and architecture decisions helps make sure that your work fits well within the larger product design. Always consider how your work influences and is influenced by other components.

When adding new functionality, you should ask:

• Could this be achieved by extending or leveraging existing core features?
• Does my design follow a consistent style to the rest of the product?
• Am I adding complexity or reducing it?

Simplicity is worth fighting for.

4.3 What's the ambition level?

It’s a good idea to define the ambition level when setting off to build something new. Are you aiming for an incremental improvement or a revolutionary new functionality?

Is this feature a USP or a basic expectation? What’s the ROI on building this feature? Does it make sense to spend the effort building this ourselves, or could we use an off the shelf library or service?

Your time is extremely valuable. Think like an owner: Would you invest your salary to build the feature, or rather on something else?

Product Engineer Mindset

If you enjoyed this checklist, you might also like the Product Engineer Manifesto on GitHub.

Consider giving the repository a star if the Product Engineering philosophy resonates with you!

Not Just Coders, but Builders

Picture this: engineers who don't just speak in code but in product design and customer problems.

Product Engineers are a special breed of engineers that don't just see themselves as coders or developers but as builders who deeply care about the products they build.

They're driven by a professional pride and desire to build great products, transcending the traditional developer role to become genuine drivers who go on to put their names behind great products they themselves played a key role in shaping.

The Counter to Hyperspecialisation

The tech world has seen for a long time a trend towards hyperspecialisation, with engineering roles becoming increasingly narrow.

We have frontend engineers, backend engineers, iOS engineers, DevOps engineers. We have developers defining their careers with a specific language: JavaScript engineers, Swift engineers, C# engineers, Python engineers. We even have developers that seemingly dedicate their entire careers to a single framework or tool: React Engineers, .NET engineers, Unity Engineers, Ruby on Rails engineers.

Feels like it's only a matter of time until we see job postings seeking for-loop engineers and variable naming engineers.

Jokes aside, specialisation isn't totally without its merits as it has allowed engineers to build deep expertise in specific technologies, a key component in building quality software. However, it also lead to silos where collaboration and broader product understanding became a nice-to-have, and often not even an expectation or focus for engineering roles.

Product Engineers stand as a counter-movement to this trend. They embody a holistic approach to engineering, where understanding the entire product and context around it is just as important as the technical skills required to build it. This broad perspective enables them to bridge gaps between different technical domains and ensure that the product serves its customers effectively.

AI: The New Playground

The rise of artificial intelligence (AI) tools in software development is rapidly setting new expectations for engineers. With AI tools becoming more sophisticated and capable, engineers are now expected to leverage these tools to enhance their work, not just in terms of speed and efficiency but also in terms of broadening their area of responsibilities.

Engineers in a post- ChatGPT and GitHub Copilot world are now expected to be able to work in more languages, incorporate more tools and libraries, and simply be able to deliver more, using a broader range of technologies and disciplines in their daily work.

It's probably not a great time to be the platform engineer whose job is to set up Jenkins pipelines for the product teams, when ChatGPT can easily help teams do it themselves.

For Product Engineers however this has unlocked a higher level of abstraction. With AI tools handling more of the routine and specialised tasks, engineers can allocate more time to creative problem-solving, ideation, and exploring new ways to meet customer needs.

This new level of abstraction allows Product Engineers to concentrate on product strategy, user experience, and overall system architecture without being bogged down by the intricacies of individual technologies.

The incorporation of AI into software development encourages a broader perspective, where the choice of technology becomes a means to an end, rather than an end in itself.

While AI didn't create the Product Engineer role (examples have been around for much longer); AI has made becoming a true Product Engineer much more accessible as a career path for software engineers.

Combining Product Thinking and Technical Execution

Product thinking involves understanding the user's needs, the market demands, and the business goals that drive a product's development. It's about seeing beyond the immediate task to grasp how each piece fits into the broader puzzle of the user experience.

Product Engineers ask the crucial questions: "How does this feature add value to the user? What impact will this improvement have on the product's overall vision? How can we measure the success of our work?"

They don't just build features based on specifications; they contribute to the design and roadmap of the product through a robust understanding of the customer's needs and business strategy.

This approach requires a balance of skills: the ability to dive deep into coding and system architecture, while also keeping an eye on the product roadmap and customer feedback.

By combining this mindset with quality technical execution, Product Engineers can uniquely ensure the product is headed to the right direction both technically and strategically, ensuring that their work directly contributes to the product's long term success.

The Manifesto: A call to Product Engineers

I've published a Product Engineer Manifesto on GitHub to more formally define the Product Engineer mindset.

Consider giving the repository a star if Product Engineering philosophy resonates with you!

Skip ahead or read on until the conclusion for a full verdict of whether rewriting our app was a good idea to begin with.

Doubling down on AWS & serverless

Good bet 👍

To be fair, AWS was already the de-facto cloud platform used by the epilot team before I joined.

My contribution was to convince the team to go all in on a serverless architecture with an ambitious strategy to rewrite and replace our proprietary Java-based 3rd party monolith running on EC2, piece by piece with functionalities leveraging serverless AWS offering.

After 3 years, there are no regrets to choosing to be fully vendor-locked with AWS for the speed, scalability and minimal operations overhead it provides.

Microfrontends with single-spa

Good bet 👍

The decision early on to take a microservices approach in our frontend was made out of purely practical reasons.

Frankly our small team who lead the work on the new platform didn't have the time, the confidence, nor the desire to make long term decisions on the frontend tech stack, tooling and important design decisions on behalf of all our other teams, present and future.

I'm extremely happy we went down this path instead of imposing one framework and toolset for everyone. The single-spa approach lets us improve our frontend game each time a new microfrontend gets added to our application, which by now has happened more than 30 times! We've even ended up replacing entire MFEs (microfrontends) a few times, which isn't as bad as it sounds when the codebases are kept relatively small.

Notably the microfrontend approach also helped us support the transition from the legacy product to the new one, as we were able to treat the legacy app as just one of our many (micro)frontends, and slowly replace functionalities one by one by shipping new microfrontends.

Using React without a framework

Bad bet 💸

I know, "skill issue", right? But honestly I've found that even the really senior (read: expensive) React engineer hires tend to struggle with performance and end up wasting a lot of time on fixing major performance issues with React applications when they start to do anything meaningful.

Starting over, I would probably spend a bit more time choosing a React framework that works well in a microfrontend environment rather than encouraging all teams to start from scratch, and establishing no-framework as the implicit default.

Controversial: I might not even pick React for most MFEs anymore. We already use Svelte in parts of our app, and turns out Sveltekit works super well in a single-page-app environment like ours!

RFCs

Good bet 👍

The practice of writing RFCs to collaborate on software design and async decision making was one of the first and best things I introduced joining epilot.

To this day we use RFCs as the main way to share ideas and be transparent about the work that our engineers do. I'd consider this one a very good bet and would absolutely recommend a culture of writing RFCs for all product organizations.

API First

Good bet 👍

We give a ton of freedom to engineers to pick whatever languages, tools and techniques they see fit, but the one thing I've always demanded from our teams has been to design using machine-readable API contracts, such as OpenAPI or GraphQL.

I would say this is internally the most important way we collaborate on our software's design, using shared types and automated tests to ensure our implementations follow the common API contracts. This approach has radically reduced bugs and allowed our engineers to work together efficiently leveraging platform features built and maintained by other teams.

On top of that, being API First and publicly building and sharing our API contracts has allowed our customers and partners to effortlessly build on our product, and integrate it deeply with their own IT landscape using our API & SDK.

Continuous production deploys from main

Bad bet 💸

Turns out our customers simply did not tolerate the frequent changes and inevitable rollbacks of this approach.

We technically still ship continuously from main to our production environment, with an average of 150 production deployments every week, (with a lot of tests!), but now ship a monthly stable snapshot release to paying customers, and heavily utilise feature flags to roll out changes.

Not using feature flags from the beginning

Bad bet 💸

We did use GitLab feature flags on launch, but they were only used to turn features on/off between stages (dev, staging, production).

Turns out this wasn't enough. Our teams really needed better tools to give us fine-grained control over rolling out features to specific customers.

Going back, I wish we had started earlier with something like LaunchDarkly.

Hiring product-focused senior engineers remotely

Good bet 👍

This is a decision I'm very proud of. Keeping an almost unreasonably high bar to getting hired and seeking out individuals who care about the product, our customers and our business has allowed us to build and retain a motivated team with zero managers and architects.

Making engineer hiring remote-first allowed us to tap into an international pool of senior talent resulting in a team of more than 30 exceptional and motivated hands-on Product Engineers from more than 10 countries working for us, and growing... ❤️

In-house DevOps / platform team

Bad bet 💸

We tried to establish a platform team on two separate occasions but both times the team struggled to produce value for our teams. The DevOps engineer hired to lead the second platform team ended up leaving us for a bigger company after a few months.

Looking back, part of the reason for this was likely a combination of leveraging managed/serverless services that don't require much in-house automation work, and hiring senior engineers who can automate things for themselves when needed.

We've recently started experimenting working with a 3rd party DevOps consultancy to join us on a project-basis to help us with non-product related engineering tasks such as optimising our CI/CD pipelines.

Regardless, we don't anymore see any need to hire and establish an in-house DevOps team, or get into Platform Engineering of any sort in the foreseeable future.

Weekly Demos

Good bet 👍

The all-hands weekly demo is still easily my favourite recurring event at epilot. It's like having a company-wide celebration of shipping things, every week!

The weekly demo brings not only a positive buzz around all the cool stuff being built, but also the expectation that engineers personally showcase concrete customer value created in their teams every week. It keeps us accountable for consistent delivery and maintains a rapid feedback loop with the rest of the company.

Open-Source Engineering Principles

Good bet 👍

Transparently laying out our Engineering Principles in a public GitHub repo helped set clear expectations for our culture internally, and perhaps even more importantly for engineer candidates, many of whom have testified making the decision to apply at epilot because of our principles resonating with their own beliefs.

Hasura

Good bet 👍

One of the early key bets was the decision to use Hasura to interface with our legacy database to break up the monolith into microservices.

Event triggers listening on DB changes and pushing messages to SQS while using the GraphQL API to implement 2-way data sync with the legacy app turned out to be a great and reliable event-driven pattern for breaking up the monolith step by step.

Since shutting down the legacy product last year, Hasura was decommissioned together with the legacy Postgres database, but it truly served its purpose well during the migration to serverless. 🫡

DynamoDB as our go-to database

Good bet 👍

We haven’t had any fundamental issues with DynamoDB single table design in our backend microservices.

Of course it’s not suitable for every use case, and we have turned to Aurora serverless, Elasticsearch, and even InfluxDB in a few special cases, such as for search and analytical needs. But for 80% of the time, DynamoDB is fast, convenient and extremely worry-free as a default application database.

AWS OpenSearch service

Bad bet 💸

We started out on Amazon OpenSearch Service (still called Elasticsearch service back then) but were struck with a pretty bad incident where our domain became completely non-responsive in our development environment and the only thing we could do was to contact AWS support to resolve the issue. This was totally unacceptable for us as we heavily rely on Elasticsearch for our main functionalities.

Elastic Cloud ended up being far more reliable and manageable for us in the end. Having a self-service reboot button on the interface makes all the difference!

However with both providers rolling out their serverless solutions, which would be extremely attractive to us, we're keeping our eyes open and are open to experimenting again with AWS OpenSearch offering.

Datadog for observability & monitoring

Good bet 👍

Datadog is one of those tools that once you get used to it, you wonder how you ever lived without it.

The value we get from Datadog's UI for logs, traces, monitors and dashboards is insane, and absolutely worth every penny.

We use Datadog RUM for browser & UX monitoring. The RUM sessions and logs are extremely helpful for tracking down frontend crashes and bugs, as well as visually understanding user behaviours.

On top of that, we use pipeline observability to analyse CI pipelines, custom metrics to track business & application KPIs, and cloud security management for compliance reporting and threat detection.

Datadog for incident management & post-mortems

Good bet 👍

Datadog Incident Management has been absolutely vital in making sure we take the right steps to analyse and learn from our incidents. We use it during incidents for coordination and communication, as well as after the fact for post-mortem and analysis of past incidents and trends.

Datadog for browser tests

Bad bet 💸

Datadog synthetic browser tests were great to get started fast but quickly became slow, expensive, hard to expand and simply not good enough for our team.

We ended up switching over to Playwright.

Redshift Serverless

Bad bet 💸

Quite slow. Very expensive. At least for our use case of generating dashboards with Spectrum querying mostly parquet files from S3.

Our team is currently investigating replacing Redshift with Clickhouse.

Microservices

It's complicated 😎

At this point I've stopped referring to our product as having a microservices architecture. I don't find it a very helpful abstraction to understand a large, event-driven, interconnected product that's mostly FaaS.

Yes, we still develop and deploy our services as independent domain-specific modules with well-defined APIs.

In reality however our application is used by the customer as one big product with all the pieces needing to connect and work together to provide a service.

Thus, we must also test and ship the product as one. (Really, we have one large shared e2e test project that blocks all pipelines. When we're shipping our monthly stable release, we take a snapshot of all our bundles and ship those in one atomic import map file.)

I simply find it more useful to think of our product as one monolith, made up of a large number of independent infrastructure & FaaS modules with APIs.

Conclusion: Was the rewrite a good idea?

If you ask me now I would never recommend any SaaS company take the decision to fully rewrite their software. It's generally just a pretty dumb and painful thing to do. 😅

For epilot sadly we were left with no other choice.

The legacy version of our SaaS was built on top of a proprietary 3rd-party low-code platform Axon Ivy, which by no means was designed to run a multi-tenant SaaS, but that the external epilot dev team from Vietnam (affiliated with Axon) had managed to wrangle to work.

The fact that epilot was able to build a successful and fast-growing business on top of what's essentially a hacked low-code business process automation tool, is the ultimate proof to me that the tech stack really doesn't determine a company's success!

But we were seriously starting to hit the limits of our software, especially when it came to scalability and speed of development. We didn't even have access to the source code so that we could take over the development of the core application. All we could do was upgrade to new versions provided by the vendor and continue to hack the software to make it do more things it wasn't designed for.

Simply put, it wasn't sustainable for a fast growing software company. No developer wants to work on software like this. Especially not the talented and ambitious ones we wanted to work with.

It took us nearly 3 years, but in the end we managed to reach feature parity, replace, migrate and shut down the legacy app, with very little churn, growing year-over-year and last year doubling our MRR while migrating the last customers over to the new platform.

Feels good to say we're only getting started with our new 360 platform.

Looking back at the last 3 years, while I would never recommend anyone pursue a rewrite for a running SaaS business, while simultaneously trying to build a healthy, ambitious engineering team and culture, I'm convinced that for epilot this was the (only) way to go, and we're much better for it.

In the end we at epilot overwhelmingly consider this a…

Good bet 👍

What's next?

To keep things short in this post, I'll publish a separate Part 2 where I share our latest bets focused on building epilot into the next SaaS Unicorn. 🦄

Mandatory recruitment disclaimer: If this post made you at all curious, please check out our docs and check our open positions for more!

In this post, I'll attempt to give a glimpse of concrete tools and practices we use in our product team to give an idea of how we apply our principles and what the engineering culture really looks like from the inside.

Team Structure

At time of writing, the epilot engineering team consists of 25 full-time remote software engineers organized into 6 squads, each with its own Product Manager and a shared design team of 4 talented product & UX designers.

Each squad is led by an engineer team lead, principally responsible for the team's delivery and engineer wellbeing.

The product leadership team is responsible for communicating our product strategy and maintains a Now-Next-Later roadmap, but ultimately it's the squads that make all delivery decisions.

Squads are formed around specific business domains within the epilot 360 platform, architected around Domain Driven Design and micro-ish services both in the backend and frontend.

Show, Don't Tell

We value real concrete working software, not roadmaps and fancy presentations.

Engineers showcase features in a weekly all-hands Product Demo session: a company-wide celebration and ruthless feedback session. 🎉

Features being demoed are deployed to production but are usually hidden behind a Feature Toggle, giving demo participants a chance to test and give feedback on features before release.

Most squads use a Kanban-style flow with a weekly planning cycle, but opt to showcase their progress every 2 weeks, with some squads hosting extra demos called "Open Houses" on off-weeks.

The engineering team also hosts an internal bi-weekly "Tech Exchange" to wind down with a beer, but also present and demo cool technical things.

API-first & serverless

The epilot backend is built as serverless microservices written mostly in Typescript, leveraging services such as AWS Lambda, Step Functions, EventBridge, API Gateway and AppSync.

DynamoDB is the most popular database solution enhanced by managed data stores such as Elasticsearch Cloud, Aurora Serverless and Redshift Serverless.

Our backend is pretty event-driven, with heavy use of EventBridge, SQS and Step Functions to implement asynchronous business logic.

Our frontend application consists of microfrontends written in React and Svelte orchestrated by the single-spa framework.

The epilot frontend is entirely built on the same APIs we expose to our customers and technical partners via our open-source SDK and public documentation.

We like open standards, so our APIs are defined using common machine-readable specs like GraphQL and OpenAPI.

We find the domain-driven architecture approach an efficient way to organize our teams to act independently while collaborating on software design with an API-first approach.

Organizing our codebase with microservices helps our fully remote engineering team focus on delivery and minimize bikeshedding over non-critical engineering decisions.

Freedom and Responsibility

Ok, I freely admit we stole this one from the Netflix culture deck. Shoulders of giants and so on. ♥

At epilot, every hire is trusted and treated as an adult from day one. We only hire smart people who we can trust and aren't afraid to terminate a relationship if that trust is broken.

We expect engineers to operate by principles and think strategically to benefit our customers and our business. (Max-the-MRR)

We intentionally make it very difficult to create company-wide policies to control which tools to use, what employees should be allowed to do and how they should organize their daily work.

It should be hard to introduce new processes but easy to get rid of existing bad ones.

The exception to this rule is security.

We encourage team members to challenge the status quo with RFCs proposing ways to improve the product and our ways of working.

RFCs - Solutions Over Problems

Among other things, RFCs written by engineers have so far led us to adopt continuous deployment, harden security, establish new teams, and even rewrite the entire product in a different tech stack.

These are often simple documents that start out by just describing a problem and expressing a wish or an idea to solve it.

RFCs at epilot have no official approval process. We expect the author to champion their proposal by garnering relevant feedback from peers and stakeholders, and then make an informed decision on how to proceed.

In most cases, the RFC author leads by example and implements the proposal themselves, usually just within her squad at first.

Good ideas spread and are adopted through all teams, not by mandate from above, but by actually improving the quality of our work.

Edit: A total of 192 RFCs have been published in our Confluence since they were introduced 2 years ago. 🔥

Every Week is Quality Week

The name of this principle came about after a group of engineers expressed concern about technical debt in their project due to pressure to deliver against deadlines.

At the time, it was already an established practice to fix bugs immediately and treat all types of bugs as critical, but many engineers still felt that they couldn't go against the wishes of product managers to follow the practice.

It was proposed to have a "quality week" at the end of every quarter to give engineers time to focus on quality topics.

As the quarter came to a close, we announced the quality week with instructions to all squads to focus solely on bugs, technical debt and other quality improvements they felt were important.

During the quality week, our engineers were extremely motivated to fix long-time issues that had affected our customers.

We made real, significant progress that week and everyone could see it.

On Monday following the first quality week, we opened this PR, making an announcement that every week from now on would be quality week:

It worked. We're still starting a new quality week every Monday.

No-Ops & Continuous Deployment

At epilot you will find a distinct lack of epilot-specific shared tooling. We actively avoid building our own tools and standardizing configurations and instead encourage using open-source tools like AWS CDK, lambda-powertools and single-spa to build our product.

We don't have dedicated DevOps engineers or a platform team to build custom tools and pipelines or run k8s clusters.

We expect squads to independently automate testing, deployment and monitoring of their features to minimize the need for manual operational tasks.

Engineers really get to pick any tools they like to do their job, as long as they weigh tech decisions against our principles.

I sometimes get engineers pushing me to leverage my position as head of department to standardize tech choices across the entire engineering team, but so far the approach of giving teams radical freedom has yielded organic adoption of best practices and tools where needed anyways.

Without extra push, all teams have converged to using the same GitLab CI templates, connecting up with Datadog to continuously deploy code with synthetic browser tests integrated into the pipeline to make sure we catch problems before they get to production.

The whole team also uses Datadog (another RFC) as the central observability solution to know what's happening in our platform at all times.

When stuff goes wrong, engineers owning the feature jump on a Slack huddle using Datadog's incident management feature to coordinate the response, cleanup and write a post-mortem.

Why principles matter

When I joined epilot, I made it my personal mission to build an engineering culture that I would personally want to join and be part of.

To me, the engineering principles are like a founding document; a source code or DNA of how to build a great engineering team.

I'm proud of how far we've come, but there's still lots to learn and improve as we grow and mature as a product team.

I fully expect that in 2 years, writing this blog post would look very different as the industry moves forward with new tools and practices. I hope that these principles will stick.

If the topics in this post resonated with you, make sure to check out our dev docs and see our open positions!

The text is written from the perspective of engineering and focuses on the challenges I faced taking on leadership of our growing team of 25+ engineers.

I fully acknowledge this is not the complete picture of epilot's growth during this time as we have many other amazing departments and individuals working to make epilot the next tech unicorn success story.

Full disclosure, I love this company and am heavily invested in the product's success after my first year and a half. 💙

(I'm the one dressed up as the doctor for our company's annual Cologne carnival celebration)

Beginnings

I joined the Cologne-based startup epilot in October 2020 as a cloud engineer to head a new platform team and aid the product team transition to serverless AWS.

The task felt right in my wheelhouse.

By that time I already had three real serverless products under my belt, had experience migrating monoliths to microservices, and had acquired more than 8 years of professional experience in total as a developer and team lead working mostly with AWS and other cloud platforms.

I knew startups. I had experience with the business domain. I knew the technology well.

This was my time to shine.

Build better software, faster.

As our CTO Szilard Toth put it in our first discussions, my job description was to help us "build better software, faster".

I took that mission statement very seriously.

Coming off a 4-year stint at Futurice, a lean service design focused consultancy, my first instinct was to make a current state analysis of my new company - epilot.

To get my bearings, I joined one of the engineering teams and got busy.

I learned about the challenges our teams were facing and got a deeper view into our legacy product, which had already been in active development for more than 3 years.

We were already doing a lot of things right: All three teams at the time were generally following agile practices, engineers were eager to discuss and learn their new tools, many had even taken the time to get AWS certified.

But there were certainly some big concerns.

At the end of my first full month, I was ready to present my learnings to the company.

First insights

I published an internal document titled "epilot Tech Review" detailing my assessment of the current state of epilot's engineering.

These were the main findings:

1. The decision made (before my time) to pick AWS, serverless and React as our foundation was a good one for what the team wanted to achieve: Low operating costs and high development speed. This was especially great for hiring due to many candidates being attracted to these modern technologies.
2. Our teams lacked experience working with these technologies. Progress was slow and unfortunate decisions were made with the very first small scale serverless projects.
3. Alarmingly, the engineering team had no viable plan or strategy to migrate the legacy product to the new serverless architecture. Everyone was excited about the new tech but we clearly had no clue how to actually build our product with it.

I emphasised in the document how important it was for epilot's success to move away from the legacy technology stack, a proprietary monolithic Java framework which had already proven to be a poor fit for our use case, difficult to run and extremely slow to develop new features on.

Our ability to deliver as an engineering team was looking pretty grim. The company had already invested a full year into going serverless with not much to show for it.

To make matters worse, some of our key engineers had just announced they were leaving the company.

In spite of our woes on the engineering side, epilot had a rare advantage:

A proven product-market fit – Even with our product leaving much to be desired, our paying customers were becoming fans!

Yes, they were sometimes frustrated with the quality of the software and our inability to deliver features they were asking for, but they were totally bought into the vision of what epilot could be and we were clearly delivering value for them.

I think this was thanks to our visionary leadership with deep industry knowledge knowing which problems to solve along with a great sales team and a MacGyver-like customer success team working around the limitations of our software.

We had the right idea, just not a great technical execution yet.

Luckily I knew I could make a difference when it comes to execution.

RFCs and API first

I went on to establish the platform team I was hired for. We started small with myself and an SRE who had been at epilot a year before me.

The first things the team tackled were improvements to developer workflows such as enabling an API Design First approach for our serverless APIs and building common CI/CD pipelines for the tools our teams were using.

I introduced RFCs for the teams as a tool to discuss designs and proposals to improve our product and work.

With key engineers leaving us, it was important to establish an open culture with teams making independent decisions while focusing on good software design and communication.

As one of the very first RFCs shared, I laid out my admittedly optimistic plan to move away from our legacy monolith app.

The radical 360 plan

I brought my plan directly to senior company leadership triggering a series of discussions in late 2020 to figure out our legacy migration strategy.

In sessions with our CEO Michel, CTO Szilard and head of sales Marlon, it became very clear this would not be just a migration from monolith to serverless. We needed to build the new version of epilot SaaS with a radically different design, starting from a new flexible data layer to power the diverse set of use cases our customers were asking for.

We called it epilot 360. And it would be my job to build it.

From the start of 2021, we established a new strike team to head the effort of building 360. I doubled as the leader of both the platform team and the new 360 team.

The 360 plan was radical. We would completely rebuild our main portal interface using frontend microservices and embed the old product into the new portal as a microfrontend to provide backwards compatibility while we rebuild and migrate features one by one in the new serverless world.

Getting the old monolith to work in the new portal was our first big technical hurdle. We developed a "headless" mode for the old app, where the UI was modified to fit into the new portal layout and implemented secure communication with the parent 360 app to pass information like the current user's session to the legacy app.

From the data side, we continued to split up the legacy SQL database into serverless microservices. We introduced Hasura to act as the gateway to synchronise data between the legacy database and our new serverless services.

By March 2021, we had migrated our first module to 360 using the new flexible data model and had shipped a new portal microfrontend for it.

We introduced Datadog for centralised logging and monitoring, as well as to provide crucial analytics and observability tools for our new serverless product.

Finally in July of 2021, we greeted all our users with a whole new 360 login and portal experience, wrapping the old familiar app in our brand new look with serverless features already replacing many legacy views.

This felt like the first big launch of 360. Although lots of work was still ahead to fully get rid of our legacy app, our users were happy to see us start to deliver on our promises with a new major release of the epilot product.

From here on the rate of new releases in our app would radically increase.

Head of Engineering

In May of 2021, I was officially promoted to head of engineering which meant I was no longer just in charge of the platform team and the 360 team. I was now in charge of the entire engineering team with our CTO shifting his focus away from engineering to product management.

In the meantime we had also ramped up our recruitment efforts. We published our in-house engineering principles and built a new recruitment landing page promise.epilot.cloud to showcase our culture. As a result, we managed to acquire some amazing new engineering talent during 2021. <3

So far it had been mostly just my small team of 4 developers working on 360. Now as the new head of department, I needed to shift the focus of all our by now 25+ engineers and 6 teams to work on the new 360 platform.

This came with some major challenges.

Scaling the team

One of the biggest challenges was the radically different underlying design of 360 with its flexible data model as opposed to our legacy product's traditional static data model.

Teams initially struggled to wrap their heads around the new flexible entities; not least because of my struggle to communicate it properly. Through some much needed visual design support and hands-on experience with the product, we slowly built up an understanding of our new 360 product together.

A major focus of the architectural design of 360 was on modularity by utilising frontend and backend microservices combined with our API First software design approach.

This was to allow teams to work independently giving them the freedom to choose their own tools and ways to solve problems.

We found this approach to be a big success as it kept our projects small with teams motivated to try out new technologies, share and learn from other teams, and quickly change approaches when needed.

After the first few months of teams working on 360 it was clear that we were able to move much faster than ever before with our new tech stack.

We adopted continuous deployment across all teams and quickly surpassed an average of >100 production releases each week. This was a huge achievement considering that back when I joined we were able to deploy only 1-2 times per week during a pre-defined maintenance window!

By the end of 2021, all our teams had shipped major features on the 360 product and we had reached enough feature parity for the first new users to be onboarded to epilot completely on 360 with no trace of the legacy product.

We had an MLP - Minimum Lovable Product!

Things would speed up even more throughout the first half of 2022 with teams getting comfortable with 360, shipping more serverless features to get us closer to the magical feature parity state with our rather large legacy product.

We opened up the 360 platform to external developers and partners with our public developer docs and open source SDK.

In the meantime our sales and customer success teams have been actively engaged with existing customers to help them migrate to 360. More and more users are now adopting the new features each week and we’re learning from them to make 360 even better.

Takeaways from the first year

As I'm writing this post in June 2022, after my first full year as head of engineering and after almost a year of running the main epilot application in production on AWS serverless and frontend microservices, I can't help but ask myself:

Was 360 a success?

While my ego would love to give a resounding "Yes!" as the answer, I can't be fully satisfied until we've fully completed the migration of all features from the legacy app and have successfully shut down the old monolith and moved on with happy users.

We're not quite there yet.

Are we building better software, faster? – Yes! The feedback from our customers is very positive and our product teams are delivering features 10x faster than we were back in 2020 when I joined. We're demoing new features every week now!

Are we on track getting rid of our legacy monolith? – Yes! At the time of writing, we're close to 90% feature parity with more users hopping over to 360 each week.

Was it way more difficult than initially thought? – Heck yes! The people side of things turned out to be much more complex than the technical side – as always.

It isn’t easy to align 5 remote product teams to work on a new product redesigned and rebuilt from the ground up.

It isn't easy to introduce big changes to our tech stack, product, and engineering culture.

It isn't easy to keep our users happy while introducing a new product with major design changes and new concepts to learn.

While none of these things are easy, they are surely worthwhile and we strive to get better every day.

Conclusion

Looking back, I've learned a lot in the past year about leadership, engineering and building products.

With our engineer teams now working well, I've been able to focus my daily work on helping our customers and technical partners hands-on with their adoption of 360. This has really helped me gain better perspective on where the 360 product stands today from our users' perspective.

I've received a ton of feedback; from customers, from colleagues, and from our leadership team.

I’m glad to say things are looking bright for epilot 360!

Still lots to learn and build.

TL;DR:

• APIs are the single most useful design tool for teams collaborating on software.
• Teams who focus on API design produce better designed software and collaborate more efficiently by parallelising their work around concrete technical contracts.
• Thinking of APIs as the core way to add features into software leads into building up a platform for developers to bootstrap new features on top of a library of existing core APIs.

Software Design & Features

As developers we intuitively think of features as the basic building blocks for the software we design and build.

Features are the capabilities our software provides. They answer the question: What does the software do?

We design software around features. Some features are user-facing while many features are purely technical and serve internal purposes to make the system as a whole work.

A software team will add new features, extend existing features and sometimes deprecate and remove old features to be succeeded by new improved ones.

Tests are used to describe features as code and make sure they follow the intended design.

This happens regardless of whether a team follows agile methods and talks about epics and user stories instead of features. These are just tools to help decide which features the software should have and how to prioritise building them.

The Problem

In a small team of up to 5-8 people working on the same project, it's still relatively easy to keep track of all features and come together to make decisions on software design.

This is no longer the case when you scale to multiple teams working on the same software.

Questions like these become increasingly harder to answer:

• Which team should be responsible for this feature?
• Can I change this feature without breaking other parts of the software?
• How was this feature intended to be used?
• Could this feature I'm building also be useful to other teams?

Technical Debt

As a result, software teams easily get into situations where the quality of the software starts to suffer; ultimately leading to unmotivated and poorly performing teams having to constantly deal with symptoms of technical debt:

• Inconsistent APIs, bad design
• Features overlapping in functionality
• Tightly coupled features that are hard to extend without breaking things
• Having to look into implementation to understand what each piece does
• "That one thing only Sam knows about"

How do we fix the problem of multiple teams and individuals collaborating on the same software?

Software Architecture

This is the point we start taking the idea of software architecture seriously. Maybe even hire some people with Architect in their job title to solve the problem.

Software architecture on the surface is about designing and organising the pieces that make up software to make it easier to understand and keep developing.

For our purposes, I would argue it's also about helping teams collaborate and organise their codebases by providing a common language and a set of abstractions to communicate about the design of the system.

Here are a few popular architectural patterns a software architect may suggest to help design your software:

• DDD (Domain Driven Design)
• Onion Architecture
• SOA (Service Oriented Architecture)
• Microservices
• Event-Driven Architecture

One thing all these patterns have in common is they attempt to abstract the software into a set of smaller, encapsulated units of software.

Most of the units expose an API. An interface to allow other parts of the software to interact with it.

API as a Design Tool

So it turns out thinking about APIs is pretty important for software architecture.

I would go as far as to say the majority of all software in existence is there to provide APIs for other pieces of software. Think software libraries, OOP interfaces, operating systems: software meant to be used by other software with APIs.

In fact APIs are so important, I'd argue API design is the single most useful design tool for teams collaborating on software design.

Not UML diagrams.

Not entity relationship diagrams.

Not abstract architecture diagrams.

API Design.

Why API Design?

If software architecture is about building elaborate systems of features provided by small encapsulated units of software, and those features are mainly manifested as APIs, how can we possibly avoid talking about API design as the centerpiece of the whole design?

That would be like designing a public transportation system without looking at the connections between lines.

Or designing a factory line without looking at how the materials move between the machines.

I imagine those things would be very difficult to do.

API First Design

Next time your team starts working on a feature, after you've drafted a design for how the feature might look like to the user, design the APIs needed for that feature.

Try designing the APIs with code. Make sure not to start implementing it yet. Use types, interfaces or technical API definition standards.

Here are some popular standard tools for designing HTTP APIs:

• Postman
• GraphQL Type Language
• OpenAPI Specification
• AsyncAPI Specification

Share the API design. Include it as part of a proposal or RFC (request-for-comments). Provide examples for how the API would be used. Invite colleagues to give feedback and make suggestions (PRs) to improve the design.

Once the comments stop coming in, run with it. This API will now become part of the beautiful software you're building together.

Efficiency

In addition to producing higher quality software, teams who focus on API design first are able to work more efficiently by parallelising work around the API contract.

Because APIs work like contracts, they can have a future start date. A promise that this API will exist in the software at a later date.

Even before a team starts implementing an API, other teams can already start building new software depending on the technical definition of the API.

Mocks can be used to test the software even before a single line of code has been written to implement the API.

Changes to the API can be proposed with concrete PRs. This is a lot easier while the API only exists as a design with no implementation.

This is a really powerful way to parallelise work and avoid blocking dependencies between teams.

Example of an API First Workflow

Let's say we have two product teams working on an Ecommerce app, both with their own backlogs and user stories they wish to deliver.

Team Alpha plans to add a price notification feature that lets users know when pricing has changed on products they saved in a watchlist.

Team Omega plans to add a new improved search feature that provides product recommendations for users based on their purchase history and products they have saved in a watchlist.

Both teams need a watchlist feature to implement their end-to-end user stories.

A Story of Collaboration

Team Alpha starts working on their price notification story by producing some UI wireframes alongside API designs as OpenAPI definition files in a git repository.

The designs are included as part of a company internal RFC document describing the solution for price notifications.

Omega team views the RFC and notices that Alpha's proposal includes the watchlist feature they need but the Watchlist API is missing a way to fetch all saved products for a user, a feature needed to come up with the recommendations for Omega's search feature.

Omega then makes an adjustment in the OpenAPI definition adding the missing feature to the API design and submits a PR, which gets merged after review from Alpha.

As both teams are happy with Alpha's proposed design, Alpha gets to work on their price notification feature.

Meanwhile Omega submits their own RFC proposal with designs for the improved search feature using the Watchlist API definition as part of the solution. After some collaborative iteration on the RFC document, Omega is ready to continue with their own feature.

Both teams work on their respective features in parallel, mocking the Watchlist API to test out their features.

Later, Alpha informs Omega that the Watchlist API has been implemented as part of their sprint. Omega tests the newly implemented API and is happy to see their search feature now working as designed.

Using this workflow, both teams efficiently collaborated on the design of the software using RFC documents to describe their solutions, including technical API designs to enable working with version control and Pull Requests, both teams working in parallel to complete their user stories.

Platform of Features

As teams focus on APIs first, they end up building up a platform of features.

Focusing on well designed APIs to build end-to-end features, the APIs will implement platform features that can be leveraged by other teams working on other parts of the software.

Smart companies will start building up a Developer Portal, a central repository for the technical API definitions and start providing tools and well written documentation to consume platform APIs. Maybe even release language-specific SDKs to make consuming versioned APIs as easy as installing software packages.

At this stage you have a level 4 API First engineering organisation (see: 4 Levels of API adoption) with teams able to iterate fast by collaborating on the software design on a detailed level, constantly expanding the platform with new APIs and tools to build and consume them.

This is a good place to be. 💙

He was right.

I just recently had to migrate a React project from a fetch based API client implementation to an axios based one where the tests heavily relied on mocking global.fetch. It very quickly became apparent why this is not good practice.

I ended up having to write my own test utility that would mock both fetch and the new API client. It never looks good when you have to change tests to prove your code didn't change anything for the user.

As a better alternative, Kent suggests using a Mock Service Worker. More specifically the msw module to essentially run a mock backend as a service worker that intercepts all outgoing API requests to handle them.

Setting up msw

Setting up a mock backend with msw for your React tests turns out to be a fairly easy process. To get a deeper picture you should check out Kent's original post, but here's all you really need to do in your test code to mock a REST endpoint:

import { rest } from 'msw';
import { setupServer } from 'msw/node';

const server = setupServer(
  rest.get('/api/pets', (req, res, ctx) => {
    const pets = [{ id: 1, name: 'Garfield', type: 'cat' }];
    return res(ctx.json({ pets }));
  }),
);

beforeAll(() => server.listen());
afterAll(() => server.close());

One of the reasons this is extremely cool is because it avoids the pain of having to start up a real local mock backend, such as an express server that needs to be bound to a specific port on the host running the test.

This helps keep your tests fast and simple to run, as they should be.

Even better with OpenAPI

As someone who works a lot with API backends that (hopefully!) provide Swagger/OpenAPI definitions, I had already been mocking my backends in React tests using OpenAPI mocks with openapi-backend. When I learned about msw, I was thrilled!

It turns out msw together with openapi-backend is the perfect combination for mocking REST apis.

To provide a full mock for an API, all I need is to create a mock backend with openapi-backend using the API definition and tell msw to use it:

import { rest } from 'msw';
import { setupServer } from 'msw/node';
import OpenAPIBackend from 'openapi-backend';
import definition from './path/to/definition.json';

// create our mock backend with openapi-backend
const api = new OpenAPIBackend({ definition });
api.register('notFound', (c, res, ctx) => res(ctx.status(404)));
api.register('notImplemented', async (c, res, ctx) => {
  const { status, mock } = api.mockResponseForOperation(c.operation.operationId);
  ctx.status(status);
  return res(ctx.json(mock));
});


// tell msw to intercept all requests to api/* with our mock
const server = setupServer(
  rest.all('/api/*', async (req, res, ctx) => api.handleRequest(
    {
      path: req.url.pathname,
      query: req.url.search,
      method: req.method,
      body: req.bodyUsed ? await req.json() : null,
      headers: { ...req.headers.raw },
    },
    res,
    ctx,
  )),
);

beforeAll(() => server.listen());
afterAll(() => server.close());

Now instead of having to write your own mock handlers for each operation, they're generated from the response schemas and examples defined in the OpenAPI document.

What's more: any time the API definition changes, all your mocks will be automatically updated giving you further confidence your app is compatible with the new API version.

Enabling Request Validation

When testing, it's often very useful to make sure your application is actually sending the correct requests to the API.

Working with OpenAPI definitions has the benefit that API operations are well defined and requests can be automatically validated using JSON schema.

To enable request validation during tests, you can simply register the validationFail handler for openapi-backend:

api.register('validationFail', (c, res, ctx) => res(
  ctx.status(400),
  ctx.json({ error: c.validation.errors }),
));

When running tests, a malformed call to an API endpoint will now result in a 400 Bad Request error from the mock backend, alongside a useful error message telling you what's wrong with the request.

Custom Handlers

In some tests it might make sense to provide a different mock than the default one as provided by openapi-backend.

Registering your own mock for an API operation in a test is as simple as calling api.register() with the operationId and a mock handler:

it('should call getPets operation', () => {
  // given
  const mockResponse = [{ id: 2, name: 'Odie' }];
  const mockHandler = jest.fn((c, res, ctx) => res(ctx.json(mockResponse)));
  api.register('getPets', mockHandler);

  // when
  // render(<MyComponent />)...

  // then
  expect(mockHandler).toBeCalled();
});

Conclusion

Finding out about msw was a major game changer for my React testing. This in combination with the auto-mocking capabilities of openapi-backend makes mocking APIs in React tests a breeze.

Thank you, Kent, and the team behind mswjs/msw! 💙