OAM best practices

What really worked for me personally, or in my experience.

Generic concepts
Teamwork
CI/CD specific
1. Pipelining
Product engineering
Management
Further readings
1. Sources

Generic concepts

Always think critically and question all the things. Especially those that don't appear to make any sense.
Don't just follow trends or advice from others. They might know better, but you will be the one dealing with the issues in the end.
Try to understand how something really works, may it be a technology, a tool or what else.
Try at least once to do manually what an automation would do in your place. Look at the source code of tools. Read the fabulous documentation. Check if they hide error messages behind successful responses.
Stay curious. Experiment. Learn and break things (in a sane and safe way). Dive deeper into what interests you.
Make the informed decision that most satisfies your current necessities.
There is no perfect nor correct solution, just different sets of tradeoff. Besides, no one will ever have all the information at the start, as some of them only come with experience and looking back at decisions one has already made gives the distorted perspective that those decisions were clearer than they really were.
Review every decision after some time. Check they are still relevant, or if there is some improvement you can implement.
Things change constantly: new technologies are given birth often, and processes improve. Also, now you know better then before.
Gain the hard skills required to solve complex problems, but only deploy complex solutions when they are actually, really, needed.
Focus on the real problem at hand.
Beware the the XY problem.
When making a business decision, it's generally good to pick the simplest, fastest, and cheapest option.
When making a career decision, it pays to be an expert in hard things.
Do not make things more complicated than they need to be. Also read Death by a thousand microservices.
Keep things simple (KISS approach) with respect of your ultimate goal and not only for the sake of simplicity.
Always going for the simple solution makes things complicated on a higher level.
Check out KISS principle is not that simple.
Modularize stuff when it makes sense, not just to avoid repetitions.
Create abstractions that do hide away the complexity behind them.
Avoid creating wrappers that would map features 1-to-1 with their not-abstracted-anymore target object, and just use the original processes and tools when in need of control.
Beware of complex things that should be simple.
E.g., check what the SAFe delusion is.
Focus on what matters, but also set time aside to work on the rest.
Check Understanding the pareto principle (the 80/20 rule).
Learn from your (and others') mistakes.
Check out the 5 whys approach.
Put in place processes to avoid repeating mistakes.
Automate when and where you can, yet mind the automation paradox and abstractions.
Keep different parts decoupled where possible, the same way interfaces are used in programming.
Allows for quick and (as much as possible) painless switch between technologies.
The one-size-fits-all approach is a big fat lie.
One'll end up with stiff, hard to change results that satisfy nobody. This proved particularly true with regards to templates and pipelines.
Stop designing systems that should work for everybody at all times. Prefer safe defaults instead.
Choose tools based on how helpful they are to you to achieve your goals.
Do not adapt your work to specific tools.
Keep track of tools' EOL and keep them updated accordingly. Trackers like endoflife.date could help in this.
Backup your data, especially when you are about to make changes to something managing or storing it.
Murphy's law is lurking. Consider the 3-2-1 backup strategy.
Branch early, branch often.
Keep a changelog.
Keep changes short and sweet.
Nobody likes to dive deep into a 1200+ lines, 356+ files pull request (PR fatigue, right?).
Consider keeping changes in behaviour (logic) separated from changes to the structure.
It allows for easier debugging by letting you deal with one great issue at a time.
Make changes easy, avoid making easy changes.
Easy changes will only build up with time and become a pain to deal with long term.
Trunk-based development and other branching strategies all work.
Consider the different pros and cons of each.
Refactoring can be an option.
Just don't default to it nor use it mindlessly.
Be aware of corporate bullshit.
DevOps, GitOps and other similar terms are sets of practices, suggestions, or approaches.
They should not roles or job titles.
They should not to be taken literally.
They need to be adapted to the workplace, not the other way around.
Amazon's leadership principles are generally good practices, but also double-edge swords.
They still create a lot of discontent even inside Amazon when used against anybody.
Keep Goodhart's law in mind:

When a measure becomes a target, it ceases to be a good measure.
Always have a plan B.
When managing permissions, consider break glass procedures and/or tools.

Teamwork

Respect what is already there, but strive to improve it.
Current solutions are there for a reason. Learn about their ins and outs and, most of all, the why. Only then, it makes any sense to try to make them better.
Don't just dismiss your teammates' customs.
E.g., use EditorConfig instead of your editor's specific setting files only.
One and one's contributors (e.g. one's teammates and other teams in one's company) should be on the same boat and should be shooting for the same goals.
Act like it. You may as well collaborate instead of fighting each other.
Prefer using standardized execution environments to avoid the it works on my machine conundrum.
This helps to ensure everybody does things the same way, (hopefully) reaching the same results.
E.g., run commands in nix or containers, use virtual environments specific to repositories, configure standard actions in tools like task or GNU make.

CI/CD specific

Keep integration, delivery and deployment separated.
They are different concepts, and as such should require different tasks.
This also allows for checkpoints, and to fail fast with less to no unwanted consequence.
Consider adopting the main must be green principle.

Pipelining

Differentiate what the concept of pipelines really is from the idea of pipelines in approaches like DevOps.
Pipelines in general should be nothing more than sequences of actions. Pipelines in DevOps (and alike) end up most of the times being magic tools that take actions away from people.
Keep in mind the automation paradox.
Pipelines tend to become complex systems just like Rube Goldberg machines.
Keep tasks as simple, consistent and reproducible as possible.
Avoid like the plague relying on programs or scripts written directly in pipelines: pipeline should act as the glue connecting tasks, not replace full fledged applications.
Most, if not all, pipeline tasks should be able to execute from one's own local machine.
This allows to fail fast and avoid wasting time waiting for pipelines to run in a black box somewhere.
Pipelines are a good central place from which make changes to critical resources.
Developers should not have the access privileges to make such changes by default, but selected people shall have ways to obtain those permissions for emergencies (break glass).
DevOps pipelines should be meant to be used as last mile steps for specific goals.
There cannot be a single pipeline for everything, the same way as the one-size-fits-all concept never really works.
Try and strike a balance between what needs to be done centrally (e.g. from a repository's origin remote) and what can be done locally from one's machine before the code reaches repositories' remotes.
. Tools like pre-commit or lefthook are a doozy for this, but can disrupt the development experience and encourage the use of the --no-verify switch. Actions that need to be enforced (e.g. automatic formatting) are usually worth done only when changes reach the central remote anyways.

Product engineering

Consider what follows for infrastructure and platform engineering as well.

Focus on creating things users will want to use.
Tools should solve issues and alleviate pain points, not create additional walls.
Focus on small audiences first. Avoid trying appealing lots of users from the very beginning.
If one does not have a user base, one's product has no reason to exist but one's will to create it.
Consider and fix users' pain points before adding new features.
If users are not happy with one's tool they'll try moving away from it, bringing the discussion back to the previous point in this list.
Avoid creating effectively useless abstractions, like templates that use variables for all their attributes.
Prefer providing one, or at most a few, simplified solution that use different adapters or interfaces in the background instead.
E.g., check out how Crossplane, Radius and KRO work.
Offer clear error messages and immediate access to them.
Consider leveraging different, more specific status codes for different global results. E.g.:
- Return 5 instead of 1 in UNIX to point out an executable could not find a required file.
- Return 422 Unprocessable Content instead of 200 OK if a request was syntactically correct, but the data it contained was wrong.
- Return 207 Multi-Status instead of 200 OK if an API fulfilled a request successfully, but something in the more global process did not quite fully go as expected.

Management

Beware the action fallacy.

18 KiB Raw Blame History

OAM best practices

Generic concepts

Teamwork

CI/CD specific

Pipelining

Product engineering

Management

Further readings

Sources

18 KiB

Raw Blame History