As expected, extremes like elite and low-performing organisations show predictable trends: low-performing orgs have higher failure rates, while elite-performing orgs have fewer.
It’s important to note that “performance” here refers to software delivery frequency, not business outcomes.
What’s interesting, however, is that medium-performing orgs have a lower failure rate than high-performing orgs.
Does it mean these orgs become more fragile over time?
Would it be fair to conclude that orgs shouldn’t step off the gas due to increased failure rates in the short term when they move from medium to high-performing in terms of software delivery?
What if DORA clusters according to historical responses of the same organisation?
I have noticed this trend of reduced failure rate due to reduced deployments at least in one organisation I’m familiar with.
Value Stream Mapping
The DORA Quick Check (https://dora.dev/quickcheck) can help guide a conversation and set this baseline measurement. Your team will need to understand what is preventing better performance.
One effective way to find these impediments is to complete a value stream mapping exercise with the team.
Value Stream Mapping is one problem solving mental model that keeps showing up and that too in so many varied scenarios
Improving Deployment Pipeline Efficiency
one story that comes to my mind is of my early days at Zomato where we had to optimise the production deployment time from 60 minutes. We didn’t knew at that time but we were drawing the value stream mapping of all the components in the deployment process and what all components are taking time and get it down to under 10 minutes or under 1 minute with some error rate (special case for production incidents).
Over the years we did value stream mapping to better improve the process of deployment pipeline, but it was very rare for us to do a total value stream mapping from the time a Asana/Github ticket was filed. This was mostly because measuring lead time metrics on top of ad hoc processes was pretty hard.
We probably could have built rough estimates but most of the time the painful problem is quite apparent.
Delivery Efficiency
like I mentioned – value stream mapping keeps coming up, one place where I have seen this is when Zomato was trying to improve the delivery efficiency during its early days. Building a detailed value stream mapping of all the components involved in the food delivery is what helped Zomato to deliver faster than competition and offer features like “On Time Or Free”
Tracing
probably too straightforward but value stream mapping isn’t that different from tracing in observability. it’s just that it’s applied in different contexts
AI Adoption
1. Writing code
2. Summarizing information
3. Explaining unfamiliar code
4. Optimizing code
5. Documenting code
6. Writing tests
7. Debugging code
8. Data analysis
Sure I have used AI in almost all of them. But personally I feel like “1. Writing Code”, “2. Summarizing Information”, “6. Writing Tests” take up most of my time with AI than the others. The others are almost negligible for me, probably because with or without AI the time spent in other parts is anyway very low for me. Since DORA consists of all sizes of organisations, even startups, it is definitely an interesting data point to see.
What exactly is a Change Failure?
We consider a deployment to be a change failure only if it causes an issue after landing in production, where it can be experienced by end users. In contrast, a change that is stopped on its way to production is a successful demonstration of the deployment process’s ability to detect errors.
DORA report considers anything that caused production impact to be a change failure, I’m not sure if that broad definition is by design i.e similar to how one classifies what is a production incident.
@Jens Rantil accurately pointed out that “change failure” is a thing of the past and SLOs are better way to define what exactly is a change failure.
But how do calculated risks fall into these? For example taking a calculated risk of shipping a feature fast even if there is uncertainty whether it contains bugs. And there is nothing wrong with it if the team is taking this risk according SLO and Error Budgets.
On one hand I do see value in a broader definition of change failure as any production impact. Similar to production incidents it leads to improvement in the software delivery process i.e improvement in catching issues that cause any SLO impact than only focussing on issues that already caused major SLO impact.
IMO DORA definition of change failure is only useful for measuring software delivery performance of a team. It’s important to understand that a team could be low in software delivery performance but the overall performance wise they could be doing very well. and even if there is a correlation between software delivery performance and overall performance, it doesn’t necessarily prove a correlation between DORAs definition of change failure rate and the overall performance.
So DORA’s change failure metric may not be highly correlated to overall performance of the team
Chatbot vs Web UI for AI
Chatbots were the most common interface through which respondents interacted with AI in their daily work (78.2%), followed by external web interfaces (73.9%), and AI tools embedded within their IDEs (72.9%). Respondents were less likely to use AI through internal web interfaces (58.1%) and as part of automated CI/CD pipelines (50.2%).
One hand, seeing lower use of AI through web interfaces shows that there is still a lot of scope for a lot of common workflows that are not simple actions like “do X”, “do Y” etc,.. Probably would require an in-depth look at web based workflows and where AI could help.
On another hand, I feel like there is still so much more leverage left in the Chat interface, personally I would feel much better if I can access tools via the ChatGPT interface or Cursor Chat rather than using a custom AI interface on a different website. It’s not much different from working with real humans, its a hassle to go through the initial barrier to start working with a new AI interface comfortably, especially if that interface is something that you work with less frequently
I wish AWS adds AI interface but I wish 10x for them to add a powerful command bar before the AI interface.
Who are getting most out of AI?
respondents who reported the largest productivity improvements from AI were security professionals, system administrators, and full-stack developers. Although they also reported positive productivity improvement, mobile developers, site reliability engineers, and project managers reported lower magnitudes of productivity benefits than all other named roles.
It is slightly (very slight) ironic that security professionals saw the largest productivity boost, even more than SREs—especially considering that many organizations’ security teams, like Samsung’s, often restrict AI usage.
Utility vs Trust
Participants’ perceptions of the trustworthiness of AI-generated code used in development work were complex. While the vast majority of respondents (87.9%) reported some level of trust in the quality of AI-generated code, the degree to which respondents reported trusting the quality of AI-generated code was generally low, with 39.2% reporting little (27.3%) or no trust (11.9%) at all.
reflects an interesting form of leverage: some developers have extreme positions on AI usage for code generation, but seems like many are using it to save time but not trusting it (healthy balance for a non-deterministic tool)
Fascinating Counter Intuitive Stats on AI
Ever since I read Leverage Points, I developed a special appreciation for the counter intuitive behaviours and stats.
Directly quoting is too verbose but consider this
Devs say AI does not help with Toilsome work
Devs say AI reduces Valuable work that they can do
Devs agree that AI is greatly helpful
Now where exactly AI is being helpful if it doesn’t help with toilsome work and also reduces the valuable work to do? – the report so elegantly captures it in the below venn diagram.
Platform Engineering is not End All
It’s not surprising that however hard one tries to build a platform with DX in consideration, in the end on an average it still reduces the delivery throughput
IMO There can be a lot of reasons for this, some of these listed in DORA report as well
Initial transformations always have a lot of friction, let’s call it Workflow Inertia, you are essentially forcing devs to change the workflows that they got comfortable over multiple months or sometimes years.
Platforms form abstractions on top of underlying tools/platforms. And as always abstractions add rigidity to existing workflows, probably why most businesses find it really hard to build a truly productive development environment for scale like 50+ microservices.
As good practice, a platform should provide methods for users of a platform to break out of the tools and automations provided in the platform, which contributes to independence, however, it comes at the cost of complexity.
Feel free to tag me on X or LinkedIn, or connect via email at sri@optiowl.cloud.
