The phrase thrown around was “collaboration headwind”, the idea was if project success depends on 1 person with a 95% chance of success, project success also had a 95% chance. But if 10 people each need to succeed at a 95% chance, suddenly the project success likelihood becomes 60%…
In reality, lazy domain owners layered on processes, meetings, documents, and multiple approvals until it took 6 months to change the text on a button, ugh
Another side of this coin is that the expected payoff from a project depends on how many unrelated projects your organization is engaging in, which is deeply counterintuitive to most people.
Every project carries with it three possibilities: that of success, where the company makes money, that of failure, where the company does not, and that of a "critical failure", where the project goes so wrong that it results in a major lawsuit, regulatory fine or PR disaster that costs the company more than the project was ever expected to make.
If you're a startup, the worst that can happen to your company is the value going to 0. From an investor's perspective, there's not much of a difference between burning all the money ($10m) and not finding product-market-fit (normal failure), or your company getting sued for $3b and going bankrupt (critical failure). The result is the same, the investment is lost. For a large corporation, a $3b lawsuit is far more costly than sinking $10m into a failed project.
You can trade off these three possibilities against each other. Maybe forcing each release through an arduous checklist of legal review or internationalization and accessibility testing decreases success rates by 10%, but moves the "critical failure rate" from 1% to 0.5%. From a startup's perspective, this is a bad tradeoff, but if you're a barely-profitable R&D project at big co, the checklist is the right call to make.
This problem is independent from all the other causes to which bureaucracy is usually attributed, like the number of layers of management, internal culture, or "organizational scar tissue." Just from a legal and brand safety perspective, the bigger your org, the more bureaucracy makes sense, no matter how efficient you can get your org to be.
"Nothing breeds conservatism like having something to conserve" was a recurring line from a founder-type, complaining individuals had become less dramatically innovative as the "startup" succeeded and prospered. Though they had colleagues who thought the line misapplied. IIRC, fuzzily.
This is really insightful, I hadn’t considered this dynamic before.
I wonder if a related intensifier is that as a company grows larger it tends to follow the letter of the law rather than the spirit of the law, which results in less buffer space between the behavior and the law (and hence higher lawsuit risk).
I like the ideas here but I think the actual chance of getting sued for $3b is so small as to be negligible in the context of costs. It's also questionable how much the additional process/overhead moves the needle on that chance. Larger companies also have various "shields" against these sorts of lawsuits. E.g. they lobby politicians, they employ lawyers, they have legal and IP protection.
Just like anything else in life you want to look at the present value and then get insurance for huge risks.
That said agree that a startup can take more risk but I don't think that is the major factor explaining why larger companies tend to be process heavy and slower.
This explanation wasn't just about lawsuits.
Other things in a similar category are:
- negative media attention (media scrutiny increases proportionally to organization size)
- doing something that upsets an influential group and may have consequences for the rest of your business (think how big the outrage would have been if Apple, Google or Microsoft tried making an "Uber" app before Uber existed)
- bringing down the service which is being worked on, potentially breaking SLAs
- failing to meet customer / legal commitments, particularly in regards to internationalization, accessibility etc.
- security incidents, which are presumably a bigger deal, as your project is connected to the rest of your infrastructure
- getting cancelled online, which causes employees (unrelated to the project) to quit
- natural, random and serious consequences that result from the fact that your project needs the company to hire additional employees. E.g. there's a certain number of people willing to commit sexual assault or financial fraud in the population, and the more people you hire, the more likely it is that you get one of them.
I work for "big tech" and have worked for others. Other than some CYA training there really isn't a lot of cross talk between engineering productivity and this sort of risk aversion.
Process, complexity, inefficiency is just something that happens for big companies and big software. Things slow down and then there's a negative feedback loop and things just go down hill. Innovators dilemma sort of stuff.
Getting sued for $3B maybe, but what about getting fined $3B? Such as by the EU.
> lazy domain owners
Interesting. As a consultant for the most of the last 25 years, my experience is the domain owners are typically invested and have strong opinions on the things that impact their jobs.
Executive leadership, on the other hand, doesn't want to actually know the issues and eyes glaze over as they look at their watches because they have a tee time.
There's a culture of "I won't approve this unless it does something for me" at Google. So now changing the text on a button comes with 2 minor refactors, 10 obvious-but-ignored bugfixes, and 5 experiments that it is actually better.
While this sounds pretty frustrating, there is at least a small upside: at least you get to the obvious-but-ignored bugfixes.
Most smaller places don’t have the bandwidth and many larger ones don’t have the desire.
I’m not sure if that makes up for bugs potentially introduced in the refactors, though.
Well, when the owner asks for a whole test suite that didn't exist to get a fix in, what most likely happens is that you just wasted your time in a draft CL that will get lost.
Do you mean the relevant code area(s) didn't have (sufficient) tests? You're being asked to backfill those missing tests in addition to your fix?
Yes. I've experienced pushback from obvious fixes with requests to formally test their code for the first time.
All because it may break someone. Even when I presented a real defect based on docs/comments and fixed it. You'd think that if they truly cared about breakages they'd already have some tests for it from where I can easily start.
I don’t think that is necessarily unreasonable. The team may have the same constraints on themselves in that they wouldn’t touch the code either until tests are written.
> All because it may break someone. Even when I presented a real defect based on docs/comments and fixed it.
It's great that you found a bug and fixed it.
The problem is, how do you know that there are no other regressions?
Code is a liability. Once you check it in, the team that owns it is responsible for it. Untested code is a liability of unknown scope. It's quite understandable why they don't want to accept someone's contributions, when the contributor isn't the one who will ultimately be dealing with any of the consequences. If you think they are being mean and lazy, imagine if the tables were reversed.
I don't accept puppies or elephants as gifts for similar reasons.
It's unfortunate that existing test coverage sucks. In this case, the best way forward should be for the team in question to improve coverage, and for you to then submit your fix + a test for it. And if they don't have budget to do this, then that sucks, but that's their call to make, and that's a signal that the project in question is abandonware.
And it's fine for a large company to have a bunch of abandonware. If it works, and produces value, the optimal amount of ongoing development effort to invest into a piece of software may, depending on the circumstances, be near-zero.
They aren't asking for you to write tests because 'it benefits them', they are asking you to write tests because as a professional engineer, you should write tests, and not just yolo it.
Look, sometimes you may have good reasons for why a test is impractical. You are allowed to push back, or look for a different reviewer. There's a hundred thousand people in the firm, you should be able to find one or two that will let you submit literally anything that compiles.
But most of the time, the reviewer is giving you advice that you should take.
If you are turning a button to a slightly different shade of blue and it's not a button you own, the owner of the button should not be asking you to write tests for the button.
It's as good an opportunity as any to improve things.
They're acting as selfish demanding you do something for them, as you are for refusing.
If the case is as simple as you describe, surely there's more than one owner of the code that can approve this, if one guy is being unreasonable.
If there is actually only one owner that can approve changes to the package, there's something really weird and unusual about that project setup, or it's someone's internal hobby project that they wrote five years ago and semi-maintain, in which case, I have to wonder why you submitting one-liner changes to it is all that important.
We're all adults, we all work together, we can all work this out. If someone absolutely insists on being an asshole, escalate. It's why you have a manager, and why they have a manager.
My experience is that very few people are unreasonable assholes.
There's always plenty of organizational, vision, strategy, and execution problem in any billion-dollar company, but 'people are unreasonable in code reviews' is not one I'd put in the top 10. It might be something that ruins your day once or twice, but that doesn't make it systemic.
> If someone absolutely insists on being an asshole, escalate.
That's doubling down on time spent on contributing back. It's usually cheaper to workaround the issue once you notice it'll be way harder than it should be (not hard at all).
I would think the person is more interesting and more relevant than the button. One doesn't create hurdles when I'm trying to work. You just don't do it.
I've allowed people to build a whole obstacle course one time. Decades later it stil has me randomly burst out in laughter. It's like hoarding technical debt until nothing in the code base makes sense or even looks familiar. You just don't do that...
Coordination Headwind: https://komoroske.com/slime-mold/
The old "If you want to go fast, go alone. If you want to go far, go together."
Also why the optimal business strategy seems to be to go as far as you can alone and then bring on other people when you're running out of steam.
Well, good management/tech leadership is about making sure that the risks coming from individual failure points (10 people in your example) are recognized and mitigated, and that the individuals involved can flag risks and conflicts early enough so that the overall project success probability does not go down as you describe...
The assumptions in that math are wrong anyway. Once you depend on 10 people, the chance that they each achieve "95% successful execution" is 0.
This is only partially down to the impossibility of having every staff member on a project be A++ players.
There is coordination RISK not just coordination overhead. Think planning a 2 week trip with your spouse with multiple planes/trains/hotels, museum/exhibit ticket bookings, meal reservations, etc. Inevitably something gets misunderstood/miscommunicated between the two of you and therefore mis-implemented.
Now add more communication nodes to the graph and watch the error rate explode.
That's what the math is reflecting. Project succeeds if all of 10 people does their job well. Each person has a 95% chance of succeeding. 0.95^10 ~= 60%, and so the chance that all 10 people do their job successfully is ~60%.
Those jobs also include things like management and product design, and so the coordination risk is reflected in the 5% chance that the manager drops the ball on communication. (As a manager, I suspect that chance is significantly more than 5% and that's why overall success rates are even lower.)
That's what I mean "only 5%" encapsulating all failure modes (comms/implementation/coordination/etc) is very low.
And that under-estimation compounds to make the top level 60% much higher than it should be.
A 7.5% rate takes top-level success odds below 50% - 46%. A not unrealistic 10% takes the top level down to 35%.
Etc.