Alternatively if your company is sane you work to ensure that there are migration periods so that pushing a new feature or replacement API doesn’t simultaneously break existing clients.
Seriously the “single version” or “not breaking clients” has nothing to do with being a monorepo, and everything to do with the development processes a company or project has. I imagine a spectrum of processes where at one end is the MS of yore, maintaining all things for all apps, for all time, and the other this kind of insane chaos. You can have a single repo with tagged versions, submission branches, etc or you can have many repos and a central service providing the submitted code for every project. Both systems can work at large scale (although mono repo perf apparently does not), and both systems can choose where on the spectrum they want to be.
That said in terms of culture I would consider producing APIs, encouraging adoption, and then mass breaking those APIs to be a sign of bad engineering. You are choosing to make a bunch of time sensitive work for a whole lot of people vs making a bit more work for yourself. This article says that they have tooling so you can make all the necessary changes to all the effected projects, but all you have to do is ensure that the clients build - working, or even passing automated tests does not appear to be a requirement - these are things that I would consider common courtesy if you /are/ in a position where a breaking change is necessary.
I definitely agree with this for the most part. Monorepo definitely helps with avoiding drift and making sure versions stay aligned, but it is by no means fundamental. It is more a visibility tool than a versioning tool.
In my experience at Google, I have not seen the wide swaths of breaking changes that the author of this post sees. I am not surprised that it could be happening somewhere in Google, but when things are configured correctly this shouldn’t be possible. To push a change for a new API, all of the code of the users should be configured such that it builds and run tests before the API change can be submitted. As such, generally the old API would be left around and labelled deprecated giving users tons of time to switch (potentially using automation to create changes). Or, the author would update all uses of the API with their change to the API (running all tests).
I can see some bad incentives for sure. If you work on V1 of a project and know how you can make a much better V2. It would almost certainly be seen as a good project for you to take on. This means that your V1 users will eventually be expected to migrate (unless they have enough priority/pull to not). So this means that multiple versions can definitely exists to some extent, but it is much much more limited than having access to large swaths of versions as seen in open source software with semver.
I’ve never worked at Google, but I’ve encountered the attitude of ‘break APIs, let downstream deal with the fallout’ from Google employees in several open-source projects and so I’d assumed that it was part of their engineering culture. Designing good APIs and maintaining them long term seems completely alien to the Google mindset, as seen from the outside.
That has less to do with breaking changes and more to do with inside/outside Google. The entire Google monorepo and ecosystem around it exists in a bubble. You can make a change and easily see every single test failure you’ve caused anywhere in any code written at Google. But there’s nothing like that for external code. Any externally visible big breaking change was probably carefully migrated internally within Google.
I can see how living in that bubble could translate to open source projects: all the tests I can see pass, ship it!
The article is utter rubbish, that’s completely untrue. Google CI runs tests for everything it builds, so you can’t push something that builds but is blatantly broken. Unless it has no tests or you explicitly press the big red “fuck these people I don’t care” button.
As bhansconnect says, perhaps the author had a bad experience in some corner of Google where no one is remotely competent. But the original article as a generalization of Google is simply wrong.
My experience was clearly different as well - not the breaking downstream because I’d never consider doing that deliberately, but the mono repo stuff. At G I never encountered the monorepo, and my external experience with G also did not involve even references to their monorepo. That said I have successfully broken so many things without breaking any tests that relying on CI to find regressions caused by changes made for arbitrary ABI/API breaks.
The last two words are the crux of this issue: monorepos deliberately centralize power. The underlying assumption is that the centralized authority makes good decisions. If not, the same efficiency that enables codebase-wide changes magnifies the impact of a bad decision.
Not to turn this into a politics thread, but this is what I find tedious about discussions of “states rights vs federal power” or “mom’n’pops vs monopolies”: whether scale is good or bad depends on who has it! If we can assume the federal government is good and competent, we don’t need states rights. If we can’t, we do. In practice, it’s usually a mixture that varies by domain. The same with big business vs small business. There’s no one right level of centralization. When things are going well, centralize and spread the benefits. When things are going poorly, decentralize and cultivate anti-fragility.
When things are going well, centralize and spread the benefits. When things are going poorly, decentralize and cultivate anti-fragility.
The key argument of decentralization is risk management. Centralisation introduces single points of failure. If the central authority is performing badly, this affects everyone. If a decentralised authority is performing badly then this has a more limited effect on the whole system. For a decentralised system to exhibit the same failure mode, all of the individual elements need to fail simultaneously.
The counter argument to this is that failures are often correlated and reasoning about them is hard. This is a big part of the reason that Katrina was such a disaster in the US, even though it did less damage than several other hurricanes: a lot of insurance companies thought that they were insuring a load of independent risks but they all had the same root cause. When Katrina hit, it turned out that they were insuring a lot of correlated risks and so several insurance companies went out of business and were unable to pay out. Often, a decentralised system fails to decentralise the risk and ends up just hiding it in a single point of failure that can trigger failure of all of the decentralised components.
I work in distributed systems, so I definitely feel this, but it sort of sounds like you’re saying “the downside of decentralized systems is that some of them may actually be centralized systems” which is true, but that’s not exactly high praise for centralized systems. Indeed, we still go through the trouble of building decentralized/distributed systems when we care about reliability even though there’s a risk we might not get it right–we don’t throw up our hands and choose explicitly centralized architectures (which sort of seems to be what you’re implying in your “counter argument” paragraph; apologies if I’m misunderstanding).
I work in distributed systems, so I definitely feel this, but it sort of sounds like you’re saying “the downside of decentralized systems is that some of them may actually be centralized systems” which is true, but that’s not exactly high praise for centralized systems.
I’d frame it more as ‘the down side of decentralised systems is that they are very hard to get right and if you get them wrong then you get the down sides of both centralised and decentralised systems’. A good decentralised system is a big win because it has inherent fault tolerance and there are a few fantastic examples of this. GMail might be down, but email as a whole is almost never down. The single point of failure for email is DNS and that is also a distributed system (the single point of failure for DNS is distributed between the root names - a single domain’s authoritative servers might be down, but DNS as a whole keeps going).
The behaviour of a centralised system depends on first-order properties of the system. The behaviour of a distributed system depends on emergent properties of the system and, in general, humans are spectacularly bad at reasoning about even second-order effects of a system.
I think in many cases there’s a systematic bias in favor of pre-mature centralization, so you probably need a culture of aggressive decentralization to balance it out. The downside risks of decentralization are usually lower: suboptimal outcomes but still better than worst case.
But looking at, e.g., the US response to COVID-19, in a hypothetical country the size and composition of the US but with a centralized authority it would have been easy to do better than we did. We had things like the COVID-19 Project from The Atlantic because even basic data reporting standards weren’t there, so the private sector had to do it. The centralized agencies we did have like CDC have no actual power to do anything so they just did pointless turf guarding to try to keep the FDA from approving rapid tests, instead of actually protecting public health by putting out clear and realistic guidelines based on clearly gathered and presented data. You couldn’t actually restrict interstate travel, so there was just a blanket policy of states being laissez-faire plus the governor being vocally pro- or anti- the restrictions that were or were not on the books and either way entirely unenforced. We sort of got the worst both worlds with what was in practice a uniform policy with low coherence. Oh well.
Going back to centralization vs decentralization, I think what that shows is you can’t just assume because there are many actors there’s an actually decentralized system. The actors need to be actually able to act independently for it to be a decentralized system that works and in the case of a virus that crosses state lines and borders, there wasn’t really independence.
The tradeoff between redundancy and efficiency is just another way to state the CAP theorem. I think intelligence is the ability to balance this tradeoff optimally.
Arrow’s theorem is just an information theoretic result, the ballot needs to have more information than ordering (also intensity). However if you do that then the issue is combining probabilistic assessments from different sources about the same thing since you get a second order probability (and third order, …, and nth order). Still, we can do much better than we do today to keep the assessments in line with our observed reality.
I do agree that it will never permanently settle (I call those kinds of steady states “utopia” or “dystopia”).
If we can assume the federal government is good and competent, we don’t need states rights
You’ve missed the point of the federation. It’s not just to curtail the power of the fed, but the states are supposed to be able to have different laws. The fed is supposed to keep the states working together, and deal with cross-state issues, not to enforce the views and laws of the most-populous states on the rest of the citizenry.
Right, but the whole reason for that is that we assume that federal law may have flaws or issues “on the ground”. If we can just make excellent, globally valid federal law, obviously that’s easier than having states’ laws. The whole system is a hedge against the central power making bad laws.
You don’t need experimentation if you already have the answers. (However note: many more people think they have the answers than do.)
It seems like the thrust of your comment was “centralize when things are going well and decentralize when things aren’t going well”, while it seems the parent is arguing that federation (federalism?) isn’t about centralizing as you describe. I thought both points were interesting, and I upvoted accordingly.
Sophistifunk seems to have the view that the Federal government in the US should just do the minimum tasks to allow decentralized control by the states. My original argument was I find that kind of fixed view about what the Federal government should or should not do silly because whether that’s good idea or not depends on whether decentralization is good or not, which is context dependent.
I think in general a problem with political discussions is that people approach them with certain meta-points labelled as good or bad and then lose the ability to argue back to basics and talk about why the meta-points are good or bad. Federal decentralization is good if it results good outcomes for governance. (I suppose you can also argue that it’s a tradition, and it’s good to stick to traditions unless they are obviously breaking down, but that’s just a way of saying you don’t want to debate the merits of the issue itself.) Just saying don’t “enforce the views and laws of the most-populous states on the rest of the citizenry” is not addressing anything. Do the most populous states want to enforce views that are good, bad, or neutral on the other states? If it’s bad or neutral then yeah, we want decentralization. If it’s good, we want them to force the small states to do the good thing! I can think of cases where the states enforced good views and cases where they enforced bad views, and if you sat in a US history class, you can too. It’s not worth debating at that level.
The difficulty is that of limiting principles. You can’t write a law at a federal level just because you think it should be done at that level, there must be a limiting principle beyond “well that’s obviously a step too far” that prevents future generations or politicians from abusing the abilities the previous group created.
“We can pass law to force X, how is that substantially different from Y? I think X was bad and Y is good, and we did it in that case…”
Limited zones of authority help prevent arbitrary and personal beliefs about what is good and what is not. Federal versus State versus County versus Local versus a lack of governance on a topic requires limiting principles up and down the chain.
Bringing it back to the original post, if all code is in a monorepo, then all people in that repo will at some point need to confront the decision of whether to make a change for someone else that, to that someone else, may feel capricious. Your social model among committers needs to be able to stand up to a total lack of limiting factors. Not every organization is ready for that. The organizations that are have to gatekeep heavily to maintain the culture.
That’s a cost. It might be acceptable, but it’s there.
If you have a polyrepo and the CTO wants to institute a new rule like “we don’t use library A in this company” it’s a little tricky because you need N teams to add a thing that enforces it. It’s much simpler with a monorepo because there’s a single chokepoint.
I work in an org without a monorepo, but we also have expectations that dependencies are kept up to date.
It’s partially automated using dependabot, if dependabot updates a dependency and the builds pass, then it gets automatically merged.
There are still lots of failures, dependency updates produce a constant stream of breaking changes. The work to update systems is exhausting.
We don’t have any of the monorepo tooling that google has, and many of the maintainers of open source dependencies don’t care much about backward compatibility.
Google also doesn’t really use a mono repo as much as people seem to believe that they do. Maybe back in the early days “everything” was in a single repo, but that is not the case today.
There are obviously some orgs in google that still talk to a single shared repo, but anything large seems to be completely independent of this. To my knowledge Android, Fuchsia (which uses git), and Chrome (also git) at least are in independent repos. So I feel like this is specific complaints about their internal tooling maybe?
That is more due to those pieces of software being open source. Google is still mostly a giant monorepo by lines of code. Some software that is focused around open source just doesn’t get to be in the monorepo. In my experience, the open source projects are much more annoying to work with than the monorepo.
Narrator voice: it does.
Imo: everything should be incentive driven. Designing a good incentive package, OKRs is a sign of strong, matured management layer in a company.
Alternatively if your company is sane you work to ensure that there are migration periods so that pushing a new feature or replacement API doesn’t simultaneously break existing clients.
Seriously the “single version” or “not breaking clients” has nothing to do with being a monorepo, and everything to do with the development processes a company or project has. I imagine a spectrum of processes where at one end is the MS of yore, maintaining all things for all apps, for all time, and the other this kind of insane chaos. You can have a single repo with tagged versions, submission branches, etc or you can have many repos and a central service providing the submitted code for every project. Both systems can work at large scale (although mono repo perf apparently does not), and both systems can choose where on the spectrum they want to be.
That said in terms of culture I would consider producing APIs, encouraging adoption, and then mass breaking those APIs to be a sign of bad engineering. You are choosing to make a bunch of time sensitive work for a whole lot of people vs making a bit more work for yourself. This article says that they have tooling so you can make all the necessary changes to all the effected projects, but all you have to do is ensure that the clients build - working, or even passing automated tests does not appear to be a requirement - these are things that I would consider common courtesy if you /are/ in a position where a breaking change is necessary.
I definitely agree with this for the most part. Monorepo definitely helps with avoiding drift and making sure versions stay aligned, but it is by no means fundamental. It is more a visibility tool than a versioning tool.
In my experience at Google, I have not seen the wide swaths of breaking changes that the author of this post sees. I am not surprised that it could be happening somewhere in Google, but when things are configured correctly this shouldn’t be possible. To push a change for a new API, all of the code of the users should be configured such that it builds and run tests before the API change can be submitted. As such, generally the old API would be left around and labelled deprecated giving users tons of time to switch (potentially using automation to create changes). Or, the author would update all uses of the API with their change to the API (running all tests).
I can see some bad incentives for sure. If you work on V1 of a project and know how you can make a much better V2. It would almost certainly be seen as a good project for you to take on. This means that your V1 users will eventually be expected to migrate (unless they have enough priority/pull to not). So this means that multiple versions can definitely exists to some extent, but it is much much more limited than having access to large swaths of versions as seen in open source software with semver.
I’ve never worked at Google, but I’ve encountered the attitude of ‘break APIs, let downstream deal with the fallout’ from Google employees in several open-source projects and so I’d assumed that it was part of their engineering culture. Designing good APIs and maintaining them long term seems completely alien to the Google mindset, as seen from the outside.
That has less to do with breaking changes and more to do with inside/outside Google. The entire Google monorepo and ecosystem around it exists in a bubble. You can make a change and easily see every single test failure you’ve caused anywhere in any code written at Google. But there’s nothing like that for external code. Any externally visible big breaking change was probably carefully migrated internally within Google.
I can see how living in that bubble could translate to open source projects: all the tests I can see pass, ship it!
The article explicitly states that an engineer’s only responsibility to clients is that they still build, not that they still work.
The article is utter rubbish, that’s completely untrue. Google CI runs tests for everything it builds, so you can’t push something that builds but is blatantly broken. Unless it has no tests or you explicitly press the big red “fuck these people I don’t care” button.
As bhansconnect says, perhaps the author had a bad experience in some corner of Google where no one is remotely competent. But the original article as a generalization of Google is simply wrong.
My experience was clearly different as well - not the breaking downstream because I’d never consider doing that deliberately, but the mono repo stuff. At G I never encountered the monorepo, and my external experience with G also did not involve even references to their monorepo. That said I have successfully broken so many things without breaking any tests that relying on CI to find regressions caused by changes made for arbitrary ABI/API breaks.
Not to turn this into a politics thread, but this is what I find tedious about discussions of “states rights vs federal power” or “mom’n’pops vs monopolies”: whether scale is good or bad depends on who has it! If we can assume the federal government is good and competent, we don’t need states rights. If we can’t, we do. In practice, it’s usually a mixture that varies by domain. The same with big business vs small business. There’s no one right level of centralization. When things are going well, centralize and spread the benefits. When things are going poorly, decentralize and cultivate anti-fragility.
The key argument of decentralization is risk management. Centralisation introduces single points of failure. If the central authority is performing badly, this affects everyone. If a decentralised authority is performing badly then this has a more limited effect on the whole system. For a decentralised system to exhibit the same failure mode, all of the individual elements need to fail simultaneously.
The counter argument to this is that failures are often correlated and reasoning about them is hard. This is a big part of the reason that Katrina was such a disaster in the US, even though it did less damage than several other hurricanes: a lot of insurance companies thought that they were insuring a load of independent risks but they all had the same root cause. When Katrina hit, it turned out that they were insuring a lot of correlated risks and so several insurance companies went out of business and were unable to pay out. Often, a decentralised system fails to decentralise the risk and ends up just hiding it in a single point of failure that can trigger failure of all of the decentralised components.
I work in distributed systems, so I definitely feel this, but it sort of sounds like you’re saying “the downside of decentralized systems is that some of them may actually be centralized systems” which is true, but that’s not exactly high praise for centralized systems. Indeed, we still go through the trouble of building decentralized/distributed systems when we care about reliability even though there’s a risk we might not get it right–we don’t throw up our hands and choose explicitly centralized architectures (which sort of seems to be what you’re implying in your “counter argument” paragraph; apologies if I’m misunderstanding).
I’d frame it more as ‘the down side of decentralised systems is that they are very hard to get right and if you get them wrong then you get the down sides of both centralised and decentralised systems’. A good decentralised system is a big win because it has inherent fault tolerance and there are a few fantastic examples of this. GMail might be down, but email as a whole is almost never down. The single point of failure for email is DNS and that is also a distributed system (the single point of failure for DNS is distributed between the root names - a single domain’s authoritative servers might be down, but DNS as a whole keeps going).
The behaviour of a centralised system depends on first-order properties of the system. The behaviour of a distributed system depends on emergent properties of the system and, in general, humans are spectacularly bad at reasoning about even second-order effects of a system.
I think in many cases there’s a systematic bias in favor of pre-mature centralization, so you probably need a culture of aggressive decentralization to balance it out. The downside risks of decentralization are usually lower: suboptimal outcomes but still better than worst case.
But looking at, e.g., the US response to COVID-19, in a hypothetical country the size and composition of the US but with a centralized authority it would have been easy to do better than we did. We had things like the COVID-19 Project from The Atlantic because even basic data reporting standards weren’t there, so the private sector had to do it. The centralized agencies we did have like CDC have no actual power to do anything so they just did pointless turf guarding to try to keep the FDA from approving rapid tests, instead of actually protecting public health by putting out clear and realistic guidelines based on clearly gathered and presented data. You couldn’t actually restrict interstate travel, so there was just a blanket policy of states being laissez-faire plus the governor being vocally pro- or anti- the restrictions that were or were not on the books and either way entirely unenforced. We sort of got the worst both worlds with what was in practice a uniform policy with low coherence. Oh well.
Going back to centralization vs decentralization, I think what that shows is you can’t just assume because there are many actors there’s an actually decentralized system. The actors need to be actually able to act independently for it to be a decentralized system that works and in the case of a virus that crosses state lines and borders, there wasn’t really independence.
The tradeoff between redundancy and efficiency is just another way to state the CAP theorem. I think intelligence is the ability to balance this tradeoff optimally.
Between CAP and Arrow’s theorem, math has proven that politics will never be permanently settled. :-)
Arrow’s theorem is just an information theoretic result, the ballot needs to have more information than ordering (also intensity). However if you do that then the issue is combining probabilistic assessments from different sources about the same thing since you get a second order probability (and third order, …, and nth order). Still, we can do much better than we do today to keep the assessments in line with our observed reality.
I do agree that it will never permanently settle (I call those kinds of steady states “utopia” or “dystopia”).
You’ve missed the point of the federation. It’s not just to curtail the power of the fed, but the states are supposed to be able to have different laws. The fed is supposed to keep the states working together, and deal with cross-state issues, not to enforce the views and laws of the most-populous states on the rest of the citizenry.
Right, but the whole reason for that is that we assume that federal law may have flaws or issues “on the ground”. If we can just make excellent, globally valid federal law, obviously that’s easier than having states’ laws. The whole system is a hedge against the central power making bad laws.
You don’t need experimentation if you already have the answers. (However note: many more people think they have the answers than do.)
Making perfect laws is in the realm of the Philosopher King
That is literally the whole point of my comment. 🙄
It seems like the thrust of your comment was “centralize when things are going well and decentralize when things aren’t going well”, while it seems the parent is arguing that federation (federalism?) isn’t about centralizing as you describe. I thought both points were interesting, and I upvoted accordingly.
Sophistifunk seems to have the view that the Federal government in the US should just do the minimum tasks to allow decentralized control by the states. My original argument was I find that kind of fixed view about what the Federal government should or should not do silly because whether that’s good idea or not depends on whether decentralization is good or not, which is context dependent.
I think in general a problem with political discussions is that people approach them with certain meta-points labelled as good or bad and then lose the ability to argue back to basics and talk about why the meta-points are good or bad. Federal decentralization is good if it results good outcomes for governance. (I suppose you can also argue that it’s a tradition, and it’s good to stick to traditions unless they are obviously breaking down, but that’s just a way of saying you don’t want to debate the merits of the issue itself.) Just saying don’t “enforce the views and laws of the most-populous states on the rest of the citizenry” is not addressing anything. Do the most populous states want to enforce views that are good, bad, or neutral on the other states? If it’s bad or neutral then yeah, we want decentralization. If it’s good, we want them to force the small states to do the good thing! I can think of cases where the states enforced good views and cases where they enforced bad views, and if you sat in a US history class, you can too. It’s not worth debating at that level.
The difficulty is that of limiting principles. You can’t write a law at a federal level just because you think it should be done at that level, there must be a limiting principle beyond “well that’s obviously a step too far” that prevents future generations or politicians from abusing the abilities the previous group created.
“We can pass law to force X, how is that substantially different from Y? I think X was bad and Y is good, and we did it in that case…”
Limited zones of authority help prevent arbitrary and personal beliefs about what is good and what is not. Federal versus State versus County versus Local versus a lack of governance on a topic requires limiting principles up and down the chain.
Bringing it back to the original post, if all code is in a monorepo, then all people in that repo will at some point need to confront the decision of whether to make a change for someone else that, to that someone else, may feel capricious. Your social model among committers needs to be able to stand up to a total lack of limiting factors. Not every organization is ready for that. The organizations that are have to gatekeep heavily to maintain the culture.
That’s a cost. It might be acceptable, but it’s there.
I don’t follow how this works if different parts of the repo have different owners who gate code submission.
If you have a polyrepo and the CTO wants to institute a new rule like “we don’t use library A in this company” it’s a little tricky because you need N teams to add a thing that enforces it. It’s much simpler with a monorepo because there’s a single chokepoint.
I work in an org without a monorepo, but we also have expectations that dependencies are kept up to date.
It’s partially automated using dependabot, if dependabot updates a dependency and the builds pass, then it gets automatically merged.
There are still lots of failures, dependency updates produce a constant stream of breaking changes. The work to update systems is exhausting. We don’t have any of the monorepo tooling that google has, and many of the maintainers of open source dependencies don’t care much about backward compatibility.
Google also doesn’t really use a mono repo as much as people seem to believe that they do. Maybe back in the early days “everything” was in a single repo, but that is not the case today.
There are obviously some orgs in google that still talk to a single shared repo, but anything large seems to be completely independent of this. To my knowledge Android, Fuchsia (which uses git), and Chrome (also git) at least are in independent repos. So I feel like this is specific complaints about their internal tooling maybe?
That is more due to those pieces of software being open source. Google is still mostly a giant monorepo by lines of code. Some software that is focused around open source just doesn’t get to be in the monorepo. In my experience, the open source projects are much more annoying to work with than the monorepo.
Google cloud customers are indeed affected by forced API upgrades and rapidly deprecated versions.