Threads for madhadron

  1. 1

    For the frequentist hypothesis testing, we don’t treat the two as the same, we treat them as indistinguishable. There would be no evidence that supporting both was necessary. Or you learn that your data is insufficient for the purpose. For the Bayesian example, if the data is sufficient for a decision when hypothesis testing gives you no significant difference, then your prior is pretty strong in one direction. A different prior would produce other results.

    There are deeper philosophical/mathematical issues. It’s been years since I looked at this, so I may get some of the details wrong. The justification for probability as belief—that is, your expectations about outcomes are captured by one number in a framework—is based on a theorem from the mid-20th century that rational actors with a prior are Bayesian. Unfortunately (see the late chapters in Berger’s ‘Statistical Inference and Bayesian Analysis’) the obvious extension of allowing classes of priors (say, all priors where half the probability mass is <0.5, or only specific points are pinned based on partial information) makes those theorems fall apart.

    Probability as a formal system has multiple interpretations. Combinatorics uses it as a means for proving the existence of certain objects, which has an entirely different philosophical underpinning. I think the proper takeaway is that probability is a tool for formulation, but not the entire basis of inference. The usual place statisticians start from today is Wald’s decision theory. In decision theory, you are interested in admissible decision procedures, i.e., those that are not everywhere worse than some other procedure. There are usually a lot of different procedures that are better in some areas than others, so you need a criterion for choosing among them. Choosing the criterion depends on what you’re doing. Unbiasedness, minimax, Bayesian, and others are all criteria. Bayesian procedures, that is, the procedures corresponding to Bayes priors) occupy an interesting place in that the set of Bayesian procedures is larger than the set of admissible procedures, but not much larger, so you can often prove things over all priors and get properties over all admissible procedures.

      1. 3

        +1 for Data and Reality, but the first edition is better, imo. IIRC, the second edition is posthumous, with a few chapters removed, and interspersed with comments from another author, which I found more distracting than enlightening. Anyway, a great reading, highly recommended, and mandatory if you’re into data modeling!

        1. 7

          IIRC, the second edition is posthumous, with a few chapters removed, and interspersed with comments from another author, which I found more distracting than enlightening.

          You’re thinking of the third edition. As Hillel wrote in that newsletter post, second edition was by Kent only.

        2. 2

          This is wildly off topic, but I guess I’m going to shoot my shot.


          Once more: we are not modeling reality, but the way information about reality is processed, by people. — Bill Kent

          I’ve been wondering for a while if the confusion about quantum physics in public discourse is because quantum physics is about modeling physics when operating at scales where measurement affects what is measured and therefore observer is inextricably linked to observation.

          Is the probabilistic nature of quantum physics just modeling that accounts for the large relative size of the tools of observation relative to the object of observation? (E.g., when you observe a billiard ball with light the mass of light is far smaller than the mass of the billiard ball so it doesn’t after the speed or positioning of the ball. If you observed a billiard ball with another billiard ball, it would affect the speed or positioning of the observed billiard ball.) Or is there something more fundamental about the nature of the model that I’m misunderstanding? Does “objective reality” exist beyond the capabilities of the model?

          (That last question is squishy as hell.)

          Nobody that I’ve talked to that’s into quantum physics seems to know much about it except as a vehicle to support their brand of metaphysics and I don’t know how to parse what you find on the internet. (Of course, the only person I personally know who I consider to be qualified to speak on the subject, a graduate level physics student, said she really didn’t know that much about it.)

          Also, I’m semi-afraid I won’t understand the answers that I receive.

          1. 2

            Coincidentally enough I have a degree in physics, though I haven’t looked at it in a while, so this answer is based on a nine-year old memory:

            It’s not a measurement error, the information literally doesn’t exist at that level of granularity. At a quantum level, particles are also waves, right? The energy of a wave correlates with its frequency, while the energy of a particle is in its kinetic energy, which correlates with momentum. The position of the particle correlates with the peaks and troughs of the wave: if the wave peaks at a certain point, there’s a higher probability the particle “is there”.

            Now, to have a precise frequency, you need a lot of wavelengths, so there’s a lot of places the particle can be. If you add together lots of different waves with different frequencies, you can get a sum wave with a single enormous peak, and the particle is localized. But now there’s many different frequencies, so the kinetic energy is uncertain.

            Mathematically, any physical observation corresponds to a transformation of the wave vector. And the position and momentum transformations don’t commute, so mathematically measuring position-then-momentum must give a different result than measuring momentum-then-position.

            It’s all weird and messy I know and I’m not doing a good job explaining it. The important thing is that this all holds regardless of how we measure it. And it leads to all sorts of weird consequences like quantum tunneling. if you put a barrier between a particle and measure the momentum very precisely, the position can get uncertain enough that it could be on the other side of the barrier.

            1. 1

              First off, thank you for taking the time to respond to this! I don’t think your reply was weird or messy at all. If you want weird and messy, keep reading!

              Now, to have a precise frequency, you need a lot of wavelengths, so there’s a lot of places the particle can be. If you add together lots of different waves with different frequencies, you can get a sum wave with a single enormous peak, and the particle is localized. But now there’s many different frequencies, so the kinetic energy is uncertain.

              This is great and I think it is core to my question. You are talking about how there is a probabilistic way to calculate the frequency, which, if I’m understanding correctly, means you can determine where the particle is as it is traveling down the path of the wave. So this is a model of the behavior of the particle.

              However, it makes intuitive sense to me that the actual, objective location of any particular particle isn’t probabilistic. (It might, however, be completely unknowable.) It just exists at some point in space at some particular time. Instead, we are using the model as a way to understand the behavior of a particle. Which is why the quote above sparked my question: are we modeling reality or the way we think about reality?

              You also said this:

              It’s not a measurement error, the information literally doesn’t exist at that level of granularity.

              Which might imply that my intuition is wrong and at the physical level particles fundamentally don’t behave the way I think they should. Or maybe you mean that the information is unknowable because of how we model the problem.

              And it leads to all sorts of weird consequences like quantum tunneling. if you put a barrier between a particle and measure the momentum very precisely, the position can get uncertain enough that it could be on the other side of the barrier.

              Again, is this a matter with the model or with reality? Are we essentially dealing with mathematical errors in the model that build up, but the particles actually have smooth trajectories in space? Or is there something more fundamentally random occurring?


              I know that at a certain point we are dealing with questions so far beyond concrete human abilities that as far as we can tell the model is reality so maybe this is really a philosophical point. Furthermore, I hope that my line of reasoning doesn’t feel totally unreasonable and unsatisfiable.


              Mathematically, any physical observation corresponds to a transformation of the wave vector. And the position and momentum transformations don’t commute, so mathematically measuring position-then-momentum must give a different result than measuring momentum-then-position.

              That’s a neat detail!

              1. 1

                However, it makes intuitive sense to me that the actual, objective location of any particular particle isn’t probabilistic. (It might, however, be completely unknowable.) It just exists at some point in space at some particular time. Instead, we are using the model as a way to understand the behavior of a particle. Which is why the quote above sparked my question: are we modeling reality or the way we think about reality?

                That’s the crazy part: when it’s not being interacted with, the particle is a wave. The particle is nowhere and the wave is everywhere. If something, anything, then interacts with the wave, it collapses back into being a particle, with either a more-defined position and a less-defined velocity or a more-defined velocity and less-defined position. It’s weird and wonderful and makes no sense whatsoever, but it’s also what’s borne out in experiment after experiment. The universe just doesn’t care about what makes sense to humans.

                @wizeman already mentioned the double-slit experiment, where a series of individual particles diffracts just like waves do. But there’s other, wilder experiments, like quantum bomb-testing. If a live bomb is guaranteed to be set off by a single photon, you can cleverly exploit wave interference to determine whether a bomb is live or not without setting it off (50% of the time).

                People have tried to reconcile QM with human intuition, and a lot of them fall under what we call Bell experiments to find more “rational” explanations. In the end the weird unintuitive QM rules always win.

                1. 1

                  However, it makes intuitive sense to me that the actual, objective location of any particular particle isn’t probabilistic. (It might, however, be completely unknowable.) It just exists at some point in space at some particular time.

                  Wouldn’t that contradict the observations of the double-slit experiment? Especially the ones that were performed by firing a single photon/electron/atom/molecule at a time.

                  1. 1

                    I don’t think I made this completely clear before, but I don’t mean that something making intuitive sense to me makes it correct.

              2. 1

                Is the probabilistic nature of quantum physics just modeling that accounts for the large relative size of the tools of observation relative to the object of observation? … Does “objective reality” exist beyond the capabilities of the model?

                This is dealt with by the field ‘foundations of quantum mechanics,’ and the experiments are fascinatingly subtle. It turns out that not only is there no such thing as negligible measurement, there isn’t a value of an observable beforehand. We generally get at that via measurements on entangled systems that we have manipulated in some clever way.

                If you really don’t want probability in the underlying theory, the usual route is to go to Everett’s interpretation, which does away with a classical observer entirely. In that case the random chance we see is because we perceive ourselves as on one lobe of a bifurcating wave function.

                1. 1

                  and the experiments are fascinatingly subtle.

                  Hence my fear of not understanding the replies!

                  If you really don’t want probability in the underlying theory, the usual route is to go to Everett’s interpretation, which does away with a classical observer entirely. In that case the random chance we see is because we perceive ourselves as on one lobe of a bifurcating wave function.

                  Oh, no! My fears have been realized!

                  My issue isn’t really with having probability in the theory.* It’s that when people talk about quantum physics I can’t tell if they are talking about the model of quantum physics or how the world actually works. I know that might sound crazy: theory is supposed to be about how the world works! But sometimes theory is an incomplete picture of how the world works. It’s a particular view that solves particular problems. That’s not necessarily how theory is always talked about and it may be completely inaccurate here.

                  You have all sorts of pop sci articles that claim that objective reality doesn’t exist. I can’t tell how seriously I should take such claims. Are they misinterpreting experiments that demonstrate the defects of the model or is there actually a deeper more fundamental question being raised here?

                  I don’t want to sit here and ignore what might be very fundamental deviations from how I view the world, but I also feel the need to clarify how I should interpret these claims before accepting them.


                  • I don’t really really have an issue here besides my own ignorance.
                  1. 2

                    talking about the model of quantum physics or how the world actually works

                    Given we haven’t (to my knowledge) got any single theory that describes all observed phenomena (from small to large scales) - I assume that anything other than experimental results is “about the model” (which is not to dismiss it!).

                    1. 1

                      I read a bit from the quantum foundations wikipedia article. It seems like all of this is still under debate and I’m not crazy.

                      From the wiki page:

                      A physical property is epistemic when it represents our knowledge or beliefs on the value of a second, more fundamental feature. The probability of an event to occur is an example of an epistemic property. In contrast, a non-epistemic or ontic variable captures the notion of a “real” property of the system under consideration.

                      There is an on-going debate on whether the wave-function represents the epistemic state of a yet to be discovered ontic variable or, on the contrary, it is a fundamental entity.

                      Thank you @madhadron for giving me the right field to look under. Thank you @hwayne for entertaining my questions. Apologies to @teymour for so thoroughly derailing the conversation.

                    2. 1

                      The pop sci articles are hard to interpret because they’re trying to bring something back to colloquial English that is deeply unfamiliar. But the objective reality does not exist parts are some of the experimental facts that any theory has to account for rather than quirks of a particular theory.

              1. 2

                Interesting. I approach it via curve fitting, and choosing the set of curves I fit to have specific interesting properties. This also generalizes nicely to things like Legendre and Laguerre polynomials and other orthonomal function bases, even as far out as things like Gegenbauer polynomials.

                1. 1

                  That is another great way to look at it. The sine and cosine waves form a basis for a certain set of functions - and infinite dimensional space. and indeed other useful spaces and bases’ exist.

                1. 3

                  I find the files that actually use Cue’s features to look foreign to me.

                  Indeed! It’s the only one that has a mental model that actually makes sense for configuration…but that mental model is quite different from templating. Once I got my head around it, though, it is one of those cases where you look back and say, “Why isn’t this how all such systems work?”

                  1. 32

                    I work at a news site. They’re going to hold the election on a certain day whether I’m ready for it or not. Obviously then the goal is to make an actionable design that can be implemented far enough in advance before the election day to do some testing and whatnot. You can call it what you like, but it’s basically a deadline.

                    Of course, lots of work I do has no deadline. The CMS is upgraded when the CMS is upgraded. So I think it’s worth distinguishing things with real externally imposed deadlines from things with internal goal dates. Goal dates can be shifted. Deadlines are not so easy to move.

                    1. 17

                      I used to work for a edtech company. If features weren’t ready by the start of the school year we’d miss our chance to get the majority of our teachers properly onboarded. Deadlines matter in software because they matter in the meatworld.

                      1. 4

                        The fact that you have to deliver a specific feature for election day is a choice made by humans, not something inevitable. The decision to develop something new before a given date instead of saying “we develop the feature and at the first useful election we use it” is a choice. Deadlines become inevitable when profit is put before software quality. Deadlines are inevitable when people that care about profit hold more power than people that care about software quality.

                        1. 24

                          Profit is more important than software quality. Without profit, people don’t get paid.

                          1. 4

                            that’s revenue. Profit is what pays your manager’s yacht. I work in a non-profit and I get paid every month.

                            Also it’s not true: in start-up economy is not really important to turn a profit. But it’s not the best driver of software quality either.

                            1. 5

                              Are you saying that deadlines happen when people care more about profit than software quality but not when people care more about revenue than software quality?

                              1. 3

                                it depends on the context and mission of the organization but in my experience, in non-profit orgs it’s much easier to reason long term compared to VC-funded startups or big corpos.

                          2. 13

                            Deadlines are also inevitable when the food is going to spoil unless we have a system for distributing it by a certain date. That doesn’t have anything to do with profits, it has to do with the food spoiling; it would spoil just as fast under any kind of system humans can come up with.

                            1. 1

                              It’s quite OT but the logistics of food greatly impact on the spoilage of food in different ways.

                            2. 7

                              “we develop the feature and at the first useful election we use it”

                              I know that Politico takes that approach because they cover lots of elections nationwide. For me, I’m only covering Pennsylvania, so there’s not really enough chance to recoup investment. If we miss 2022, there’s no telling if we’ll even be using the same CMS in 2024. It would be like trying to get the kids ready for school on Sunday morning: you can do some things, for sure, but lots of it really can only be done on Monday morning.

                              1. 3

                                I generally agree, it’s part of the system the engineer is operating inside of…they are linked. It’s the reality we often brush off or ignore. Instead, we tell ourselves (and our customers) this software was “made with 💖” and “we’re making the world a better place.” Fine attitude for a pet project you control, but when you work for a company that’s beholden to VC money or the public stock market, good luck.

                              2. 3

                                So I think it’s worth distinguishing things with real externally imposed deadlines from things with internal goal dates.

                                Failure to do this effectively has been the cause of so many Dilbertesque own-goals - including some of my own - over the several decades I’ve been part of the industry.

                                It’s doubly important for senior leaders. I’ve seen a “gee it would be nice if we had this by $DATE” turn through a series of retellings into a hard deadline, and then eventually horrify the person who spoke off-the-cuff when she discovered how that comment had been taken.

                                1. 2

                                  Indeed – this seems to.me like the difference between a deadline and good old wishful thinking!

                                  1. 2

                                    Distinguishing deadline vs goal date is a good clarification.

                                    If you look at what the article is espousing, it’s exactly what you would want to do with a real deadline: check in regularly on where you are, decide what gets done and what doesn’t get done, and bulldoze everything else out of the way.

                                    Invoking the halting problem’s a bit problematic, because we write many, many programs where we do know when they will finish. We have whole languages (Datalog) that are guaranteed to terminate.

                                    Though most of what I see in articles like this is what the agile community figured out long ago. I still haven’t figured out what the balance of what happened to agile was among sabotage, misunderstanding, or Gresham’s law.

                                  1. 15

                                    This ongoing dispute seems to consist of:

                                    Company: “I received value from something you posted for free with no warranty, so you have an obligation to make sure I keep receiving value.”

                                    Programmer: “No, I don’t. I have no relationship with you.”

                                    It’s rather like how in feudal societies, if the peasants gave the lord five loaves of bread on a special occasion one year, the lord would show up again the next year demanding them as his annual due. If companies want support, they should pay for it. Of course they don’t want to in general, and would prefer to browbeat people into treating them as a feudal lord, but the health of our society demands that they, collectively, be put in their place.

                                    1. 12

                                      There are some more subtle cases. In particular, often the company is willing to pay but the developer doesn’t have time. Sometimes in this case the company pays someone else to do the work and the developer doesn’t then want to take the patches. Sometimes the company wants to pay but has procurement processes that make it hard to pay the original developer in a way that they can accept (GitHub sponsorship can sometimes help here).

                                      1. 2

                                        the developer doesn’t then want to take the patches.

                                        They don’t have any obligation to take patches.

                                        1. 4

                                          I never said that they did, but it leaves downstream in the position of being forced to either:

                                          • Maintain a private fork
                                          • Try to run a public fork
                                          • Live without the feature

                                          All of these can be expensive. When you are in a situation where one or both parties in an exchange are unable to get what they want, you will have friction. Fault is not implied in either party.

                                          1. 3

                                            All of these can be expensive.

                                            The expense was always there, it was just being borne by others. Now it’s being borne by the user.

                                        2. 1

                                          That doesn’t seem particularly subtle. If I were selling the software and entering into contracts, I might well say, “No, I don’t want to do business with you.” How is this different?

                                      1. 2

                                        I’ve been thinking about this, and I think that there is a single structure that makes for sustainable development of decentralized community software like Matrix and Mastodon. First, no companies are involved in the core development and governance. There is a foundation that employs full time or part time engineers and funds work. The foundation’s governing board comes from instances of the software that pay membership for the board to fund its work. Member instances have to meet a set of criteria to prevent tragedy of the commons and commercial capture, e.g., must be self funding without outside patronage or ads, must be democratically controlled by its user base, must have a mature moderation system in place… The software is not open source. It is free for noncommercial, self governing communities. Any other group must pay an annual licensing fee to the foundation, but is not eligible to be on the governing board.

                                        This seems really strange if you grew up in the open source world starting in the 1990’s, but if the goal is to maintain the critical chain required for communities to continue to exist, prevent them being destroyed by malignant, moneyed interests, and not end up in tragedy of the commons, I don’t see another way.

                                        1. 2

                                          The software is not open source

                                          I don’t know of many popular programming languages with closed source compilers. If another model has worked for the majority of programming languages, why can’t it work for communication software too? What about Signal, is that not working? I seems to be funded and is still open source.

                                          there is a single structure that makes for sustainable development

                                          What does ‘sustainable’ mean in this context? It sounds to me like Matrix isn’t considered sustainable. Is IRC sustainable? Is Emacs sustainable? Or are they too simple to be considered in this discussion?

                                          1. 2

                                            If another model has worked for the majority of programming languages, why can’t it work for communication software too?

                                            Why would you expect it to? Much of the development of our open source compilers is done by companies that want to use them (Apple, Google, Facebook, etc.). Their incentives are generally not too far misaligned with the other users of them. The incentives do not align for social software.

                                            For Signal, that’s a weird one. They get enough donations to keep their small team afloat. Meanwhile developers at Facebook and Google that wrap their software and rebrand it get paid many times what Signal’s developers get paid. If Google decided they wanted Signal doing something very different, they could put enough money in easily to corrupt that direction. For now it’s a cheap resource for them to mine.

                                            What does ‘sustainable’ mean in this context? It sounds to me like Matrix isn’t considered sustainable. Is IRC sustainable? Is Emacs sustainable? Or are they too simple to be considered in this discussion?

                                            Sustainable here refers to expecting that the software can have a fairly stable team of people with departures replaced for the foreseeable future without requiring anyone to sacrifice themselves. Matrix, by their own admission, is not sustainable right now. IRC isn’t a piece of software, it’s many, and those projects have been born and died and continue to do so. Emacs is a program where the community of users are also developers of the program (to greater and lesser degrees), it has entrenched itself as a piece of identity and a status symbol for much of the community…and it also had periods in its long history when its development looked like it might be unsustainable.

                                        1. 7

                                          Just because you’re a volunteer does not mean you’re not a supplier.

                                          1. 34

                                            Just because you’re a volunteer does not mean you’re not a supplier.

                                            We have this friend Lisa who helps my family with grocery shopping (we’re blind). Last week she randomly showed up at our door and gave us a couple grocery items for free. They’re things we always have trouble finding due to … supply chain issues, post covid. She saw them at a store, bought them, and gave them to us gratis.

                                            Now that’s basically what free software is all about. It’s a gift, in a gift culture. The free software license typically describes what the recipient may do with that gift.

                                            To use my analogy above, making demands of free software developers would be like me demanding that Lisa come over to my house and cook the free food she bought us. Lisa owes us absolutely nothing whatsoever.

                                            You want work done by free software developers? You can ask nicely. You can offer to do the work yourself and contribute it to them. In both cases, they are well within their rights to say no, because they owe you nothing. You should not make demands unless you are in a contractual relationship that permits such things.

                                            On the other hand, there are demands made of us all through the social contract. If you want to put constraints on free software, this is the language that might get more traction. If you want to encourage free software developers to observe certain practices, use the language of community and social contract, not the capitalistic language of suppliers and customers. Language and the way a thing is framed really can matter. I absolutely loathe the term “human resources”. People aren’t timber forests or oil wells or gold mines. Free software developers are people, not suppliers.

                                            1. 15

                                              making demands of free software developers would be like me demanding that Lisa come over to my house and cook the free food she bought us

                                              This. Only, unfortunately, the mentality of many people has changed. The gratitude, that a developer makes open source software for free in his spare time, gives way more and more to a universal attitude of entitlement, and many users even seem to have the mistaken attitude that the developer even has to be grateful to them for using his software.

                                              1. 17

                                                It’s not an accidental change, it’s a direct result of large projects being maintained by companies, who have some sort of business model around it. This creates the expectation that there will be regular updates, prompt security fixes, good docs, etc. This in turn makes people think that software grows on trees.

                                                There is also a flipside to it: certain software is ostensibly open source (e.g. chromium), but in actuality it’s an oligopoly/monopoly that people are pretty much forced to use or rely upon. This in turn puts an enormous responsibility on those projects to actually do a good job, or they are basically squandering that mandate. They can hide behind the “it’s open source” fig leaf easily, when they absolutely shouldn’t.

                                                Volunteer developers tend to get the worst of both: they are also subject to upstream changes, but don’t have the resources to provide similar maintenance downstream. It also doesn’t help that most people don’t think about why a volunteer dev is working on a particular project:

                                                • as a learning experience
                                                • as a social club
                                                • as a loss leader for job opportunities
                                                • as a platform for delivering services

                                                So you actually need to understand a fair amount about a project to know how to approach it.

                                                Case in point, I had an exchange with a Chrome GPU dev who flippantly responded to a request with a dismissal that showed they had no idea what I was even talking about (misidentifying a 2009 feature as the latest and greatest tech). This person’s work will define what the entire web ecosystem will be able to do GPU-wise for the next decade at least. You should in fact give them an earful for that.

                                                1. 12

                                                  Ugh, this unlocked a relevant memory. I now recall asking Android developers why they did not reuse existing standard libraries like Cairo or Pixman when writing their graphics subsystem; their excuse at the time was bullshit and it was later revealed that Google ordered them to rewrite everything to avoid GPL (slides 36, 51). (Yes, I know Cairo/Pixman are MIT-licensed, but do they?) I also asked them whether they had armored their latest GPU-exporting API (“Native Client”, I think) against buffer-reuse attacks which exfiltrate data from GPUs, and their response was similar to the one that you describe. By avoiding software in the commons, Google made their product slower and less secure.

                                                  The worst part is that there’s nearly no way to contribute labor to make it better. I spent a few months working on Chromium OS and all I improved was one small chunk of graphics testing.

                                                2. 9

                                                  I don’t think you intended offense. I see you referred to Oss devs as “him” I know OSS devs that go by “her” and “they” and I don’t want to quietly erase their work. Can you use singular they in the future?

                                                3. 8

                                                  Now that’s basically what free software is all about. It’s a gift, in a gift culture. The free software license typically describes what the recipient may do with that gift.

                                                  It’s kinda weird if you think about it. If you were to explain to a lay person that companies make a living by sticking together many “free gifts” and sell the result as a product, they might declare you mad.

                                                  Nothing else in the world works this way, AFAIK. I think this is also why we’re still struggling to make sense of the weird distributions of power and wealth surrounding FOSS, because there’s no analogue that fits well enough to adopt wholesale. You could say FOSS is an economic anomaly of sorts.

                                                  1. 6

                                                    Many fields work somewhat like this, it’s just that usually the costs are different. Electronics is full of handy designs written down and distributed for free. Chemical engineering is full of handy reactions written down and distributed for free. Writing fiction is full of handy tropes written down and distributed for free. Creating and using these sorts of things are usually managed by copyrights and patents, and those are the cases where the patent system usually kinda sorta works like it’s supposed to. Lots of trivial BS gets patented, but it also does a halfway decent job of making sure that the people who put 8 or 9 figures into developing something can then get paid for it reliably.

                                                    The thing about software is that the cost to using/distributing/copying it is tiny, while in all those other fields you generally need to either physically buy and assemble components to use the design, or put in significant amounts of work massaging it into your own design, or both. So you have the fixed costs of designing and building A Thing, and then the variable costs of actually manufacturing and distributing it, and for software the variable costs are miniscule because it’s just copying around bytes. So while designing and building software is just as much work as designing and building anything else, actually “manufacturing” and transporting it is so, so cheap that you might as well give it away. Even companies routinely give away fairly large amounts of storage and bandwidth, as seen by the free tiers of Cloudflare, Github, and all sorts of other things. On the flip side there’s a number of open source hardware projects of one kind or another that happily give away schematics for free, but also just straight-up sell manufactured parts as well. Sure I could manufacture my own Pinebook Pro mainboard from the designs if I really wanted to, but it would be a metric shitload of work to just find and get all the parts, let alone assemble it. In comparison, I can download and build the firmware for said board in about three minutes.

                                                    There’s other things that go into this. Anyone can write software of arbitrarily high quality by sitting down with some relatively cheap tools and getting to work. I don’t think there will ever be a gift economy in, say, 100 ton mining trucks, because while many people can forge and weld steel in their garages, most of those hobbyists don’t have the space or tools to build machines the size of a house. I don’t think there will ever be a (large) gift economy in hand-made woodworking art either, ‘cause those things basically can’t be replicated efficiently – once you start scaling up production it ceases to be hand-made, so there’s no economy of scale to take advantage of. In comparison, I would need very, very little physical infrastructure to sit down and write the next curl or sqlite or ninja.

                                                    This is changing! I can go to Shapeways or some other 3D printing design repo and, indeed, stick together many “free gifts” and sell the result as a product. And, frankly, there are already plenty of companies that sell products that they don’t manufacture, they just do the “value add” of things like retail distribution, customer support, etc so that the manufacturer doesn’t have to.

                                                    So yeah. There’s nothing fundamentally different about software compared to any other artifact made by humans. The difference is all in the costs to different parts of the process of building and distributing it.

                                                    1. 1

                                                      Electronics is full of handy designs written down and distributed for free. Chemical engineering is full of handy reactions written down and distributed for free. Writing fiction is full of handy tropes written down and distributed for free.

                                                      I’d argue our equivalent to such things is algorithms and design patterns, rather than ready-to-reuse code. These are covered by software patents (ugh) and sometimes copyright, as well. The “copy and reuse” in writing is perhaps more like fairy tales and such that get repeated verbatim and tweaked here and there. But there the origins are lost in the mist of time, such that there’s no one clear author and if there was, they’d be long deceased by now anyway.

                                                      1. 2

                                                        The key difference is that reactions and circuit designs need no ongoing maintenance to remain useful.

                                                        Chemical reactions also make a very poor analogy because simply knowing a reaction is just the beginning of the beginning of the process. Development of equipment for conducting that reaction at a commercially viable scale is a whole different problem.

                                                        1. 3

                                                          The key difference is that reactions and circuit designs need no ongoing maintenance to remain useful.

                                                          That’s why I equated them to algorithms and design patterns, which don’t need any maintenance either. They’re a high-level description of how to do things, without actually implementing the thing in a concrete language.

                                                    2. 4

                                                      Actually I think you’re a little wrong that nothing works anything like this. The NGO sector works at least something like this.

                                                      It’s not that uncommon - especially in some non-western cultures, especially historically- for exchanges of gifts to result in loosely defined reciprocal relationships. Similarly, the trope in business circles is that in Japan contracts are much more flexible than in the west.

                                                      I do agree that this is something very odd in our current culture, and definitely a sign that our culture needs to change.

                                                      1. 2

                                                        It is worth noting that that anomaly is not an accident. The Free Software Foundation and associated parties were enacting their ideology from the start, part of which was economic in nature. That history gets forgotten a lot today, because frankly the people who felt most strongly about it weren’t good at communicating those ideals, but it was there from the outset.

                                                        1. 2

                                                          That’s a great point. I wonder how things would’ve been different if they had foreseen the current situation, and if FOSS would have become as popular as it has if they had chosen a different approach.

                                                          I mean, if you think about it, the ability to run almost any OS you like and still be able to work practically is a huge luxury. Twenty years ago, using Linux as a daily driver was like pulling teeth because there was just so much software you couldn’t run and file formats you couldn’t open. Mostly it was the web that made this possible though, the very thing that makes FOSS a paper tiger (excepting AGPL, but that’s not very popular).

                                                          If you had told people like RMS “in 40 years you will be able to run any software you like, which you can mostly inspect and modify as you wish, but you’ll mostly be using it as a viewscreen into other people’s computers running closed-off software you cannot even inspect”, I wonder how they’d respond.

                                                          1. 1

                                                            Fascinating question. I don’t claim to know but it’s interesting to think about. My own take-away is mostly that the fight continues, a lot of ground has been won but there’s a lot yet to do.

                                                      2. 3

                                                        I bet you’d be pissed if she gave you something that had a safety recall on it. From that point on, I think you’d be on notice that you can’t rely on her not to do that, and take appropriate steps to guard against that.

                                                      3. 13

                                                        The article is arguing that it does mean that. It offers several arguments. You are just stating the opposite without addressing any of those arguments. What do you think your comment accomplishes?

                                                        1. 3

                                                          I don’t think it argues any such thing, it just rests on the assumption that supply requires a relationship.

                                                          1. 4

                                                            That’s not an assumption, but a stipulation about the word ‘supplier’ as it is used in the article. The entire point is that there are legal consequences to the relationship that they describe as ‘being a supplier’.

                                                            You seem to just be bluntly applying a dictionary definition of the word ‘supplier’. But the dictionary definition of the word is irrelevant here.

                                                            1. 2

                                                              You can redefine any word to mean anything. Sure, under the author’s made up definition they’re not a supplier. Under the definition used by speakers of English in general, and especially people who talk about software supply chains, the author is a supplier.

                                                              Having clarified all of this about the word “supplier” has achieved nothing. The question is what kind of obligations, formal, informal, and legal, a person has if they make software available to other people, and what obligations the receivers have, in their capacity as both end users and as onward suppliers.

                                                              1. 3

                                                                Under the definition used by speakers of English in general,

                                                                Which is irrelevant to the discussion.

                                                                and especially people who talk about software supply chains, the author is a supplier.

                                                                That’s exactly what they are arguing they are not and you’re still just contradicting without engaging any of their arguments.

                                                                The question is what kind of obligations, formal, informal, and legal, a person has if they make software available to other people

                                                                Yes, and that is exactly what they are addressing and you aren’t.

                                                                1. 3

                                                                  You may be a volunteer, and even a supplier, but you may be free to ignore any requests from those that you supply. Not because you’re volunteer, that’s irrelevant. Because you have no legal relationship with the people using your software.

                                                          2. 11

                                                            That’s playing semantics a bit too much.

                                                            We can argue either way, but whatever you call oss maintainers, they ain’t getting paid to have the responsibilities that the “software supply chain” discourse kinda pins on them.

                                                            1. 1

                                                              Ok but your comment is way more insightful than the whole article. That said, I think the analogy is helpful: if you receive a defective product that you’ve paid for, or is supplied for for a particular purpose, your remedy is against the one who supplied it to you, not the remoter parts of the supply chain.

                                                              We need to take seriously that our incoming supply chains are primarily made of software received for free and with no warranty of fitness for purpose. That doesn’t imply oss maintainers should be more responsible - it implies that their users need to take responsibility for how they source their input software, in the same way as people shopping for responsibly sourced foods.

                                                              At the same time, it helps us to think about what kind of non contractual obligations a supplier of goods or software might have. For example, even if you’re selling hammers, you probably should be held responsible if you knowingly supply hammers contaminated with anthrax. If you put out hammers with their component clearly and correctly described, and it is well known that there are limitations to the hammer head that will make it scuff things (say), is the hammer supplier responsible or the user for economic loss from hammer scuffs?

                                                              In the OSS context, knowingly distributing malware seems off, for example, but not obviously something that users should be able to assume won’t happen. Or what about open source software that is for a specific purpose regardless of what the licence says? If I distribute an oss software library known to run on specific medical hardware, and advertise that, and the software does something bad, can I completely disclaim any responsibility for foreseeable physical harm occasioned by using an affected device? And does the person who supplied the device to the end user have any responsibility? Are indemnity clauses in the licenses effective?

                                                              I think these are all worthy questions, and point to the final supplier of software being responsible for auditing their supply chain, and choosing to use software that can be patched to use patched libraries, and be able to rebuild their whole set of dependencies from source, or accept the consequences of being dependent on third party builds.

                                                              Incidentally, Amazon does have the ability to build all their OSS dependencies from source, and I assume that the big monorepo companies (Facebook, Google) have that ability. As an aside - if you don’t have that ability but you think you have a monorepo, I think you’re just a FAANG fanboy playing dress up.

                                                              1. 6

                                                                you probably should be held responsible if you knowingly supply hammers contaminated with anthrax.

                                                                Let’s remove the knowingly and change selling to giving away hammers for free. That’s a much more accurate analogy for OSS maintainers, and in that case, I don’t think the hammer giver is responsible for anything in anyway whatsoever.

                                                                If I distribute an oss software library known to run on specific medical hardware, and advertise that, and the software does something bad, can I completely disclaim any responsibility for foreseeable physical harm occasioned by using an affected device?

                                                                Legally? Yes, 1000%. Why the everliving fuck did a manufacturer of medical devices put software in it that they didn’t fully audited? It doesn’t matter that I wrote the buggy code, I didn’t make a promise to patients that the device would work.

                                                                Now, ethically, that is shaky, specially the advertising bit. But honestly, do you see this happening, ever? Non-experts writing code for critical domains and advertising it and making money of it? I can’t think of s single example.

                                                                And to be clear, by critical I mean actually critical. Medicine, nuclear power, etc. Doesn’t matter how big your web SaaS is, if people don’t die when it fails, it ain’t critical.

                                                                Every time OSS maintainers bring up the fact that big corporations leech of their free work, people start whatabouting that maintainers have responsibilities too. It’s a valid conversation, but feels like when people say “What about Palestine?” when Jews are talking about antisemitism[^1]. Yeah, that’s fucked up, but it’s a different conversation, that has potentially nothing to do with the current one (lots of Jews disapprove of Israel’s government), and smells like an attempt at victim blaming.

                                                                [^1]: know I’m violating Goodwin’s law here, but it’s the best analogy that comes to mind.

                                                                1. 1

                                                                  Ok what if you’re giving away anthrax contaminated hammers recklessly - ie you could know if you did a normal level of diligence for a hammer supplier? Does that seem like something that should be free of liability?

                                                                  1. 2

                                                                    I found a bunch of hammers in an abandoned lot, took them home, and put them in my front yard with a sign saying “free hammers”.

                                                                    There is no normal level of due diligence that can be expected of me. I’m not a supplier of anything. I’m just giving away shit for free. Hell, I’m probably dying of anthrax myself, if you wanna push the analogy.

                                                                    Flippancy aside, though: there absolutely are OSS maintainers that might fall more in the “actually somehow a supplier, even if the product is free” side of things, but:

                                                                    1. They are the exception, not the rule
                                                                    2. There is still can’t be no legal obligation or liability to them from people that use their stuff for free.

                                                                    I’ll give you that maybe, MAYBE, if you can PROVE malicious intent, and if the maintainer in question derives significant monetary and/or societal benefits from the software they maintain, and if the damage caused was significant, then MAYBE it would be valid to assign them legal liability.

                                                                    But that never happened, ever, in the history of ever. And is still whataboutism, because that’s not the situation anyone is talking about (other than you). The article is not philosophizing about what set of rules should govern all possible relationships between OSS maintainers and the users of the software they maintain (which would be pointless, by the way: to much variance for a general case to exist). It’s talking about the here and now of how the majority of these relationships is happening.

                                                            2. 8

                                                              But they’re not volunteers either. A volunteer is someone who forms an explicit relationship with an organization to provide effort and resources for the organization’s ends. A programmer who puts code online that an organization picks up and uses is not a volunteer for that organization.

                                                              All these words attempt to imply the existence of a relationship that does not exist.

                                                              1. 2

                                                                Words are complex and have complex meanings, and the more common they are, the more meanings they have. It’s honestly mind boggling when people assume that a common word has one, and only one, narrow and exact definition.

                                                                Merriam-Webster seems to think your definition of volunteer is way to narrow, by the way (some irrelevant parts omitted for brevity):

                                                                : a person who voluntarily undertakes or expresses a willingness to undertake a service: such as […] b (1): one who renders a service or takes part in a transaction while having no legal concern or interest

                                                                I mean, the comment you are replying to has less legs to stand on than a snake, but your argument is almost worse.

                                                                1. 1

                                                                  No, if you’re giving away code for free you’re a volunteer.

                                                                  1. 3

                                                                    Not everyone giving away something for free is a volunteer.

                                                                    On the other hand a true volunteer (i.e. a person who works for an organization without being paid) is usually not allowed to give the products of this company (i.e. which includes his work) away for free.

                                                                    1. 1

                                                                      Volunteer for what exactly? Just saying ‘volunteer’ doesn’t give anyone the obligation to do whatever you think they should do. Dictionary definition:

                                                                      a person who freely offers to take part in an enterprise or undertake a task.

                                                                      (New Oxford American Dictionary)

                                                                      A person who publishes free software online can be said to be a volunteer for the enterprise or task of publishing free software. They cannot be said to be volunteering for the task of providing ongoing support and maintenance to any specific user for said software. Especially given that the software license clearly delineates that fact.

                                                                      When you throw around words that try to place obligations on others, perhaps you should consider the words carefully.

                                                                  2. 6

                                                                    Suppliers usually have contractual relationships with their customers. Just because someone uses open source software doesn’t mean there is a contractual relationship; e.g. GPL states “You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Program or its derivative works.” A user can therefore acquire and use an open source application or library without concluding a contract.

                                                                    Acceptance of the offer to conclude the open source license agreement thus generally only occurs when the user wishes to make use of the special powers arising from an open source license. In the vast majority of cases, acceptance of the agreement is “implied”, i.e. by conclusive action. However, this does not impose any obligations on the developer, neither a delivery obligation nor a maintenance obligation, and certainly not an obligation for a certain functionality or quality. This is not what is meant by supplier or supply chain.

                                                                  1. 8

                                                                    The key point in this article, buried at the end:

                                                                    humans suck at maintaining invariants. All of us. But we are capable of building tools that can help us doing that. And focusing our efforts on that has an upfront cost, but that cost is well worth it.

                                                                    This ranges from the near invisible level of our languages providing functions and stack frames so we don’t live in a sea of gotos, to the micro level of not putting your data in a corrupt state to maintaining invariants of concurrent systems to substituting modules in different members of a program family seamlessly (e.g., I want to attach a toy implementation of X for local dev or testing instead of the full X without having to do lots of plumbing in my code).

                                                                    1. -1

                                                                      People are still paying attention to OpenAI? They’ve demonstrated pretty conclusively that the pure statistical approach to natural language processing is a dead end (which was a valuable thing), but at this point any further work from them in that vein isn’t worth our attention.

                                                                      1. 2

                                                                        They’ve demonstrated pretty conclusively that the pure statistical approach to natural language processing is a dead end

                                                                        where can i read a full explanation of this?

                                                                      1. 5

                                                                        The two books I recommend, as they were two that fundamentally changed how I code:

                                                                        • Writing Solid Code - yes, it’s from Microsoft Press and geared for C, but it does contain a lot of advice that can apply to other languages, especially around designing APIs.
                                                                        • Thinking Forth - yes, it’s geared towards Forth, but again, the advice can apply to any language, and if it seems like extreme programming/agile, it’s because it is, about a decade before it hit the mainstream.
                                                                        1. 1

                                                                          ‘Thinking Forth’ is an amazing book. I read it as a teenager and still recommend it to people today even if they have no interest in Forth.

                                                                        1. 12

                                                                          This year I read Debugging - The 9 Indispensable Rules The book is more than twenty year old at this point and use as example, among other thing, a lisp server connected to Mac System via serial cable. But you are not here for that, you are here for the 9 rules. I loved it so much that I have the 9 rules on a post-it under my screen and I glance at them everytime I have to investigate something in production. I think this is an underated book, but I would agree that the advice do end up being a bit obvious/generic

                                                                          Data and Reality By William Kent is an all time favorite of mine, but the 3rd edition has been badly butchered. If you can find the 2nd edition somewhere I would go with that.

                                                                          The Mythical Man Month by Fred Brook need no introduction. What I like about the edition I have is that it also contains not only his famous “No Silver Bullet” essay, but also a commentary on “No Silver bullet” that preemptively addressed most modern take on the essay that try to disprove it. Found it interesting.

                                                                          I still love Domain Driven Design, and it still fill me with insight when I reread it. But I wish Eric Evans would update it to add the “missing” chapter on business events and also to deemphasize the OOP patterns, as a lot of people tend to focus on them rather than the more important concept of a shared language between the business and techical people.

                                                                          I have high hope that Designing Data Intensive Application will stand up to the test of time, but its too early to tell. Still a big “recommend” from me, and it will absolutely remain relevant for the next 5-10 years at least

                                                                          I have a harder and harder time recommending Clean Code, but there are still some good advice in it. Just skip the endless chapter where Martin refactor a class, the result is… not that great.

                                                                          1. 4

                                                                            Data and Reality By William Kent is an all time favorite of mine, but the 3rd edition has been badly butchered. If you can find the 2nd edition somewhere I would go with that.

                                                                            The 2nd edition is out of print and completely unavailable on ABE Books or Amazon. But you can find the 2nd edition in PDF form here and there online.

                                                                            1. 1

                                                                              +1 for Debugging. I’ve bought a few copies of it and given to people. It deserves more attention than it gets.

                                                                              1. 1

                                                                                Dang, I’ve always meant to work through Data and Reality - what’s so bad about the 3rd edition?

                                                                                1. 6

                                                                                  The first problem is that whole portion of the text were removed because they were”outdated”. This is not a book about a specific technology, so it doesn’t matter that he speak about old stuff.

                                                                                  But the most annoying part is that the modern commentary by Steve Oberman is interspersed with the original text, and it end up altering the tone of the book a lot. Kent was meditating about software, asking question to the reader and making him reflect on how his software diverge from real life.

                                                                                  Oberman is more your traditional consultant telling you his data modeling method. He “know” how to answer Kent question. This is really annoying. I believe it is the real reason he removed portion of the text: it didn’t fit his view of how things should be done.

                                                                                  A much better way to update the book would have been to put the commentary at the end of each chapter, without altering Kent text. This way, we would have the original meaning and could then compare with Oberman own take on the matter.

                                                                                  1. 1

                                                                                    I’ve only read the third edition and came away thinking it fair for an introduction to thinking about these things but otherwise lightweight. Hoberman’s interjections added useful context or explanation in a couple of places but were by and large unnecessary. Comparing it with the second edition, it’s three chapters lighter and huge swaths of others are also missing; chapter 7, Models, is almost halved.

                                                                                  2. 1

                                                                                    It really is appalling. The guy who did the ‘revision’ adds all this commentary inline…but it’s clear that he truly doesn’t understand what the author is talking about.

                                                                                1. 9

                                                                                  For simple, stable stuff I have settled on plain Javascript and MVC. It removes the need for name, a bundle, a framework, and all the rest.

                                                                                  1. 2

                                                                                    I would be curious/interested to see an example of good practices around this concept of simple. Depending of the level of complexity, I am also waiting to see where _hyperscript ends up.

                                                                                    1. 4

                                                                                      Here’s a collection of pedagogical examples: https://github.com/madhadron/mvc_for_the_web An example in the wild would be the hex editor I embedded in this page: https://madhadron.com/programming/serializing_data.html

                                                                                  1. 3

                                                                                    I’m confused about the graphic showing that you would want a Hadoop cluster for 1TB of data. That’s something you do on one machine, and not even that beefy a machine today.

                                                                                    1. 17

                                                                                      Coming from a Haskell background, many of the complaints here are things that strike me as strange. For example,

                                                                                      What really bites is when you need to change the type signature of a load-bearing interface and find yourself spending hours changing every place where the type is used

                                                                                      Or, you can make changes to load bearing interfaces and the compiler tells you everywhere you need to update something, which makes it mechanical work and removes the possibility of human error.

                                                                                      when I’m building a new feature I usually don’t have all the data types, APIs, and other fine details worked out up front. I’m often just farting out code

                                                                                      If you have a strong type system, you tend to fart around in the type system instead.

                                                                                      The expected load on this service was going to be on the order no more than a few queries per second, max, through the lifetime of this particular system

                                                                                      So why was this a microservice at all? Why not just a nice monolith where you can take advantage of the type system strongly?

                                                                                      1. 30

                                                                                        Basically, the problems that Rust is designed to avoid can be solved in other ways — by good testing, good linting, good code review, and good monitoring.

                                                                                        This sounds like a false economy to me. In my experience catching problems already at compilation stage is pretty much always a saving over catching bugs later in the life cycle – especially if it goes into production and is caught via monitoring. It might initially feel like higher velocity, but as the code complexity grows, it really isn’t.

                                                                                        1. 16

                                                                                          I’m reminded of the facetious adage from science that “a year in the library can save a day in the lab.”

                                                                                          1. 11

                                                                                            You have reminded me of one of my favorites, which is, “weeks of programming can save you hours of planning.”

                                                                                        1. 29

                                                                                          Why Twitter didn’t go down … yet

                                                                                          I was hoping for some insights into the failure modes and timelines to expect from losing so many staff.

                                                                                          This thread https://twitter.com/atax1a/status/1594880931042824192 has some interesting peeks into some of the infrastructure underneath Mesos / Aurora.

                                                                                          1. 12

                                                                                            I also liked this thread a lot: https://twitter.com/mosquitocapital/status/1593541177965678592

                                                                                            And yesterday it was possible to post entire movies (in few-minute snippets) in Twitter, because the copyright enforcement systems were broken.

                                                                                            1. 5

                                                                                              That tweet got deleted. At this point it’s probably better to archive them and post links of that.

                                                                                              1. 11

                                                                                                It wasn’t deleted - there’s an ongoing problem over the last few days where the first tweet of a thread doesn’t load on the thread view page. The original text of the linked tweet is this:

                                                                                                I’ve seen a lot of people asking “why does everyone think Twitter is doomed?”

                                                                                                As an SRE and sysadmin with 10+ years of industry experience, I wanted to write up a few scenarios that are real threats to the integrity of the bird site over the coming weeks.

                                                                                                1. 12

                                                                                                  It wasn’t deleted - there’s an ongoing problem over the last few days where the first tweet of a thread doesn’t load on the thread view page.

                                                                                                  It’s been a problem over the last few weeks at least. Just refresh the page a few times and you should eventually see the tweet. Rather than the whole site going down at once, I expect these kinds of weird problems will start to appear and degrade Twitter slowly over time. Major props to their former infrastructure engineers/SREs for making the site resilient to the layoffs/firings though!

                                                                                                  1. 2

                                                                                                    Not only to the infra/SREs but also to the backend engineers. Much of the built-in fault-tolerance of the stack was created by them.

                                                                                                2. 2

                                                                                                  https://threadreaderapp.com/thread/1593541177965678592.html

                                                                                                  I have this URL archived too, but it seems to still be working.

                                                                                                  1. 1

                                                                                                    hm, most likely someone would have a mastodon bridge following these accounts RT-ing :-)

                                                                                                  2. 2

                                                                                                    FWIW, I just tried to get my Twitter archive downloaded and I never received an SMS from the SMS verifier. I switched to verify by email and it went instantly. I also still haven’t received the archive itself. God knows how long that queue is…

                                                                                                    1. 2

                                                                                                      I think it took about 2 or 3 days for my archive to arrive last week.

                                                                                                  3. 2

                                                                                                    oh, so they still run mesos? thought everyone had by now switched to k8s…

                                                                                                    1. 13

                                                                                                      I used to help run a fairly decent sized Mesos cluster – I think at our pre-AWS peak we were around 90-130 physical nodes.

                                                                                                      It was great! It was the definition of infrastructure that “just ticked along”. So it got neglected, and people forgot about how to properly manage it. It just kept on keeping on with minimal to almost no oversight for many months while we got distracted with “business priorities”, and we all kinda forgot it was a thing.

                                                                                                      Then one day one of our aggregator switches flaked out and all of a sudden our nice cluster ended up partitioned … two, or three ways? It’s been years, so the details are fuzzy, but I do remember

                                                                                                      • some stuff that was running still ran – but if you had dependencies on the other end of the partition there was lots of systems failing health checks & trying to get replacements to spin up
                                                                                                      • Zookeeper couldn’t establish a quorum and refused to elect a new leader so Mesos master went unavailable, meaning you didn’t get to schedule new jobs
                                                                                                      • a whole bunch of business critical batch processes wouldn’t start
                                                                                                      • we all ran around like madmen trying to figure out who knew enough about this cluster to fix it

                                                                                                      It was a very painful lesson. As someone on one of these twitter threads posted, “asking ‘why hasn’t Twitter gone down yet?’ is like shooting the pilot and then saying they weren’t needed because the plane hasn’t crashed yet”.

                                                                                                      1. 8

                                                                                                        Twitter is well beyond the scale where k8s is a plausible option.

                                                                                                        1. 2

                                                                                                          I wonder what is the largest company that primarily runs on k8s. The biggest I can think of is Target.

                                                                                                          1. 3

                                                                                                            There’s no limit to the size of company that can run on kube if you can run things across multiple clusters. The problem comes if you routinely have clusters get big rather than staying small.

                                                                                                            1. 1

                                                                                                              Alibaba, probably.

                                                                                                              1. 1

                                                                                                                Oh, I didn’t realize that was their main platform.

                                                                                                              2. 1
                                                                                                                1. 2

                                                                                                                  I was thinking about that too, but I’m guessing that CFA has a fraction of the traffic of Target (especially this time of year). Love those sandwiches though…

                                                                                                            2. 2

                                                                                                              Had they done so, I bet they’d already be down :D

                                                                                                              1. 1

                                                                                                                I work at a shop with about 1k containers being managed by mesos and it is a breath of fresh air after having been forced to use k8s. There is so much less cognitive overhead to diagnosing operational issues. That said, I think any mesos ecosystem will be only as good as the tooling written around it. Setting up load balancing, for instance . . . just as easy to get wrong as right.

                                                                                                            1. 11

                                                                                                              For those who don’t want to watch the video, the trick is to nest blocks, where the title of a given tree level is in the element and the subtree consists of more details elements outside the summary, and use the rule

                                                                                                              details > *:not(summary) { padding-left: 2em; }
                                                                                                              
                                                                                                              1. 4

                                                                                                                The phrase “software crisis” rubs me the wrong way. I’m reminded of this classic from Bertrand Meyer which points out how much working software we take for granted.

                                                                                                                It looks like this book might be worthwhile though.

                                                                                                                1. 2

                                                                                                                  For it to be a crisis, it should also be something that’s new and temporarily, right? As far as I know, issues with software quality have been there forever.

                                                                                                                  But Bertrand Meyer is also being a bit disingenuous - yes, we have a lot of software that’s nice, but if that’s the (say) 10% that survives and thrives there’s still 90% of projects that fail in some way or another (if it’s not technically, then financially/time-wise). It’s survivorship bias to say that everything’s fine.

                                                                                                                  1. 3

                                                                                                                    The software crisis was originally coined because electrical engineers were convinced that software should be the trivial, easy part of computing, and reality kept refusing to bow to their preconceptions.

                                                                                                                1. 3

                                                                                                                  As others have pointed out, this is not yet a container. This is a chroot’d environment. You also need to isolate the process tree into its own kernel namespaces (see https://en.wikipedia.org/wiki/Linux_namespaces).