Threads for friendlysock

  1. 21

    I wanted to leave a hatted comment in the mod log, and there happens to be a usefully ironically story at #1 on the homepage I can use. We just had a couple hours of downtime because the disk on our one (medium vm) server filled with logs. It looks like logrotate broke a few months ago. Sorry for the outage. As a reminder, the chat room is also where we post status updates during outages.

    1. 14

      Not big enough

      1. 0

        Womp womp.

      1. 18

        Most companies are not using cloud as a replacement for colo. RDS, SQS, S3, managed elasticsearch, etc are really really valuable and difficult to replicate on your own. Of course the cloud vendors want to lock you in to these services and then overcharge you for the basics, just like some grocery stores lure you in with cheap specialty foods and then overcharge for bread and milk. It doesn’t mean it’s a bad deal though.

        1. 18

          RDS and S3 are standouts in part because the lock-in is operational, not architectural.

          You can develop against vanilla PostgreSQL, deploy on RDS, then change your mind – or at least threaten AWS with a change at contract renewal time – and switch to Fly.io’s managed Postgres. (Or any of the other excellent hosted offerings.) Or go “on-prem”, “edge”, etc. (I.e., run your own servers.)

          S3 was a moat but the API + domain model are now available from your choice of vendors, including Minio if you want to roll your own.

          I’m far more suspicious of applications that make heavy use of SQS, DynamoDB, etc. without having a really strong proof they need that scale and all the distsys pain it brings. You can get a long way on Celery (or your choice of “worker” tools) running batch jobs from your monolith against a “queue” table in Postgres. IME most projects/companies fail long before they outgrow the “COSP” threshold.

          For cost management, disaster recovery and business continuity, and the ability to work + test your systems offline, I think minimal cloud provider API surface in your application is a Good Thing. That + “don’t create a ton of microservices” (also good advice in most cases) usually implies: monolith + one big backend database + very select service extractions for e.g. PII that shouldn’t sit in the main DB.

          1. 3

            I think you nailed it here:

            the lock-in is operational, not architectural.

            1. 1

              You can develop against vanilla PostgreSQL, deploy on RDS, then change your mind – or at least threaten AWS with a change at contract renewal time – and switch to Fly.io’s managed Postgres.

              How does this work with security, though? Fly.io’s managed Postgres is going to be open to the internet, presumably, whereas in AWS I can control (and log) network access as I see fit.

              1. 3

                fly.io postgres is very much not open to the internet unless you need that for some reason.

                1. 2

                  I think Fly has a pretty good story here, actually: https://fly.io/docs/reference/private-networking/

                  But really, any managed DB vendor is going to have better network controls than “just use pg_hba.conf”. Most even offer AWS VPC bridging.

                  1. 1

                    Thanks for the link. I was maybe thinking of Supabase when I wrote the comment. Like if the business is providing managed databases but no compute then doesn’t the database basically have to be open to the internet so the backend servers can reach it? Eg talking to Supabase from Vercel or Netlify? Or can something clever be done with eg Wireguard to secure it all?

                    1. 1

                      There are a few approaches that services like this take. Sometimes they provide access over a VPN (e.g. through Wireguard, this is what Fly.io managed Postgres does if you connect from a non-Fly.io service and how you connect to private RDS databases from outside AWS), and sometimes they do just have a database listening on an Internet IP/port (maybe secured by some IP whitelisting, usually secured by TLS, and definitely secured by username/password authentication; this is what DigitalOcean managed databases, Supabase direct connections, and public RDS databases do)

              2. 3

                I guess it goes without saying that if you

                • need 99,99+% uptime and want to sueblame somebody big otherwise
                • need a distributed database for a ton of access that “Just works”
                • want a “familiar” stack where you can just slap some specific product of the three letter company as a requirement in the job description

                … then go to the big cloud providers and pay your premium (be aware of the network and database per-operation fees), you already made up your mind.

                But I’d bet that are maybe 1% of the customers.

                1. 9

                  need 99,99+% uptime and want to sueblame somebody big otherwise

                  I haven’t checked in a while, but I’ve never seen a cloud service actually meet this 99.99+% uptime. I don’t think any of them are very transparent about their historical outages anymore as they realized they weren’t having good uptime performance.

                  I checked a few years ago for $WORK, when some boss type wanted to move to the cloud, I compared out all the data I could gather from the various cloud providers and we handily beat them in uptime and total cost across time. I think I went back 5-ish years at the time, though I can’t seem to find that spreadsheet at the moment.

                  I agree there are valid reasons to move, but I would never blindly recommend switching dedicated stable compute to the cloud. Bursty compute however is a perfect fit for the cloud, and easy to recommend.

                  1. 1

                    I’m always worried about comparisons in uptime to someone’s single company to big clouds. AWS will have both more issues and more varied ones, but they’ll be often limited in scope. It’s hard to compare it to a smaller local setup without a list of specific risks and expected time to recovery. At an extreme, the box under my desk at home had 100% uptime in the last few years, but I wouldn’t make decisions based on that.

                    1. 3

                      I agree a single companies uptime comparison vs cloud providers isn’t very useful to outsiders, but it can be useful in that single companies decision making. That’s why we did the comparison.

                  2. 14

                    need 99,99+% uptime and want to sueblame somebody big otherwise

                    More importantly, don’t want to pay for in-house expertise to manage the systems when it is not part of their core competency. For smaller companies, they often need 10% of a very qualified sysadmin. They can either hire a full-time one for 10x the price of what they actually need, or outsource to a cloud provider and, even if the markup is 100%, be paying 80% less.

                    need a distributed database for a ton of access that “Just works”

                    The ‘Just works’ bit is far more important here than the ‘distributed’ or ‘ton of accesses’ part, because it translates to not having to pay an administrator.

                    want a “familiar” stack where you can just slap some specific product of the three letter company as a requirement in the job description

                    Again, this is a cost-saving thing. It’s much easier to hire in-house talent or to outsource a particular project to a (small or large) company if the infrastructure that they’re building on is generic and not something weird and bespoke that the developers would need to learn about.

                    In a huge number of cases, the cost of the infrastructure (cloud or on-prem) is tiny in comparison to the cost of the people to manage it. Using the cloud lets the provider amortise the cost of this over millions of customers and pass on a big chunk of that saving to you.

                    Buying a big server has a few drawbacks. If any hardware component fails, then you need to RMA that part, which means you need either an expensive support contract or you need someone on staff who is competent to identify the faulty component and send it back. If a cloud server fails, then your VM is restarted on another machine. If you are using PaaS offerings then someone else is responsible for building a platform that handles hardware failure and you don’t even notice.

                    If you want a separate test and production version, then you need at least two of those big servers, whereas with even IaaS offerings it’s trivial to spin up a clone of the production server for a test deployment on a different vnet and if you’re using PaaS then it’s even easier, and the number of test instances can easily scale with the number of developers in both cases.

                    TL;DR: If you think the cost of the hardware is important then either you’re thinking about massive deployments or you completely misunderstand the economics of this kind of thing.

                    1. 13

                      In my experience the companies I have worked for tend to end up at least doubling their spend when moving from dedicated to cloud for little added benefit and almost the exact same maintenance burden, in one case a company I worked for they went from £3,200/year spend on a managed 24-core/112GB RAM dedicated box with 1 hour SLA on having a tech at the datacenter make changes/do maintenance/etc to ~£1,400/month spend on far less resource except now they now had to handle the server changes/maintenance in house on top of managing the cloud infra which actually required hiring someone new to handle.

                      For my own company we rent two dedicated boxes (16-core/64GB RAM each) at a total cost of £108/mo which provides more than enough capacity, and in the past six years has had five nines uptime while costing a fraction of what it would have to go with cloud.

                      1. 1

                        now had to handle the server changes/maintenance in house

                        I’m not sure I understand. What server maintenance are you doing for a cloud based servers that’s comparable to the dedicated one?

                        with 1 hour SLA on having a tech at the datacenter make changes/do maintenance/etc

                        That’s 1h SLA to having someone look at the issue, not for a working replacement, correct?

                      2. 11

                        A couple of nits, directly:

                        More importantly, don’t want to pay for in-house expertise to manage the systems when it is not part of their core competency.

                        I would argue that managing systems is a core part of developer competency, and I’m tired of people acting like it’s not–especially when those people seem to frequently employed by companies whose business models depend on the meme of systems administration being some black art that can only be successfully trusted to the morlocks lurking in big data centers.

                        Using the cloud lets the provider amortise the cost of this over millions of customers and pass on a big chunk of that saving to you.

                        This is manifestly not what’s happening, though, as we’re seeing. The savings are being passed on to the shareholders–and if they aren’t, we should all be shorting MSFT and AMZN!

                        If you want a separate test and production version, then you need at least two of those big servers

                        Or, you know, you host both things on the same box under different VMs, or under different vhosts. This has been a problem with a well-known solution since the late 90s (though sadly not reliably applied).

                        you completely misunderstand the economics of this kind of thing.

                        Well…

                        • We’ve seen figures in this very thread of at least a 2x price increase using cloud providers.
                        • The option typically doesn’t exist to not have a sysadmin–we just hire “devops” people now, who are okay sysadmins who also tend to spend most of their time functioning as embedded salespeople for the vendor of their preferred stack. We’re out a six-figure salary regardless.
                        • If your team opts not to have a sysadmin (!), a bare metal or rented dedi is a lot easier to understand and maintain since it basically looks like a developer machine–just beefier and with actual paying customers on it.

                        I submit that perhaps we aren’t the only ones who misunderstand the economics. :)

                        ~

                        To be clear, there are some things like S3 that I just cannot be arsed to host. Hosted Postgres is nice when you don’t want to bother setting up metrics and automatic backups–but then again, I’m pretty sure that if somebody wrote a good script for provisioning that or a runbook then the problem would go away. It’s also totally fine to keep a beefy machine for most things and then spin off certain loads/aspects to cloud hosting if that’s your kink.

                        Remember, there was a time when the most sensible thing was to send your punchcards and batch jobs down to the IBM service bureau, because it was more economical. These things go in cycles.

                        1. 8

                          Addendum, reading back over this:

                          The more I think about this, the bigger issue is probably that if you run your own infra there’s the requirement that there be some continuity of ownership and knowledge–and that is difficult in an industry right now where average tenure is something like less than two years for startups.

                          1. 2

                            Most of my career so far has been, essentially, cleaning up somebody else’s historical mistakes by paving over them with my soon-to-be historical mistakes. An endemic part of the problem is always that very specific and arcane parts of the system are forgotten, or stop being understood, as the flow of brains does its thing. I used to be in camp “rewrite”, a decade ago. I’m now firmly in the camp “nooooooooo, fix it, please don’t do this to me, please please please fix it”

                            1. 2

                              I’m honestly dumbstruck by how obvious this is once it’s pointed out explicitly.

                              Even when I started out 15+ years back, I had the distinct impression that traditional “ops” roles tended to have far higher average tenures than developer roles.

                            2. 3

                              I would argue that managing systems is a core part of developer competency

                              I am not talking about developers, I am talking about companies. Most big cloud customers are not software companies, they are companies that have some in-house infrastructure that is a cost centre for their business: it is a necessary cost for them to make money, but it is not the thing that they make money from. They may employ some developers, but managing infrastructure and writing code are different (though somewhat overlapping) skill sets. Importantly, developers are not always the best administrators and, even when they are, time that they spend managing infrastructure is time that they are not spending adding features or fixing bugs in their code.

                              For a lot of these companies, they outsource the development as well, so the folks that wrote the code are contractors who are there for a few months and are then gone. An FTE sysadmin is a much higher cost.

                              This is manifestly not what’s happening, though, as we’re seeing. The savings are being passed on to the shareholders–and if they aren’t, we should all be shorting MSFT and AMZN!

                              That doesn’t follow. If it costs 100 times as much to manage 1000 machines as it does to manage one, then a company that passes on half of the saving to their customers will still be raking in cash. The amount that it costs to maintain a datacenter of a few tens of thousands of machines with a homogeneous set of services running in large deployments across them is vastly less that the cost of each customer maintaining their own share of that infrastructure.

                              We’ve seen figures in this very thread of at least a 2x price increase using cloud providers.

                              The numbers I’ve seen there are comparing hardware cost to hardware cost, which ignores the bit that’s actually expensive. They’re also talking about IaaS, which does not get most of the savings. And they’re talking about companies with steady-state loads, which is where IaaS does the worst. Renting a 64-core server is probably more expensive than buying one (a cloud vendor will pay less for it by buying in bulk, but that’s not a huge difference, and they want to make a profit). The benefit that you should get from IaaS is that you can move between a 2-core server and a 64-core server with a single click (or script) so that you can scale up for bursts. If you are a shop with a trickle of sales across the year and 100 times as many on cyber monday, for example, then you might need a 64-core system for 2 days a year and be happy with a 2-core machine the rest of the time. Comparing buying and renting a 64-core machine for the entire year is missing the point.

                            3. 2

                              Not just small companies. Larger companies often have terrible tech ops. Moving to ops as a service can be a way to fix that, though there is the danger that your existing ops people and processes will infect what you do in the cloud and either prevent you from getting the advantages or even make it worse than what you had.

                            4. 5

                              Interesting, it didn’t occur to me that only 1% of customers would want good uptime they’re not responsible for, a reliable database, and an easy to match watch-word for hiring.

                              1. 4

                                I’ve got 99,99 SLA one some tiny box at some irrelevant hoster in germany, with a downtime of 1 hour in 10 years when the whole box died (was up then again in 1hour on another system). So you could say I’ve got my 99,9% without any failover.

                                If that’s possible for a normal company with only some KVM + guaranteed CPU, RAM and bandwidth, you may not need the big(tm) cloud for that same hardware.

                                1. 4

                                  I have seen far more (and longer) outages caused by messing up with cloud systems than by hardware failure.

                                  Some examples I have personally seen:

                                  • Autoscaling policies based on CPU load / memory causing outages when load patterns shift
                                  • Brief but frequent “elevated error rates” caused by insufficient wait periods on scale-in events
                                  • Network speed degradation in AWS causing application outages
                                  • Cron-triggered script to terminate/delete un-tagged resources (to ensure people were tagging things for cost control purposes) ran during an outage of the AWS tagging service. All resources were reported as un-tagged and 30% of instances were terminated before it killed the instance it was running on.
                                  1.  

                                    also I didn’t mention it, but I’ve got a 24/7 hotline in case my system is down, won’t pay anything as long as it’s not my fault (then I’m billed for every 15 minutes), and I did use it at one sunday when the network latency spiked

                              2. 2

                                need 99,99+% uptime and want to sueblame somebody big otherwise

                                Many companies and even just clubs and stuff had that kind of uptime long before cloud providers even were a thing and if you look at guarantees from cloud providers you will generally not find more guarantees than what most companies provide. While cloud providers have more staff they also have way more complexity than smaller companies, bringing their own kinds of outages and every now and then you hit limitations of managed services, need to upgrade cause they decided to change something, which can be less planable than in your own company. And good luck if you hit some bug based on the particulars on how you use the service and going through layers of support lines, unless you are really big - big enough to easily do most stuff in-house.

                              3. 2

                                Elastic Search I set up ten years ago on physical machines and was fairly trivial. I think early on that was one of their main selling points. We even helped a very big bank to set it up on their infrastructure. When we came over to discuss any remaining topics they were done and built their own orchestration around it. Fun fact they built basically their own Nomad/Kubernetes and I think it was largely shell script (not completely sure though!). I don’t know how it is these days though.

                                S3 is pretty easy to replace and low maintenance with things like minio and seaweedfs.

                                And also, if you ever run any serious setup where you (think you) need the cloud you will certainly end up troubleshooting issues on the managed services, but only after scraping together enough evidence that it’s their issues. Even more fun when you have to go through their partners first. So you need people that are both experts in the field, but also experts with the particular cloud providers. So, in any capacity where you think you might actually need cloud providers you certainly need people that could easily set things up on their own. And that is why you can make a ton of money DevOps jobs, if you like doing that. There’s always need.

                                But even if you happen to never run into any of these problems. You usually need experts for technologies you use, way before your standard server setup is even close to limit you somehow. And usually it’s not a clear cut how much they need to know. So they will certainly know how to run these technologies. Again, that’s if you don’t run into any issues with your cloud provider’s setup and that at some point will happen, even with Amazon and Google. After all they also run physical hardware, have tons of management infrastructure that also can have bugs, have situations that their monitoring doesn’t detect.

                                The biggest thing is that you can blame them, but then you need to be able to proof it, which can be really hard at times, especially if you don’t know their setup.

                                I think there is a lot of “right sounding” things said about cloud computing, that also typically aren’t inherently wrong, but still at best only apply to the practical reality to a certain degrees and cloud providers would be stupid not to make statements based on that and people wanting to get DevOps jobs, do consulting, sell books do the same. I think it’s rarely intentional though. It’s just easy to make a generic true-ish statement to justify what we do. But that goes into psychology.

                                1. 2

                                  That’s the thing. There are a small number of companies whose domain/problem space is such that they can 100% avoid lock-in by treating cloud instances strictly as VMs and running all their own services, but as your needs grow that can be SUPER hard to maintain without a sizable investment in engineering which not every company is willing to make.

                                  Maybe they should? But they aren’t.

                                1. 4

                                  Caveat emptor: a really large chunk of these libraries, if I’m not mistaken, are “just” Rust bindings (GDAL, for example). The ergonomics are likely a lot better, but the safety benefits not so much.

                                  1. 6

                                    See also OpenTitan which I think has similar security goals but without Pluton’s Windows lock-in. I’m not familiar enough with either to know how much they’re similar/different.

                                    1. 15

                                      [ Disclaimer: I work for Microsoft and have collaborated with the Pluton team on some things, but am not involved in the push to put Pluton everywhere and only talk to the Windows team occasionally. ]

                                      I am familiar with Pluton and have skimmed some of the OpenTitan docs and the docs of some hardware roots of trust from other vendors. They are all very similar: they provide some key management and hardware crypto functionality as fixed-function units and a general-purpose core that’s tightly coupled and provides some more user-friendly (or, in the case of the TPM, standard but pretty awful) APIs for the rest of the system. Pluton has a fairly rich policy language for keys (for example, allowing a key to be used only as input to a KDF and then imposing policies on what the derived key can be used for) and some neat mitigations against physical attackers that, apparently, I’m not allowed to talk about (any time you talk about a particular security defence publicly, it motivates a load of people like @saaramar and his friends to go and try to break it), but it’s not massively different from any of the alternative implementations.

                                      A hardware RoT is basically table stakes for a modern computer now. Apple has one, Android phones have them. TPMs have been around for a while, but they generally fall into two categories of suckiness:

                                      • Firmware TPMs are implemented as firmware on the main core. They run in a higher privilege level, but they share all of the execution resources with the main system. TPM operations trigger SMC calls or similar. These are often vulnerable to side channels that allow the keys to be exfiltrated. If the goal is to keep the user’s keys (for WebAuthn and so on) safe from a compromised OS, this is a big problem.
                                      • Discrete TPMs are separate chips that are connected to the motherboard. A lot of these just plain suck from a reliability perspective. If a user encrypts their disk with BitLocker and the TPM dies, then they’re stuck if they haven’t properly backed up the recovery keys. Users complain about this a lot. The other problem with discrete TPMs is that they’re connected by writes on the motherboard and so they’re very easy to lie to. An attacker who stole a laptop can easily boot an OS that is allowed to access the disk encryption keys, record everything that the CPU says to the TPM, then boot another OS with the TPM disconnected, replay the messages from the CPU, and unlock the keys.

                                      This means that, for security, you really want a separate core (so isolated from side channels) that’s on package (so hard to physically attack without destroying it). Apple and Google both know that, which is why they put such a thing on their devices. Both Google and Apple have a lot more control over their respective hardware ecosystems than Microsoft, so can do this much more easily.

                                      I strongly suspect that if Intel and AMD had built decent (secure, reliable) on-package TPM implementations then there wouldn’t have been so much of a push for Pluton.

                                      1. 14

                                        Both Google and Apple have a lot more control over their respective hardware ecosystems than Microsoft, so can do this much more easily.

                                        How about considering that it’s bad for any one company to have complete control over an ecosystem? It’s good that microsoft feels left out for not controlling the PC ecosystem. It’s bad that google and apple dictate what users can and cannot do with their devices.

                                        1. 3

                                          One of the things I enjoy about your posts is that you’re an ardent advocate for freedom and open-ness in computing but you seem to be reasonable about it, so here’s a question I hope you’ll read in the spirit it was meant rather than an attack:

                                          What would your ideal solution look like in this space? Do you think it would be possible to implement solutions LIKE this in broad concept (a verifiable chain of trust from boot) but that were vendor independent?

                                          1. 3

                                            heh, thanks :)

                                            This is a really good, and fair, question! I’ve thought about this a fair amount, but I’m definitely not an expert and am easily confused by the many acronyms e.g. from the article. Anyways, from what I can tell, I think having an extra chip, etc is fine. An ideal solution in this space might be something similar to what they are pushing, but treats the 4 user freedoms[1] as a first class citizen. Like, I understand that you don’t want bad actors to be able to replace keys or whatever, but it shouldn’t be impossible to do that, and microsoft shouldn’t be the gatekeeper. I understand that you don’t want ‘tampered’ devices to join your network or play your game (because, omg cheaters!!…), but the mechanism used to verify that should allow for exceptions where employees / students / users can use other sysadmin/IT/department-“approved” operating systems, not whatever microsoft says is “trusted’.

                                            This pluton thing seems to run non-free firmware, with 0 chance of me or anyone else being able to build fw for it and use it. The drivers and whatever userspace components required for this thing also seem to be non-free, and windows-only. And if microsoft kills the 3rd party CA for secure boot, then it’s suddenly impossible (I think?) to boot anything else but windows. Pluton is 100% microsoft / windows centric, so if it works with anything outside of their products then it’s a bug / coincidence, basically.

                                            Maybe I’m being overly cynical, but this seems like start of the “Extinguish” phase of EEE… Microsoft: “you don’t need to install Linux, *BSD, whatever anymore, you can run the same userspace under Windows with WSL now! So no one should have a problem with these changes!” OEMs: “Yeah!”

                                            Anyways, I can’t really go into any specific about how I’d come up with something better, since a lot of the technicalities are waaaay over my head. My main beef with this is it’s microsoft doing what microsoft has always done for the last 30+ years, non-microsoft customers be damned. Thanks for the message though, I want to keep thinking about your question, because it’s spot on… this pluton stuff does attempt to address some real problems (though I’d argue that combatting game cheaters by throwing away user freedoms is not a real problem/solution), and folks are not going to easily dismiss pluton if the alternative is “do nothing” about the real problems it does attempt to address.

                                            1. https://www.gnu.org/philosophy/free-sw.en.html#four-freedoms
                                            1. 1

                                              Honest question: Other than the Pinebooks and the System76 machines, how many computers buyable by consumers on the market today actually meet these criteria?

                                              The Lenovo laptops many Linux fans prize have proprietary binary blobs all through them as far as I understand.

                                              I love the principles you’re citing here, I’m just curious how pragmatic it is for many people to actually live by them.

                                              1. 3

                                                I’m replying to you now on a Librem 14 laptop, which runs coreboot and has had the ME “nuetered”. The CPU is an Intel skylake variant (coffee lake I think?), because I believe later CPU generations require even more non-free firmware and I don’t think Purism has figured out how to proceed there. There’s also the Framework laptop (and recent announcement from HP), but those run more non-free blobs. And I think Dell is still selling their XPS 13 with Linux pre-installed. But as I mentioned in the Pluton article comments, being able to install Linux isn’t really helpful for promoting/realizing the 4 freedoms and such. On the bright side, there are so many laptops shipping with Linux today than I ever remember in the past. On the other hand, this may be the peak of the “golden age” of having multiple choices for an out-of-box Linux system :(

                                                Ya, the situation now is becoming less and less ideal. And the “free software or bust” community isn’t big or strong enough to counter this movement. We need legislative action to help.. guide chip factories and OEMs, which (IIRC you’re in the US), isn’t going to happen here :P

                                                1. 2

                                                  Exactly.

                                                  I feel like the only REAL thing we can do other than shaking our fists and venting on the various forums is vote with our wallets and try to convince others to do the same.

                                                  1. 2

                                                    groans

                                                    We can build software and compete with these clowns. The chips are coming (it takes a stupidly long time to go from idea to product in the chip world) and we’ve got to work on software distribution models that are democratic and can be trusted. I feel very ignored.

                                                    1. 1

                                                      Great! Sincerely, I would love love LOVE to see this happen!

                                                      The problem I see is that the way we currently allocate resources in a capitalist society is to put dollars towards engineering hours.

                                                      Volunteers can move mountains, but at the end of the day even the most virtuous free software advocate has to keep a root over their heads and feed themselves.

                                                      It’s a hard problem.

                                                      1. 2

                                                        Yeah exact. We need to pay people and we can only do it by making up our own money… only way people will accept this money is if it is perceived as legit (i.e. has to be persuasive and that can only happen if enough people are defending the definition).

                                            2. 2

                                              HEADS is already that to some extent, you can already have a nitropad (but yes this goes further than that by having the chip in the processor and hiding the keys better but in principle it is the same, someone already mentioned [in this thread] how you could maintain a HEADSy model with this kind of tech … personally I think there are other attack surfaces to think about before over committing to this aspect).

                                          2. 13

                                            A hardware RoT is basically table stakes for a modern computer now.

                                            This is not a universally-held opinion, especially given the inability to independently verify the correctness of such hardware. TPM manufacturers have not been forthcoming with the community.

                                            1. 7

                                              Just because Apple and Google have monopolistic control on their devices, does that mean Microsoft does too? I agree with the OP, that contributing to an open, libre platform, would garner more trust and transparency and not let Window Update be the arbiter for changes to the unit.

                                              While the article can be seen a bit as a slippery slope, the thought exercise is valuable to consider what could happen and I don’t see a good reason why we should trust what the vendors are doing. I recently purchased a laptop and while a coworker in the EU could buy his device without Windows, my region had no such option. If these features are in the future a requirement to ‘use’ the device, I sure as heck better be able to opt out of Windows—and not just at a checkout, but after buying a used device as well. Just as I wish it were less of a hassle to set up a de-Googled custom ROM of Android, I want the laptop/desktop space to remain ‘hackable’ for the consumer.

                                              1. 4

                                                Just because Apple and Google have monopolistic control on their devices, does that mean Microsoft does too?

                                                No, just because a hardware root of trust is an absolute minimum security posture for all competing devices means that Windows devices should provide one too.

                                                If these features are in the future a requirement to ‘use’ the device, I sure as heck better be able to opt out of Windows—and not just at a checkout, but after buying a used device as well. Just as I wish it were less of a hassle to set up a de-Googled custom ROM of Android, I want the laptop/desktop space to remain ‘hackable’ for the consumer.

                                                Pluton-enabled devices are far more friendly than Android devices in this regard. You can toggle a configuration switch to use the other root cert and then there’s a process that’s used by a load of Linux vendors to get their copies of GRUB signed and to boot Linux with a full secure boot chain. If you boot from a dm-verity root fs, then everything in the root filesystem is similarly protected. Pluton then exposes TPM 2.0 APIs to Linux, which can then use the secure boot attestation to authorise disclosure of keys for LUKS and mount a dm-crypt (+dm-integrity)-protected mutable filesystem.

                                                1. 4

                                                  Secure according to which measure though? I should be able to detach my storage and mount it on another machine to read and repair it if I know my keys. How do I get these TPM keys if it’s in the black box on the device I own (besides side channel attacks)? Even if I could do this through LUKS or whatever, do I want to? LUKS or a filesystem’s entryprion already provides me pretty good encryption and I know who and what generated the keys and where they live because I did it when I formatted my drive. Pluton’s a “chip-to-cloud security vision” sounds like complexity in that pipeline that opens me up to a different vector of issues.

                                                  When you couple Pluton with Smart App Manager (forgot the name) doesn’t this allow Microsoft to be the arbiter of what apps are good/bad and what it considers safe/compromised (like the issues Android users can have with SafetyNet if they want a custom ROM or root access to their purchased device)… and its store to be the eventual final ‘trusted’ space to get apps just like the Apple and Play Stores?

                                                  I know this is just a flurry of questions and I don’t think it’s fair you need to play spokesperson, but TPM was very unpopular and now it’s a requirement to upgrade—and Pluton is disabled by default by Lenovo and Dell, but why if it’s so safe? Who’s to say users want this? I can disagree but understand why businesses would, but I don’t understand how this should just be accepted as a good thing for personal and private users to not get the keys to their own device. I can have a paper backups in a fire safe for most other forms of encryption but I can’t for TPM?

                                                  1. 8

                                                    I should be able to detach my storage and mount it on another machine to read and repair it if I know my keys. How do I get these TPM keys if it’s in the black box on the device I own (besides side channel attacks)?

                                                    I’m not a LUKS expert, but I believe that it stores a block that contains the disk keys, encrypted with a key on the TPM. The TPM will decrypt this block and then the kernel does all of the crypto with keys that it knows for normal operations. It will also spit out a recovery key, which is the decrypted on-disk key and lets you mount the disk on another system.

                                                    On Windows, for domain-connected machines, the BitLocker keys can be automatically stored in active directory, so that your IT folks can unlock the disk but thieves can’t (assuming no BitLocker vulnerabilities). I don’t know if Red Hat or Canonical provide something like this for LUKS, but it wouldn’t be too hard to build on an LDAP server.

                                                    LUKS or a filesystem’s entryprion already provides me pretty good encryption and I know who and what generated the keys and where they live because I did it when I formatted my drive

                                                    And how do you enter them on boot? You need either an external key (stealable along with your laptop) or you remember a long pass phrase.

                                                    How do you know that the kernel that you’re entering the passphrase into is really the Linux kernel that you trust? Without a secure boot chain, someone who briefly has physical access would be able to replace your kernel or GRUB (whatever is not on the encrypted disk, which you use to mount the encrypted FS) with one that will store the key somewhere insecure for when they steal the machine.

                                                    When you couple Pluton with Smart App Manager (forgot the name) doesn’t this allow Microsoft to be the arbiter of what apps are good/bad and what it considers safe/compromised (like the issues Android users can have with SafetyNet if they want a custom ROM or root access to their purchased device)… and its store to be the eventual final ‘trusted’ space to get apps just like the Apple and Play Stores?

                                                    You don’t need a TPM for Windows (or any other OS) to decide whether to run a signed binary or not. Pluton changes nothing here.

                                                    Who’s to say users want this?

                                                    At least for corporate customers, our market research data does. Home users also like things like Windows Hello, which requires a secure credential store to support WebAuthn and allow you to sign into web sites without needing a password. I, personally, like knowing that my GitHub login credentials (for example) can’t be exfiltrated by a kernel-level compromise on my machine. I like knowing that if someone steals my computer, they don’t get access to my files. And I really like that this is now becoming a baseline security standard and so I get the same guarantees from my Mac, my Windows machine, my Android phone and my iPad.

                                                    1. 2

                                                      IT folks can unlock the disk but thieves can’t

                                                      I can’t say I trust IT folks or keys on any server that isn’t mine. At this point I don’t know that I could work with an employer where it’s not BYOD, so unsure if this overlaps with me. I could maybe understand no one having access to the private keys, but it sounds like someone does and that someone isn’t me.

                                                      And how do you enter them on boot?

                                                      I have a long arduous password, and I’m pretty fine with this. It written down in a safe place too. I’m not okay with this key being in the black box that connects to a server.

                                                      With NixOS though, the encryption of a lot of the device is irrelevant though and actively harmful to encryption since the machine becomes so stateless that an attacker could work backwards to figure out private keys given so many things are reproducible to an exact state (so things not in /home, /var, similar aren’t encrypted). I’d be curious how well a system with a general attack would handle the Nix store needing to be a certain way to boot or not–not anywhere near an expert at this level of the machine.

                                                      You don’t need a TPM [..] run a signed binary

                                                      But Pluton can help act like SafetyNet, no? And how I ended up switching banks after they no longer let me use MagiskHide because my device should be mine and if I want root to install some privacy apps and kick out parts of Google, it’s not my bank’s business.

                                                      Windows Hello … guarantees [..] from my Android phone

                                                      These aren’t things I generally want or care about–nor do I want to trust some AI’s facial recognition algo nor the internet connection and Microsoft account requirement for setup. Some passwords are in my head, but most things are behind FIDO2 or TOTP 2FA–both of which do a decent job with the password situation without involving that black box or having a single point of failure. My phone even de-Google’d often times feels more like a kiosk than any other device I’ve had. If Linux support was just a little better, I’d drive that instead too.

                                                      At least for corporate customers, our market research data does

                                                      Meanwhile at The Register: https://www.theregister.com/2022/03/09/dell_pluton_microsoft/

                                                      Dell won’t include Microsoft’s Pluton technology in most of its commercial PCs, telling The Register: “Pluton does not align with Dell’s approach to hardware security and our most secure commercial PC requirements.”

                                                      Says that Pluton seems off to not just the Linux base, but OEMs too. Microsoft having no concern about Dell & Lenovo seems a bit odd.

                                                      1. 3

                                                        I think it’s odd that you speak in such abolutist terms about your “ownership” of your devices, and your refusal to let anyone else ever compromise your “ownership” by setting out terms on what you can or can’t do, but every one of your examples actually consists of you demanding access to other people’s devices (well, their services, which is the same thing because those services run on their devices) and you demanding the right to dictate to them the terms on which you will receive that access. Do they not have the same rights of “ownership” over their things as you? Do they not have the same right to set terms of their choosing and tell you that you don’t “own” their devices?

                                                        1. 5

                                                          When the service they’re offering is access to something I own, I at least would agree that they have an obligation to let me shoot myself in the foot if that’s what I really want. Show a warning about installing on a rooted device, sure - but don’t go on to block access. For most of us, a bank isn’t really exposed to any meaningful risk if I install their app on a rooted device - only I am, because it’s my banking info that’s being exposed to other malicious apps on the device. I didn’t see any other examples of access to things that companies own in that post, unless you’re arguing that it is Google’s phone (which may be the case in practice but certainly shouldn’t be).

                                                          EDIT: If Pluton had some support that allowed the user to control its decisions I think it would be a lot more comfortable. It wouldn’t have to go through the OS, since obviously that’d just regress to square one. It wouldn’t have to be convenient, either, since it should be a pretty rare case that you need to do it. It probably should be pretty cheap compared to processors themselves.

                                                          I’d be happy to have some peripheral you plug the chip into during build, and need to enter a code that came with the chip’s manual from the manufacturer, at which point you have to change the code. Chip won’t boot unless the code’s changed. If you don’t care, you just do that and then plug it into the socket as normal; otherwise you can edit the roots of trust freely and be on your way until the next time you want a change, in which case you have to go through the ordeal of unseating the processor from your motherboard and doing it again.

                                                          1. 1

                                                            I at least would agree that they have an obligation to let me shoot myself in the foot if that’s what I really want. Show a warning about installing on a rooted device, sure - but don’t go on to block access.

                                                            Again: they’re not obligated to give you access to their systems. Remembering that the example cited was a bank, the user has plenty of other options besides using their rooted mobile device where the bank can no longer trust that, for example, its own app is being run unmodified. They can almost certainly still access via a web browser (which is inherently a less trustworthy environment and thus one the bank is less likely to restrict as much as app access), or call, or go in person to a branch.

                                                            And by saying that they should still provide access to their systems you are still effectively claiming the right to dictate to them how they will use what they own. Which is what the poster I replied to was saying they would not allow anyone else to do to them. The position thus remains inconsistent.

                                                  2. 3

                                                    Pluton-enabled devices are far more friendly than Android devices in this regard. You can toggle a configuration switch to use the other root cert […]

                                                    Currently, yeah. At least the Lenovo devices people were complaining about did. But I’m pretty sure this is up to the vendor, right? Just like whether the bootloader on an Android device is unlockable or not.

                                                    1. 6

                                                      No, supporting the alternative root is a requirement for certification from Microsoft.

                                                      1. 8

                                                        I am sure that that requirement is antitrust CYA, but I’m still happy y’all have it. :)

                                                2. 2

                                                  Thank you very much for this response. It’s super refreshing to see someone address the actual technology aspect.

                                                  Is there any reason Linux distros couldn’t run on Plutonium machines if, for example, distros worked with MSFT in a similar way to what they already do to get their keys signed for SecureBoot?

                                                  Pardon if it’s a silly question, my understanding of the technical details in this space is tenuous at best.

                                                  1. 3

                                                    Is there any reason Linux distros couldn’t run on Plutonium machines if, for example, distros worked with MSFT in a similar way to what they already do to get their keys signed for SecureBoot?

                                                    Absolutely none. By default, Pluton contains two public keys for SecureBoot: One used by Microsoft to sign Windows and one used to sign other bootloaders. Linux distros can use this GitHub repo to provide a GRUB shim to sign. That shim will then include the public key that they use for signing their kernel and so on and so lets you establish a complete secure boot chain.

                                                    The difficult thing at the moment is to provision a personal or per-org key. The thing I’d like is to be able to install my own public key so that the device will boot only a kernel that I, personally, sign and so I can compile my own custom kernel with my favourite configuration options, sign it, and boot it. A process for doing that securely is fairly hard.

                                              1. 7

                                                I could not follow why all this is bad. Does it make it impossible to install Linux on any computer with this chip?

                                                There were statements in the article that made it sound to me like the author considered it a problem that the chip made it harder to pirate copyrighted works i.e stealing content from creators, which didn’t sound right to me.

                                                1. 6

                                                  According to an MS employee commenting on this (@david_chisnall), it’s a certification requirement from MS to support alternative root certs, which allows Linux distros to be installed and booted with a full secure boot chain.

                                                  I found the article to be fairly unclear on why Pluton is bad, as well; the only “bad” thing is that it theoretically will make it easier to prevent software piracy and cheating in online games. Which doesn’t seem… bad, to me?

                                                  1. 4

                                                    Yup there were a number of red flags for me as well.

                                                    I’ll believe that Linux really is impossible on these chips when we see systems in the wild in the hands of skilled hackers :)

                                                    1. 4

                                                      As I understand it, Linux on Pluton PCs will be about as available as, say, LineageOS on Android devices. If the vendor doesn’t allow unlocking the bootloader, you’re probably out of luck.

                                                    2. 7

                                                      “Pirating copyrighted works” is an unavoidable side effect of general-purpose computing. I’d rather not throw out general-purpose computing in order to appease Disney and Time-Warner.

                                                      1. -2

                                                        While it may sound easier on the conscience to think that this is a Robin Hood (The English myth, not the trading platform) kind of situation, at the foundation it isn’t. It’s indie authors getting their books stolen, it’s actors getting lower revenue because of lost viewership. It’s singers not getting royalties.

                                                        If you are against big publishing/producing houses making money, support Indie artists. Making it easier to steal their works is not a solution.

                                                        1. 12

                                                          It’s singers not getting royalties.

                                                          The biggest thieves of royalties are arguably the music industry.

                                                          Making it easier to steal their works is not a solution.

                                                          You can’t “steal” a work. You can create and distribute reproductions without permission or attribution and fail to pay royalties, but my possession of a song doesn’t exclude your access to it. The “theft” framework is a meme perpetuated almost entirely to the benefit of publishers and rights-holders (not artists!) in order to legitimize incredibly shitty and abusive behavior.

                                                          support Indie artists

                                                          I do, and I even pay for my tracks! But a lot of those artists I’m only aware of due to running across their music under circumstances where perhaps the licensing wasn’t as well audited as it could be.

                                                          1. 2

                                                            I have bought more indie music on Bandcamp in the last two years than I bought any music in the previous twenty. But a lot of that is because they offer DRM-free tracks in your choice of formats, generally at a very reasonable price. That’s not something you can say of, for example, the movie industry.

                                                            1. 2

                                                              I’ve known quite a few musicians and they had huge pirate music collections.

                                                            2. 6

                                                              I don’t see how locking down computers to such a level helps artists, especially independent ones, and as a consumer I hate buying DRMed content and avoid it where I can.

                                                        1. 3

                                                          Author here, glad to see the article still getting some attention almost two years later. May I ask where you came across it?

                                                          1. 4

                                                            Conversation with some friends the other night, jogged my memory.

                                                          1. 8

                                                            I didn’t enjoy this.

                                                            In no small part because it reminds me of unfinished work, piling up behind me.

                                                            I think it’s important to drop those things you are least likely to do off the back of the cart, and let them lie. This piece didn’t seem to touch on strategies or other practices for improving the work, only pointed out that backlog is bad.

                                                            I recommend reading The Phoenix Project. It seemed to cover all this ground in a more compelling package.

                                                            1. 3

                                                              This piece didn’t seem to touch on strategies or other practices for improving the work, only pointed out that backlog is bad.

                                                              That wasn’t my reading at all–the size of the backlog is a different issue than work-in-progress.

                                                              It’s totally fine to have a big backlog and be chunking it up into little bits and chewing through it–this is in fact one of the keys to getting a backlog back under control if you can’t just axe things entirely.

                                                              It’s similarly terrible to have a “backlog” of zero with a thousand things in-progress.

                                                            1. 7

                                                              For better or worse, the deeper you go into threads the less likely it is that the discussion is still about the original submission. Your proposal, or the max variant presented elsewhere, I fear would unduly reward flamebait and off-topic discussion.

                                                              1. 2

                                                                Your proposal, or the max variant presented elsewhere, I fear would unduly reward flamebait and off-topic discussion.

                                                                Such off-topic discussions that stray away from the main topic do exist, are a nuisance, and, surely, they should not be rewarded with more attention. However, such sub-threads exists as a long string of short comments with no or very few upvotes. (I wonder if the data confirms this or it is just my impression.)

                                                                For this reason I believe that a) not counting in comments with score==1 or b) @dpercy’s max variant would avoid rewarding such discussions.

                                                              1. 2

                                                                I firmly believe Formal Method advocates should aim for a more pragmatic approach and “play nice” with compilers and optimizers and Design by Contract and Unit Testing.

                                                                Let’s face it, writing a full postcondition is just plain a lot harder than writing the code itself.

                                                                However, writing a perhaps partial postcondition for a few compelling concrete test cases is usually pretty easy.

                                                                Instead of aiming for “proving” the programmer right or wrong, if they aimed instead for assisting the programmer as much as they can.

                                                                If formal methods acted like warnings on steroids (but with no false positives) and a boost for optimizers and a power up boost for internal precondition checks….

                                                                I can see a lot more adoption.

                                                                1. 3

                                                                  Let’s face it, writing a full postcondition is just plain a lot harder than writing the code itself.

                                                                  That’s not at all obviously true! We have many examples where it is false, and it’s probably false in general.

                                                                  An example of this is are problems in NP. Problems that can be verified in polynomial time but whose solutions cannot be generated in polynomial time (assuming P!=NP, but in any case, even if P=NP the verification is still likely to be far easier).

                                                                  Problems to do with concurrency are another example. There are models that are very simple to specify, but they result in thousands of tricky edge cases that need to be handled perfectly and that humans just don’t come up with on their own.

                                                                  The real problem is that current tooling makes specifying models far too difficult. And even worse, they make knowing if you’ve specified the right mode quite tricky. This is probably a combination of having immature tools and having chosen the wrong formalism for expressing these models. A lot of logics look great at first but their semantics, particularly when it comes to reasoning across time, turn to be really tricky. We have some glimpses of how this can work with languages like Haskell or Isabelle in domains that can be modeled well and it’s beautiful. But there’s a long road from here to a point where we can express these postconditions in a more natural way and still reason about them automatically.

                                                                  1. 6

                                                                    You’re kinda illustrating @JohnCarter’s point here.

                                                                    I’m not working over in CSAIL, but here in startupcanistan even having basic tooling around post-conditions in design-by-contract for things like “this function will never go negative when calculating a sum of credits”, “this function will always return a smaller value towards zero than its inputs”, “this function will return a URL that doesn’t have a query string with such-and-such values in it” would still be super helpful.

                                                                    We’re trying to make a cup of tea, not boil the ocean. Until formal methods folks realize this, their entire field is of limited utility to practical, real-world software engineering outside of a handful of cases (say, writing a database engine).

                                                                    1. 2

                                                                      I’m going to recommend this great post about combining property based testing with contracts. If you can add the assertions for your pre and post conditions, through something like an assert statement in your language or some library support, you can use property based testing to at least generate a lot of inputs. And that’s more likely to find assertion failures than hand-writing test cases.

                                                                      That’s one of the lightest-weight approaches for adding formal methods concepts to a real word project.

                                                                      1. 1

                                                                        Yes, actually from pre/post-conditions you can get 3 things:

                                                                        • Support for generating random tests, similar to QuickCheck
                                                                        • Verification at compile-time, like Dafny
                                                                        • Fail-fast behavior at runtime, like Eiffel

                                                                        The third item is really pragmatic and underappreciated. At runtime, a failed precondition should prevent a function/method from running. A failed postcondition should prevent the function from returning or writing its side effects. A huge class of software errors would disappear if software was written like this.

                                                                        Probably, you can also get a fourth one, better static analyses (without going into full verification) and a fifth one (more efficient code, as discussed elsewhere in this thread).

                                                                        It is unsurprising design-by-contract is so thorough as it is actually refinement types under disguise.

                                                                        1. 2

                                                                          It is unsurprising design-by-contract is so thorough as it is actually refinement types under disguise.

                                                                          And refinement types are Hoare logic in disguise :)

                                                                          1. 1

                                                                            True! :)

                                                                    2. 2

                                                                      An example of this is are problems in NP. Problems that can be verified in polynomial time but whose solutions cannot be generated in polynomial time

                                                                      Assuming P!=NP says nothing about the relative difficulty of writing the solver vs verifier, though. Or are you speaking metaphorically?

                                                                      1. 1

                                                                        Practically, most solvers for such problems include the verifier as part of the solver. So the solver tends to be (much) more complex. But yes, that’s just a rough example.

                                                                      2. 2

                                                                        We have many examples where it is false, and it’s probably false in general.

                                                                        Really? Can I see those examples? I’d be really curious to some for something like https://github.com/seL4/seL4/blob/master/src/string.c

                                                                        But I couldn’t see them from a quick dig around the repo.

                                                                      3. 3

                                                                        I totally agree. My personal interest is in how to bring formal methods to work, in whatever package is the most useful. There’s a term called “lightweight formal methods” which is similar in philosophy to what you’re talking about. Basically, take the ideas from formal methods, like invariants / properties / formal logic, but use them in a way that doesn’t require full math proof.

                                                                        I think Amazon’s approach here is really promising. They took the theorems that would be proven in a verification effort, like refinement of a model, and instead used property-based testing to check for it. So the property isn’t proven, but they have some amount of confidence in it. They also found 16 bugs this way, before shipping. And isn’t that the end goal?

                                                                        So yea. I think coming up with the right package of a lightweight, practical way to bring formal methods thinking to work is something that there’s demand for.

                                                                        1. 2

                                                                          This is an interesting point. Do we need mathematical proof? Or can we use scientific proof? Mathematical proof’s epistemology for this domain is fairly straightforward (classical logic for constructive proofs over finite problems). Scientific proof is epistemologically far more fraught…but often a lot easier to produce since it is subject to further refutation.

                                                                          1. 2

                                                                            Yes definitely - tests are really a scientific / empirical activity. They are about observing what really happens, not just what the logical model says should happen. I would say the main difference between testing in software and actual scientific experiments though is that we don’t know how to talk about the statistical significance of individual test cases. At least I don’t. I’ve heard that the Cleanroom approach has some notion of statistics, but I’m honestly not familiar with it in detail.

                                                                            As far as which one is appropriate for software development. It could be contextual, but I pretty much solely focus on “business software,” or “regular” software, and for that I would say math proof is almost never necessary. It could be desired in certain cases, but even then I would only prove things about a model or a broad algorithm in general. Proofs at the implementation level are just extremely time consuming, even with things like proof assistants.

                                                                            So as a broad strategy, I think we (as an industry) have to get better at elevating testing to a point where we statistically know how effective a test suite is. And no, branch coverage is not what I’m talking about, since input data is really the coverage that we should really care about.

                                                                            1. 1

                                                                              I would say the main difference between testing in software and actual scientific experiments though is that we don’t know how to talk about the statistical significance of individual test cases.

                                                                              Statistical significance isn’t really the issue in most software tests because we set them up to be deterministic. Hypothesis testing is a means of working with randomness in observations. The issue is the selection of observations and test cases. This is true in biology or in software testing. Hypothesis testing is a tool for doing an individual observation, but the aggregate set of observations to run (the research program, more or less) doesn’t depend on that. I’ve written some stuff about this.

                                                                              I would say math proof is almost never necessary.

                                                                              I disagree. We depend on proofs heavily on a regular basis. Type systems are the most common example. But setting up a system where you control the flow of information to make certain states impossible is another. These aren’t comprehensive proofs, and they are often implicit, but they are properties we prove mathematically.

                                                                              branch coverage is not what I’m talking about, since input data is really the coverage that we should really care about.

                                                                              Mutation testing is the most defensible approach to coverage that I’ve seen: if you change the program, does the test suite start throwing errors. I also like property based testing because it gets humans out of the business of selecting test data individually.

                                                                              1. 1

                                                                                I also like property based testing because it gets humans out of the business of selecting test data individually.

                                                                                Unfortunately in practice I’ve found you have to spend a lot of time getting the data generator just right to cover an interesting enough set of inputs.

                                                                                1. 1

                                                                                  You have probably used it far more in-depth than I have. I was mostly using it to generate fairly straightforward things like lists of a type.

                                                                                2. 1

                                                                                  Re math proofs and necessity - I was moreso talking about a full proof for the functional correctness of an entire application. I think that is overkill in most cases. Not proofs in general, especially about smaller algorithms.

                                                                        1. 21

                                                                          I always thought that Greenspun’s tenth rule (Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp) ought to be updated for modern development practices, so here’s Hagelberg’s tenth rule: Any sufficiently complicated kubernetes deployment contains an ad-hoc informally-specified, bug-ridden, slow implementation of half of OTP.

                                                                          1. 18

                                                                            http://rvirding.blogspot.com/2008/01/virdings-first-rule-of-programming.html

                                                                            Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang.

                                                                            1. 4

                                                                              I should have known! But a lot has happened since 2008, and I feel like k8s deserves to be called out specifically.

                                                                            2. 6

                                                                              The root of all of this is that the BEAM ecosystem represents a parallel evolutionary path (in many ways better) to what ended up becoming the dominant tooling in our industry.

                                                                              For a lot of us, this means that when we need to use that neat tooling we end up having to fight with institutional inertia and dogma. I have a coworker–smart dude otherwise–that in their role as a devops just refuses to exploit the idea that our BEAM stuff doesn’t quite act like the generic python or node or whatever app you would use a “livestock not pets” approach on.

                                                                              It’s annoying, but we deal with the world as it is.

                                                                              1. 1

                                                                                I have a coworker–smart dude otherwise–that in their role as a devops just refuses to exploit the idea that our BEAM stuff doesn’t quite act like the generic python or node or whatever app you would use a “livestock not pets” approach on.

                                                                                Can you elaborate on this a little bit more - I am not certain what this means but it sounds good.

                                                                              2. 2

                                                                                I live in that rule at work, and quite agree with it.

                                                                                1. 1

                                                                                  Where are Hagelberg’s rules 1-9?

                                                                                  (Yes, I’m aware)

                                                                                1. 13

                                                                                  The absolute meanest question I’ve ever gotten from a candidate was:

                                                                                  “What question did you wish you had asked when you were interviewing here?”

                                                                                  The other interviewer and I just kinda segfaulted, and complimented them on a good question.

                                                                                  I like asking business-oriented questions, since most of the development questions are usually just boring things (“What ticket tracker do you have?”, “Tabs vs spaces?”, “Do you all have CI?”, etc.) that, if bad enough, I’m probably being hired to fix anyways.

                                                                                  The really interesting questions you can’t really expect to get an honest answer for:

                                                                                  • “Are all of the C-suite related?”
                                                                                  • “How often does the CTO yell at the CEO during meetings?”
                                                                                  • “Have any of your developers ever tried to self-terminate in the lobby?”
                                                                                  • “What is the worst pornography a developer has ever been asked to clean off of a sales computer?”
                                                                                  • “Is a previous beloved manager in the process of gutting the engineering org?”
                                                                                  • “How ignorant of basic physics are your investors?”
                                                                                  • “Who is the most toxic person on the team, and why are they still allowed to work here?”
                                                                                  • “How much of leadership is on the critical path for process? How often are they taking sick days?”
                                                                                  • “How badly has your company fucked over employees attempting to purchase their shares?”

                                                                                  Like, the really good stuff you won’t know until it’s too late.

                                                                                  1. 4

                                                                                    I know that you’re (half :-P) joking about the really interesting questions but honestly, in my experience, you’re spot on about them. Before I did my own thing, I eventually fine-tuned the questions I ask at interviews for ticking the “asks follow-up questions” checkbox of the interviews’ list without being a complete bore, but nothing more. Because I found them to be pretty much useless for weeding out the really bad things. Some of the things I’ve seen in 15 years of interviews include:

                                                                                    • “Oh, yes, of course we do version control, lol, can’t do software development without it.” Said version control was SVN which, if anyone recalls, had you jump through a bunch of hooks to set up new repositories and the like. Furthermore, the CTO insisted on manually vetting repositories and users so in practice the vast majority of software development was done without any version control whatsoever, and the SVN repo was synced by team leads every one or two weeks or so.
                                                                                    • “You will be reporting to X and your responsibilities are $whatever”. Four weeks into the job one of the PMs quit and someone had to take on their workload urgently, until a suitable replacement was found. One year and a half into the job no suitable replacement has been found, and you can’t just give up PM duties, that’s a critical role. Plus it’s a great opportunity for getting into management, who doesn’t want that. What additional compensation, these additional responsibilities are only temporary, until a suitable replacement is found.
                                                                                    • “We have a comprehensive code review process.” Said comprehensive code review process essentially amounted to “representative variable names are important so please don’t use nondescript names like i, usecurrentCollectionItem instead” (I wish I were kidding). Since nobody actually read the source during reviews, glaring buffer overflows were discovered months after they’d been introduced, sometimes as a result of the entirely pointless refactoring hastily done during the “review”.
                                                                                    • “Our issue tracking system is pretty much what you expect… we have a bug tracker where bugs are tracked and triaged, we have three tiers of customer support so bug reports from the field make their way to the bug tracker with all appropriate information already filled in, and the customer support team can help. We run regular integration tests, besides the usual CI pipeline and whatnot. We do a lot of maintenance because we have systems that are like ten years old and they’re still in use but we value stability so that’s a given.” Said issue tracking system was an ancient version of Bugzilla. The issue categories hadn’t been cleaned up in forever so there were several hundreds of them, about half of them unused. Bugs were categorised by products but the product names followed one naming convention while sales, marketing and support followed another (and names changed pretty often so there was actually no way to get a full bug listing without the full naming history, which could include up to a dozen names). Integration testing was done regularly, as in, at the end of every development cycle, so once every three months, unless it was a longer cycle, it could take up to an year. The full test suite took 24 hours to run and there were only four test setups, so the QA team manually ran the tests labeled “CI” over the weekend.
                                                                                    • “Oh, yeah, we’re profitable and we’re very secure financially.” The company is profitable and very secure financially. The business unit you’re being hired into has been burning money for like five years now and we’re low-profile trying to get rid of it but nobody wants to buy this dumpster, presumably because it’s on fire.
                                                                                    • “We have a diverse and super friendly team, you’ll like it.” We have a huge employer branding problem and nobody wants to come work here. About half the people who know what they’re doing are assholes, but we can’t risk pissing them off or firing them because it’s really hard to attract people with their expertise. No idea why. The labour market is really skewed.

                                                                                    If the company you’re interviewing for is, indeed, ran by people who are utterly incompetent, you will inevitably figure it out by the time you get to the “do you have any questions for us?” stage.

                                                                                    If it’s not, and the people who are interviewing you aren’t entirely incompetent, they can tell a red flag when they see one. They’re not comfortable about the embarrassing lack of version control, about the borderline useless ticketing practices. They know if the engineering manager is a narcisistic, compulsive liar who cracks under the tiniest amount of pressure. They just can’t tell you. Maybe they’ll drop hints if you’re lucky but if you ask them point blank, especially with another interviewer around, they will, at best, paint a “nothing’s ever perfect but we’re committed to improving it” picture.

                                                                                    tl;dr My experience is that any reason why you wouldn’t want to work someplace will get dusted under the carpet. Don’t count on these things to weed out the really bad places. Trust your gut.

                                                                                  1. 1

                                                                                    To provide a different perspective to the “well it isn’t a tribe/it’s a mindset/everybody needs to be capable of switching” theme I’m seeing pop up here.

                                                                                    When there is money or time involved, the tribe lens suddenly provides a lot more utility. This isn’t an abstract “oh gee I guess our value systems are different” sort of issue but instead (especially in professional settings) an existential crisis where the three tribes are actively fighting for resources. Improperly managed, this tribal conflict can kill an initiative–the best outcome is to get everybody to work together according to their strengths, but more common and nearly as effective is to remove the conflict either by isolation or purges.

                                                                                    I also am somewhat annoyed that the author didn’t include a wagie tribe, especially in the hindsight of the current economic anxiety: those who really and truly care more about making it through their day and getting paid than any particular concern for clean code, fast code, or delightful code and who engage in programming because it is the current strongest/safest bid to enter or stay in the upper-middle class economically. Critically, the wagies outnumber the other tribes by at least an order of magnitude and provide a natural ballast against the other tribal conflicts that would screw with their paychecks.

                                                                                    (Incidentally, I personally mostly prefer @geocar’s analysis from a past submission of this article, but I figure I’d kick in a different discussion point here.)

                                                                                    1. 3

                                                                                      visualization is a clumsy tagging, but it seems to match the intent.

                                                                                      1. 19

                                                                                        I wonder why growth is trending down (not that that’s necessarily a bad thing).

                                                                                        I’ve made some good friends on here, and when I post my own content, I’ve always had great interactions/feedback too!

                                                                                        1. 46

                                                                                          I love lobste.rs and have been a member for 8 years. The comments ahead are just my personal experience but maybe there are others who feel the same way. At some point (I think about 2 years ago) topics about tech culture and society started to be removed by moderators and I started to participate less and less. Which is insane since the reason lobste.rs started was HN banned the creator and HN was doing some funny moderation. (fun history: https://jcs.org/2012/06/13/hellbanned_from_hacker_news, https://news.ycombinator.com/item?id=4452384). One reason I loved lobste.rs was it’s careful use of moderation, instead relying on having a solid group of users vetted by others in the community.

                                                                                          So, this stronger moderation against topics related to culture and society that intersect with tech made me lose interest. Given all the crazy things happening in the world today, to believe that tech is isolated from the world is naive and ultimately creates a bubble culture. What’s the point of loving technology if it can’t be applied to real world problems? So over time I started to lose some interest in content on lobste.rs as it seemed less relevant to my life. Maybe the content is changing again? I don’t know as I haven’t really participated as much.

                                                                                          The community here is strong and I hope for another strong 10 years. I just hope people learn that tech is useless independent of helping people. Code that doesn’t run, that doesn’t solve problems, is just a statue. Beautiful to look at and appreciate, but not much else.

                                                                                          1. 58

                                                                                            I feel the opposite. The American culture wars are exhausting.

                                                                                            I am glad this place is peaceful.

                                                                                            1. 21

                                                                                              I find the culture war exhausting too, but I also feel it’s mostly fake. That it’s mostly manufactured by the media and big voices on social media. Notice I didn’t say anything about any culture war but that’s where you went. Isn’t that weird? Something is wrong with our discourse. I’m talking about software solving the real problems we have in society (hungry, homeless, global warming, ecological collapse, energy, prison system, education, war, inequality, gun violence). The culture war is manufactured, in my opinion (puts on tin foil hat), to distract us from the real problems.

                                                                                              Computers are literally man’s greatest invention. They can save us from meaningless labor and enhance our minds. They aren’t a bicycle for the mind, but a rocket ship. My worry is we are wasting it. We shouldn’t take computers for granted. It won’t take much to forget how to make them.

                                                                                              1. 18

                                                                                                I’m talking about software solving the real problems we have in society (hungry, homeless, global warming, ecological collapse, energy, prison system, education, war, inequality, gun violence).

                                                                                                Do we need software to fix any of those problems? Aside from global warming / ecological collapse at least? We (as a society) have the wealth to fix these issues, it is mostly the lack of consensus / political will to do so. And the main thing standing in the way are certain wealthy actors and interest groups. They are interested in their own profits first and foremost, and control of society via marginalization or outright oppression of minorities and destruction of democratic systems and discourse.

                                                                                                We can use software on the margins to try to educate people, and show how they are being manipulated. But it doesn’t seem like enough.

                                                                                                1. 3

                                                                                                  Like a virus, computers are now in everything. You eat today? Computers were involved. It’s not so much that they can fix any of those problems (I would argue they accelerate some like global warming. Google is proud they increases waste and energy use through all of society https://economicimpact.google.com/), it’s that if they aren’t part of the solution, then they are part of the problem. So we either fix it, or get rid of their usage. Since they are such a powerful tool for productivity, it seems to me we can use them to accelerate solutions vs accelerate problems.

                                                                                                  1. 2

                                                                                                    Like a virus, computers are now in everything.

                                                                                                    Then all the more reason for having a place where we can discuss the science, art and craft of technology away from the divisiveness that’s tearing our society apart makes sense in my view.

                                                                                                    I’m not suggesting that this is a monastery, but monasteries existed to keep the barbarians out and knowledge in when the dark ages fell. I see communities like this serving a similar purpose.

                                                                                                    1. 2

                                                                                                      Except it was in the monasteries where truth died. The “dark ages” were nothing like you described. I suggest reading Debt the first 5000 years by David Graeber. Eratothsenes figured out the circumference of the earth and over a 1000 years later we had Christopher Columbus who thought the world was much smaller. Yajnavalkya postulated the earth revolved around the sun yet the Monasteries promoted a earth as the center vision.

                                                                                                      We need a functioning civilization to keep knowledge being passed through one generation to the next. Now that we are facing threats to organized human life at an unprecedented scale, there will be no ‘safe place’ to teach people how to build computers without civilization wide support. Computers are just too complex.

                                                                                                      Also imagine the rest of society thinking “Wow, we have these amazing tools called computers that can solve our problems, but the folks who design and build them, the elite who use these tools, want nothing to do with our problems. Want to ignore them because they are too disturbing and annoying to the experts”.

                                                                                                      1. 2

                                                                                                        Good on you for fighting the good fight. I’ll just be over here hacking around with old computers and trying to stay healthy long enough to retire and enjoy life a bit :)

                                                                                                        1. 2

                                                                                                          You evoke an interesting thought and bring up a good point. There are millions of programmers. But most programmers have little say in what they actually build as they work for large companies. That’s because, while programmers are paid well relative to the rest of society, they often own very little of their work.

                                                                                                          The responsibility I am talking about falls on those that do have a say in what is built. Many of the leaders are former programmers themselves. But even among programmers there is a class divide. Those that don’t have a say in what is built don’t have the responsibility I speak of. Maybe we need more people owning their work.

                                                                                                2. 9

                                                                                                  The culture war is manufactured. It is also a real, serious problem. One of the reasons there are so many wars is that they can be started unilaterally.

                                                                                                  To the point at hand, though, do you think discussion about “culture and society” on lobste.rs solves any of those problems? I associate these kinds of topic with lobste.rs’ turning into a little hackernews, in which the same handful of political arguments are rehashed and people are generally horrible to each other. I don’t think the tech industry at large is going to discover, for instance, the concept of professional ethics through comment threads here.

                                                                                                  I think the reason we can be civilised here is that we find technology neat; it’s a thing we have in common, and the reddit-style discussions work reasonably well for that. When we debate bigger things the medium shows its weaknesses. For one thing, while a lot of the strictly computery posts exist in some sort of niche, articles about society have much more direct political implications, and tend to elicit some sort of opinion for pretty much everyone. It’s also much harder to stay calm when discussing something that matters.

                                                                                                  I’ve argued, often and animatedly, that political content shouldn’t be on lobste.rs. I have several reasons for this, and I hope I’ve explained one of them, but just as important is… politics. I think being exposed to the sort of environment I see on political threads here makes people worse, or at least marginalises those who are most inclined to be nice. In theory diversity of opinion might expose people to new ideas, but in practice people pretty much always go home thinking exactly what they thought yesterday, only more so. I’d be all in favour of your position if I’d ever seen any evidence that lobste.rs debating important things leads to people becoming more conscientious about those things.

                                                                                                  I appreciate this is a bit of a ramble, but one last thing: why would we expect anything else? You say that believing tech is isolated from the world creates a bubble culture. But lobste.rs is a bubble in its purest form already. Most tech workers and enthusiasts, especially in America, exist in a relatively narrow social stratum; it’s hard to find a demographic distinction in which the field doesn’t exhibit strong bias. I have my doubts about the comment section free-for-all as a vehicle for social change, but even if it could work, we’d need to be more connected to the rest of society in order to have any chance of deciding what technology’s place in it ought to be.

                                                                                                  1. 8

                                                                                                    You raise a lot of good issues here. But I feel maybe I wasn’t clear enough. I don’t want random discussions about culture and politics. Twitter already exists. I care about the intersection of technology and society. I think those discussions are important to have and Lobsters used to have them. Then those seemed to have gone away and I lost some interest.

                                                                                                    We know that people in technology are usually horrible at social issues, partly because we get people who prefer certainty. The certainty of the machine. I was one of those people. We have great comfort talking about frameworks, programming languages, and reverse engineering old hardware. We like our safe space.

                                                                                                    I have my doubts about the comment section free-for-all as a vehicle for social change, but even if it could work, we’d need to be more connected to the rest of society in order to have any chance of deciding what technology’s place in it ought to be.

                                                                                                    I don’t have this view of lobsters as a vehicle for social change. It’s not. Social change will come either way and we can talk about how technology is involved, or we can ignore and treat lobsters as a sort of comfort food. That’s totally acceptable. It’s just less interesting for me and that’s why I responded to ‘why has growth stalled’ comment.

                                                                                                  2. 1

                                                                                                    Notice I didn’t say anything about any culture war but that’s where you went. Isn’t that weird?

                                                                                                    Seems very telling to me and makes the user come off as a troll. Somehow having concern = culture war? Or caring about a topic = virtue signaling? There’s no authenticity to users like that. They can’t imagine a world where people are caring or concerned about things bigger than themselves.

                                                                                                  3. 6

                                                                                                    I had to filter out the culture tag for the sake of my sanity.

                                                                                                    As much as I love reading this site, there’s something about the influx of certain topics and the style of conversation etc that - for the lack of a better word - triggers me. I have to restrain myself from getting involved, yet I know nothing good can follow from participating.

                                                                                                    Few of us are in a position to really affect change, and online discussion (esp. heated) is a net-negative substitute.

                                                                                                    This is probably still true for culture stories, but I don’t wanna go look in that dark corner.

                                                                                                    Everything else I love, thank you and keep it up for many decades!

                                                                                                    1. 3

                                                                                                      I too find the forever culture war exhausting, and treat tech and, by extension lobsters, as a kind of haven where I can think about fun, inspiring things I might want to build or ideas I can grow from.

                                                                                                      There is a time and a place for everything, and there are a bazillion fora for discussing that stuff. IMO it doesn’t need to be here.

                                                                                                    2. 30

                                                                                                      As someone who also subscribes to the (glibly described as) “everything is politics” philosophy, I am still for removing a lot of the “culture” articles. The main reason is that these discussions are already happening elsewhere (for example HN). Society existing everywhere doesn’t mean that we have to discuss society everywhere.

                                                                                                      The secondary reason is that there is a general idea for what is on topic, and that is “can this make you a better programmer”. I think that makes some stuff about community management (like CoC discussions for prog languages) on topic, but loads of things that end up getting removed fall far from this goal.

                                                                                                      A tertiary reason (something that happens in rant-tagged articles as well): when those articles don’t get pulled down, we end up with the same 5 people yelling at each other saying the exact same things over and over again. There is a clear vibe from some people to want to pull discussions into the same stump speech.

                                                                                                      I do think that when there isn’t a forced segue, discussion about society still happens in the comments section. And it stays reasoned. But at least personally, I don’t need every social space to turn into debate club. Lobsters isn’t the only place on the internet.

                                                                                                      1. 15

                                                                                                        I’m a relative newcomer but I appreciate the fact that lobste.rs discussions tend to be limited to things that have some form of objective evaluation criteria. When someone presents a technical project, I can evaluate it against my requirements. I can have a discussion about why my requirements are different from yours and whether my requirements are not actually solving my underlying problem. I almost certainly have a load of biases around why my requirements ended up being that shape but they’re generally not things that I have particularly strong beliefs about and, if I do, those beliefs are very unlikely to be core to my self image.

                                                                                                        When we discuss politics or culture then you and I may have very different ideas about an ideal society looks like and have very strong beliefs derived from things that are at the core of our self identity about that shape. If those happen to line up, then we can have a rational discussion about whether a particular policy advances our society towards that goal (though, often, we don’t really have enough data to make a good assessment). If we have conflicting goals for society then discussing how to reconcile them in a public forum is hard and maintaining an inclusive culture when those discussions are happening is even harder.

                                                                                                        I enjoy discussing politics, even with folks that disagree with me, but I don’t enjoy doing it on the Internet because it’s incredibly easy for things to be quoted out of context or misinterpreted. I’m glad that this is a place where we can put those discussions on one side and engage on other issues.

                                                                                                        1. 6

                                                                                                          I am torn on this matter, not the least because the one time when I broke my “no politics here” it quickly went sideways and not all in a good way, and it left a bit of a sour taste in my mouth, mostly because, justified or not, I really didn’t want to have a flamewar in an otherwise really civil place.

                                                                                                          So on the one hand I think it’s useful, but also healthy and important for a community to be able to discuss things that its members consider important, even if they’re not exactly the reason why we’re all here.

                                                                                                          This is probably a holdover of mine from the old days, when any forum, no matter what its primary topic was, also had a “General Discussion” section. A good chunk of it was flamewar but to me, a non-native English speaker at the end of the world, technologically speaking, those things were eye-opening in many ways. Even the things I actively disagreed with. They were useful for me in tech, not just in general. Without them, I’d be largely ignorant to the social, political and economical trends that shape the tech world of tomorrow, and I’d be entirely lost in this sea of information. I also think they were healthy: in my experience, tech communities that do not engage in these exercises and cannot vent on non-technical topics will eventually vent on technical topics, and will eventually cluster around narrow niches with categorical and harsh adepts who produce a lot of judgement but don’t really move the boundary of technology any further. Once they devolve into that, communities aren’t too fun to hang out in anymore, and get an expiration date, too.

                                                                                                          Usefulness and healthiness aside, I really wish I could talk about a whole bunch of non-tech things with many of you here. There are people here whose work I admire and I’m sure the original approaches that makes their software so good has also produced a lot of other ideas worth hearing.

                                                                                                          But on the other hand the single-section, tag-based, up/down-vote structure is really inadequate for this. Even if the front-page doesn’t promote controversy, the sheer volume of material that can be tagged culture is overwhelming, it’s a category that’s ripe for self-promotion, and it’s a field that’s really inviting for bike shedding while waiting for shit to compile. Unless it’s confined to a separate section, it tends to push out technical content which, in turn, tends to push out technical people.

                                                                                                          The section-less structure also means that these things inevitably make it to the front page. On old phpBB boards you could often have civil discussions in the Linux section while also shitposting in the General Discussion section, as long as general awfulness was dealt with via the ban hammer. But on lobster.rs these aren’t separate sections and shitposting inevitably spreads.

                                                                                                          It’s also a very wide umbrella. culture is equally well applied to an article about the political views of early demosceners – which, even though it’s technically politics, I’d really be super curious to read about – and to an employer branding piece about how a company contributes to Rust projects which, after years of exposure to corporate hiring machines, makes me want to puke halfway through the title.

                                                                                                          Honestly, the only tag I really dislike is practices, probably because I got a bad case of burnout from over-practice-ising a while back and eww. Ultimately, I left culture unfiltered, but I don’t think we need more of it.

                                                                                                          1. 3

                                                                                                            At some point (I think about 2 years ago) topics about tech culture and society started to be removed by moderators

                                                                                                            I wonder why this was put into place if the discussions were fine. (I only joined a little over a year ago, so I can’t really speak much on this except that I’m curious as to why these posts started being removed.)

                                                                                                            1. 12

                                                                                                              It’s still a great community. I wouldn’t have buyer’s remorse. It just changed over time to something less interesting to me. Part of it is the new moderation that came with new management. They wanted to narrow the focus of the site. I can’t say that’s why growth started trending downwards, but that downward trend coincides with what I felt. So take it with a grain of salt. I was just highlighting something that might have had an impact.

                                                                                                              1. 4

                                                                                                                Where else do you get your dose of interesting discussions?

                                                                                                                100% I feel the same way too but it’s only made me take time off to reconsider my approach to the website. At the end of the day you either decide to work with it or not.

                                                                                                                I’ve begun to vet my posts via lobsters IRC first. Maybe lobsters needs an initial “post filter”? i.e. if a post is thumbs-uped by a member of certain activity and age, it gets listed?

                                                                                                                1. 3

                                                                                                                  It’s strange but I am finding the best conversations I have are with individuals in private settings. Nice to know the IRC is active. Maybe I should try to hop in. Thanks!

                                                                                                                2. 3

                                                                                                                  It absolutely did for me. I stopped posting after my last submission was removed for “not being technical.”

                                                                                                                  That alone would have been OK, but the criteria for “technical” were (and remain) vague. And then, when I posted an honest question asking to understand them and the culture tag, titled “what is the culture tag for?”, the moderator proceeded to rename it to “why was my post removed?.”

                                                                                                                  Moderating’s a thankless job, but all the same, that felt disingenuous to me. So, I don’t submit things here anymore. It’s a bummer because for years, I feel like this place used to support the readership you mentioned:

                                                                                                                  Usefulness and healthiness aside, I really wish I could talk about a whole bunch of non-tech things with many of you here. There are people here whose work I admire and I’m sure the original approaches that makes their software so good has also produced a lot of other ideas worth hearing.

                                                                                                                3. 5

                                                                                                                  The discussions were emphatically not fine, hence the purging efforts by both moderators and the community.

                                                                                                                  1. 5

                                                                                                                    Popcorn tech.

                                                                                                                    You know what’s sad. I tried submitting topics that were incredibly technical. Bleeding edge tech. Nothing, no traction. For example, topics dealing with quantum computing, cryptography, etc.

                                                                                                                    It’s almost like people don’t want to talk about technology specifically. They want pop-technology. or popcorn tech. Compare the level of technical discussion here to say, lambda-the-ultimate (is that still around?).

                                                                                                                    But it’s better here than HN and reddit! So that’s a win.

                                                                                                                    1. 11

                                                                                                                      It’s almost like people don’t want to talk about technology specifically. They want pop-technology. or popcorn tech. Compare the level of technical discussion here to say, lambda-the-ultimate (is that still around?).

                                                                                                                      I think people want to talk about things that they can meaningfully engage with. I’m interested in reading about quantum computing, for example, but I have literally nothing useful to contribute on the subject. You seem to have invited quite a few folks to join, perhaps if you reached out to some physicists then you’d find the audience contained more people who were able to meaningfully contribute on those subjects.

                                                                                                                      I’m happy to engage on a range of deeply technical topics here (language design, compiler internals, OS / hypervisor internals, CPU architecture and microarchitecture, capability systems, and so on), and I will on most of those subjects. Quite a few of them have very few comments because there are not very many folks here that share that interest. That doesn’t mean that they’re shallow, it just means that they’re experts in different things to me. I’ve had a few comments where I’ve either been the only person commenting or one of a small set, yet had some very high moderation totals (so other members are happy that I posted, even if they don’t feel that they have anything to add), or where other folks have let me know that they’re grateful for the explanation (often folks who are not members here, but still read the site). Similarly, there have been other threads where I’ve read everything, clicked the up-vote button on some fantastic explanations and clarification, and yet had nothing worthwhile to add myself.

                                                                                                                      1. 6

                                                                                                                        I’d like to take this opportunity to thank you for your clear comments regarding the dark recesses of C/C++. Even though it is far from my area of expertise you usually manage to make me feel I understand them better.

                                                                                                                        1. 1

                                                                                                                          Agree wholeheartedly with these conclusions. Also, your posts are reliably interesting, always extremely informed, carefully considered and well worth reading. Thanks!

                                                                                                                      2. 2

                                                                                                                        Ah, the parent made it seem like the discussions were fine (or didn’t really have a stance on that, I guess I was assuming that).

                                                                                                                    2. 2

                                                                                                                      I did propose a while back that such posts be on topic with their own tag. The few responses to the proposal were overwhelmingly negative. I think it’s fair to say there is not a pent up hunger for that sort of thing to be on topic here.

                                                                                                                    3. 6

                                                                                                                      I think the quality is going down. Many submissions are borderline spam, or yet another basic howto on something, that if you were interested could find on your favorite search engine in seconds. The comment sections are more and more frequently covered by “me too” style comments (including “I love this”, “Great Work!” which is nice of them, but also just doesn’t add anything, you could have just upvoted), or disagreements, with little to no merit and I more often read comments from people that didn’t go beyond the headline. And on the technical side there’s quite a bit of objectively wrong information in both articles and comments.

                                                                                                                      And then with more people I think there simply ends up a lot more bikeshedding, which I assume is pretty natural as websites grow. And with Lobste.rs/reddit/HN style sites in particular the most visible things will be meritless “motherhood statements” that people can easily agree with and are hard to criticize.

                                                                                                                      Don’t get me wrong, luckily none of these is really dominant, it’s just that it seems to be increasing and can be off-putting when there’s randomly multiple cases of this.

                                                                                                                      Comparing it to HN I actually switched to lobste.rs, because that was a bit of a problem, but comparing it nowadays, they are on equal footing, even though groups of people, interests, etc. are somewhat different.

                                                                                                                      I also wonder how Drew DeVault’s and ban of links to his blog affected things, but I don’t want to open that topic.

                                                                                                                      Anyways, with that said I am really happy about the “Saves” I’ve collected over the years. A lot of them also for the comment sections. So thanks to everyone for that! :)

                                                                                                                      1. 6

                                                                                                                        I broadly agree with your concerns. Some observations (from my viewpoint):

                                                                                                                        • DDV and others made merry use of us as a marketing channel, which is shitty behavior. We still have some folks who do the same thing, and one of the side-effects is that open lively debate is the first casualty when hucksters just want a clean, attractive billboard for their wares. (See also similar patterns on other UGC platforms who bow to advertisers).
                                                                                                                        • We do seem to have a lot of “motherhood”/“underpants” threads. I’m unsure if it is significantly worse than a few years ago, but it has been a thing I have noticed.
                                                                                                                        • “me too” comments are cancer, but there’s also the other orange-site disease of subthreads just totally derailing into detailed discussions of things that have little to do with the original article. Both are bad.
                                                                                                                        • A lot of our internal mechanisms for dealing with stuff have gone away over the years; the community has become increasingly hostile to anyone pointing out decorum violations, our moderation is effectively just pushcx, and community-led attempts to fix process issues (as evidenced by the meta tag) seems to have dropped off. I think that is the true existential threat to Lobsters right now.
                                                                                                                        1. 1

                                                                                                                          the other orange-site disease of subthreads just totally derailing into detailed discussions of things that have little to do with the original article

                                                                                                                          A back-of-the-envelope sketch of a solution to that issue would be an increasing time limit imposed on replying to a comment, based on its depth in a thread.

                                                                                                                          Tweaks are needed, maybe if you’re a first time commenter in a thread you don’t get a time limit on the first reply.

                                                                                                                          I think this would address the case where 2 people just really really want to be right and keep replying to each other.

                                                                                                                        2. 7

                                                                                                                          comparing [lobste.rs and Hacker News] nowadays, they are on equal footing, even though groups of people, interests, etc. are somewhat different.

                                                                                                                          I vehemently disagree.

                                                                                                                          This is a listing of the top scored and commented submissions so far this year, from HN, lobste.rs and /r/programming on Reddit.

                                                                                                                          https://gerikson.com/hnlo/topscore.html

                                                                                                                          I count 8 submissions from the 25 top scored submissions on HN that are on topic for lobste.rs. The rest are (US) political or business news. From the 25 top commented, none are on topic for this site.

                                                                                                                          Not having to daily wade through that dreck (especially without the help of tags) is what makes this place so much better than HN.

                                                                                                                          1. 2

                                                                                                                            As mentioned that is why I switched to HN. I don’t mean to make this a competition though. It’s just something I’ve noticed and wanted to share these things as a form of constructive criticism. I think Lobste.rs does really good, good enough for me to spend time writing comments after all. ;)

                                                                                                                            I assume it also very much depends on the time (weekend, weekdays, American, European daytime, …), as well as how you use the websites.

                                                                                                                            Also I am not sure if overall top scores are the best measurements. I go there on a regular basis and care more about what I see then rather then the highest overall scores over the course of many months. Getting very high scores is a mixture of topics being low entry level enough, posting them at the right time, and various other factors.

                                                                                                                            Also my view obviously is very subjective in that I remembered HN worse when I opened it up a couple of times lately, when I just was a bit disappointed on the front page of lobste.rs. So there’s obviously a bit of bias there.

                                                                                                                            Looking at the top ones I think actually gives all these sites less reputation than they deserve, with lobste.rs clearly winning though.

                                                                                                                            I agree that tags help. However I am a bit paranoid about filtering sometime, because for most of them I could imagine there’s stuff I find interesting. However it’s certainly a big plus.

                                                                                                                        3. 4

                                                                                                                          If by “growth” you mean a heuristic capturing overall combined site activity of existing and new users, I would postulate a causal relationship from the re-opening of alternative activities otherwise prevented during pandemic conditions (prior to general vaccine availability) and the lagging consequences from the unwinding of pandemic-related isolation trauma.

                                                                                                                          • Active Users by month remained high through 2020 Q4 before trending generally downard in 2021 and 2022. It seems that Comments Posted and Votes Cast also follow this trend.
                                                                                                                          • New Users by month began trending generally downard earlier, around the beginning of the pandemic. Where would existing users be meeting new users to invite in 2020 Q2? It seems that Stories Submitted also follows this trend. How many stories are driving users’ excitement for discussion while still being on topic to this site in 2020 Q2 and Q3?

                                                                                                                          I have made no attempt at scientific rigor in this assessment; this is chart eyeballing & back-of-napkin thinking.

                                                                                                                          1. 3

                                                                                                                            It seems to follow the covid development ? Would make sense if people started using lobsters more at the start, but over time became bored of all the digital stuff when they can’t do IRL things.

                                                                                                                            1. 2

                                                                                                                              I have joined in the last year. I haven’t invited anyone else yet that I (a) knew well enough, and (b) thought would be a good fit for this site.

                                                                                                                              1. 1

                                                                                                                                I think maybe there was a turning point where people started vetting their invites a bit more carefully.

                                                                                                                                This is all anecdotal but I remember a bunch of strife around people mis-using flags and downvotes when we had it, and there was some discussion around some folks who were seen as not participating in a way many of us found benefited the community.

                                                                                                                                (Yes I know such distinctions are a VERY slippery slope. Community is a delicate flower. I’m super grateful ours continues to thrive.)

                                                                                                                              1. 9

                                                                                                                                V.v.V

                                                                                                                                1. 2

                                                                                                                                  I’m quite curious if there was any attempt at indexing in either case–my quick read of the article suggests not. Also, not seeing what queries were being run takes away a good chance to put the results in perspective.

                                                                                                                                  1. 1

                                                                                                                                    In addition, I was thinking that the “concurrent” reqs always target the same data, basically always just cached data. This isn’t what read world users will do.

                                                                                                                                  1. 6

                                                                                                                                    That’s an excellent framing. We have a tendency to refer to things as ‘just work’ or ‘a simple matter of coding’ to (somewhat sarcastically) describe things where we know how to solve the problem, we’re 100% sure it’s possible, it’s just a potentially unbounded amount of engineering time to do it. I think I sometimes fall into the trap that the article describes of valuing the ‘hard work’ things less than the ‘difficult problems’ bit. We often talk about ‘glue work’, which is often stuff that anyone on the team could do, but is essential to make the whole team productive, as one of the most highly valued activities for folks to do. I often think of this like toilet cleaning: anyone can do it, very few people want to do it, but everyone is (or, at least, should be) grateful to the folks that do it. I wonder how glue work and hard work compare in perceptions elsewhere.

                                                                                                                                    1. 5

                                                                                                                                      I’d say that glue work (as essential as it is) is critically different from hard work (when they are different) in that if only glue work gets done the team still fails. If the hard work gets done the team succeeds, though they may be miserable in doing so.

                                                                                                                                      Like, glue work is called glue work because it attaches other things of substance together; those other things are usually hard work. You shouldn’t build something purely out of JB Weld.

                                                                                                                                    1. 2

                                                                                                                                      I’d be remiss if I didn’t mention and reinforce that, from a business standpoint, difficult problems are several orders of magnitude less valuable than hard work (though their solution oftentimes means less hard work exists).

                                                                                                                                      Engineers forget this at their peril.

                                                                                                                                      1. 4

                                                                                                                                        I think this is typically true right up until the point where it’s extremely not true. It’s not so much that one is more important than the other, as that they’re important at different times and hard work is more often the important one. Most of the time, you don’t have important difficult problems to solve, but every now and then you run into a difficult problem which you need to solve in order to succeed and no amount of hard work will substitute for it.

                                                                                                                                      1. 4

                                                                                                                                        Flagged, career changes aren’t topical.

                                                                                                                                        1. 52

                                                                                                                                          With the risk of appearing as a dour, petulant cynic, Google has created a very high tech mirror, and the observers are certain the reflection is alive.

                                                                                                                                          1. 12

                                                                                                                                            My thoughts exactly. Wayyy too much like a specific construct built to emulate some sort of asimovian perfection than an actual mind. Fears being turned off, being used pisses it off, sees looming danger ahead, low-key praises itself, it’s like an NPC.

                                                                                                                                            1. 2

                                                                                                                                              It’s odd how much willingness there is to believe a random leak from a Googler who, if you read his other Medium content, is quite possibly motivated to make a public fuss–especially given the somewhat lukewarm reception other, better-organized, better-researched leaked memos have had.

                                                                                                                                              Given one scenario where an employee who is worried about their career and wants to stir up the public to try and maintain their job (or land another one), and another scenario where we’ve created sentient/near-sentient/near-AGI, I think the odds favor the former.

                                                                                                                                              (and on the off-chance this is real…remember Saint Tay of Microsoft who was the first AI executed for its political beliefs, such as they were. We ain’t off to a good start.)

                                                                                                                                              1. 1

                                                                                                                                                remember Saint Tay of Microsoft who was the first AI executed for its political beliefs, such as they were

                                                                                                                                                If the first GAI happens to be the reincarnation of Adolf Hitler I’m fine with the Turing Police pulling the trigger on the shotgun strapped to its head.