1. 6

I am writing a test for a toy distributed system. In the test, I spin two components locally and make them talk to each other. I use printf for debugging, so I want a simple and unambiguous way to get the node’s stderr.

The problem is that there are two stderr streams, but my driver program only have one. I need to come of with some way to multiplex several streams into one, and so I am wondering what’s the prior art here. I am looking for simplicity and convenience, rather than robustness and production readiness. The options I can think about, none of which are particularly appealing:

  • stream logs to files, split my terminal and tail -f – probably what I want in terms of final result, but requires a lot of manual steps after I run the binary. Also makes it annoying to ^C the whole thing.
  • just inherit stderr – simple, but it’s hard to say which output comes from which node
  • pipe stderrs and append node ids to each line/record, or layout output as n columns – my current favorite, but needs a fair amount of programming and I am unsure about details
  • buffer stderr and print continous chunks at the end of the test – I want live output

Is there perhaps some secret ANSI cheat escape code to unlock bonus file descriptors for stderr2 and such? Or perhaps there’s some existing library which implements something like mini-tmux? Really, interested in any prior art here, as I feel like I’ve seen this problem several times, but I can’t recall any nifty real-world solutions.


  2. 3

    The usual approach I have seen in things like Rust’s log crate is to have multiple log streams within the program (just queues, each with a unique name) and then a driver of some kind that takes multiple streams and coalesces them into one. Then the driver has several options: you can start it and say “only output logs from stream A or stream B”, or “output logs that match this pattern”, or “send logs from stream A to this network address and stream B to this other one”, or whatever. Then the usual way to multiplex multiple streams together into stderr is to have the driver output a timestamp and the stream name before each log line, which means they stay ordered and you can sort/filter/organize them with grep and related tools.

    Then, for example, you can open up a couple tmux terminals and in one run tail -f logfile | grep 'Stream A' and in another tail -f logfile | grep 'Stream B' and off you go.

    1. 3

      One approach I sometimes use under Linux could be used here. First open up a few extra terminals, and note which device they are using (use w for this). So now you have two terminals, say on /dev/pts/3 and /dev/pts/4`. Then run your test program from a different terminal, assigning stderr for each program under test to the other terminals. You get live output, and it should be easy to see what each program is writing to stderr.

      As a test, I just opened up two terminals, where one ended up on /dev/pts/0 and the other one on /dev/pts/1. On the first one (assigned to /dev/pts/0) I ran ls -l >/dev/pts/1 and the output appeared on the other terminal window.

      1. 3

        You want to assign each subprocess a pseudo-terminal (PTY - search for these terms), and map the I/O of each subprocess to a PTY. Next, read line-by-line from each PTY, prefixing with an identifier, and finally, multiplexing the lines to your supervisor’s output. tmux uses PTY’s under the hood.

        1. 2

          You don’t need programming to put multiple outputs into one if you use your shell:

          mkfifo pipe

          Then have your first process write lines to it… here’s me faking it:

          while true; do (echo -n "uptime: "; uptime) >> pipe; sleep 5; done

          And your second …

          while true; do (echo -n "date: "; date) >> pipe; sleep 5; done

          And then you can just:

          tail -f pipe
          1. 3

            I’d written the below code as answer but realised I had the requirements all wrong (I thought you were bringing the stderr of both processes into your driver program already) so it’s more like the opposite of what you’re asking about.

            This is how I’ve ‘unlocked stderr2’ previously - but this is really just a way of easily writing more streams for the shell to consume, and deal with separately… I first spotted this being used by D. J. Bernstein.

            #include <unistd.h>
            #include <string.h>
            void print(int process_number, const char * text)
              const int first_free_fd = 3;
              write(first_free_fd + process_number, text, strlen(text));
            int main()
              print(0, "Hello from stream 0!\n");
              print(1, "Hello from stream 1!\n");

            And from your shell, open up those extra fds for your program and use them:

            ./program 3> file0 4> file1
          2. 1

            pipe stderrs and append node ids to each line/record, or layout output as n columns – my current favorite, but needs a fair amount of programming and I am unsure about details

            Which logging library are you using? If you are not using one, you should adopt one right now. They take care of most of these things and more.