Cout and flushing behavior

schveiguy Joined: 08/02/07 Posts: 128

Posted: 01/30/08 20:55:35 I wanted to get other people's opinion on this. In the current implementation of Cout, the the system checks to see if the output is to a console. If it is, every newline flushes the buffer. If it is not, the buffer is flushed when it is full. I see the benefit of having this behavior, but the issue I have is that when I create a pipeline of applications, the D ones tend to be very bursty, which isn't what I want. For example, I have an application that takes a long time to run, and so every 10 percent through the process, I print the percentage completed. However, if I redirect this through another application, no percentage is printed until the application is 100% finished. Now I understand that it is possible to configure Cout to always flush even if the file descriptor is not a console, but I don't always remember to do this, and by the time I figure out what is wrong, I've already built a system with the incorrect value. I'm assuming the current behavior is based on the expectation that the user most of the time wants the non-flushing behavior when redirected and flushing behavior when not redirected. What I have found is that my desire for a program to behave the same way whether it is redirected or simply run on the console far overrides my desire to have Cout be more efficient in its printing (actually the desire for the latter is slightly above zero). Anyone else have this opinion (or the opposite)? -Steve

Author	Message
larsivi Joined: 03/27/04 Posts: 614	Posted: 01/30/08 21:30:47 This way to do it was implemented last spring after lengthy discussions on various topics. I don't longer remember, but I think that the always flushing versions was very much slower only was one aspect of it. You may try to search tickets for console and redirect? As it is, I very much doubt you will find support to change it again.
schveiguy Joined: 08/02/07 Posts: 128	Posted: 01/31/08 16:18:20 I can't seem to find any tickets on this except for #370, which does not appear to have effected the change. The change occurred in changeset [2421] as an experimental. Then later in [2474] the change was tentatively instated as the default. I don't see anywhere that a discussion was had on the forums before this. There was a forum post about redirecting console when the experimental change was made, but that was just an 'oh by the way, I'm making this change, it might help you', not a productive discussion. Let me bring up some reasons why not having consistent behavior is bad (at least for me): Debugging. If I'm outputting debug info, and I want to see the context around the crash, the output may be longer than the screen. In this case I output the debug printouts to a file. If the crash is a hard one (i.e. segfault), any unflushed content at that point is lost. If Cout is behaving differently because it's being redirected, then the output to the file is different than the output to the screen. Piping. If I'm piping the output of a program to another program, I want that program to behave the same. This is similar to my previous argument. If I'm expecting to have the reading program react based on the timing of the output from the original program, then I want it to work like it does without redirection. What if the program accepting the outputis a remote agent, and the output is a prompt for the user on the other end of the agent? The prompt won't appear because D has decided it's better to not flush for performance reasons, where now we have a blank screen and the user is confused as to whether the program is even running. Confusion. I've dealt with this problem several times, and each time, I'm getting closer to understanding what the problem is sooner, but I still scratch my head when it happens. I think a newbie is going to be totally confused. Performance. A program like unix's cat would not be programed to flush on newlines because what's the point? It doesn't need to have each line flushed, whether it is going to a console or a file or another application. A program that prompts the user for info, or outputs data between long pauses is not going to care about output stdout performance, and so will want newlines to flush automatically. I can't think of a case where a program wants to flush automatically on newlines when it is attached to a console, and doesn't want that behavior when not attached to a console. In all environments, it wants it one way or the other. IMHO, the behavior currently invoked when an application is redirected should be manually invoked only. In other words, if you want Cout not to flush on newlines, it should have to be specifically told to do that. Otherwise, it should flush on newlines. Failing that, Cout should by default not flush on newlines, and can be told to flush on newlines manually. I think the current behavior of deciding what it should do based on whether Stdout is a console is error prone and will cause hidden bugs. Either default to one or the other, and let the coder decide what he wants to do. -Steve
kris Joined: 03/27/04 Posts: 581	Posted: 02/02/08 03:27:21 the special case here is interactive console output ... the non-redirected variety. To make the console consistent with the rest of the I/O package one would remove the automatic flush behavior, and then users would have to explicitly flush the output as needed. We tried this, very early on, and got a lot of grief for it. If you have a way to make the flush explicit while making everyone happy, we'd love to apply it
schveiguy Joined: 08/02/07 Posts: 128	Posted: 02/04/08 15:22:46 Hm... there probably is no way to please everybody. I was under the impression that the code always flushed on a newline until it was changed to switch behavior on redirection. I didn't know that originally there was no flush on a newline. Maybe the best answer here might be just to add another method that explicitly flushes whether the system is connected to a console or not. That way, existing code is unaffected, and those who are bothered by the current implementation can simply call the new method. Maybe something like flushln, or endl (to be consistent with C++?) -Steve
kris Joined: 03/27/04 Posts: 581	Posted: 02/04/08 16:48:27 And that's why the original code did not have implicit shortcuts such as formatln() etc ... nothing ever flushed until you explicitly took that action. Now, with what you're suggesting, there would be a version of formatln which flushed and one that did not? :)
schveiguy Joined: 08/02/07 Posts: 128	Posted: 02/04/08 17:07:56 No. Only one version of formatln, which does what the current version does. Nothing in Stdout/Cout would change, the only difference would be a new function that did both newline and flush, regardless of whether the output was redirected. If you want the current behavior, use newline or formatln. If you want to guarantee consistent behavior, use the new method. If you want to use the new method with formatted output, use Stdout.format(...).newFlushingNewlineMethod; -Steve
kris Joined: 03/27/04 Posts: 581	Posted: 02/04/08 17:21:16 -- Modified: 02/04/08 17:24:20 by kris That seems terribly confusing to me. Stdout.formatln().flush is currently supported, and would appear to be more explicit?
schveiguy Joined: 08/02/07 Posts: 128	Posted: 02/04/08 18:50:21 Yeah, I think you are right. My suggested solution is really not helpful at all. The real problem that I see is the behavior of deciding what to do depending on whether the output is a console or redirected. The fact that newlines flush or not is not the problem, the problem is the decision being made based on the environment. Whether flushing is on or off by default is not as important to me as removing the decision to turn it on or off based on the redirection. I would prefer flushing be on by default, but I could live with it being off by default, because at least it would be consistent. The issue now is that it does what I want while testing (i.e. type the command on a terminal), and then it doesn't do what I want when I put it in a script and pipe the output. -Steve
kris Joined: 03/27/04 Posts: 581	Posted: 02/04/08 19:38:29 -- Modified: 02/06/08 06:02:17 by kris the proper way to do this, and be consistent across the board, is to use an explicit flush where appropriate. The interactive console mode is purely to support (arguably) lazy programmers :) Edit: the autoflush mode enabled on an interactive console is really a concession to a more concise manner of controlling a buffered stream -- effectively, you don't need to invoke .flush in typical cases. In all other uses of buffered output, an explicit flush is required to transmit a partially full buffer. One way to resolve this anomaly would be to remove buffering from the console, though that would negatively impact performance in almost all cases. The autoflush behavior can be disabled via a call to Stdout.flush(false) somewhere during application startup, which reverts Stdout to having standard buffered behavior instead. My personal take on this is that a trade-off was deemed necessary, though (like you) I don't care for the asymmetry it introduced. The only conciliation might be that an explicit .flush still operates as one would expect in all cases