Streams provide a way to read and write potentially unlimited information and working with them is very different than data types representing a single variable.
This episode continues the explanation of streams. There’s more than just reading and writing to them. One very important aspect is buffering which is like batching. Buffering allows you to save writes and when there’s been enough, then they all get written to the stream destination at once. One example that comes to mind is not really accurate at first but I’ll explain why it’s not a good example and then start changing it until it’s better. Sometimes coming up with something that’s not quite right is a great way to understand a topic as long as you know why.
This episode describes some other features of streams that you’ll want to make sure to listen to. You can also read the full transcript below.
Transcript
This episode continues the explanation of streams. There’s more than just reading and writing to them. One very important aspect is buffering which is like batching.
Buffering allows you to save writes and when there’s been enough, then they all get written to the stream destination at once. One example that comes to mind is not really accurate but I’ll explain it anyway and then explain why it’s not a good example. Sometimes coming up with something that’s not quite right is a great way to understand a topic as long as you know why. Let’s say you want to organize a camping trip but the expenses are too much for you to consider going alone. So you start asking friends and people you know if anybody is interested to go with you. You tell them that unless you get enough people, the trip will be canceled. If you get enough people, then the costs are divided and the trip becomes affordable. This is a good example of buffering because you and your friends are delayed for a bit but eventually go on the trip that would have been too expensive for anybody to afford alone.
The reason this is not a good example of buffering is because it can be canceled. Stream buffering will delay writes until there are enough to justify the costs of writing the information to the stream. But it wouldn’t be used very much if the writes could be abandoned. A better example is similar. But instead of telling your friends that the trip will be canceled, the message is to try getting as many people to join in order to reduce costs and that everybody should be ready to go anyway even if not enough people join.
This fits the way buffering works. Once you’ve committed to go on the trip, just like once some data has been written to a stream, then you know you’ll eventually make it to the destination. And who knows, if you get enough people to sign up, then you might be able to make two camping trips or more. Streams usually have a buffer size that will hold writes until it fills up. The information will be sent to the destination as many times as the buffer fills up.
Some streams may not be buffered though. This would be like a camping trip where the camp sites can only hold one person. You might as well proceed to the destination right away if there’s only room for one person anyway.
That’s a much better example and I’ll add more to it right after this message from our sponsor.
It’s possible for a stream to change from buffered to unbuffered and back. I don’t remember ever needing to do this myself. I can see benefits in an adaptive stream that changes it buffer size according to outside conditions though. Normally, when you create a stream, you’ll specify if it should be buffered or not. And when you’re using an existing stream, you’ll want to know if it’s buffered or not.
I mentioned earlier that you don’t have to worry about data written to a stream reaching the destination once it’s been written but this is under normal conditions. And this is a good reason why you might want to consider turning a stream buffer off. Buffering will normally give you better performance just like how your friends are able to lower the costs of a camping trip. But there is a tradeoff. Maybe the travel company collects payments in advance for your group trip. If you pay by yourself and go on the trip, then you pay more but get to go. What happens if you’ve already paid for your portion of the trip but while waiting for more friends to join you, the travel company goes bankrupt.
The same thing can happen in a stream. If you write data to an unbuffered stream, then it goes to the destination right away even if that can slow things down in the long run. If you try to speed up the whole application by buffering the stream, then you do get benefits until the power fails one day. If the computer loses power while data is sitting in a buffer, then it’s gone. Had it instead reached the destination, then it might be safe. It depends on the destination.
You have an option that can help make things better. You can start with a buffered stream and after writing some important data to the stream, you can flush the stream. This causes the buffer to be written to the destination right away even if it wasn’t yet ready. This approach gives you the ability to take advantage of the buffering for data while you’re building up to some final piece. Once you write that last piece to the stream, then you can flush the stream and feel as safe as possible that the data will reach the destination.
For streams that are tied together like a stream that’s both input and output, then writing to one usually causes the other one to be flushed. This helps to synchronize the streams. Otherwise, let’s say you write some text to a stream going to the display but it gets buffered and doesn’t appear to the user yet. If you then try to read a response from the input stream, then that output stream needs to get written to the display or the user may not even realize that the computer is waiting for a reply.
There’s still a lot of details to explain about streams but most of these are not suited to explaining in a podcast.
There are error conditions such as end-of-file, or EOF, that you should know about. And yes, this applies even for streams that have nothing to do with files. It really just signals that you’ve reached the end of the stream.
There are different modes that you can open a file such as append vs. truncate that you should know about. These are details that I think you should be able to look up when needed now that you know how streams behave in general.
But there’s one more concept I want to explain in this episode and that’s something called manipulators. You see, you can send more than just data to a stream. Some of the things you can send to a stream affect the behavior of the stream itself. Either how it writes information to the destination or how it reads information from the source.
Some common manipulators allow you to control how bools are written either as text or as numbers, whether or not to skip leading whitespace when reading data, add a newline character to the output, and to set the fill character, precision, and width of numbers.
You use these manipulators just like you would read or write data. They blend right in to the overall system while still remaining separate and distinguishable from the data.