Streams provide a way to read and write potentially unlimited information and working with them is very different than data types representing a single variable.

This episode continues the explanation of streams. There’s more than just reading and writing to them. One very important aspect is buffering which is like batching. Buffering allows you to save writes and when there’s been enough, then they all get written to the stream destination at once. One example that comes to mind is not really accurate at first but I’ll explain why it’s not a good example and then start changing it until it’s better. Sometimes coming up with something that’s not quite right is a great way to understand a topic as long as you know why.

Let’s say you want to organize a camping trip but the expenses are too much for you to consider going alone. So you start asking friends and people you know if anybody is interested to go with you. You tell them that unless you get enough people, the trip will be cancelled. If you get enough people, then the costs are divided and the trip becomes affordable. This is a good example of buffering because you and your friends are delayed for a bit but eventually go on the trip that would have been too expensive for anybody to afford alone.

The reason this is not a good example of buffering is because it can be cancelled. Stream buffering will delay writes until there are enough to justify the costs of writing the information to the stream. But it wouldn’t be used very much if the writes could be abandoned. A better example is similar. But instead of telling your friends that the trip will be cancelled, the message is to try getting as many people to join in order to reduce costs and that everybody should be ready to go anyway even if not enough people join. This fits better with the way buffering works. Once you’ve committed to go on the trip, just like once some data has been written to a stream, then you know you’ll eventually make it to the destination. And who knows, if you get enough people to sign up, then you might be able to make two camping trips or more. Streams usually have a buffer size that will hold writes until it fills up. The information will be sent to the destination as many times as the buffer fills up.

Some streams may not be buffered though. This would be like a camping trip where the camp sites can only hold one person. You might as well proceed to the destination right away if there’s only room for one person anyway. Normally, you don’t have to worry about data written to a stream reaching the destination once it’s been written but this doesn’t always apply in error conditions. And this is a good reason why you might want to consider turning a stream buffer off.

Buffering will normally give you better performance just like how your friends are able to lower the costs of a camping trip. But there is a tradeoff. What happens if you’ve already paid for your portion of the trip but while waiting for more friends to join you, the travel company goes bankrupt?

The same thing can happen in a stream. If you write data to an unbuffered stream, then it goes to the destination right away. If you try to speed up the whole application by buffering the stream, then you do get benefits until the power fails one day. If the computer loses power while data is sitting in a buffer, then it’s gone. Had it instead reached the destination, then it might be safe. It depends on the destination.

You have another option that can help make things better. You can start with a buffered stream and after writing some important data to the stream, you can flush the stream. This causes the buffer to be written to the destination right away even if it wasn’t yet ready. This approach gives you the ability to take advantage of the buffering for data while you’re building up to some final piece. Once you write that last piece to the stream, then you can flush the stream and feel as safe as possible that the data will reach the destination.

This episode describes some other features of streams that you’ll want to make sure to listen to.