BSD-3-Clause licensed by michaelt
This version can be pinned in stack with:streaming-bytestring-0.1.7@sha256:5b53960c1c5f8352d46a4f1a604d04340a08dcf7ff27ca3ed31253e02c01fd03,2968

streaming-bytestring

Build Build Status Hackage

This library enables fast and safe streaming of byte data, in either Word8 or Char form. It is a core addition to the streaming ecosystem and avoids the usual pitfalls of combinbing lazy ByteStrings with lazy IO.

This library is used by streaming-attoparsec to enable vanilla Attoparsec parsers to work with streaming “for free”.

Usage

Importing and Types

Modules from this library are intended to be imported qualified. To avoid conflicts with both the bytestring library and streaming, we recommended Q as the qualified name:

import qualified Streaming.ByteString.Char8 as Q

Like the bytestring library, leaving off the Char8 will expose an API based on Word8. Following the philosophy of streaming that “the best API is the one you already know”, these APIs are based closely on bytestring. The core type is ByteStream m r, where:

  • m: The Monad used to fetch further chunks from the “source”, usually IO.
  • r: The final return value after all streaming has concluded, usually () as in streaming.

You can imagine this type to represent an infinitely-sized collection of bytes, although internally it references a strict ByteString no larger than 32kb, followed by monadic instructions to fetch further chunks.

Examples

File Input

To open a file of any size and count its characters:

import Control.Monad.Trans.Resource (runResourceT)
import qualified Streaming.Streaming.Char8 as Q

-- | Represents a potentially-infinite stream of `Char`.
chars :: ByteStream IO ()
chars = Q.readFile "huge-file.txt"

main :: IO ()
main = runResourceT (Q.length_ chars) >>= print

Note that file IO specifically requires the resourcet library.

Line splitting and Stream interop

In the example above you may have noticed a lack of Of that we usually see with Stream. Our old friend lines hints at this too:

lines :: Monad m => ByteStream m r -> Stream (ByteStream m) m r

A stream-of-streams, yet no Of here either. The return type can’t naively be Stream (Of ByteString) m r, since the first line break might be at the very end of a large file. Forcing that into a single strict ByteString would crash your program.

To count the number of lines whose first letter is i:

countOfI :: IO Int
countOfI = runResourceT
  . S.length_                   -- IO Int
  . S.filter (== 'i')           -- Stream (Of Char) IO ()
  . S.concat                    -- Stream (Of Char) IO ()
  . S.mapped Q.head             -- Stream (Of (Maybe Char)) IO ()
  . Q.lines                     -- Stream (ByteStream IO) IO ()
  $ Q.readFile "huge-file.txt"  -- ByteStream IO ()

Critically, there are several functions which when combined with mapped can bring us back into Of-land:

head     :: Monad m => ByteStream m r -> m (Of (Maybe Char) r)
last     :: Monad m => ByteStream m r -> m (Of (Maybe Char) r)
null     :: Monad m => ByteStream m r -> m (Of Bool) r)
count    :: Monad m => ByteStream m r -> m (Of Int) r)
toLazy   :: Monad m => ByteStream m r -> m (Of ByteString r) -- Be careful with this.
toStrict :: Monad m => ByteStream m r -> m (Of ByteString r) -- Be even *more* careful with this.

When moving in the opposite direction API-wise, consider:

fromChunks :: Stream (Of ByteString) m r -> ByteStream m r

Changes

0.1.7 (2020-10-14)

Thanks to Viktor Dukhovni and Colin Woodbury for their contributions to this release.

Added

  • The skipSomeWS function for efficiently skipping leading whitespace of both ASCII and non-ASCII.

Changed

  • The ByteString type has been renamed to ByteStream. This fixes a well-reported confusion from users. An alias to the old name has been provided for back-compatibility, but is deprecated and be removed in the next major release.
  • Modules have been renamed to match the precedent set by the main streaming library. Aliases to the old names have been provided, but will be removed in the next major release.
    • Data.ByteString.Streaming -> Streaming.ByteString
    • Data.ByteString.Streaming.Char8 -> Streaming.ByteString.Char8
  • An order-of-magnitude performance improvement in line splitting. #18
  • Performance and correctness improvements for the readInt function. #31
  • Documentation improved, and docstring coverage is now 100%. #27

Fixed

  • An incorrect comment about Handles being automatically closed upon EOF with hGetContents and hGetContentsN. #9
  • A crash in group and groupBy when reading too many bytes. #22
  • groupBy incorrectly ordering its output elements. #4

0.1.6

  • Semigroup instance for ByteString m r added
  • New function lineSplit

0.1.5

  • Update for streaming-0.2