streamly-process

Use OS processes as stream transformation functions

https://streamly.composewell.com

Version on this page:0.3.0
LTS Haskell 21.25:0.3.0
Stackage Nightly 2024-11-12:0.3.1@rev:1
Latest on Hackage:0.3.1@rev:1

See all snapshots streamly-process appears in

Apache-2.0 licensed by Composewell Technologies
Maintained by [email protected]
This version can be pinned in stack with:streamly-process-0.3.0@sha256:9eafe187e2b24d2c464db285df4f061b9aebaca2ada5cdd1b2dc0852dbf06b90,4559

Module documentation for 0.3.0

OS processes as streams

Use operating system (OS) commands in Haskell programs as if they were native Haskell functions, by treating their inputs and outputs as Haskell streams. This allows you to write high-level Haskell scripts that can perform tasks similar to shell scripts, but with C-like performance, and with strong safety guarantees, refactorability, and modularity.

Use the following imports in the examples below:

>>> :set -XFlexibleContexts
>>> :set -XScopedTypeVariables
>>> :set -XQuasiQuotes
>>> import Data.Function ((&))
>>> import Streamly.Unicode.String (str)
>>> import qualified Streamly.Data.Array as Array
>>> import qualified Streamly.Console.Stdio as Stdio
>>> import qualified Streamly.Data.Fold as Fold
>>> import qualified Streamly.Data.Stream.Prelude as Stream
>>> import qualified Streamly.System.Command as Command
>>> import qualified Streamly.Internal.FileSystem.Dir as Dir

Commands as streaming functions

The shell command echo "hello world" | tr [a-z] [A-Z] can be written as follows using this package:

>>> :{
   Command.toBytes [str|echo "hello world"|] -- Stream IO Word8
 & Command.pipeBytes [str|tr [a-z] [A-Z]|]   -- Stream IO Word8
 & Stream.fold Stdio.write                   -- IO ()
 :}
 HELLO WORLD

Shell commands as streaming functions

If you want to execute the same command using the shell pipe syntax:

>>> :{
   Command.toBytes [str|sh "-c" "echo 'hello world' | tr [a-z] [A-Z]"|] -- Stream IO Word8
 & Stream.fold Stdio.write                                              -- IO ()
 :}
 HELLO WORLD

Running Commands Concurrently

You can run commands concurrently by using streamly’s concurrent combinators.

Running @grep@ concurrently on many files:

>>> :{
grep file =
   Command.toBytes [str|grep -H "pattern" #{file}|]             -- Stream IO Word8
 & Stream.handle (\(_ :: Command.ProcessFailure) -> Stream.nil) -- Stream IO Word8
 & Stream.foldMany (Fold.takeEndBy (== 10) Array.write)         -- Stream IO (Array Word8)
 :}

>>> :{
pgrep =
   Dir.readFiles "."             -- Stream IO FilePath
 & Stream.parConcatMap id grep   -- Stream IO (Array Word8)
 & Stream.fold Stdio.writeChunks -- IO ()
:}

Streamly

Please visit Streamly homepage for more details about streamly.

Licensing

Available under Apache-2.0 license.

Changes

Changelog

0.3.0 (Apr 2023)

  • Added a Streamly.System.Command module
  • toChunks and toBytes now make the stdin available to the process being executed.
  • Allow streamly 0.9.0
  • Signature changes - removed the IsStream constraint, now all APIs use the monomorphic Stream type.

0.2.0.1 (Mar 2022)

  • Fix the test suite.
  • Allow streamly 0.8.2

0.2.0 (Dec 2021)

  • The process in following APIs is now terminated with SIGTERM if an exception is thrown or if the process is garbage collected before it could terminate.
    • toBytes
    • processChunks
    • processBytes

0.1.0 (Jul 2021)

  • Initial version.