As part of my series of write-about-personal-projects, my latest obsession is writing shell scripts with Michael Snoyman’s Conduit.
Here is my package, shell-conduit. It’s still in the experimental phase, but I don’t forsee any changes now for a while.
I hate writing scripts in Bash. Until now, it was the easiest way to just write unix scripts. Its syntax is insane, incredibly error prone, its defaults are awful, and it’s not a real big person programming language.
If you’re going to go as far as using a real programming language, why bother with these dynamically typed messes? Go straight for Haskell.
I had an inkling a while back that conduits mirror the behaviour of bash pipes very well. I knew there was something to the idea, but didn’t act on it fully for a while. Last week I experimented somewhat and realised that the following Haskell code
does indeed accurately mirror
And that also the following
is analogous to
We’ll see examples of why this works later.
Another trick I realised is to write some template Haskell code which will calculate all executables in your PATH at compilation time and generate a top-level name that is a Haskell function to launch that process. So instead of writing
you could instead just write
There are a few thousand executables, so it takes about 10 seconds to compile such a module of names. But that’s all.
Again, we’ll see how awesome this looks in a minute.
My choice of modeling the typical shell scripting pipe handles is by having a type called
Left values are from
Right values are either being pulled from
stdin or being sent to
stdout. In a conduit the difference between
stdout is more conceptual than real.
When piping two commands, the idea is that any
Left values are just re-yielded along, they are not consumed and passed into the process.
Putting the previous model into practice, we come up with a type for launching a process like this:
Meaning the process will be launched, and the conduit will accept any upstream
Right values), and send downstream anything that comes from the actual process (both
I defined two handy functions for running process conduits:
One to launch via a shell, one to launch via program name and arguments. These functions can be used in your shell scripts. Though, we’ll see in a minute why you should rarely need either.
First we want something to consume any remainder chunks after a script has finished. That’s
This simply consumes anything left in the pipeline and outputs to the correct file handles, either
Now we can write a simple
First it yields an empty upstream of chunks. That’s the source. Then our script
p is run as the conduit in between, finally we write out any chunks that remain.
Let’s try that out:
Looks good. Standard output was written properly, as was stderr.
Let’s take our earlier work of generating names with template-haskell. With that in place, we have a process conduit for every executable in
PATH. Add to that variadic argument handling for each one, we get a list of names like this:
The real types when instantiated will look like:
We can now provide any number of arguments:
We can pipe things together:
Results are outputted to stdout unless piped into other processes:
Live streaming between pipes like in normal shell scripting is possible:
--line-buffered if it is to output things line-by-line).
By default, if a process errors out, the whole script ends. This is contrary to Bash, which keeps going regardless of failure. This is bad.
In Bash, to revert this default, you run:
And the way to ignore erroneous commands on case-by-case basis is to use
Which means “do foo, or otherwise ignore it, continue the script”.
We can express the same thing using the Alternative instance for the
OverloadedStrings so that you can use
Text for arguments, then also enable
ExtendedDefaultRules, otherwise you’ll get ambiguous type errors.
But this isn’t necessary if you don’t need to use
Text yet. Strings literals will be interpreted as
String. Though you can pass a value of type
Text or any instance of
CmdArg without needing conversions.
Quick script to reset my keyboard (Linux tends to forget these things when I unplug my keyboard):
Cloning and initializing a repo (ported from a bash script):
import Control.Monad.IO.Class import Data.Conduit.Shell import System.Directory main = run (do exists <- liftIO (doesDirectoryExist "fpco") if exists then rm "fpco/.hsenvs" "-rf" else git "clone" "firstname.lastname@example.org:fpco/fpco.git" liftIO (setCurrentDirectory "fpco") shell "./dev-scripts/update-repo.sh" shell "./dev-scripts/build-all.sh" alertDone)
Script to restart a web process (ported from an old bash script I had):
import Control.Applicative import Control.Monad.Fix import Data.Conduit.Shell main = run (do ls echo "Restarting server ... ?" killall name "-q" <|> return () fix (\loop -> do echo "Waiting for it to terminate ..." sleep "1" (ps "-C" name $= discardChunks >> loop) <|> return ()) shell "dist/build/ircbrowse/ircbrowse ircbrowse.conf") where name = "ircbrowse"
Right. Shelly’s fine. It just lacks the two killer things for me:
mv "x" "y".
Also, Shelly cannot do live streams like Conduit can.
You might want to import the regular Conduit modules qualified, too:
Which contains handy functions for working on streams in a list-like way. See the rest of the handy modules for Conduit in conduit-extra.
Finally, see the Conduit category on Hackage for other useful libraries: http://hackage.haskell.org/packages/#cat:Conduit
All of these general purpose Conduits can be used in shell scripting.
So far I have ported a few small scripts to shell-conduit from Bash and have been happy every time. I suck at Bash. I’m pretty good at Haskell.
The next test is applying this to my Hell shell and seeing if I can use it as a commandline shell, too.
My friend complained that having to quote all arguments is a pain. I don’t really agree that this is bad. In Bash it’s often unclear how arguments are going to be interpreted. I’m happy just writing something predictable than something super convenient but possibly nonsense.
I set out a week ago to just stop writing Bash scripts. I’ve written a bunch of scripts in Haskell, but I would still write Bash scripts too. Some things were just too boring to write. I wanted to commit to Haskell for scripting. Today, I’m fairly confident I have a solution that is going to be satisfactory for a long while.