Running Shell Commands from Julia

Aug 8, 2013   #julialang  #projects  #tutorials  #code 

Here are some examples of starting and interacting with other programs from Julia. The official documentation is pretty good, but I want something with more (basic) examples and fewer words. I do reccommend reading that to see some of the fancier tricks you can pull (and for up-to-date documentation, when this blog post gets stale).

Running Other Programs

The easiest, most basic way to run a shell command is run.

julia> help(run)
Base.run(command)
    
   Run a command object, constructed with backticks. Throws an error
   if anything goes wrong, including the process exiting with a non-
   zero status.

You can’t communicate with the process at all, and run will block until the command finishes running.

julia> run(`echo hello!`)
hello!

Notice that Command literals are written with backticks, not single quotes (').

You can run two commands in parallel using &:

julia> run(`echo hello` & `echo world`)
hello
world

If you try to cat a file that doesn’t exist, you’ll get an error.

julia> run(`cat test.txt`)
cat: test.txt: No such file or directory
ERROR: failed process: Process(`cat test.txt`, ProcessExited(1)) [1]
 in error at error.jl:22
 in pipeline_error at process.jl:430
 in run at process.jl:413

The first line is cat printing an error message. The rest is Julia throwing an error. Note that the command is containedin backticks, not normal single quotes.

Redirecting


Base.|>

   Redirect standard input or output of a process.

   Example: run(`ls` |> "out.log")
   Example: run("file.txt" |> `cat`)

Use |> to redirect the STDOUT of a command to a file or to write a file to a command’s STDIN.

julia> run(`echo hello, world` |> "test.txt")

julia> run(`cat test.txt`)
hello, world

julia> run("test.txt" |> `cat`)
hello, world

You can also use |> to redirect a process’s output to another process.

julia> run(`echo hello` |> `cat`)
hello

julia> run(`echo $("hello\nhi\nhello, world")` |> `cat` |> `grep -n o`)
1:hello
3:hello, world

String Interpolation

You can use $ to interpolate into Command literals, in the same way you can into string literals.

julia> filename = "test\nnewline.txt"
"test\nnewline.txt"

You can see what the interpolation expands to by just printing the Command literal. For example, this doesn’t run or do anything, it just shows you what the command looks like.

julia> `cat $filename`
`cat 'test
newline.txt'`

If you run the command without having that file around, you’ll get an error because cat doesn’t exit cleanly.

julia> run(`cat $filename`)
cat: test
newline.txt: No such file or directory
ERROR: failed process: Process(`cat 'test
newline.txt'`, ProcessExited(1)) [1]
 in error at error.jl:22
 in pipeline_error at process.jl:430
 in run at process.jl:413

Creating the file resolves the error:

julia> run(`echo the name is $filename` |> filename)

julia> run(`cat $filename`)
the name is test
newline.txt

The rest of this section’s examples are mostly from the official documentation.

julia> names = ["foo","bar","baz"]
3-element ASCIIString Array:
 "foo"
 "bar"
 "baz"

Command interpolation is more specialized than just plain string interpolation. Lists will become escaped and space separated, which is more reasonable in shell commands than the normal square-brackets-and-commas notation.

julia> `grep xylophone $names`
`grep xylophone foo bar baz`

If you interpolate a list as part of a word, the resulting space separated list will be versions of the complete word:

julia> `grep xylophone $names.txt`
`grep xylophone foo.txt bar.txt baz.txt`

julia> `grep xylophone hi$names.bye`
`grep xylophone hifoo.bye hibar.bye hibaz.bye`

julia> `grep xylophone hi$(names)bye`
`grep xylophone hifoobye hibarbye hibazbye`

You can do even cooler things if you have two arrays. If you interpolate two arrays into the same word, then you’ll get all combinations of elements in the two lists.

julia> exts = ["aux","log"]
2-element ASCIIString Array:
 "aux"
 "log"

julia> `rm -f $names.$exts`
`rm -f foo.aux foo.log bar.aux bar.log baz.aux baz.log`

Getting Output

The easiest way to call a command and get it’s output as a String is readall.

julia> readall(`echo hello?`)
"hello?\n"

This words just as well with chains of commands and redirections because the chain will become a single Cmd value when the redirectin operators are done.

julia> readall("test.txt" |> `cat`)
"hello, world\n"

If you just want the contents of a file as a string, you don’t need to use cat or any other command. Just use readall on the file:

julia> readall("test.txt")
"hello, world\n"

However, sometimes you’d like to be able to read the output in over time, rather than all at once. For that, you want readsfrom:

julia> help(readsfrom)
Base.readsfrom(command)

   Starts running a command asynchronously, and returns a tuple
   (stream,process). The first value is a stream reading from the
   process' standard output.

julia> (st,pr) = readsfrom(`cat poem`)
(Pipe(active, 0 bytes waiting),Process(`cat poem`, ProcessRunning))

This use of readsfrom isn’t that interesting, since cat exits right away.

julia> pr
Process(`cat poem`, ProcessExited(0))

You can use any reading stream functions you’d like: readall, readline, etc.

julia> readline(st)
" There is a place where the sidewalk ends\n"

The pipe is closed because the process has exited.

julia> st
Pipe(closed, 718 bytes waiting)

This way, you can process the data one line (or whatever increment you like) at a time.

julia> readline(st)
"And before the street begins,\n"
    
julia> readline(st)
"And there the grass grows soft and white,\n"

julia> st
Pipe(closed, 646 bytes waiting)

When you run out of data in the pipe, readline will start returning empty strings: "".

Want to ls, but get the result as an array of file and directory names? You want to use readdir()

julia> help(readdir)
# methods for generic function readdir
readdir(path::String) at file.jl:169
readdir() at file.jl:198

julia> readdir()
6-element String Array:
 "new\nline"      
 "poem"           
 "snowmansnowman"
 "sp ace"         
 "t\tab"          

Sending Input

Sometimes, you might want to communicate with another process in a more complex way than just commandline args.

Write to STDIN and then wait for it to finish

The writesto function will give you a Pipe hooked up to the STDIN of the process and a Process. Do not read from the Pipe.

julia> (si,pr) = writesto(`cat`)
(Pipe(open, 0 bytes waiting),Process(`cat`, ProcessRunning))

If you don’t give cat any filenames (or if you pass in a -), then cat will read from STDIN until you send CTRL-D.

julia> write(si,"hello")
5

We can keep writing to cat for as long as we want. As you can see, the process is still running:

julia> pr
Process(`cat`, ProcessRunning)

When you’re done writing to the process, close the Pipe.

julia> close(si)

This will make cat exit, as it’s done doing it’s work:

julia> pr
Process(`cat`, ProcessExited(0))

If you want to wait for the process to be down doing work, you can call wait on the Process. This will block until the process exits.

julia> wait(pr)
0

Read AND Write on the same process

Using the readandwrite function, you can get two Pipes and a Process. The Pipes are the STDOUT and STDIN of the process. The Process is a Julia value representing the asynchronous process started to run the command.

You can read from the STDOUT of the process, but don’t write to it. You can write to the STDIN of the process, but don’t read from it.

julia> (so,si,pr) = readandwrite(`cat`)
(Pipe(active, 0 bytes waiting),Pipe(open, 0 bytes waiting),Process(`cat`, ProcessRunning))

julia> write(si,"hello\ngood\nbye\n")
15

julia> so
Pipe(active, 15 bytes waiting)

julia> si
Pipe(open, 0 bytes waiting)

julia> close(si)

julia> so
Pipe(closed, 15 bytes waiting)

julia> pr
Process(`cat`, ProcessExited(0))

julia> readall(so)
"hello\ngood\nbye\n"

julia> so
Pipe(closed, 0 bytes waiting)

REPL specialness

In the REPL, you can use a ; to run shell commands easily.

julia> ;ls
new?line  NGrep.jl  poem  snowman☃snowman  sp ace  t?ab  test.txt

julia> ;mkdir folder

julia> ;ls
folder	new?line  NGrep.jl  poem  snowman☃snowman  sp ace  t?ab  test.txt

julia> ;cd folder/
/home/leah/src/lambdajam/folder

The name of the directory (folder/) tab completed! :)

julia> ;ls

julia> ;touch my_new_file

julia> ;ls
my_new_file

API Summary

Setting up the process

julia> help(readsfrom)
Base.readsfrom(command)

   Starts running a command asynchronously, and returns a tuple
   (stream,process). The first value is a stream reading from the
   process' standard output.

julia> help(writesto)
Base.writesto(command)

   Starts running a command asynchronously, and returns a tuple
   (stream,process). The first value is a stream writing to the
   process' standard input.

julia> help(readandwrite)
Base.readandwrite(command)
    
   Starts running a command asynchronously, and returns a tuple
   (stdout,stdin,process) of the output stream and input stream of the
   process, and the process object itself.

Interacting with streams

julia> help(write)
Base.write(stream, x)

   Write the canonical binary representation of a value to the given
   stream.

julia> help(readall)
Base.readall(stream)

   Read the entire contents of an I/O stream as a string.

julia> help(readline)
Base.readline(stream)

   Read a single line of text, including a trailing newline character
   (if one is reached before the end of the input).

julia> help(close)
Base.close(stream)

   Close an I/O stream. Performs a "flush" first.

Interacting with processes

julia> help(wait)
Base.wait(x)

   Block the current task until some event occurs, depending on the
   type of the argument:

   * "RemoteRef": Wait for a value to become available for the
     specified remote reference.

   * "Condition": Wait for "notify" on a condition.

   * "Process": Wait for the process to exit, and get its exit code.

   * "Task": Wait for a "Task" to finish, returning its result
     value.