Shell Deep Dive: Easy Unix Daemons (on macOS too)

I've been setting up my new work machine recently, and while doing so I encountered an interesting challenge.

I wanted to setup a proxy to the company's VPN as a daemon. Here was my feature wishlist:

• Runs in the background.
• Starts at startup or login.
• Can be manually killed / relaunched.
• Output visible (e.g. via log files).
• Restarts automatically if it stops for some reason.

Now, there are daemons systems build into most OSes. For macOS this is launchd (for "launch daemon"). On Linux it's most likely than not systemd, though are other systems, like the venerable sysvinit and OpenRC.

Even before starting however, I knew (from reputation and past personal experiences) that those systems sucked mightily. The less I had to deal / rely on them, the better. But I'll expand on that later.

In the end I did actually use launchd to launch at startup. I'm not providing details for linux, but look here for pointers about systemd (and by the same occasion note that I'm right: this is a mess).

Running in the Background

Your daemon system will run jobs in the background for you. Nevertheless it's quite interesting to take a small detour to see how one can run a job in the background from the user shell. In fact this is how I stumbled upon the backbone of my solution.

The obvious way is to append & at the end of your command. This runs in the background, but the command is now a child of the current shell, and will be terminated along with the shell.

The traditional approach in this case is to do something like nohup <command> & disown. Here is a very thorough explanation of that command. But basically: disown just removes the command from the list of the shell's job while nohup redirects the terminal output to a file called nohup.out in the current directory (or you can just redirect it yourself), as well as closes terminal input. Using either disown or nohup (or both) will prevent the command from being killed when the shell is killed (it prevents the sending of the SIGHUP signal ("signal hang up"), which is "a signal sent to a process when its controlling terminal is closed").

So this works rather nicely.

But I have an even better solution: use screen.

In short, screen lets you start a virtual shell that you can "attach" to a real shell, or "detach" from. So you can use it to run a command in its own shell, which you can recall at will, in any real shell.

The advantages of screen is that you can use it for job control. We couldn't have used normal shell job control (using the command jobs) because it is shell-specific (not shared between different shell instances). So you can't use it as soon as you close the original shell used to run the command. And if you want to kill the command at that point, you need to find out its PID and use pkill.

But with screen, you can do this, from any shell:

$ screen -dm <command>           # start command in background
$ screen -dm -S <name> <command> # ... and give it a specific name
$ screen -ls                     # list all running screens
$ screen -r <name>               # (re)attach to the given screen
$ screen -S <name> -X quit       # kill the given screen

Since some of those are not super intuitive, I propose the use of the following aliases and functions:

alias sjobls='screen -ls'
alias sjoba='screen -r'
alias sjobmk='screen -dm -S'

sjobk() {
    screen -S $1 -X quit
}

Final tip: when you attach to the screen, it's ctrl+a d (or cmd+a d on mac) to detach.

Repeating the Command

To repeat a command in bash, with 3 seconds of delay between each execution, you'd normaly do:

while true; do <command>; sleep 3; done

Which you can turn into the following function:

repeat() {
    while true; do $1; sleep 3; done
}

IMPORTANT POINT: Making a function was a mistake. This is also valid for other functions that will be presented later. It should be a script to be put somewhere on the path instead. Why?

1. By making it a script, you can use a shebang (#!) to specify the shell (bash, zsh, ...) used to run it, making it usable whatever shell you decide to run as a user.

2. Functions are normally not inherited by sub-shells. In Bash, it's actually possible to export them anyway by using export -f <function>. And as we'll see later, we really want to use subshells.

This remark, aside, there is a couple improvements we might want to make to our function.

First, we'd like to be able to pass an optional argument specifying the duration of the delay between two command runs.

Second, we'd like the command to be not only a simple command, but also a whole pipeline (including pipes | and redirects >). But if we try the naive way, we'll rune into a parser issue.

Consider the command echo "x" | cat -n. This prints 1 x to standard output (-n is the option for line numbering). Then try running repeat echo "x" | cat -n. This should output 1 x multiple times. Instead it prints 1 x, then 2 x, then 3 x, etc. This is because the command is interpreted as (repeat echo "a") | cat and not repeat (echo "a" | cat) (note the use of parentheses here is not valid Bash syntax).

How to pass a pipe then? We have to quote it, then pass it as a parameter to bash -c. e.g. repeat 'echo "a" | cat -n' This is by no means perfect — it gets annoying when dealing with commands that already have multiple level of quotations: for instance echo "'a'" should have been quoted as 'echo "\'a\'"'. But it's still a good step forward.

With all the considerations factored in, we get the following function:

repeat() {
    local OPT OPTIND OPTARG T=3
    while getopts ":t:" OPT; do
        case $OPT in
            t)  T=$OPTARG;;
            \?) echo "Usage: repeat [-t <Time interval in seconds>] <command>";
                return;;
        esac
    done
    shift $((OPTIND - 1))
    while true; do bash -c "$*"; sleep $T; done
}
export -f repeat

For a brief description of argument handling: the option string :t: says that to disable getopt's normal error output (first colon :): we'll do it ourselves with the \? option. The t: part says we expect an optional argument t that takes a value (the second colon :). The variable OPT receives the option letter in the loop, while OPTARG received the option value (if any). OPTIND is set the index of the next (unprocessed) argument. The loop stop when all options have been processed. The shift $((OPTIND - 1)) removes all processed options from the list of arguments (accessible via $*, $@ and $1, $2, ...). Check this tutorial for more information.

An important note on bash -c "$*": "$*" will expand to all remaining arguments, quoted as a single parameter. In this way, repeat can be used with or without quoting whenever supported: repeat echo "x" or repeat 'echo "x" | cat -n' (using pipes while not quoting still doesn't work).

The alternative to "$*" is "$@" which expands to the same arguments, but quoted individually. Since bash -c expects a single argument, this wouldn't work when used without quotes. We'll however use "$@" later!

Getting The Process' Output

A simple way to get the process' output even though it runs in the background is simply to redirect it. Because of how we built it, this is even compatible with our repeat function.

But if we're going to use screen, we can simply attach to the screen and see what's going on.

This has, however, two slight issues:

• The logs are lost in case the machine shuts down. Here, simply using the tee command (manpage) works: <command> | tee logfile enables us to benefit from both logs and screen output.

• If the command terminates, then the screen shuts down and the output is lost. Not necessarily a problem if you log, but it is if you don't.

(At this point, I'll note that for my proxy I don't really care about long-time logs, I just want to be able to know what's happening right now, sometimes. Also proper log management requires some thought, as "just append forever" might work in practice, but it makes fussy me shudder.)

To solve that second issue, I introduced a function called remain, which would take a command as parameter (much like repeat) and run a shell once that command exited.

You can actually achieve that quite easily with just <command>; bash, as long as the command exits by itself. By hitting ctrl+c in the screen (and thus sending SIGINT), you'll shut down the whole chain, just like in a regular shell.

But imagine you want to kill the command, while still being able to peruse the output in the shell? (Again, quite relevant to my use case: stop trying to connect to the proxy, but let me see what's going on.) Well that's possible too:

remain() {
    trap "echo Interrupted by user" SIGINT
    bash -c "$*"
    exec bash
}
export -f remain

This traps SIGINT, making it echo a message rather than letting it kill the screen. We run our command using bash -c (again, to enable passing whole pipelines to the function). We end by exec bash which makes the process becomes bash. We could just have written bash there instead, this is just a tiny bit more economical.

Quoting Woes

It's time for a little confession: repeat and remain are not entirely robust as-is.

Consider that we want to use remain with repeat:

remain repeat 'echo "x" | cat -n'

What happens is that the bash -c "$*" line in remain expands to something equivalent to bash -c 'repeat echo "x" | cat -n' — the outer quotes have been stripped! This will misbeheave in repeat, producing the wrong 1 x, 2 x (etc) output rather than straight 1 x each time.

We cannot "just" use "$@" instead of "$*" either: bash -c "$@" would expand to bash -c 'repeat' 'echo "x" | cat -n' — but bash -c expects a single argument!

Important note: I'm slighlty simplifying here. In particular here, bash -c "$@" would not literally expand to bash -c 'repeat' 'echo "x" | cat -n'. Instead the shell would perform expansion and the result would be an interpretation like "invoke bash with three parameters of value -c, repeat and repeat' 'echo "x" | cat -n on which no further expansion must be performed". Otherwise, "$@" wouldn't work with parameters containing single quotes!

The solution is to re-insert the quotes manually. for this we introduce a function called quote_args, which takes a series of arguments and sets ARGS to the concatenation of all those arguments with quotes inserted around each of them (excepted when there is a single argument).

quote_args() {
    ARGS=''
    # single argument: output directly
    if [[ $# -eq 1 ]]; then
        ARGS="$1"
        return
    fi
    # multiple arguments: quote each and output
    for ARG in "$@"; do
        ARGS+="'$ARG'"
    done
}
export -f quote_args

Why add exception for single arguments? Consider the following examples given that we always quote even on single args:

always_quote_args echo hello

bash -c "$ARGS"
# 1.    "expansion": bash -c "'echo' 'hello'"
#       output: "hello"

bash -c $ARGS
# 2.    "expansion": bash -c 'echo' 'hello'
#       output: ""

always_quote_args 'echo hello'

bash -c "$ARGS"
# 3.    "expansion": bash -c "'echo hello'"
#       output: bash: echo hello: command not found

bash -c $ARGS
# 4.    "expansion": bash -c 'echo hello'
#       output: "hello"

(Expansion is quoted because again, bash does not expand to a textual representation, but to an internal representation tracking "words".)

We would like behaviour 1 and 4, meaning we need to know the number of arguments to know whether to quote $ARGS. So we move this choice into quote_args itself and now we should just always quote $ARGS.

This is not always "correct" depending on what you need to do, but it's good for commands like bash -c that expect a command as a single argument. As always, the key is to understand what this is doing so you can reason about it.

Now that we have quote_args, we can patch repeat and remain with it:

repeat() {
    local OPT OPTIND OPTARG T=3
    while getopts ":t:" OPT; do
        case $OPT in
            t)  T=$OPTARG;;
            \?) echo "Usage: repeat [-t <Time interval in seconds>] <command>";
                return;;
        esac
    done
    shift $((OPTIND - 1))
    quote_args "$@"
    while true; do bash -c "$ARGS"; sleep $T; done
}
export -f repeat

remain() {
    trap "echo Interrupted by user" SIGINT
    quote_args "$@"
    bash -c "$ARGS"
    exec bash
}
export -f remain

So in our initial example (remain repeat 'echo "x" | cat -n'), we end up with ARGS containing 'repeat' 'echo "x" | cat -n'. Using "$ARGS" within remain then produces a single argument where the individual arguments are properly quoted: "'repeat' 'echo "x" | cat -n'". (Note that in this case we are literally expanding to that, and that quote_args is not meant to be used with arguments that contain literal single quotes!) Within repeat, the single argument is note quoted again and passed to bash -c directly. It works!

Putting the Pieces Together

Now that we have a repeating mechanism and one to be able to preserve the output, we still need to combine those things together, and then to combine them with screen.

Let's consider a simpler but useful case first. If we want to use screen for job control, it'd make sense to have a command that:

1. Calls remain so that we can check on the output after the job is "done".
2. Can take pipes just like we did in repeat and remain, and not just a single command. Additionally, screen can't normally take Bash function calls as parameter (that's because it passes its arguments to exec, not bash). Our fix will allow that.

For this simple(r) case, I'm making a function called sjob (for "screen job") that takes an optional -n option to set the job name (equivalent to screen's own -S parameter) and (a) parameter(s) to specify the command, similar to remain and repeat.

sjob() {
    local OPT OPTIND OPTARG N=sjob
    while getopts ":n:" OPT; do
        case $OPT in
            n)  N=$OPTARG;;
            \?) echo "Usage: sjob [-n <Name>] <command>";
                return;;
        esac
    done
    shift $((OPTIND - 1))
    quote_args "$@"
    screen -dm -S $N bash -c "remain $ARGS"
}
export -f sjob

You can see all our previous tricks: option handling, quote_args and a direct call to remain.

Now for the full shebub, we also want to throw repeat in the mix. Since this is going to be run by some kind of init system (launchd, systemd, ...), we also want to make sure that the command is only run once even if the function is called multiple times. We'll use a unique name + screen -ls to ensure that.

daemon() {
    local OPT OPTIND OPTARG T=3 N=daemon
    while getopts ":n:t:" OPT; do
        case $OPT in
            t)  T=$OPTARG;;
            n)  N=$OPTARG;;
            \?) echo "Usage: daemon [-n <Name>] [-t <Time interval in seconds>] <command>";
                return;;
        esac
    done
    shift $((OPTIND - 1))
    # Only run if the daemon by this name is not already running.
    (screen -ls | grep -q $N) && return
    sjob -n $N repeat -t $T "$@"
}
export -f daemon

Note that this doesn't use quote_args: sjob can handle that, and we use "$@" meaning the arguments we receive (minus the processed options) will be passed as "as-is" and not unduly split because of stripped quotes.

Calling at Startup (mac-only)

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>EnvironmentVariables</key>
    <dict>
      <key>PATH</key>
      <string>/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:</string>
    </dict>
    <key>Label</key>
    <string>com.startup</string>
    <key>Program</key>
    <string>/Users/norswap/bin/startup</string>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <false/>
    <key>LaunchOnlyOnce</key>
    <true/>
    <key>StandardOutPath</key>
    <string>/tmp/startup.stdout</string>
    <key>StandardErrorPath</key>
    <string>/tmp/startup.stderr</string>
  </dict>
</plist>

#!/bin/bash
P=/Users/norswap/startup
for file in $P/*
do
    $file
done

#!/bin/bash -l
daemon -n proxyvpn 'echo <password> | openconnect --script-tun \
    --script "ocproxy -D 9999" -u <username> -passwd-on-stdin <vpn address>'

launchctl load -w ~/Library/LaunchAgents/com.startup.plist

Service could not initialize: 19D76: xpcproxy + 15636
com.apple.xpc.launchd[1] (com.startup[2355]): Service exited with abnormal code: 78