Guides on Cmd.io Documentation

Overview

Tue, 31 Jan 2017 18:00:00 -0600

Cmd.io is a service that remotely runs commands and scripts. The primary interface to Cmd.io is SSH, meaning you can use Cmd.io commands from any host with an SSH client. A couple of use cases for Cmd.io include:

CLI utilities as a service. Example, bring jq wherever you go, without installing it.
Avoid installing big CLI programs. Example, latex is huge and difficult to build but used infrequently.
Scripts you can use from anywhere. Trigger and orchestrate other systems using any language.
Share and control access to automation. Build tools for your team or Rickroll your friends.

Cmd.io commands can run any x86 64-bit program in an Linux Docker container. However, it’s designed for commands and scripts as opposed to long-running daemons. In fact, commands can only run for a limited time before they timeout. So Cmd.io is also not suitable for replacing your shell. Kudos for such a clever idea, though.

Commands are unable to listen on routable ports, which is not common with most commands anyway. In the rare cases you need to, such as debugging with netcat, consider using a tool like ngrok.

Another important constraint of commands is that they are stateless. Besides configured environment variables, nothing persists across command runs. If the command makes a file, it will not be there the next time you run it. Utilities that expect files on the filesystem will need to be wrapped so they can receive the file(s) via STDIN.

Setup

Tue, 31 Jan 2017 18:00:00 -0600

User Notes

We’d love it if you took notes about your experience getting started and then using Cmd.io. You can then share on UserNotes and we can use them as living documents to improve the experience. Also, if something is wrong on this page or any other, feel free to edit this wiki!

Setup

This document is currently for the alpha release channel (alpha.cmd.io) … any accidental references to the cmd.io domain should be read as alpha.cmd.io

Cmd.io uses your GitHub user for authentication. It also relies on the public keys stored with your GitHub account. If you haven’t uploaded a public key to GitHub, you can easily add one in Settings under SSH and GPG keys.

Now you can connect to Cmd.io over SSH:

$ ssh progrium@alpha.cmd.io
Usage:
  ssh <user>@cmd.io [command]

Available Commands:
  cmd-add             Install a command
  cmd-ls              List installed commands
  cmd-rm              Uninstall a command

Use "[command] --help" for help about a meta command.

If you’re using a local user with the same username as your GitHub username, SSH will use it by default. If not, you can also avoid specifying a username with some configuration added to your ~/.ssh/config file:

Host alpha.cmd.io
    User progrium

Builtin Commands

Tue, 31 Jan 2017 18:00:00 -0600

Cmd.io is about running and managing user commands. Builtin commands let us add and remove user commands. They’re prefixed with cmd-:

$ ssh progrium@alpha.cmd.io
Usage:
  ssh <user>@cmd.io [command]

Available Commands:
  cmd-add             Install a command
  cmd-ls              List installed commands
  cmd-rm              Uninstall a command

Use "[command] --help" for help about a meta command.

Installing a command

We can install a command with cmd-add. We can see how with --help:

$ ssh alpha.cmd.io cmd-add --help
Install a command

Usage:
  ssh <user>@cmd.io cmd-add <name> <source>

Although this help could be more helpful, it does at least tell us the arguments to cmd-add. The first, name, is a name you choose for the command. The second, source, is where to get the command. Right now, the only supported sources are public Docker registries like Docker Hub. Let’s use this mysterious Docker image, progrium/welcome:

$ ssh alpha.cmd.io cmd-add demo progrium/welcome
Command installed
$ ssh alpha.cmd.io
Usage:
  ssh <user>@cmd.io [command]

Available Commands:
  cmd-add             Install a command
  cmd-ls              List installed commands
  cmd-rm              Uninstall a command
  demo

Use "[command] --help" for help about a meta command.

Although it lacks a description, the command demo is now available. Run it:

$ ssh alpha.cmd.io demo

Hello! This is a cmd.io command. All it does is display this message.
However, cmd.io commands can do lots more. They can pretty much do
anything you can do in a Docker container, except for long-running
processes like daemons.

You can install cmd.io commands from a number of sources, including
anything off Docker Hub. Once you have a command installed, you can
configure it and share access to it. Anybody that has access to your
command can run it from anywhere they have an SSH client.

Ignoring the fact this message is lying to you (Docker Hub is currently the only source for commands), you’ve run your first Cmd.io command!

The container images you install don’t need to be specific to Cmd.io. Pretty much any CLI tool in a container can be installed. Here’s a netcat container we can use to show that cmd.io port 22 is open:

$ ssh alpha.cmd.io cmd-add nc gophernet/netcat
Command installed
$ ssh alpha.cmd.io nc -z -v cmd.io 22
cmd.io (159.203.159.60:22) open

Removing commands

This is pretty straightforward:

$ ssh alpha.cmd.io cmd-rm nc
Command uninstalled

Authoring Commands

Tue, 31 Jan 2017 18:00:00 -0600

You’ll probably be using your own commands more than off-the-shelf commands. Right now, since Docker Hub is the only source for commands, making and publishing commands is as easy as any Docker container. After you’ve gone through this once, you might be surprised at how quickly you can make and update Cmd.io commands. It’s literally build, push, use.

Up to this point, we haven’t needed to use or install Docker. For the time being, you’ll need Docker to make Cmd.io commands. We highly recommend Docker for Mac if you’re running macOS.

You’ll also need a Docker Hub account and be sure to login with Docker (docker login).

Commands based on existing utilities

The recommended way to build commands from existing open source utilities is to install them via a package manager. To keep your experience snappy, and since Cmd.io may enforce a size limit on images, we highly encourage you to use Alpine Linux for all command containers.

Alpine combines the small size of Busybox (~5MB) with a large package index optimized for small disk footprints. You can search for packages based on name or based on contents. If you can’t find a package for a utility, you can try using ubuntu-debootstrap, which is a minimal Ubuntu image with apt-get. However it starts at ~90MB and easily bloats from there.

Here is jq for Alpine v3.4, so we can make a container for it with a simple Dockerfile that uses the apk package tool:

FROM alpine:3.4
RUN apk add --update --no-cache jq
ENTRYPOINT ["/usr/bin/jq"]

The directives used in this example are nearly all that make sense to use for Cmd.io commands, but here is the full Dockerfile reference.

Now we can build this with Docker, assuming we’re in the directory with the Dockerfile. Immediately after building, we can push to Docker Hub. Replace progrium with your Docker ID.

$ docker build -t progrium/jq .
...
$ docker push progrium/jq
...

At this point you can now install this command on Cmd.io like before. If you push new versions of the image to Docker Hub, Cmd.io will pull it just before the next run.

Commands based on scripts

Making a container for a script is not that different from making it for an existing utility. You’ll want to install the interpreter and any other utilities the script depends on the same way as before. But you’ll also be adding your script and making it the entrypoint.

Create a file called netpoll and make sure it’s executable with chmod +x netpoll. Inside it, put this Bash script that uses netcat to poll an address and port for roughly 10 seconds or until the port accepts a connection. If it connects it returns. If it times out it returns non-zero. A rather handy little script.

#!/bin/bash
for retry in $(seq 1 ${TIMEOUT:-10}); do
  nc -z -w 1 "$1" "$2" && break
done

In the same directory create a Dockerfile like this:

FROM alpine:3.4
RUN apk add --update bash netcat-openbsd
COPY ./netpoll /bin/netpoll
ENTRYPOINT ["/bin/netpoll"]

Build, push, and install with Cmd.io. Let’s say I installed it as netpoll. I can run it against cmd.io port 22 and it returns immediately with status 0. Run against cmd.io port 23 and it blocks for at least 10 seconds before giving up and returning status 1.

$ ssh alpha.cmd.io netpoll cmd.io 22; echo $?
0
$ ssh alpha.cmd.io netpoll cmd.io 23; echo $?
1

Meta Commands

Tue, 31 Jan 2017 18:00:00 -0600

Some of the real power in Cmd.io comes from what you can do with meta commands. Meta command are commands that operate on a command. They look like <command>:<metacommand>. Let’s use the :help meta command on our netpoll command from before to see what we can do:

$ ssh alpha.cmd.io netpoll:help
Usage:
  ssh <user>@cmd.io netpoll:[command]

Available Commands:
  :access     Manage command access
  :admins     Manage command admins
  :env        Manage command environment

Use "[command] --help" for help about a meta command.

You can explore :access and :admins on your own. In short, access lets you share a command with others by adding and removing GitHub usernames. They can run it prefixed with your username and a slash. For example, if I shared netpoll with you, you could run it with ssh cmd.io progrium/netpoll. Access also lets you make a command public, letting any user run it, or make it private again.

The difference between access and admins is currently that admins have access to these meta commands just like you. Users with access don’t. A useful dynamic this allows is storing credentials in configuration, and users with just access can use your commands that use those credentials, but aren’t able to see the credential themselves.

However, if there is a way to display the environment in your command, those credentials will be plainly visible. Use at your own risk.

Setting environment variables

Let’s look at the :env meta command:

$ ssh alpha.cmd.io netpoll:env --help
Manage command environment

Usage:
  demo :config [command]

Available Commands:
  set         Manage command environment
  unset       Manage command environment

Use "[command] --help" for help about a meta command.

What this help is not telling you is that running :env without --help will list current configuration values. However if you do that now, there will be no output since there is no configuration. Let’s set configuration on netpoll.

Configuration is exposed to commands as environment variables, so you can usually just think of :config as managing environment variables. If you look back, our netpoll script actually uses a variable if set called TIMEOUT, which defaults to 10. We can change that value by setting TIMEOUT:

$ ssh alpha.cmd.io netpoll:env set TIMEOUT=30
Config updated.
$ ssh alpha.cmd.io netpoll:env
TIMEOUT=30

Now if you use netpoll against a port that’s not accepting connections, it’s going to loop 30 times instead of 10. This will apply to any user that has access to this command.

Run API

Tue, 31 Jan 2017 18:00:00 -0600

You can run commands without SSH as well. The Run API gives you an HTTP and WebSocket API.

Authentication

Since we can’t use your SSH key easily for HTTP authentication, it’s best to use tokens. You can also use your GitHub username and password via Basic Auth in development, as we ensure the connection uses TLS. However this should never be used in production integrations.

Tokens

The builtin command cmd-tokens has a number of commands for managing access tokens that you can give access to like a regular user.

$ ssh alpha.cmd.io cmd-tokens
Usage:
  ssh <user>@cmd.io cmd-tokens [command]

Available Commands:
  ls          List tokens
  new         Create a token
  rm          Delete a token

Additional help topics:

Use "[command] --help" for help about a meta command.

Creating a token will generate a UUID based token:

$ ssh alpha.cmd.io cmd-tokens new
Token created: 84e67956-0ec3-4e49-a588-8788719dea95

Now we can add this to a command:

$ ssh alpha.cmd.io welcome:access add 84e67956-0ec3-4e49-a588-8788719dea95
Access granted.

Be careful with these tokens. This token now lets anybody that has it run your command over SSH as well as the Run API. When using over SSH, the token is used as the username and no key or password is required.

HTTP API

Now that you have a token you can use the run API endpoint:

https://alpha.cmd.io/run/<user>/<command>

The user path part is the owner of the command, in this case, your username. Performing a GET request with the token will capture all the output (combined STDOUT and STDERR) and return it in the response:

$ curl https://84e67956-0ec3-4e49-a588-8788719dea95@alpha.cmd.io/run/progrium/welcome

Hello! This is a cmd.io command. All it does is display this message.
However, cmd.io commands can do lots more. They can pretty much do
anything you can do in a Docker container, except for long-running
processes like daemons.

You can install cmd.io commands from a number of sources, including
anything off Docker Hub. Once you have a command installed, you can
configure it and share access to it. Anybody that has access to your
command can run it from anywhere they have an SSH client.

With the query parameter ?stream=1, the output will be streamed in chunked transfer encoding. There is no support for STDIN or separate STDOUT and STDERR output via this API.

WebSocket API

The same endpoint can be upgraded to a WebSocket connection. Output will be streamed back as text.

Advanced

Tue, 31 Jan 2017 18:00:00 -0600

Handling `git push`

There are a number of patterns and recipes to cover elsewhere in this wiki, but I’m going to share this one here. You can use Cmd.io commands over SSH as Git remote endpoints to implement and react to git push from your repositories. Specifically, you can use one command for Git remotes: git-receive-pack.

You see, all that happens when you git push to an SSH remote like git@github.com:gliderlabs/cmd.git is it translates to the SSH command ssh git@github.com git-receive-pack 'gliderlabs/cmd.git'. This opens a session to github.com and runs the command git-receive-pack with the repo path argument. The actual git-receive-pack command then reads packed Git data over STDIN and applies it to the bare repository on the filesystem that was provided as an argument.

What Heroku and later gitreceive, Dokku, Flynn, etc all do in some form effectively is wrap git-receive-pack and install a pre-receive hook into the repository (perhaps created on the fly) that does some task. Git is designed to display the output of that hook back to the user during the push, so from that hook script you can git archive to get a tar of what was pushed and do whatever you want with it.

Currently, you can do this with Cmd.io by creating a command named git-receive-pack, which will handle all Git pushes to Cmd.io that authenticate with your username. Cmd.io is doing nothing special to make this work, but now you need a command that will properly handle the push.

Here’s an example to get you started. It involves three files: Dockerfile, git-receive, and pre-receive. Remember the last two need to have chmod +x run on them.

Dockerfile

FROM alpine:3.4
RUN apk add --update --no-cache git sed bash
COPY ./git-receive /bin/git-receive
COPY ./pre-receive /hooks/
ENTRYPOINT ["/bin/git-receive"]

Note that anything your pre-receive script is going to use also needs to be installed in this Dockerfile.

git-receive

#!/bin/bash
repo="$1"
if [[ "$repo" != /* ]]; then
  repo="/$repo"
fi
git init --quiet --bare "$repo"
cp /hooks/pre-receive $repo/hooks
git-shell -c "git-receive-pack '$repo'"

This normalizes the repository argument, creates a bare repo in that location, installs pre-receive as a hook for that repository, and performs the actual git-receive-pack. This will then trigger pre-receive.

pre-receive

#!/bin/bash

main() {
  # reads git push header data into variables
  read old new ref

  # use archive to tarpipe pushed branch files to a working directory
  git archive "$new" | (cd /tmp && tar -xpf -)

  # go to that directory
  cd /tmp

  # do something with the files!
  # exit non-zero and the push will fail.
}

delete-remote-prefix() {
  # this removes "remote: " that git prefixes hook output with client side
  sed -u "s/^/"$'\e[1G'"/"
}

main | delete-remote-prefix

This is just a template but from here you could deploy, do builds, run checks, or something more creative.

git remote

Once you’ve made a command from the above and installed it as git-receive-pack, you can add a remote to the repositories you want to push from like this:

$ git remote add cmd ssh://progrium@cmd.io/repo/path

The name of the remote can be anything you like, here it’s cmd. You can also drop the username if you were previously. The repo path can be anything as well, perhaps use it to determine what to do in your pre-receive script.

Unfortunately, git-receive-pack is not sharable. Perhaps in the near future there will be more integrated support in Cmd.io for this pattern.

Guides on Cmd.io Documentation

Overview

Setup

User Notes

Setup

Builtin Commands

Installing a command

Removing commands

Authoring Commands

Commands based on existing utilities

Commands based on scripts

Meta Commands

Setting environment variables

Run API

Authentication

Tokens

HTTP API

WebSocket API

Advanced

Handling git push

Dockerfile

git-receive

pre-receive

git remote

Handling `git push`