Running headless with Ubik

Ubuntu Server on Windows 10 with Cygwin/cmder, VirtualBox, and Ubik

Cory Salveson
Apr 7, 2017 7 min read Technology

Background

I ran Linux as my primary desktop for years, but now I find myself, as in a forest dark, running Windows 10. Bash on Ubuntu on Windows (BUW) is a commendable project, but it’s incomplete: BUW has a fully stocked /bin directory, for example, but ICMP is broken, so ping and traceroute don’t work. Some Python scripts also fail in ways unique to the BUW Linux environment. So, while I do hope the project goes on to get future updates, I need something a bit more powerful and flexible in the meantime.

Running a Linux virtual machine (VM) is my solution. Being a “real” Linux server, and one that I can fully control, it should be able to do anything I need for local (non-cloud) tasks. Meanwhile, the performance and resource costs of running a complete but lightweight VM, compared to BUW, are acceptable given the benefits. Unfortunately, support for things like mouse scrollback and clipboard sharing don’t work in VirtualBox (the free virtualization platform I prefer) when the guest is just a shell. I’ve also fallen in love with cmder, a persistent drop-down console environment that supports arbitrary shells, that I use constantly. Having to switch to a special, dedicated window for running commands on the VM is nearly a deal-breaker.

Running a headless VM (i.e. no display, real or virtual, attached), to which I can SSH from a shell in cmder, is therefore my ideal setup, with one key problem left to sort out: how to secure command-line SSH access to the Linux VM without BUW (which this effort undertakes to eliminate). Cygwin, “a distribution of popular GNU and other Open Source tools running on Microsoft Windows,” solves this problem while also providing additional benefits. So, starting from a setup where I was previously running a Windows command prompt as well as BUW in Cmder; with the occasional manual bootup of a VirtualBox VM in a separate window; I now have just one shell in Cmder provided by Cygwin, and—with help from a shell script I’m calling “Ubik”—an easy to use, “real” Linux server for when I need it, all in the same Cmder window.

Setup

Cmder is a big deal to me, but not required for the basic approach taken here, so I’ve listed it in the “Optional Steps” below.

Prerequisites

Setting this up requires a recent version of Windows (I’m on Windows 10) and a basic level of comfort with Linux and virtual machine management concepts. For example, I don’t detail how to install the software below; rather, I show how to configure everything to work nicely together. If there’s demand for a more step-by-step intro to this content, I’d be happy to put something together—just let me know!

Required software (versions I used listed for reference)

Optional software

Step 1: Configure port forwarding on the Linux VM

This enables SSH access to the VM.

  1. Shut down and close the VM if it isn’t closed already.
  2. Open up the Settings panel for the VM, navigating to Settings > Network.
  3. In the “Attached to:” dropdown, select NAT if not already selected.
  4. Expand the Advanced section, then click on Port Forwarding to open the Port Forwarding Rules dialog.
  5. Add a new port forwarding rule. Give it a handy name (“ssh”), leave Protocol set to “TCP”, leave Host IP blank, assign a Host Port (I’m using 3022), leave Guest IP blank, and set Guest Port to 22. Alternatively, you can accomplish this via command-line like so (my VM’s name is “Ubuntu”):
VBoxManage modifyvm Ubuntu --natpf1 "ssh,tcp,,3022,,22"

Step 2: Streamline launch and login of the Linux VM

VirtualBox understands the concept of running a “headless” server, i.e. running the machine without a (virtual) monitor attached, and this can be controlled via command-line as follows (my machine name is “Ubuntu”):

VBoxManage startvm Ubuntu --type headless

You can also stop a VM with the “save state” option via command-line; this is a feature for similar to putting your laptop to sleep by closing the lid, in which the machine state is saved in a snapshot:

VBoxManage controlvm Ubuntu savestate

To streamline this process, I’ve created a shell script I call ubik after Philip K. Dick’s novel of the same name. In the novel, the main character comes to question whether his reality is simulated or real after a series of distortions and other events. A product called Ubik, distributed in an aerosol can, is a mysterious substance that he and other characters discover to help stabilize the boundaries between different realities. Riffing on this idea, the shell script first checks which “reality” it’s running from (whether it’s being run from the guest or host), then transitions the user to the other one, starting or stopping the VM as needed and tidying up after itself. You can download it from GitHub here: https://github.com/argotechnica/ubik.

Here’s some example usage of ubik:

Cory@LYRETOME /cygdrive/c/CMD
$ ubik
Ubik ... Safe when taken as directed.
Waiting for VM "Ubuntu" to power on...
VM "Ubuntu" has been successfully started.
Attempting to SSH to VM "Ubuntu"...
Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-71-generic x86_64)

    * Documentation:  https://help.ubuntu.com
    * Management:     https://landscape.canonical.com
    * Support:        https://ubuntu.com/advantage

0 packages can be updated.
0 updates are security updates.


Last login: Fri Apr  7 13:10:18 2017 from 10.0.2.2
cory@ubuntu:~$ ./ubik
Connection to localhost closed.
Return the VM to cryonic suspension? y
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%

Cory@LYRETOME /cygdrive/c/CMD
$

If you put a copy of this script into your Cygwin and VM home users' PATH directories; grant execution permission to the script with chmod +x ubik; and edit the script to incorporate your relevant parameters, you can just type ubik from the command line, and it’ll take care of the rest.

Optional Steps

Configure Cmder with Cygwin

If a drop-down console sounds appealing to you, install Cmder and then check out this guide for instructions on configuring Cygwin with ConEmu, the upstream project from which Cmder is built: https://github.com/nadav-dav/Cygwin-guide. Note that the author says to use the path C:\cygwin\Cygwin.bat, but your path may actually be C:\cygwin64\Cygwin.bat if you installed the 64-bit version of Cygwin.

Configure key-based SSH authentication

This saves you from having to type passwords all the time. Canonical maintains a good writeup here. Essentially, in Cygwin, you can just do this:

mkdir ~/.ssh
chmod 700 ~/.ssh
ssh-keygen -t rsa

Press “Enter” for every option to get a key that doesn’t require a passphrase. Then, copy the content from one of the resulting files, id_rsa.pub, into another file, authorized_keys, located on the VM. You can do this in one command (changing your port, username, and hostname as needed) like this:

cat ~/.ssh/id_rsa.pub | ssh -p 3022 cory@localhost "cat >> ~/.ssh/ authorized_keys"

You should be prompted for the password of the destination user account on the virtual machine for this command, but if it worked correctly, you shouldn’t have to enter a password to SSH ever again.

Start Cmder automatically with Windows

Create a shortcut to the Cmder executable and put the shortcut in your Startup folder in the Start Menu. Anything placed in this folder starts automatically when your computer boots up. Windows has worked like this for years, but for some reason the folder is hidden in Windows 10. Here’s the full path, so you don’t have to hunt for it:

%APPDATA%\Microsoft\Windows\Start Menu\Programs\Startup

Discussion

Over the years, I’ve found that investing a few hours in optimizing my tech setup pays for itself many times over in future efficiencies—and opens the door for more creativity. (When I know the cost of doing x is just a few seconds and an easy-to-use command, rather than an unknown number of minutes and a memory exercise, I’m more likely to experiment with x.) In that spirit, the method proposed here sought to balance the best of three approaches for bringing Linux into Windows: the lightweight, compiled-for-Windows tools of Cygwin, the power of a “real” (virtual) Linux server, and (optionally) a fancy console interface called Cmder—with a bit of ubik to help it all gel.

References

technical tools Linux VirtualBox
Cory Salveson
Cory Salveson
Researcher, Technologist, Writer

R&D what’s next in capital markets. Passionate about tech + human learning and knowledge.