Although Cloudflare generally has pretty good documentation. In this specific case, the documentation is spread over multiple pages and actually just wrong in a couple spots. I am going to try to consolidate this process here.

My goal here was to enable programmatic SSH access to an arbitrary number of devices deployed to client networks without the need for a VPN. I use Chef, so there are some examples of Chef recipes to accomplish some tasks.

Argo

I probably do not need to go too deep here as the official description is likely much better.  Argo is a smart routing technology that connects a server to the Cloudflare network and only exposes ports and services within the Cloudflare network.  This means that you can host something inside of a network boundary without having to open up firwall ports and expose the services directly to the internet. This is done by creating a tunnel into the Cloudflare network.

Access

Cloudflare Access is an identity aware proxy (IAP) that can site in from of any application protected  by or hosted within the Cloudflare network.  Because Argo Tunnels terminate within the Cloudflare network, that means that Access can be used to protect those applications and services.

Coupling these two technologies together means that you can limit exposure of your origin server or service using Argo and then protect that service with Access (which integrates with several SSO providers, complete with MFA).

Within the last year, Cloudflare has also added support for SSH (beyond typical web services) and support for signed SSH keys.

Unfortunately, tying all of this together is a bit more complicated that I would like. I will try to break down the process here.

Documentation

Here are the pages that I referenced frequently:

• Get an Argo Tunnel set up on your origin server
• cloudflared downloads (essentially the Argo agent)
• Expose SSH via Argo and Access
• Create an Access policy
• Utilize short-lived certificates for SSH key-signing
• Automatically start an Argo Tunnel
• Argo configuration file format
• Argo FAQ

Expose SSH From Your Origin

Get an Argo Tunnel set up on your origin server

Step 1: Download cloudflared

First download the appropriate binary to your origin server from the link above.  I was doing this via automation so I opted for the binary file to simplify the process across multiple operating systems.

# Get  cloudflared binary
remote_file '/tmp/cloudflared.tgz' do
  source 'https://bin.equinox.io/c/VdrWdbjqyF/cloudflared-stable-linux-amd64.tgz'
end

# Extract the binary
execute 'extract_cloudflared' do
  cwd '/tmp'
  command 'tar xzvf cloudflared.tgz'
  not_if { ::File.exist?('cloudflared') }
end

Step 2: Login and download certificate

Next run cloudflared tunnel login on your origin and select the appropriate domain. This process generates and downloads a private key certificate to associate your origin server with the Argo tunnel. This certifcate is scoped to the primary domain (example.com) but is used to created tunnels for any number of subdomains as we will see shortly. The command will output the location of the certificate.

# Create global service directory
directory '/etc/cloudflared' do
  owner 'root'
  group 'root'
  mode '0770'
  action :create
end

# Copy in the private key for the Argo Tunnel
cookbook_file '/etc/cloudflared/cert.pem' do
  source 'cloudflare-cert.pem'
  owner 'root'
  group 'root'
  mode '770'
end

Automation

As an automation consideration, you can generate this cert on any machine and move it around. From an automation standpoint, I generated one cert and added it to my deployment process so that all of my servers could use the same primary domain. You can add this certificate to /etc/cloudflard/cert.pem on any server to achieve this goal. You will likely have to create the /etc/cloudflared directory in your automation platform.

🚨NOTE: this is a minor security risk as the exposure of this certificate would allow the holder to create arbitrary tunnels and expose services on your domain ie. istoleyoursubdomain.example.com.  

Step 3: Create an Access policy

Create an Access policy

The link basically says it all. Just be aware that you can use any email address here. Cloudflare will send an email with a TOTP code to the user if they have appropriate access.

Alternatively, Access can be configured to use an SSO provider for even greater security.

Although you could automate this step via the Cloudflare API, I have not done so yet.

Step 4: Create Configuration File

Argo configuration file format

In order to install Argo as a service, you first have to have a configuration file in place.  If you did you prior testing with your target hostname, then the file is already in place. But if you are tring to automate this across multiple devices, then you have to put it in place on your own.

In Step 2, /etc/cloudflared should have already been created. Now you just have to add a config.yml file to that directory. The format is pretty simple:

hostname: myhost.example.com
url: ssh://127.0.0.1:22

Because I was trying to do this with automation, each host was going to have to have a different subdomain. As such, in Chef, I used a template file and a random word generating library to accomplish this. The steps should be similar for any other deployment automation platform.

hostname: <%= @hostname %>.example.com
url: ssh://127.0.0.1:22

# Install random word gem
chef_gem 'random-word' do
  version '2.1.1'
  compile_time true
end

require 'random-word'

ruby_block 'exclude_words' do
  block do
    RandomWord.exclude_list << /.*-.*/
    RandomWord.exclude_list << /.*_.*/
  end
end

# Create config file
template '/etc/cloudflared/config.yml' do
  source 'cloudflare-config.erb'
  owner 'root'
  group 'root'
  mode '770'
  action :create_if_missing
  variables(hostname: lazy { "#{RandomWord.adjs.next}-#{RandomWord.nouns.next}" })
end

Step 5: Install The Service

With the certifcate and the config file in place in /etc/cloudflared, you can now install the service. The service will look in this directory for the files and start an Argo Tunnel.

sudo cloudflared service install

Or via Chef:

# Install the service
execute 'install_cloudflared_service' do
  cwd '/tmp'
  command './cloudflared service install'
end

Step 6: Connect

At this point, you should be able to SSH to your device via password or key authentication with a small addition to your SSH config file.

Add SSH Config Options

You can easily generate the needed config with: cloudflared access ssh-config --hostname somehostname.example.com. But I found it easier to just use a wildcard.

Host *.example.com
	ProxyCommand /usr/local/bin/cloudflared access ssh --hostname %h

Either way, copy that into ~/.ssh/config.

Then just SSH as normal: ssh [email protected]

What is going on?

You have now connected your origin to the Cloudflare network. Subsequently, the SSH ProxyCommand connects your client to the Cloudflare network. The network "Smart Routes" the traffic between the two.  SSH is never exposed to the internet and you can actually tunnel out of other networks to expose the service.

Troubleshooting

Argo FAQ

Per the FAQ, Argo requires outbound access on TCP ports 80, 443, and 7844 to route1.argotunnel.com and route2.argotunnel.com. ⚠️THIS IS INCORRECT⚠️

The ports are correct but the correct domains are region1.argotunnel.com and region2.argotunnel.com.

Utilize Access Short-Lived Certificates for Authentication

Utilize short-lived certificates for SSH key-signing

Now that SSH is exposed, you might be done. But if you have a bunch of different people accessing your devices, managing accounts and SSH keys could become quite the burden. Enter Short-Lived Certificates.  The article above explains it much better than I ever could.