In Remoteler 8, we introduced the TLS Routing feature that can multiplex all client connections on a single TLS/SSL port.
Recently we’ve added support for TLS Routing for Database Access when Remoteler is deployed behind an AWS Application Load Balancer (ALB). In this article, we will take a deep look at the problem with Remoteler behind an ALB and how we solved it.
By Wikipedia definition, load balancing is “the process of distributing a set of tasks over a set of resources (computing units)”.
AWS offers a few types of Elastic Load Balancers. An AWS Application Load Balancer, as the name suggests, functions at the application layer, the 7th layer of the Open Systems Interconnection (OSI) model. ALBs can make routing decisions based on the host, path, headers, etc. of an HTTP request, and can terminate HTTPS traffic with TLS certificates managed by AWS.
A Network Load Balancer (NLB), on the other hand, functions at the 4th layer of the OSI model. Hence, for load balancing UDP or non-HTTP TCP traffic, NLB is a clear choice.
How about HTTP traffic, which is technically TCP? Well, I would say ALBs are usually preferred since they have visibility into the HTTP requests, so you can make HTTP-based routing rules or inspect certain aspects of the HTTP requests in the ALB access logs. That being said, there are situations in that you would choose an NLB over an ALB, like when latency performance is critical, or when static IPs are required for the load balancer, etc.
A load balancer is usually used to achieve High Availability (HA), and Remoteler is no exception. But Remoteler is complicated. It runs more than just HTTP.
Naturally, one would just put Remoteler behind a NLB and call it a day. Well, that works, if only TCP listeners are used. If you need the NLB to terminate TLS with AWS-managed certificates, you are out of luck.
For starters, TLS listeners for NLBs do not support mutual TLS (mTLS). And when it comes to Application-Layer Protocol Negotiation (ALPN) support, only http1.1 and h2 can be served. What is worse, Server Name Indication (SNI) and ALPNs are all stripped out from the Client Hello when the load balancer performs the TLS handshake with TLS targets so the info is lost. Neither does the load balancer validate the certificates or server name of the TLS targets during the handshake.
ALBs are also found to have the same problems. And likely some of these restrictions also apply to other load balancer and reverse proxy implementations.
Then why are we even talking about ALBs? Well in a typical Remoteler setup today, an ALB can be deployed alongside of an NLB (with TCP listeners), so at the minimum HTTPS traffic like Remoteler’s webapp and web APIs can be terminated by the ALB with AWS-managed certificates. Let’s see if we can find a general solution to make the best use of it.
Ok, Remoteler uses fancy TLS. Can we do that over HTTP? Or even better, is it possible to carry an arbitrary network protocol over HTTP?
Wait a minute. Isn’t there one such protocol that ALBs natively support?
WebSockets!
How do they work?
The WebSocket connection starts with a HTTP connection upgrade request that features a couple of headers:
And if the server is happy about it, it returns:
Once the upgrade connection is established, the client and server can exchange WebSocket data frames in full-duplex mode.
This is great! Let’s see if we can generalize this process for a different protocol.
So here is the plan. Instead of an Upgrade for websocket, we will use a custom upgrade type between our server and client. In our TLS Routing case, we named it alpn but it’s really an arbitrary name.
Once the upgrade handshake is done, the client will initiate a TLS handshake to start the TLS Routing flow, inside the upgraded connection. And yes, now this TLS handshake can have all the goodies like ALPN, SNI, client certificates, etc.
It is important to note that what’s encapsulated inside the upgraded connection is exactly what the client would normally send to the Remoteler Proxy server, when there is no such ALB in the middle! Thus there is minimal change required for the original TLS Routing implementation. All that’s needed is to bootstrap it with the custom connection upgrade.
Now we know how to get through the ALB, but we also need a way to detect if an ALB is present to avoid the overhead of the connection upgrade when not necessary.
We do this by testing a custom (non-HTTP) ALPN in a TLS handshake with the remote server the client is connecting to.
It is found that an ALB will complete this TLS handshake with no negotiated protocol. That’s when we know we can proceed to do a connection upgrade.
As an ALB is usually stationary, there is no need to repeat the test on every single connection. We currently test once per client session, which should more than suffice.
In this article, we have explored a method to use a custom connection upgrade to tunnel an existing network protocol over HTTP.
For Remoteler starting v10.2.4, the tsh proxy db client command will perform a custom connection upgrade to enable TLS Routing for database protocols, when ALB is detected. And we will surely look into porting this solution to other Remoteler features like Server Access, Kubernetes Access, Application Access, and more in the future.