Skip to content

Setting up EKS Hybrid Node

This is a step-by-step guide for turning arbitrary VMs into EKS hybrid nodes that can be seamlessly integrated into a customer's EKS cluster.

The process is unfortunately not yet fully automated i.e. the steps below have to be performed manually, some only once per cluster, others once per node.

Warning

Some of the AWS resources created in the process of setting up hybrid nodes incur costs just by existing, i.e. whether any nodes are in use or not. It is thus important to delete resources that are no longer required. How to do this is not explicitly documented here, refer to the official AWS docs and use your best judgement.

The aws_eks_hybrid_node Ansible Role

Turning an existing VM into an EKS hybrid node is fully handled via our aws_eks_hybrid_node role. In order to use it, manually create a temporary inventory containing the hybrid node and a playbook that executes the role.

Adding the node to an existing EKS cluster requires setting up a VPN connection to the EKS cluster's VPC and running nodeadm ⧉ once on the hybrid node.

The owner of the cluster has to perform various other setup steps before this such as setting up the VPN connection and configuring routing tables. This is not yet fully integrated into our service offering and thus has to be performed manually. Since this role depends on values generated during this initial setup step it too has to currently be invoked manually by some handcrafted playbook.

This will hopefully be remedied in the future such that we can offer seamless integration of our VMs into customers' EKS clusters. Until then the required setup steps are described here at a high level so that they are not lost and it becomes clear what this role is actually good for.

Initial AWS-Side Setup

This assumes an EKS cluster already exists and that we have sufficient permissions to access it (TODO(Timo Nicolai): Note down exactly which permissions are necessary for all setup steps).

Here we will first describe how to gather some relevant information about the cluster. Then we'll go over setup steps that are only required once per cluster (or per VPC which is the same things unless you run multiple clusters in one VPC) and finally discuss per-hybrid-node setup.

1. Gather Information

First we need to obtain a number of facts about the EKS cluster and associated AWS resources:

Cluster name and region

This should already be known to us, export this as $CLUSTER_NAME and $CLUSTER_REGION.

Cluster Endpoint

export CLUSTER_ENDPOINT=$(aws eks describe-cluster
  --name $CLUSTER_NAME
  --region $CLUSTER_REGION
  --query 'cluster.endpoint'
  --output text)

Cluster VPC ID

export CLUSTER_VPC_ID=$(aws eks describe-cluster
  --name $CLUSTER_NAME
  --region $CLUSTER_REGION
  --query "cluster.resourcesVpcConfig.vpcId"
  --output text)

Cluster VPC CIDR

export CLUSTER_VPC_CIDR=$(aws ec2 describe-vpcs
  --vpc-ids $CLUSTER_VPC_ID
  --query 'Vpcs[*].CidrBlockAssociationSet[*].CidrBlock'
  --output text)

Cluster k8s Service CIDR

export CLUSTER_K8S_SERVICE_CIDR=$(aws eks describe-cluster
  --name $CLUSTER_NAME
  --region $CLUSTER_REGION
  --query 'cluster.kubernetesNetworkConfig.serviceIpv4Cidr'
  --output text)

Cluster Routing Table IDs

export CLUSTER_ROUTING_TABLE_IDS="$(aws ec2 describe-route-tables
  --filters "Name=vpc-id,Values=$CLUSTER_VPC_ID"
  --query 'RouteTables[*].RouteTableId'
  --output text)"

Node Security Group ID

export NODE_SECURITY_GROUP_ID=$(aws eks describe-cluster
  --name $CLUSTER_NAME
  --region $CLUSTER_REGION
  --query 'cluster.resourcesVpcConfig.clusterSecurityGroupId'
  --output text)

2. Set up LyceumEKSHybridNodeRole (Once/Once Per Cluster)

LyceumEKSHybridNodeRole is an AWS role that we supply with enough permissions to create hybrid node activations that nodeadm can use to securely register each hybrid node. If you want to use hybrid nodes accross several EKS clusters you can reuse it and only need to repeat the final step (creating an access entry per cluster).

Create the role

First create the following policy document:

cat > lyceum-eks-hybrid-node-role-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ssm.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

Then create the role:

aws iam create-role \
  --role-name LyceumEKSHybridNodeRole \
  --assume-role-policy-document file://lyceum-eks-hybrid-node-policy.json \
  --region $CLUSTER_REGION

Attach Role Policies

Attach the following role policies which together encompass the minimum set of permissions we require:

aws iam attach-role-policy \
  --role-name LyceumEKSHybridNodeRole \
  --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

aws iam attach-role-policy \
  --role-name LyceumEKSHybridNodeRole \
  --policy-arn arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy

aws iam attach-role-policy \
  --role-name LyceumEKSHybridNodeRole \
  --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly

Create an Access Entry

Finally we need to allow an LyceumEKSHybridNodeRole to access the EKS cluster, indicating that it is intended to be used with hybrid nodes via --type HYBRID_LINUX:

AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

aws eks create-access-entry \
  --cluster-name $CLUSTER_NAME \
  --region $CLUSTER_REGION \
  --principal-arn arn:aws:iam::${AWS_ACCOUNT_ID}:role/LyceumEKSHybridNodesRole \
  --type HYBRID_LINUX

3. Create a VPN Gateway (Once Per Cluster/VPC)

We need to create a VPN gateway which is the AWS side of the VPN connection we want to establish to the hybrid node. One VPN gateway can support VPN connections for all our hybrid nodes.

Create the Gateway

Run the following and export VpnGatewayId from the output to VPN_GATEWAY_ID.

aws ec2 create-vpn-gateway \
  --type ipsec.1 \
  --region $CLUSTER_REGION

Attach the Gateway to the Cluster's VPC

Run the following, this may take a bit to complete after the command finishes.

aws ec2 attach-vpn-gateway \
 --vpn-gateway-id $VPN_GATEWAY_ID \
 --vpc-id $CLUSTER_VPC_ID \
 --region $CLUSTER_REGION

4. Create a VPN Connection

Create the Customer Gateway

A customer gateway is an AWS resource which provides information to AWS about the "customer gateway device" we run on-prem. Of course in our case on-prem means a single VM somewhere in the cloud which will become our hybrid node and we don't actually own any Cisco gateway device that we can insert between AWS and this VM.

The customer gateway will thus reference a virtual gateway (one per hybrid node) created with stronSwan ⧉. This is a large part of what this role does.

Run the following and export CustomerGatewayId from the output to CUSTOMER_GATEWAY_ID. Here $HYBRID_NODE_OUTSIDE_IP is the public IP of our hybrid node VM. Note that we use static routing but aws requires a value for --bgn-asn. We pass 64512 (the smallest possible value) + 1.

aws ec2 create-customer-gateway \
  --type ipsec.1 \
  --public-ip $HYBRID_NODE_OUTSIDE_IP \
  --bgp-asn 64513  \
  --region $CLUSTER_REGION
Create the VPN Connection

We can now create the VPN connection, specifying our VPN gateway and customer gateway as endpoints. Run the following and export VpnConnectionId from the output to VPN_CONNECTION_ID.

aws ec2 create-vpn-connection \
  --type ipsec.1 \
  --customer-gateway-id $CUSTOMER_GATEWAY_ID \
  --vpn-gateway-id $VPN_GATEWAY_ID \
  --options '{"StaticRoutesOnly":true}' \
  --region $CLUSTER_REGION
Create VPN Connection Routes

We have to tell AWS to route traffic to our hybrid node via the VPN connection. We can somewhat freely choose the IP address ranges to give to both our hybrid nodes and pods running on them. This is important because we do not use BGP and instead only route statically. At a later point we might want to switch to BGP to enable e.g. routing from pods on one hybrid node to pods on another. We use the following scheme:

  • All hybrid nodes receive an IP address 10.255.0.X/32. This is part of the private range 10.0.0.0/8 but unlikely to already be in use because 10.255.* is somewhat unusual. This implies that no two hybrid nodes are ever in the same network. Export this to $HYBRID_NODE_CIDR.

  • We give pods across all hybrid nodes IP address in 10.254.0.0/16. This is large enough to allow for up to 256 hybrid nodes with up to 256 pods each which should be plenty. AWS must however be aware which pod IPs need to be routed through which VPN connection. To this end every new node should be assigned pod IPs in 10.254.{n+1}.0./24 where the last created node used IPs in 10.254.{n}.0./24. Export this to $HYBRID_NODE_POD_CIDR and export 10.254.0.0/16 to $HYBRID_POD_CIDR. Cilium is responsible for pod IP assignment so it is worth double checking that it indeed looks like this after a new node has been created. A more flexible solution that does not require manual routing might be possible via EKS Hybrid Nodes gateway.

Run the following:

aws ec2 create-vpn-connection-route \
  --vpn-connection-id $VPN_CONNECTION_ID \
  --destination-cidr-block $HYBRID_NODE_CIDR \
  --region $CLUSTER_REGION

aws ec2 create-vpn-connection-route \
  --vpn-connection-id $VPN_CONNECTION_ID \
  --destination-cidr-block $HYBRID_NODE_POD_CIDR \
  --region $CLUSTER_REGION

Export VPN Connection Configuration

Finally, run the following to dump settings we will need to supply to this role later to a file:

aws ec2 describe-vpn-connections \
  --vpn-connection-ids $VPN_CONNECTION_ID \
  --query 'VpnConnections[0].CustomerGatewayConfiguration' \
  --region $CLUSTER_REGION \
  --output text > vpn-config.xml

5. Update Cluster Routes and Security Rules

We now have to add make our hybrid node networks known to the EKS cluster and add routes for its EC2 instances. Additionally we have allow traffic from our hybrid node to these EC2 instances.

Update Cluster config

aws eks update-cluster-config \
  --name $CLUSTER_NAME \
  --remote-network-config "remoteNodeNetworks=[{cidrs=[\"$HYBRID_NODE_CIDR\"]}],remotePodNetworks=[{cidrs=[\"$HYBRID_POD_CIDR\"]}]" \
  --region $CLUSTER_REGION

Warning

If setting up multiple hybrid nodes for on EKS cluster all of them need to be specified in remoteNodeNetworks whenever this is run in the process of adding a new one.

Create Static Routes

for ROUTING_TABLE_ID in $CLUSTER_ROUTING_TABLE_IDS; do
  echo "Creating routes in routing table: $ROUTING_TABLE_ID"

  aws ec2 create-route \
    --route-table-id "$ROUTING_TABLE_ID" \
    --destination-cidr-block "$HYBRID_NODE_CIDR" \
    --gateway-id "$VPN_GATEWAY_ID" \
    --region "$CLUSTER_REGION"

  aws ec2 create-route \
    --route-table-id "$ROUTING_TABLE_ID" \
    --destination-cidr-block "$HYBRID_POD_CIDR" \
    --gateway-id "$VPN_GATEWAY_ID" \
    --region "$CLUSTER_REGION"
done

Configure Security Group

Note: -1 means "all protocols" i.e. ICMP, TCP, UCP, ...

aws ec2 authorize-security-group-ingress \
  --group-id $CLUSTER_SECURITY_GROUP \
  --protocol -1 \
  --cidr $HYBRID_NODE_CIDR \
  --region $CLUSTER_REGION

aws ec2 authorize-security-group-ingress \
  --group-id $NODE_SECURITY_GROUP \
  --protocol -1 \
  --cidr $HYBRID_NODE_CIDR \
  --region $CLUSTER_REGION

6. Create Activation and Run Ansible

We are now ready to actually configure the hybrid node itself. First we must create an activation. We do this by asking AWS for a one-time activation code:

aws ssm create-activation \
  --iam-role LyceumEKSHybridNodesRole \
  --registration-limit 1 \
  --region $CLUSTER_REGION

This will return an activation ID and an activation code, export these as $ACTIVATION_ID an $ACTIVATION_CODE.

We then supply this role with the following information, using the pseudo-syntax $FOO here to signify that the respective environment variable set in one of the steps above should be inserted. The tunnel-specific values are taken from the exported VPN connection configuration in vpn-config.xml. Note that AWS mandates that there are two tunnels for redundancy.

aws_eks_hybrid_node_cluster_name: $CLUSTER_NAME
aws_eks_hybrid_node_cluster_region: $CLUSTER_REGION
aws_eks_hybrid_node_cluster_endpoint: $CLUSTER_ENDPOINT
aws_eks_hybrid_node_k8s_version: "1.33" # Example, adjust to your cluster
aws_eks_hybrid_node_nodeadm_version: "1.0.11" # Example, adjust to your cluster
aws_eks_hybrid_node_aws_vpc_cidr: $CLUSTER_VPC_CIDR
aws_eks_hybrid_node_k8s_service_cidr: $CLUSTER_K8S_SERVICE_CIDR
aws_eks_hybrid_node_hybrid_node_outside_ip: $HYBRID_NODE_OUTSIDE_IP
aws_eks_hybrid_node_hybrid_node_inside_ip: $HYBRID_NODE_CIDR
aws_eks_hybrid_node_hybrid_node_pod_cidr: $HYBRID_NODE_POD_CIDR
aws_eks_hybrid_node_activation_id: $ACTIVATION_ID
aws_eks_hybrid_node_activation_code: $ACTIVATION_CODE

# From vpn-config.xml:
aws_eks_hybrid_node_tunnel1_aws_outside_ip: ...
aws_eks_hybrid_node_tunnel1_aws_inside_ip: ...
aws_eks_hybrid_node_tunnel1_hybrid_node_inside_ip: ...
aws_eks_hybrid_node_tunnel1_psk: ...
aws_eks_hybrid_node_tunnel2_aws_outside_ip: ...
aws_eks_hybrid_node_tunnel2_aws_inside_ip: ...
aws_eks_hybrid_node_tunnel2_hybrid_node_inside_ip: ...
aws_eks_hybrid_node_tunnel2_psk: ...

After Ansible has completed successfully you should be able to see the hybrid node with kubectl get nodes, its name will have an mi- prefix. The node will still be NotReady at this stage if Cillium has not yet been installed and configured, see the next step.

7. Install and Configure Cilium

We are finally done with creating AWS resources but a bit of work remains: EKS does not support just any CNI for use with hybrid nodes. Thus we have to install Cilium. We assume here that the user does not already use Cilium (in which case this would have to be adapted) and configure it to only run agent pods on hybrid nodes. Some minor networking-related kubectl adjustments are then required in order for Cilium to not break existing cluster communication.

Update kubeconfig

We will install Cilium with helm which means that we first need to ensure we have our kubeconfig pointed to the EKS cluster:

aws eks update-kubeconfig \
  --name $CLUSTER_NAME \
  --region $CLUSTER_REGION

Install Cilium

Fist create the following configuration file.

Here $HYBRID_POD_MASK_SIZE can be adapted to control how many pod IP addresses are available per hybrid node (and thus also how many hybrid nodes can be created in total). Set this to 24 to allow up to 256 pods per hybrid node, thus up to 256 hybrid nodes.

Furthermore, $HYBRID_NODE_DEFAULT_INTERFACE should be set to the hybrid node's primary interface, e.g. eth0 (check with ip route | grep default).

cat > cilium-values.yaml << 'EOF'
---
k8sServiceHost: "$CLUSTER_ENDPOINT"
k8sServicePort: "443"

# Controls where Cilium agent pods run (only match hybrid nodes)
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: eks.amazonaws.com/compute-type
          operator: In
          values:
          - hybrid

# Don't manage pods that existed before Cilium was installed.
operator:
  unmanagedPodWatcher:
    restart: false

# Do not replace kube-proxy because EC2 nodes are still using it.
kubeProxyReplacement: "false"

ipam:
  mode: cluster-pool
  operator:
    clusterPoolIPv4PodCIDRList:
    - $HYBRID_POD_CIDR
    clusterPoolIPv4MaskSize: $HYBRID_POD_MASK_SIZE

# This prevents Cilium from trying to masquerade traffic through virtual tunnel
# interfaces. We don't want that because we insert our own SNAT iptables rules
# to do so.
egressMasqueradeInterfaces: $HYBRID_NODE_DEFAULT_INTERFACE
EOF

Then install cilium with:

helm install cilium cilium/cilium --namespace kube-system -f cilium-values.yaml

Disable SNATing

When AWS tries to SNAT traffic to hybrid nodes/pods strange things happen when e.g. the target is a ClusterIP. This can be prevented by disabling SNATing for our node and pod CIDRS:

kubectl -n kube-system \
  set env daemonset/aws-node \
  AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS="$HYBRID_NODE_CIDR,$HYBRID_POD_CIDR"

Warning

If setting up multiple hybrid nodes for on EKS cluster all of them need to be specified in AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS whenever this is run in the process of adding a new one.

Adjust CoreDNS Configuration

Because we've been experiencing some issues with CoreDNS pods once Cilium has been installed we perform some additional tweaks to make sure that there are at least two CoreDNS pods running on AWS and that pods on EC2 instances prefer these over those running on hybrid nodes and vice-versa.

First, annotate hybrid nodes with the (arbitrary) lyceum-<n> zone label (a separate one per hybrid node) and patch kube-dns to prefer CoreDNS pods in the same zone, e.g.:

kubectl label node <hybrid-node-1-name> topology.kubernetes.io/zone=lyceum-1
kubectl patch service kube-dns -n kube-system -p '{"spec":{"trafficDistribution":"PreferClose"}}'

Then increase the number of CoreDNS pods with (here representatively to 3, in reality chose something like number of hybrid nodes + 2):

kubectl scale deployment coredns \
  -n kube-system \
  --replicas=3

Warning

This does not actually guarantee that a CoreDNS replica will be scheduled on each hybrid node and we do need this to be the case in order for DNS to work on these notes. Simply increasing the number of replicas until one is scheduled works but a more robust solution needs to be found.