Installing External IP Controller

This document describes how to expose Kubernetes services using External IP Controller.

Introduction

One of the possible ways to expose k8s services on a bare metal deployment is using External IPs. Each node runs a kube-proxy process which programs iptables rules to trap access to External IPs and redirect them to the correct backends.

So in order to accessa k8s service from the outside we just need to route public traffic to one of k8s worker nodes which has kube-proxy running and thus has needed iptables rules for External IPs.

Deployment scheme

Description

External IP controller is a k8s application which is deployed on top of a k8s cluster and which configures External IPs on k8s worker node(s) to provide IP connectivity.

For further details please read External IP controller documentation

Ansible Playbook

The playbook is utils/kargo/externalip.yaml and the ansible role is utils/kargo/roles/externalip.

The nodes that have externalip role assigned to them will run External IP controller application which will manage Kubernetes services’ external IPs. Playbook labels such nodes and then creates DaemonSet with appropriate nodeSelector.

External IP scheduler will be running as a standard Deployment (ReplicaSet) with specified number of replicas.

ECMP deployment

Deployment model

In this sample deployment we’re going to deploy External IP controller on a set of nodes and provide load balancing and high availability for External IPs based on Equal-cost multi-path routing (ECMP).

For further details please read Documentation about ECMP deployment

Inventory

You can take inventory generated previously for Kargo deployment. Using utils/kargo/externalip.yaml playbook with such inventory will deploy External IP controller on all kube-node worker nodes.

Custom yaml

Custom Ansible yaml for ECMP deployment is stored here: utils/jenkins/extip_ecmp.yaml. Here is the content:

# Type of deployment
extip_ctrl_app_kind: "DaemonSet"
# IP distribution model
extip_distribution: "all"
# Netmask for external IPs
extip_mask: 32
# Interface to bring IPs on, should be "lo" for ECMP
extip_iface: "lo"

Deployment

Just run the following ansible-playbook command:

export ws=/home/workspace
ansible-playbook -e ansible_ssh_pass=vagrant -u vagrant -b \
--become-user=root -i ${ws}/inventory/inventory.cfg \
-e @${ws}/utils/jenkins/extip_ecmp.yaml \
${ws}/utils/kargo/externalip.yaml

This will deploy the application according to your inventory.

Routing

This application only brings IPs up or down on a specified interface. We also need to provide routing to those nodes with external IPs. So for Kubernetes cluster with Calico networking plugin we already have calico-node container running on every k8s worker node. This container also includes BGP speaker which monitors local routing tables and announces changes via BGP protocol. So in order to include external IPs to BGP speaker export we need to add the following custom export filter for Calico:

cat << EOF | etcdctl set /calico/bgp/v1/global/custom_filters/v4/lo_iface
if ( ifname = "lo" ) then {
  if net != 127.0.0.0/8 then accept;
}
EOF

Please note that this will only configure BGP for calico-node. In order to announce routing to your network infrastructure you may want to peer Calico with routers. Please check this URL for details:

Kargo docs: Calico BGP Peering with border routers

Uninstalling and undoing customizations

Uninstall k8s applications by running the following commands on the first kube-master node in your ansible inventory:

kubectl delete -f /etc/kubernetes/extip_scheduler.yml
kubectl delete -f /etc/kubernetes/extip_controller.yml

Remove custom Calico export filter:

etcdctl rm /calico/bgp/v1/global/custom_filters/v4/lo_iface

Also remove external IPs from lo interface on the nodes with the command like this:

ip ad del 10.0.0.7/32 dev lo

Where 10.0.0.7/32 is external IP.