This document describes how to expose Kubernetes services using External IP Controller.
One of the possible ways to expose k8s services on a bare metal deployment is using External IPs. Each node runs a kube-proxy process which programs iptables rules to trap access to External IPs and redirect them to the correct backends.
So in order to accessa k8s service from the outside we just need to route public traffic to one of k8s worker nodes which has kube-proxy running and thus has needed iptables rules for External IPs.
External IP controller is a k8s application which is deployed on top of a k8s cluster and which configures External IPs on k8s worker node(s) to provide IP connectivity.
For further details please read External IP controller documentation
The playbook is utils/kargo/externalip.yaml and the ansible role is utils/kargo/roles/externalip.
The nodes that have externalip role assigned to them will run External IP controller application which will manage Kubernetes services’ external IPs. Playbook labels such nodes and then creates DaemonSet with appropriate nodeSelector.
External IP scheduler will be running as a standard Deployment (ReplicaSet) with specified number of replicas.
In this sample deployment we’re going to deploy External IP controller on a set of nodes and provide load balancing and high availability for External IPs based on Equal-cost multi-path routing (ECMP).
For further details please read Documentation about ECMP deployment
You can take inventory generated previously for Kargo deployment. Using utils/kargo/externalip.yaml playbook with such inventory will deploy External IP controller on all kube-node worker nodes.
Custom Ansible yaml for ECMP deployment is stored here: utils/jenkins/extip_ecmp.yaml. Here is the content:
# Type of deployment
extip_ctrl_app_kind: "DaemonSet"
# IP distribution model
extip_distribution: "all"
# Netmask for external IPs
extip_mask: 32
# Interface to bring IPs on, should be "lo" for ECMP
extip_iface: "lo"
Just run the following ansible-playbook command:
export ws=/home/workspace
ansible-playbook -e ansible_ssh_pass=vagrant -u vagrant -b \
--become-user=root -i ${ws}/inventory/inventory.cfg \
-e @${ws}/utils/jenkins/extip_ecmp.yaml \
${ws}/utils/kargo/externalip.yaml
This will deploy the application according to your inventory.
This application only brings IPs up or down on a specified interface. We also need to provide routing to those nodes with external IPs. So for Kubernetes cluster with Calico networking plugin we already have calico-node container running on every k8s worker node. This container also includes BGP speaker which monitors local routing tables and announces changes via BGP protocol. So in order to include external IPs to BGP speaker export we need to add the following custom export filter for Calico:
cat << EOF | etcdctl set /calico/bgp/v1/global/custom_filters/v4/lo_iface
if ( ifname = "lo" ) then {
if net != 127.0.0.0/8 then accept;
}
EOF
Please note that this will only configure BGP for calico-node. In order to announce routing to your network infrastructure you may want to peer Calico with routers. Please check this URL for details:
Uninstall k8s applications by running the following commands on the first kube-master node in your ansible inventory:
kubectl delete -f /etc/kubernetes/extip_scheduler.yml
kubectl delete -f /etc/kubernetes/extip_controller.yml
Remove custom Calico export filter:
etcdctl rm /calico/bgp/v1/global/custom_filters/v4/lo_iface
Also remove external IPs from lo interface on the nodes with the command like this:
ip ad del 10.0.0.7/32 dev lo
Where 10.0.0.7/32 is external IP.