Skip to content

Commit e1b1a31

Browse files
aaurenaauren
committed
doc(ipv6.md): update for increased dual-stack support
1 parent e2f0f18 commit e1b1a31

File tree

1 file changed

+136
-75
lines changed

1 file changed

+136
-75
lines changed

‎docs/ipv6.md

+136-75
Original file line numberDiff line numberDiff line change
@@ -1,114 +1,175 @@
1-
# Ipv6 support in kube-router
1+
# IPv6 / DualStack Support in kube-router
22

3-
This document describes the current status, the plan ahead and general
4-
thoughts about ipv6 support in `kube-router`.
3+
This document describes the current status, the plan ahead and general thoughts about IPv6 / Dual-Stack support in
4+
kube-router.
55

6-
Ipv6-only is supported (in alpha) in `Kubernetes` from version 1.9 and
7-
support for ipv4/ipv6 dual stack is on the way
8-
([KEP](https://github.com./leblancd/community/blob/dual-stack-kep/keps/sig-network/0013-20180612-ipv4-ipv6-dual-stack.md)). It
9-
is desirable for `kube-router` to keep up with that development.
6+
Dual-Stack (e.g. IPv4 and IPv6) has been supported in Kubernetes since version `v1.21`:
7+
[IPv4/IPv6 dual-stack documentation](https://kubernetes.io/docs/concepts/services-networking/dual-stack/)
108

11-
The idea is to implement ipv6-only function-by-function;
9+
kube-router's current approach is to implement dual-stack functionality function-by-function:
1210

1311
* CNI `--enable-cni`
1412
* Proxy `--run-service-proxy`
1513
* Router `--run-router`
1614
* Network policies `--run-firewall`
1715

18-
It is important to always keep dual-stack in mind. The code must not
19-
be altered to handle ipv6 instead of ipv4 but must be able to handle
20-
both at the same time in the near future.
2116

22-
To use ipv6 is usually not so hard in golang. The same code is used,
23-
only the addresses differ. This is also true for `iptables` and
24-
`ipvsadm`. This makes support for ipv6 a bit easier to implement.
17+
## Current status (January 22, 2023)
2518

26-
## Testing
19+
Support for dual-stack in kube-router is under active development and is currently the main focus of development for the
20+
kube-router maintainers. Currently, we are targeting a v2.0.0 release of kube-router which will see all controllers
21+
updated for dual-stack compatibility.
2722

28-
Test and development has so far used
29-
https://github.com./Nordix/xcluster/tree/master/ovl/kube-router-ipv6
30-
which is an easy way to get a ipv6-only Kubernetes cluster.
23+
We are currently running this work off of the
24+
[prep-v2.0 branch](https://github.com./cloudnativelabs/kube-router/tree/prep-v2.0) and, as of the time of this writing,
25+
have released a [release candidate]() with this some dual-stack functionality built into it.
3126

32-
To setup an ipv6-only Kubernetes cluster is usually no simple task,
33-
see for instance https://github.com./leblancd/kube-v6
27+
Functions that currently support dual-stack on the v2.0.0 release line:
3428

35-
No automatic tests exist yet for ipv6.
29+
* CNI
30+
* Router / BGP (`--run-router`)
31+
* Network Policies (`--run-firewall`)
3632

33+
## How Can You Help?
3734

38-
## Current status (Thu Oct 11 2018)
35+
The work to make dual-stack support workable in kube-router has involved a lot of work and several large contributions
36+
to the code base. If you have access to a development or staging environment where you would be willing to run the
37+
latest v2.0.0 release candidate and give us feedback, it would be much appreciated!
3938

40-
Support for ipv6 in the the CNI function in `kube-router` is under
41-
development. The local BGP routers peers with ipv6;
39+
While the most helpful feedback will come from users that are able to run kube-router in dual-stack mode, there have
40+
been enough changes, that even users that are only able to run kube-router in a single-stack mode will be able to give
41+
valuable feedback concerning any bugs or regressions.
4242

43-
```
44-
# gobgp neighbor
45-
Peer AS Up/Down State |#Received Accepted
46-
1000::1:c0a8:101 64512 00:00:37 Establ | 0 0
47-
1000::1:c0a8:102 64512 00:00:37 Establ | 0 0
48-
1000::1:c0a8:103 64512 00:00:40 Establ | 0 0
49-
```
43+
If any users that find bugs have Golang experience, please consider helping us squash some of our dual-stack related
44+
bugs by filing PRs against the project.
5045

51-
The CNI configuration is also updated with ipv6 addresses;
46+
All PRs related to the new v2.0.0-based functionality will be given priority in terms of review as we work to get this
47+
released and out into the wild so that we can work on other features and issues.
5248

53-
```
54-
# jq . < /etc/cni/net.d/10-kuberouter.conf | cat
55-
{
56-
"bridge": "kube-bridge",
57-
"ipam": {
58-
"subnet": "1000::2:b00:100/120",
59-
"type": "host-local"
60-
},
61-
"isDefaultGateway": true,
62-
"isGateway": true,
63-
"name": "ekvm",
64-
"type": "bridge"
65-
}
66-
```
49+
## Roadmap
6750

68-
This means that pod's gets assigned ipv6 addresses.
51+
The next big item to add to make kube-router fully dual-stack functional will be the **Proxy (`--run-service-proxy`)**
52+
functionality. At this point, kube-router is not able to proxy traffic for IPv6 service VIPs at all until we get this
53+
functionality added.
6954

70-
The announcement of the pod CIDRs does not work yet. So pods on other
71-
nodes than the own cannot be reached.
55+
After that, we'd like to give kube-router some time to run in the wild for a bit so that we can be sure that there
56+
aren't any large bugs or regressions before we tag an official v2.0.0 release.
7257

73-
To get this working the routes must be inserted in the RIB for
74-
`gobgp`. Checking the ipv4 rib gives an error;
58+
## Important Notes / Known Limitations / Etc.
7559

76-
```
77-
# gobgp -a ipv4 global rib
78-
invalid nexthop address: <nil>
79-
```
60+
This represents a major release for kube-router and as such, user's should approach deploying this into an established
61+
kube-router environment carefully. While there aren't any huge bugs that the maintainers are aware of at this time,
62+
there are several small breaks in backwards compatibility. We'll try to detail these below as best we can.
8063

81-
While the ipv6 rib is empty;
64+
### How To Enable Dual-Stack Functionality
8265

83-
```
84-
# gobgp -a ipv6 global rib
85-
Network not in table
66+
In order to enable dual-stack functionality please ensure the following:
67+
68+
* kube-router option `--enable-ipv4=true` is set (this is the default)
69+
* kube-router option `--enable-ipv6=true` is set
70+
* Your Kubernetes node has both IPv4 and IPv6 addresses on its physical interfaces
71+
* Your Kubernetes node has both IPv4 and IPv6 addresses in its node spec:
72+
73+
```shell
74+
$ kubectl describe node foo
75+
...
76+
Addresses:
77+
InternalIP: 10.95.0.202
78+
InternalIP: 2001:1f18:3d5:ed00:d61a:454f:b886:7000
79+
Hostname: foo
80+
...
8681
```
8782

88-
A guess is that `kube-router` tries to insert ipv6 addresses in the
89-
ipv4 rib.
83+
* Add additional `--service-cluster-ip-range` and `--service-external-ip-range` kube-router parameters for your IPv6
84+
addresses. Note, as mentioned before `Proxy` functionality still isn't working, but this is important for a future
85+
where `Proxy` functionality has been enabled.
86+
* If you use `--enable-cni`, `kube-controller-manager` has been started with both IPv4 and IPv6 cluster CIDRs (e.g.
87+
`--cluster-cidr=10.242.0.0/16,2001:db8:42:1000::/56`)
88+
* `kube-controller-manager` & `kube-apiserver` have been started with both IPv4 and IPv6 service cluster IP ranges (e.g.
89+
`--service-cluster-ip-range=10.96.0.0/16,2001:db8:42:1::/112`)
9090

91-
When the bgp announcement of ipv6 cidr for pods work the support for
92-
ipv6 in the `kube-router` CNI is done (I hope).
91+
### Tunnel Name Changes (Potentially Breaking Change)
9392

93+
In order to facilitate both IPv4 and IPv6 tunnels, we had to change the hashing format for our current tunnel names. As
94+
such, if you do a kube-router upgrade in place (i.e. without reboot), it is very likely that kube-router will not clean
95+
up old tunnels.
9496

95-
## Roadmap
97+
This will only impact users that were utilizing the overlay feature of kube-router to some extent. Such as if you were
98+
running kube-router with `--enable-overlay` or `--overlay-type=full` or `--overlay-type=subnet` (it should be noted that
99+
these options default to on currently).
100+
101+
If you are upgrading kube-router from a pre v2.0.0 release to a v2.0.0 release, we recommend that you coordinate
102+
your upgrade of kube-router with a rolling reboot of your Kubernetes fleet to clean up any tunnels that were left from
103+
previous versions of kube-router.
104+
105+
### kube-router.io/node.bgp.customimportreject Can Only Contain IPs of a Single Family
106+
107+
Due to implementation restrictions with GoBGP, the annotation `kube-router.io/node.bgp.customimportreject`, which allows
108+
user's to add rules for rejecting specific routes sent to GoBGP, can only accept a single IP family (e.g. IPv4 or IPv6).
109+
110+
Attempting to add IPs of two different families will result in a GoBGP error when it attempts to import BGP policy from
111+
kube-router.
112+
113+
### kube-router.io/pod-cidr Deprecation
96114

97-
There is no time-plan. Help is welcome.
115+
Now that kube-router has dual-stack capability, it doesn't make sense to have an annotation that can only represent
116+
a single pod CIDR any longer. As such, with this release we are announcing the deprecation of the
117+
`kube-router.io/pod-cidr` annotation in favor of the new `kube-router.io/pod-cidrs` annotation.
98118

99-
After the CNI the next function in line may be the service
100-
proxy. `ipvs` has full support for ipv6. The dual-stack KEP states
101-
that to get dual stack support for a service two services must be
102-
specified, one for ipv4 and another for ipv6. The implementation
103-
should get the protocol from a global setting for ipv6-only and later
104-
from some attribute in the service object.
119+
The new `kube-router.io/pod-cidrs` annotation is a comma-separated list of CIDRs and can hold either IPv4 or IPv6 CIDRs
120+
in string form.
105121

106-
Since the same `gobgp` is used for the CNI and the router functions is
107-
may be fairly simple to implement ipv6 support.
122+
It should be noted, that until `kube-router.io/pod-cidr` is fully removed, at some point in the future, it will still
123+
be preferred over the `kube-router.io/pod-cidrs` annotation in order to preserve as much backwards compatibility as
124+
possible. Until `kube-router.io/pod-cidr` has been fully retired, user's the use the old annotation will get a warning
125+
in their kube-router logs saying that they should change to the new annotation.
108126

109-
Ipv6 support for the firewall function is not investigated. Ipv6 support
110-
for `ipset` is implemented already for the CNI.
127+
The recommended action here, is that upon upgrade, you convert nodes from using the `kube-router.io/pod-cidr` to the new
128+
`kube-router.io/pod-cidrs` annotation. Since kube-router currently only updates node annotations at start and not as
129+
they are updated, this is a safe change to make before updating kube-router.
111130

131+
If neither annotation is specified, kube-router will use the `PodCIDRs` field of the Kubernetes node spec which is
132+
populated by the `kube-controller-manager` as part of it's `--allocate-node-cidrs` functionality. This should be a sane
133+
default for most users of kube-router.
112134

135+
### CNI Now Accepts Multiple Pod Ranges
113136

137+
Now that kube-router supports dual-stack, it now also supports multiple ranges in the CNI file. While kube-router will
138+
still add your pod CIDRs to your CNI configuration via node configuration like `kube-router.io/pod-cidr`,
139+
`kube-router.io/pod-cidrs`, or `.node.Spec.PodCIDRs`, you can also customize your own CNI to add additional ranges or
140+
plugins.
141+
142+
A CNI configuration with multiple ranges will typically look something like the following:
143+
144+
```json
145+
{
146+
"cniVersion": "0.3.0",
147+
"name": "mynet",
148+
"plugins": [
149+
{
150+
"bridge": "kube-bridge",
151+
"ipam": {
152+
"ranges": [
153+
[
154+
{
155+
"subnet": "10.242.0.0/24"
156+
}
157+
],
158+
[
159+
{
160+
"subnet": "2001:db8:42:1000::/64"
161+
}
162+
]
163+
],
164+
"type": "host-local"
165+
},
166+
"isDefaultGateway": true,
167+
"mtu": 9001,
168+
"name": "kubernetes",
169+
"type": "bridge"
170+
}
171+
]
172+
}
173+
```
114174

175+
All kube-router's handling of the CNI file attempts to minimize disruption to any user made edits to the file.

0 commit comments

Comments
 (0)