2019独角兽企业重金招聘Python工程师标准>>> 
##网络及版本信息
docker1 centos7 192.168.75.200
docker2 centos7 192.168.75.201
物理网络 192.168.75.1/24
Docker version 1.10.3, build 3999ccb-unsupported ，安装过程略
# calicoctl version
Version: v1.0.0-12-g0d6d228
Build date: 2017-01-17T09:01:03+0000
Git commit: 0d6d228
##1.安装etcd
####下载安装etcd
# ETCD_VER=v3.0.16
# DOWNLOAD_URL=https://github.com/coreos/etcd/releases/download
# curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
# mkdir -p /tmp/test-etcd && tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /tmp/test-etcd --strip-components=1
# cd /tmp/test-etcd && cp etcd* /usr/local/bin/
启动etcd
# etcd --listen-client-urls 'http://192.168.75.200:2379' --advertise-client-urls 'http://192.168.75.200:2379'
查看etcd信息
# etcdctl --endpoint 'http://192.168.75.200:2379' member list
8e9e05c52164694d: name=default peerURLs=http://localhost:2380 clientURLs=http://192.168.75.200:2379 isLeader=true
##2.下载安装calico
修改网络内核参数
# sysctl -w net.netfilter.nf_conntrack_max=1000000 # echo "net.netfilter.nf_conntrack_max=1000000" >> /etc/sysctl.conf
下载calicoctl
# cd /usr/local/bin/ && wget http://www.projectcalico.org/builds/calicoctl
# chmod 755 calicoctl
设置etcd环境变量
# export ETCD_ENDPOINTS=http://192.168.75.200:2379 && echo "export ETCD_ENDPOINTS=http://192.168.75.200:2379" >>/etc/profile
安装运行calico node
# calicoctl node run
Running command to load modules: modprobe -a xt_set ip6_tables
Enabling IPv4 forwarding
Enabling IPv6 forwarding
Increasing conntrack limit
Removing old calico-node container (if running).
Running the following command to start calico-node:docker run --net=host --privileged --name=calico-node -d --restart=always -e ETCD_AUTHORITY= -e ETCD_SCHEME= -e NODENAME=docker1 -e CALICO_NETWORKING_BACKEND=bird -e NO_DEFAULT_POOLS= -e CALICO_LIBNETWORK_ENABLED=true -e CALICO_LIBNETWORK_IFPREFIX=cali -e ETCD_ENDPOINTS=http://192.168.75.200:2379 -v /run/docker/plugins:/run/docker/plugins -v /var/run/docker.sock:/var/run/docker.sock -v /var/run/calico:/var/run/calico -v /lib/modules:/lib/modules -v /var/log/calico:/var/log/calico calico/node:latestImage may take a short time to download if it is not available locally.
Container started, checking progress logs.
Waiting for etcd connection...
Using auto-detected IPv4 address: 192.168.75.200
No IPv6 address configured
Using global AS number
Calico node name: docker1
CALICO_LIBNETWORK_ENABLED is true - start libnetwork service
Calico node started successfully
在docker1查看calico node状态，发现与docker2（192.168.75.201）连接已建立
# calicoctl node status
Calico process is running.IPv4 BGP status
+----------------+-------------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+----------------+-------------------+-------+----------+-------------+
| 192.168.75.201 | node-to-node mesh | up | 01:57:54 | Established |
+----------------+-------------------+-------+----------+-------------+IPv6 BGP status
No IPv6 peers found.
##3.配置calico pool
查看默认pool
# calicoctl get pool
CIDR
192.168.0.0/16
fd80:24e2:f998:72d6::/64
删除默认pool，在任意一台node上操作
# calicoctl delete pool 192.168.0.0/16
Successfully deleted 1 'ipPool' resource(s)
# calicoctl delete pool fd80:24e2:f998:72d6::/64
Successfully deleted 1 'ipPool' resource(s)
创建新的ipPool，在任意一台node上操作
# vi /etc/calico/ippool_10.1.0.0_16.cfg
apiVersion: v1
kind: ipPool
metadata:cidr: 10.1.0.0/16
spec:ipip:enabled: truenat-outgoing: truedisabled: false
# calicoctl create -f /etc/calico/ippool_10.1.0.0_16.cfg
Successfully created 1 'ipPool' resource(s)
##4.配置docker，创建docker network
修改集群中每台docker启动参数，重启docker
添加--cluster-store=etcd://192.168.75.200:2379/calico 指定docker集群使用的存储，否则下一步不会成功创建network
# vi /etc/sysconfig/docker
OPTIONS='--selinux-enabled --log-driver=journald --cluster-store=etcd://192.168.75.200:2379/calico'
集群中任意一台上docker创建网络
# docker network create --driver=calico --ipam-driver=calico-ipam net1
0501f1b788756d122568e7aed2d7c56fe2de9138f9bd00f6628c4b66c81c7c9b
# docker network create --driver=calico --ipam-driver=calico-ipam net2
4b636bf63b23dee13b817c911335823a84ad6d55771a44e89fb81c16f76663ad
# docker network ls
NETWORK ID NAME DRIVER
54a450c39848 net1 calico
8fdcdecdb0bc net2 calico
e0d1a688fef8 none null
0e987140865a host host
b5122ac5e20e bridge bridge
##5.测试网络连否连通
docker1启动net1，net2各一个container
[root@docker1 bin]# docker run -itd --net=net1 --name=testnet1 centos /bin/bash
579c509e293e25340f10cc188a91136f99ed9021b99f795a9056a683b6b46864
[root@docker1 bin]# docker run -itd --net=net2 --name=testnet2 centos /bin/bash
c8777a2ff6add64e6abf454828820a6cfee332086a58c769a6cf1e5e0fda8760
docker2启动net1，net2各一个container
[root@docker2 bin]# docker run -itd --net=net1 --name=testnet3 centos /bin/bash
8bb7be8d86a04631a442a9f43e6be9576a891f704b91042550c5fe632fa11f06
[root@docker2 bin]# docker run -itd --net=net2 --name=testnet4 centos /bin/bash
422f4466db503b380f646d6eaee14a2f695550669fd4987fadefff438f456a36
container ip信息如下
testnet1 10.1.174.193
testnet2 10.1.174.194
testnet3 10.1.166.129
testnet4 10.1.166.130
####testnet1上ping其他container
testnet1容器只和docker2上的testnet3容器能通，因为两个container都属于net1网络
[root@579c509e293e /]# ping 10.1.166.129
PING 10.1.166.129 (10.1.166.129) 56(84) bytes of data.
64 bytes from 10.1.166.129: icmp_seq=1 ttl=62 time=0.400 ms
^C
--- 10.1.166.129 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.400/0.400/0.400/0.000 ms
[root@579c509e293e /]# ping 10.1.166.130
PING 10.1.166.130 (10.1.166.130) 56(84) bytes of data.
^C
--- 10.1.166.130 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3000ms[root@579c509e293e /]# ping 10.1.174.194
PING 10.1.174.194 (10.1.174.194) 56(84) bytes of data.
^C
--- 10.1.174.194 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2000ms
遇到的问题：
1.docker异常后无法restart testnet3，4容器
docker: Error response from daemon: service endpoint with name testnet3 already exists.
解决方案:
etcd中endpoint信息未删除，手动删除吧，查找方法如下
54a450.....是network id，可通过docker network ls查找到
遍历下/calico/docker/network/v1.0/endpoint/54a450c3984853b3942738163cfaaa7dd247686ccc10b8f395dfb807df11e2bb/的所有数据就能找到对应的数据手工删除
# etcdctl --endpoint 'http://192.168.75.200:2379' get /calico/docker/network/v1.0/endpoint/54a450c3984853b3942738163cfaaa7dd247686ccc10b8f395dfb807df11e2bb/5d9cad95e7193e47177eb6d8bdfa25ebc878d8565c48227861^Cf6700136a10c
{"anonymous":false,"disableResolution":false,"ep_iface":{"addr":"10.1.174.198/32","dstPrefix":"cali","mac":"ee:ee:ee:ee:ee:ee","routes":["169.254.1.1/32"],"srcName":"temp5d9cad95e71","v4PoolID":"CalicoPoolIPv4","v6PoolID":""},"exposed_ports":[],"generic":{"com.docker.network.endpoint.exposedports":[],"com.docker.network.portmap":[]},"id":"5d9cad95e7193e47177eb6d8bdfa25ebc878d8565c48227861f6f6700136a10c","locator":"","myAliases":null,"name":"testnet1","sandbox":"bc9abf7c29a9532500aeb9618b22254eab9e73aecc9d4b6c3bf488b6d173791e"}
2.node访问其他node上的container不通
默认net1和net2的profile是允许tag相同的访问endpoint，但是calico node默认无法访问，需要修改profile
# calicoctl get profile net1 -o yaml > /etc/calico/profile_net1.yaml
# vi /etc/calico/profile_net1.yaml
- apiVersion: v1kind: profilemetadata:name: net1tags:- net1spec:egress:- action: allowdestination: {}source: {}ingress:- action: allowdestination: {}source:tag: net1
#下面是新加的rule- action: allowdestination: {}source:net: 192.168.75.0/24- action: allowdestination: {}source:net: 10.1.174.192/32- action: allowdestination: {}source:net: 10.1.166.128/32
# calicoctl create -f /etc/calico/profile_net1.yaml
Successfully created 1 'policy' resource(s)
10.1.174.192/32和10.1.166.128/32是docker1和docker2的tunl0的ip，手工配置这个还是比较繁琐，应该写成脚本做这个工作
再在集群中任意一台node上ping另外一台node上随便一台net1下的container都能ping通了
![org.jeecgframework.core.common.exception.MyExceptionHandler]java.lang.NullPointerException](http://pic.xiahunao.cn/org.jeecgframework.core.common.exception.MyExceptionHandler]java.lang.NullPointerException)
