1、练习场景说明
在OpenStack Yoga版安装笔记（五）中，glance已经在controller node虚拟机上安装完成，并且已经成功拍摄了快照。
此时，controller node虚机已经安装了keystone、keystone DB、glance、glance DB、OpenStackClient、memcache等软件：
将controller node虚机转到此快照然后开机，在这台虚机上执行openstack image list命令，观察OpenStack组件之间完整交互过程。
2、安装wireshark
controller node:
controller node root用户下执行：
apt update
apt install wiresharkroot@controller:~# wireshark --version
Wireshark 3.6.2 (Git v3.6.2 packaged as 3.6.2-2)Copyright 1998-2022 Gerald Combs <gerald@wireshark.org> and contributors.
License GPLv2+: GNU GPL version 2 or later <https://www.gnu.org/licenses/gpl-2.0.html>
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.Compiled (64-bit) using GCC 11.2.0, with Qt 5.15.2, with libpcap, with POSIX
capabilities (Linux), with libnl 3, with GLib 2.71.2, with zlib 1.2.11, with Lua
5.2.4, with GnuTLS 3.7.3 and PKCS #11 support, with Gcrypt 1.9.4, with MIT
Kerberos, with MaxMind DB resolver, with nghttp2 1.43.0, with brotli, with LZ4,
with Zstandard, with Snappy, with libxml2 2.9.12, with libsmi 0.4.8, with
QtMultimedia, without automatic updates, with SpeexDSP (using system library),
with Minizip.Running on Linux 5.15.0-113-generic, with Intel(R) Core(TM) i7-10710U CPU @
1.10GHz (with SSE4.2), with 7901 MB of physical memory, with GLib 2.72.4, with
zlib 1.2.11, with Qt 5.15.3, with libpcap 1.10.1 (with TPACKET_V3), with c-ares
1.18.1, with GnuTLS 3.7.3, with Gcrypt 1.9.4, with nghttp2 1.43.0, with brotli
1.0.9, with LZ4 1.9.3, with Zstandard 1.4.8, with libsmi 0.4.8, with
LC_TYPE=C.UTF-8, binary plugins supported (0 loaded).
root@controller:~# 建议win11（10.0.20.1）电脑上下载MobaXterm，root登录controller node(10.0.20.11)，执行wireshark。
controller node 安装完成后，关闭虚机，拍摄快照，方便回退。
3、OpenStack绑定不同IP地址
controller node上安装了OpenStack不同组件，比如：keystone、keystone DB、glance、glance DB、memcache等，为了方便查看不同组件交互，可以将他们分别绑定不同的IP。
3.1 ens33配置多个相同网段地址
# This is the network config written by 'subiquity'
network:ethernets:ens33:addresses:- 10.0.20.11/24 --> 规划给keyston- 10.0.20.17/24 --> 规划给database- 10.0.20.18/24 --> 规划给glance- 10.0.20.19/24 --> 规划给memcachenameservers:addresses:- 10.0.20.2search: []routes:- to: defaultvia: 10.0.20.2ens34:dhcp4: falseversion: 2
root@controller:~#netplan apply3.2 配置/etc/hosts文件
root@controller:~# vi /etc/hosts
root@controller:~# cat /etc/hosts
127.0.0.1 localhost
# 127.0.1.1 os-1# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters10.0.20.11 controller
10.0.20.12 compute110.0.20.17 controller-db <--新增
10.0.20.18 controller-glance <--新增
10.0.20.19 controller-memcached <--新增3.3 database配置修改
root@controller:~# vi /etc/mysql/mariadb.conf.d/99-openstack.cnf
root@controller:~# cat /etc/mysql/mariadb.conf.d/99-openstack.cnf
[mysqld]
bind-address = 10.0.20.17 <--修改default-storage-engine = innodb
innodb_file_per_table = on
max_connections = 4096
collation-server = utf8_general_ci
character-set-server = utf8
root@controller:~#3.4 memcache配置修改
root@controller:~# vi /etc/memcached.conf
...
-l 10.0.20.19 <--修改
...3.5 keystone连接database信息修改
root@controller:~# vi /etc/keystone/keystone.conf [database]
# connection = sqlite:var/lib/keystone/keystone.dbconnection = mysql+pymysql://keystone:openstack@controller-db/keystone <--修改3.6 glance连接database信息修改
root@controller:~# vi /etc/glance/glance-api.conf
[database]
# connection = sqlite:var/lib/glance/glance.sqlite
connection = mysql+pymysql://glance:openstack@controller-db/glance <--修改[keystone_authtoken]
memcached_servers = controller-memcached:11211 <---修改3.7 修改db信息后，第一次重启controller node
root@controller:~#reboot重启后，在controller node上使用openstackclient，检查keystone、glance连接db、memcached是否正常：
可以执行：
openstack user list
openstack image list (需要正常连接keystone db，keystone, glance，glance db，memcached才能正常显示）3.8 修改glance service的endpoint
root@controller ~(admin/amdin)# openstack endpoint list --service image
+----------------------------------+-----------+--------------+--------------+---------+-----------+------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+-----------+--------------+--------------+---------+-----------+------------------------+
| 56d1454571064cb3ad3c749d18fb57e6 | RegionOne | glance | image | True | admin | http://controller:9292 |
| ae98462c8f4346f3991d693232a108d3 | RegionOne | glance | image | True | internal | http://controller:9292 |
| e047382ce61740bf9dc4219c97eff2f8 | RegionOne | glance | image | True | public | http://controller:9292 |
+----------------------------------+-----------+--------------+--------------+---------+-----------+------------------------+
root@controller ~(admin/amdin)# openstack endpoint delete 56d1454571064cb3ad3c749d18fb57e6
root@controller ~(admin/amdin)# openstack endpoint delete ae98462c8f4346f3991d693232a108d3
root@controller ~(admin/amdin)# openstack endpoint delete e047382ce61740bf9dc4219c97eff2f8root@controller ~(admin/amdin)# openstack endpoint create --region RegionOne \
> image public http://controller-glance:9292
+--------------+----------------------------------+
| Field | Value |
+--------------+----------------------------------+
| enabled | True |
| id | b700cc0526e34aaba7579fe7fc1e7bf3 |
| interface | public |
| region | RegionOne |
| region_id | RegionOne |
| service_id | d6a3dadf92e542289c5ebd37e3553cdd |
| service_name | glance |
| service_type | image |
| url | http://controller-glance:9292 |
+--------------+----------------------------------+
root@controller ~(admin/amdin)# openstack endpoint create --region RegionOne \
> image internal http://controller-glance:9292
+--------------+----------------------------------+
| Field | Value |
+--------------+----------------------------------+
| enabled | True |
| id | 19e20f4f9f9144118f91fff2cfb2d17b |
| interface | internal |
| region | RegionOne |
| region_id | RegionOne |
| service_id | d6a3dadf92e542289c5ebd37e3553cdd |
| service_name | glance |
| service_type | image |
| url | http://controller-glance:9292 |
+--------------+----------------------------------+
root@controller ~(admin/amdin)# openstack endpoint create --region RegionOne \
> image admin http://controller-glance:9292
+--------------+----------------------------------+
| Field | Value |
+--------------+----------------------------------+
| enabled | True |
| id | 58b83e744b9945c8930aabfac21f207a |
| interface | admin |
| region | RegionOne |
| region_id | RegionOne |
| service_id | d6a3dadf92e542289c5ebd37e3553cdd |
| service_name | glance |
| service_type | image |
| url | http://controller-glance:9292 |
+--------------+----------------------------------+
root@controller ~(admin/amdin)#3.9 修改glance endpoint信息后，第二次重启controller node
root@controller:~#rebootcontroller node重启后，需检查endpoint能否正常显示：
root@controller ~(admin/amdin)# openstack endpoint list
+----------------------------------+-----------+--------------+--------------+---------+-----------+-------------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-------------------------------+
| 19e20f4f9f9144118f91fff2cfb2d17b | RegionOne | glance | image | True | internal | http://controller-glance:9292 |
| 58b83e744b9945c8930aabfac21f207a | RegionOne | glance | image | True | admin | http://controller-glance:9292 |
| a82bd043e133464fa863fba98b400460 | RegionOne | keystone | identity | True | admin | http://controller:5000/v3/ |
| ac109817862344c4854b783d642be412 | RegionOne | keystone | identity | True | internal | http://controller:5000/v3/ |
| b700cc0526e34aaba7579fe7fc1e7bf3 | RegionOne | glance | image | True | public | http://controller-glance:9292 |
| eaf4ddaab1b74aa59403f9fbf5fc4ac1 | RegionOne | keystone | identity | True | public | http://controller:5000/v3/ |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-------------------------------+
root@controller ~(admin/amdin)# openstack image list
+--------------------------------------+--------+--------+
| ID | Name | Status |
+--------------------------------------+--------+--------+
| 429decdd-9230-49c0-b735-70364c226eb5 | cirros | active |
+--------------------------------------+--------+--------+
root@controller ~(admin/amdin)#3.10 关闭虚机，拍摄快照
4、Wireshark抓包
4.1 抓包目的
通过执行openstack image list查看OpenStack各组件之间的完整交互情况。
4.2 抓包方式
1、建议win11（10.0.20.1）上用MobaXterm软件，登录10.0.20.11，运行wireshark，抓取“Loopback: lo”端口流量：
2、win11(10.0.20.1)用SecureCRT软件ssh登录10.0.20.11，监控TCP建立情况：
1、显示keystone、memcached、glance监听端口的进程信息，如进程ID和进程名称：root@controller:~# ss -lp | grep -E '(:5000|:11211|:9292|:3306)'
tcp LISTEN 0 1024 10.0.20.19:11211 0.0.0.0:* users:(("memcached",pid=816,fd=22))
tcp LISTEN 0 4096 10.0.20.18:9292 0.0.0.0:* users:(("glance-api",pid=1226,fd=4),("glance-api",pid=1225,fd=4),("glance-api",pid=1223,fd=4),("glance-api",pid=1222,fd=4),("glance-api",pid=1210,fd=4))
tcp LISTEN 0 511 *:5000 *:* users:(("apache2",pid=874,fd=6),("apache2",pid=873,fd=6),("apache2",pid=867,fd=6))
root@controller:~# 2、查看与端口5000、11211、9292和3306相关的TCP连接，需要在执行openstack image list时，同时执行此命令：
root@controller:~# ss -p | grep -E '(:5000|:11211|:9292|:3306)'3、执行openstack image list命令
win11(10.0.20.1)用SecureCRT软件新开窗口，ssh登录10.0.20.11，在osclient上，执行：openstack image list：
root@controller:~# . admin-openrc
root@controller ~(admin/amdin)# openstack image list
+--------------------------------------+--------+--------+
| ID | Name | Status |
+--------------------------------------+--------+--------+
| 429decdd-9230-49c0-b735-70364c226eb5 | cirros | active |
+--------------------------------------+--------+--------+
4、停止抓包，进行分析
4.3 抓包分析
4.3.1 TCP连接建立信息
记录执行openstack image list命令过程中，TCP连接信息，用于查看TCP连接的两端信息。
root@controller:~# ss -p | grep -E '(:5000|:11211|:9292|:3306)'
tcp ESTAB 0 0 10.0.20.11:53046 10.0.20.11:5000 users:(("glance-api",pid=1225,fd=7))
tcp CLOSE-WAIT 1 0 10.0.20.11:53034 10.0.20.11:5000 users:(("openstack",pid=1467,fd=3))
tcp ESTAB 0 0 10.0.20.11:43508 10.0.20.19:11211 users:(("glance-api",pid=1225,fd=6))
tcp ESTAB 0 0 10.0.20.19:11211 10.0.20.11:43508 users:(("memcached",pid=816,fd=23))
tcp ESTAB 0 0 10.0.20.11:59804 10.0.20.18:9292 users:(("openstack",pid=1467,fd=4))
tcp ESTAB 0 0 10.0.20.18:9292 10.0.20.11:59804 users:(("glance-api",pid=1225,fd=5))4.3.2 抓包结果
为观察数据流方便，过滤掉大量的kestone和keyston db，glance和glance db交互的MySql数据包，只保留每次login信息和每次最后一个response packet。
OpenStackClient、keystone api、glance api之间主要信息都是http，过滤掉大量的维护tcp连接的packets。
使用filter:
http or tcp.port == 11211 or mysql.user == "keystone" or mysql.user == "glance" or frame.number == 3784 or frame.number == 4684 or frame.number == 5431 or frame.number == 5477
抓包显示了OpenStackClient、keystone、keyston db、glance、glance DB之间的交互情况，参考前面的TCP连接的端口信息和PID信息，可以得出连接顺序为：
- OpenStackClient（tcp:53024) --> keystone(tcp:5000)，keystone-->keystone db，认证user admin身份，取得user admin的token
- OpenStackClient(tcp:59804) --> glance(tcp:9292)，请求image list
- glance(tcp:43508) --> MEMCACHE，检查是否有user admin的token
- glance(tcp: 53046) --> keystone(tcp:5000), keystone-->keystone db，认证uer glance身份，取得user glance的token
- glance(tcp:53046) --> keystone(tcp:5000), keystone-->keystone db，使用验过的glance身份（user glance的token）去认证uer admin的token，并成功
- glance(tcp:43508) --> MEMCACHE，存储admin token
- glance --> glance db，查询glance DB，取得image元数据
- glance(tcp:9292) --> OpenStackClient(tcp:59804)，返回查询结果
4.3.3 抓包的流量图
这里使用的是keystone密码认证，用户通过用户名和密码向 Keystone 提交凭证（Credentials），Keystone 验证用户的用户名和密码，验证通过后会生成一个令牌（Token）并发送给用户，用户拿到令牌后，可以使用这个令牌访问 OpenStack 的其他服务，示意如下：
用户 -> (用户名/密码) -> Keystone -> (令牌) -> 用户
比如，这里user admin取得令牌后，访问OpenStack glance service。
glance service要继续对admin token进行验证，首先glance service自己作为用户向keystone认证，获得glance token，然后使用glance token，访问keystone服务，验证admin token。admin token验证通过后，再执行后续动作（查询glance db，返回查询结果给OpenStackClient）
5、Mermaid绘制时序图
可以用mermaid绘制时序图，方便查看:
附mermaid code:
sequenceDiagramautonumberparticipant C as OpenStackClientparticipant K as keystoneparticipant KD as keystone DBparticipant G as glanceparticipant GD as glance DB participant M as memcachedC ->> K: user admin scope to project admin, 发送username:admin/password:xxxxNote left of C: 命令行中输入: <br/> openstack image listK ->> KD: Keystone 查询user admin的身份KD ->> K: okK ->> C: user admin身份验证成功，Keystone生成admin tokenC ->> G: user admin向 glance service请求image list，请求中包含之前从 Keystone 获取的admin tokenG ->> M: glance服务接收到请求，去memcache检查请求中的admin tokenNote right of M: 先检查memcached是否有admin tokenM ->> G: 没有，继续去keystone检查G ->> K: user glance scope to project service，发送username:glance/password:xxxxNote right of G: glance首先要认证自己的身份！K ->> KD: Keystone 查询user glance的身份KD ->> K: okK ->> G: glance的身份验证成功，Keystone 生成glance tokenG ->> K: glance服务将admin token发送回 Keystone，请求验证admin token的有效性，请求中包含之前从 Keystone 获取的glance tokenNote right of G: glance拿着自己的token，去验证admin tokenK ->> KD: Keystone 查询admin tokenKD ->> K: okK ->> G: admin token有效，Keystone 返回给glance serviceG ->> M: please store admin tokenNote right of M: 保存一份到memcacheM ->> G: storedG ->> GD: glance service根据用户的权限和角色，检索glance dbGD ->> G: 返回镜像列表G ->> C: 返回镜像列表
![BUU [PASECA2019]honey_shop](https://img-blog.csdnimg.cn/img_convert/e5326bd45cf53322422a68f6a3eff439.png)
