MongDB集群部署和升级方案

******集群部署部分******

主机信息mongodb集群
192.168.1.46
192.168.1.47
192.168.1.48
192.168.1.49
192.168.1.50
192.168.1.51

规划
sharding 1
192.168.1.46
192.168.1.47
192.168.1.48
端口 28111

sharding 2
192.168.1.49
192.168.1.50
192.168.1.51
端口 28112

config 副本集
192.168.1.46
192.168.1.47
192.168.1.48
端口 28600

mongos 集群
192.168.1.46
192.168.1.47
192.168.1.48
端口 28700

做一个LSB 各个负载均衡IP+port
mongo:192.167.63.22:40005

[
目前DB使用的lvs主机:
192.167.3.169
192.167.3.170
]

数据库版本
3.4.24

下载
wget https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-rhel70-3.4.24.tgz
wget https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-rhel70-3.6.17.tgz[升级需要的版本]

目录规划 mkdir -p src/{a,b,c,d}
mkdir -p /DATA/mongo_user_sale/{data,etc,logs,config,mongos}

部署
第一部分
1.先部署sharding副本集
参数
# for documentation of all options, see:
# http://docs.mongodb.org/manual/reference/configuration-options/

# where to write logging data.
systemLog:
destination: file
logAppend: true
logRotate: reopen
path: /DATA/mongo_user_sale/logs/mongo_usersale.log

# Where and how to store data.
storage:
dbPath: /DATA/mongo_user_sale/data
journal:
enabled: true
directoryPerDB: true
engine: wiredTiger
#########storage.wiredTiger Options
wiredTiger:
engineConfig:
cacheSizeGB: 24
journalCompressor: "zlib"
directoryForIndexes: true
collectionConfig:
blockCompressor: "zlib"
indexConfig:
prefixCompression: true

##########operationProfiling Options
operationProfiling:
slowOpThresholdMs: 100
mode: "all"

########ProcessManagement Options
processManagement:
fork: true
pidFilePath: /DATA/mongo_user_sale/data/mongod.pid

# network interfaces
net:
port: 28111
bindIp: 0.0.0.0
maxIncomingConnections: 18500
#serviceExecutor: adaptive #4.2版本
#security:
#keyFile: /DATA/mongo_user_sale/etc/keyfile

#operationProfiling:

##########replication Options
replication:
replSetName: usersale1Rs
oplogSizeMB: 50240
#secondaryIndexPrefetch: all

sharding:
clusterRole: shardsvr #configsvr or shardsvr

## Enterprise-Only Options

#auditLog:

#snmp:

setParameter:
enableLocalhostAuthBypass: true
replWriterThreadCount: 32
wiredTigerConcurrentReadTransactions: 128
wiredTigerConcurrentWriteTransactions: 128

第一次部署将security注释
注意:其他sharding部署注意副本集名称和端口

2.在第一个sharding 192.168.1.46操作
(1)编辑参数# pwd
/DATA/mongo_user_sale/etc
vim mongo_usersale.conf
(2)启动
[root@node10-133-1-46 etc]# /usr/local/mongodb-3.4.24/bin/mongod -f /DATA/mongo_user_sale/etc/mongo_usersale.conf
about to fork child process, waiting until server is ready for connections.
forked process: 47385
child process started successfully, parent exiting
说明启动成功

(3)配置单机副本集
# /usr/local/mongodb-3.4.24/bin/mongo 192.168.1.46:28111/admin
> config={_id:"usersale1Rs",members:[{_id:1,host:"192.168.1.46:28111",priority:136,tags:{'use':'user-sale-rs1-1'}}]}
> rs.initiate(config)
{ "ok" : 1 }
usersale1Rs:SECONDARY>
usersale1Rs:PRIMARY>

(4)创建账号
a. 管理账号:
usersale1Rs:PRIMARY> use admin
switched to db admin
db.createUser(
{
user: "admin",
pwd: "tk123456",
roles:
[
{role: "userAdminAnyDatabase", db: "admin"},
{ role: "readAnyDatabase", db: "admin" },
{ role: "dbOwner", db: "admin" },
{ role: "userAdmin", db: "admin" },
{ role: "root", db: "admin" },
{ role: "clusterMonitor", db: "admin" },
{ role: "dbAdmin", db: "admin" },
]
}
)

b. mongodb只读账号
use admin
db.createUser(
{
user: "readany",
pwd: "tk123456",
roles:
[
{ role: "readAnyDatabase", db: "admin" },
]
}
)

c. 监控用户
use admin
db.createUser(
{
user: "mongodb_exporter",
pwd: "tk123456",
roles:
[
{ role: "readAnyDatabase", db: "admin" },
{ role: "dbAdmin", db: "admin" }
]
}
)

d. 授予抓取Oplog权限
db.createRole({role:'sysadmin',roles:[], privileges:[ {resource:{anyResource:true},actions:['anyAction']}]})
db.grantRolesToUser( "admin" , [ { role: "sysadmin", db: "admin" } ])
db.grantRolesToUser( "readany" , [ { role: "sysadmin", db: "admin" } ])
第一个admin和readany是用户名
db.system.users.find().pretty() 查询看一下是否授权成功

(5)重启并开启认证模块
a. 添加认证文件
vim keyfile
MIIEowIBAAKCAQEAxJ4E9nfpSaEtBUUP/ypkVmzr2ORD4BUSzQXp+EMXv0oeIeW0
vU6Jv/mvAPD0+9veCGK/LFmZGVSQZbjyYnFxx1xLCgzqkkvgX8HqaChPKySta3wi
YHQLgBDJ4boXZoLIFlGO9UWjJTMXZSbD0Tdu4g2DL3qZLGENDLFam4gt0t2HnZ+g
iY/9C9PKHnKqUpjfJNnWU8h09Zc4erZdM50KWxEdC4iFMKWTSOZr/AxzXC5cOYC/
y8ALfBLlJstavp1KMG2XoQoGRRnLSX3bWqIRt60oFUq35x8slknHxOKkXxhX7583
BfLHfWxrCU8viOp9z1yb0URUR6fotykpUz9/QwIDAQABAoIBADkCADpeRjvdCaqv
Fl3BDttoQ5Munt7pPNljSvwxnee3ZWBRwmQqhQo85OEYcIRsXMoG2ceW71HgTlwK
fpHLXyDIksUNUNujcbX9152/a94Fd4FU45GpcxojDwPfS+pG8GObOvNopSZzEivS
Az21uTZ51zxwv5Jwi0cs3vDLq9/HmkPGUFpIuRq/CdZGmOh4m0HWWaKohPFHQPJ8
lfMjAaECgYEA7RWeg6nnRzMHhNiMRw2bBURdYhDu13gorWY2wr+Xse8+oxHDrz0d
baHbF0UWkUlTpeiQK30ygWy82/bm8v6lds12iElkY13Ua+xMxHdZkY1h2k69n/cE
en18ESaapcX1DjWNzWCtP89g9x2p/MMljDJ/tr2xt6KxiEcxVpToo0sCgYEA1E3e
pCLflIGbN2x3fUJkoM2aVpbaffsbLHkf8SglV9OJI7ROAvZ29xmXa7sYkN7OdXMu

# chmod 600 keyfile 权限必须是600
b. 将mongod shutdown
# /usr/local/mongodb-3.4.24/bin/mongod -f /DATA/mongo_user_sale/etc/mongo_usersale.conf --shutdown
将如下注释去掉
security:
keyFile: /DATA/mongo_user_sale/etc/keyfile
c. 启动 /usr/local/mongodb-3.4.24/bin/mongod -f /DATA/mongo_user_sale/etc/mongo_usersale.conf
进入后需要认证
db.auth('admin','tk123456')
usersale1Rs:PRIMARY> db.auth('admin','tk123456')
1
usersale1Rs:PRIMARY> rs.status() 查看状态
"members" : [
{
"_id" : 1,
"name" : "192.168.1.46:28111",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 363,
"optime" : {
"ts" : Timestamp(1586659765, 1),
"t" : NumberLong(2)
},
"optimeDate" : ISODate("2020-04-12T02:49:25Z"),
"syncingTo" : "",
"syncSourceHost" : "",
"syncSourceId" : -1,
"infoMessage" : "",
"electionTime" : Timestamp(1586659413, 1),
"electionDate" : ISODate("2020-04-12T02:43:33Z"),
"configVersion" : 1,
"self" : true,
"lastHeartbeatMessage" : ""
}
],
"ok" : 1
}

(6)复制1.46的参数文件和认证文件到47/48主机，并启动
/usr/local/mongodb-3.4.24/bin/mongod -f /DATA/mongo_user_sale/etc/mongo_usersale.conf

(7)添加节点
rs.add({_id:2, host:"192.168.1.47:28111", priority:100,tags:{'use':'user-sale-rs1-2'}})
rs.add({_id:3, host:"192.168.1.48:28111", priority:190,tags:{'use':'user-sale-rs1-3'}}) #添加完，该节点为primary
注意，我设置了优先级，可以不设置

3.日志切分--每天生成一个
mongodb日志切换
mongodb日志切换路径保留日志30天
vim /etc/logrotate.d/mongo_28111
/DATA1/mongo_28116/logs/mongo_28111.log {
daily
rotate 30
dateext
compress
missingok
notifempty
sharedscripts
copytruncate
postrotate
/bin/kill -SIGUSR1 `cat /DATA/mongo_user_sale/data/mongod.lock 2> /dev/null` 2> /dev/null || true
endscript
}

启动
logrotate -vf /etc/logrotate.conf

4.压测
./ycsb load mongodb -s -P /DATA/soft/ycsb-0.12.0/workloads/workloada -threads 1024 -p recordcount=0 -p operationcount=0 -p insertstart=0 -p insertcount=330000000 -p readproportion=0.1 -p updateproportion=0.9 -p insertproportion=0 -p fieldcount=8 -p fieldlength=250 -p mongodb.url=mongodb://admin:tk123456@192.168.1.48:28111/admin?waitQueueMultiple=100 -p table=userinfo > /dev/null 2> /DATA/soft/lsbwrite &

监控
/usr/local/mongodb-3.4.24/bin//mongostat --host=192.168.1.48 --port=28111 -u admin -p "tk123456" --authenticationDatabase admin

5.部署第二个sharding
参考第一个部署步骤
端口和副本集修改
> config={_id:"usersale2Rs",members:[{_id:1,host:"192.168.1.49:28112",priority:136,tags:{'use':'user-sale-rs2-1'}}]}
> rs.initiate(config)
rs.add({_id:2, host:"192.168.1.50:28112", priority:100,tags:{'use':'user-sale-rs2-2'}})
rs.add({_id:3, host:"192.168.1.51:28112", priority:90,tags:{'use':'user-sale-rs2-3'}})

第二部分
部署config副本集
1.配置参数
# fordocumentation of all options, see:
# http://docs.mongodb.org/manual/reference/configuration-options/

# where to writelogging data.
systemLog:
destination: file
logAppend: true
logRotate: reopen
path: /DATA/mongo_user_sale/logs/configsvr.log

# Where and howto store data.
storage:
dbPath: /DATA/mongo_user_sale/config
journal:
enabled: true
directoryPerDB: true
engine: wiredTiger
#########storage.wiredTigerOptions
wiredTiger:
engineConfig:
cacheSizeGB: 4
directoryForIndexes: true
indexConfig:
prefixCompression: true

##########operationProfilingOptions
#operationProfiling:
#slowOpThresholdMs: 50
#mode: "all"

########ProcessManagementOptions
processManagement:
fork: true
pidFilePath: /DATA/mongo_user_sale/config/config.pid

# networkinterfaces
net:
port: 28600
bindIp: 0.0.0.0
maxIncomingConnections: 15000
security:
keyFile: /DATA/mongo_user_sale/etc/keyfile

#operationProfiling:

##########replicationOptions
replication:
replSetName: configRS
oplogSizeMB: 20480
#secondaryIndexPrefetch: all

sharding:
clusterRole: configsvr #configsvr or shardsvr

2.启动
# /usr/local/mongodb-3.4.24/bin/mongod -f /DATA/mongo_user_sale/etc/config.conf
3.配置副本集
> rs.initiate( {_id: "configRS",
... configsvr: true,
... members: [
... { _id: 1, host: "192.168.1.46:28600", priority:100, tags:{'use':'config-1' }}
... ]
... } )
{ "ok" : 1 }
configRS:SECONDARY>
configRS:PRIMARY>

3.创建账号
管理账号:
usersale1Rs:PRIMARY> use admin
switched to db admin
db.createUser(
{
user: "admin",
pwd: "tk123456",
roles:
[
{role: "userAdminAnyDatabase", db: "admin"},
{ role: "readAnyDatabase", db: "admin" },
{ role: "dbOwner", db: "admin" },
{ role: "userAdmin", db: "admin" },
{ role: "root", db: "admin" },
{ role: "clusterMonitor", db: "admin" },
{ role: "dbAdmin", db: "admin" },
]
}
)

4.启停止后，将认证参数打开
5.添加节点
rs.add({_id:2, host:"192.168.1.47:28600", priority:90, tags:{'use':'config-2'}})
rs.add({_id:3, host:"192.168.1.48:28600", priority:30, tags:{'use':'config-3'}})

【或者
rs.initiate( {_id: "configReplSet",
configsvr: true,
members: [
{ _id: 1, host: "192.168.1.46:28600", priority:100, tags:{'use':'config-1' }},
{ _id: 2, host: "192.168.1.47:28600", priority:90, tags:{'use':'config-2' }},
{ _id: 3, host: "192.168.1.48:28600", priority:30, tags:{'use':'config-3' }}
]
} )
】

第二部分
部署mongos
1.参数配置
systemLog:
destination: file
path: /DATA/mongo_user_sale/logs/mongos.log
logAppend: true
processManagement:
fork: true
pidFilePath: /DATA/mongo_user_sale/mongos/mongos.pid
net:
port: 28700
bindIp: 0.0.0.0
sharding:
configDB: configRS/192.168.1.46:28600,192.168.1.47:28600,192.168.1.48:28600
security:
keyFile: "/DATA/mongo_user_sale/etc/keyfile"

2.启动
/usr/local/mongodb-3.4.24/bin/mongos -f /DATA/mongo_user_sale/etc/mongos.conf

说明:
mongos和config创建的账号密码一致，可以和sharding不一样

/usr/local/mongodb-3.4.24/bin/mongo 192.168.1.46:28700/admin

集群添加副本集*****
>use admin
db.runCommand({addshard:"usersale1Rs/192.168.1.46:28111,192.168.1.47:28111,192.168.1.48:28111",name:"usersale1_tag"});
db.runCommand({addshard:"usersale2Rs/192.168.1.49:28112,192.168.1.50:28112,192.168.1.51:28112",name:"usersale2_tag"});
如果有同名的数据库则添加失败:
"errmsg" : "can't add shard 'sale2Rs/192.168.1.49:28116,192.168.1.50:28116,192.168.1.51:28116' because a local database 'ycsb' exists in another sale1_tag",

>sh.status()
databases:
{ "_id" : "ycsb", "primary" : "usersale1_tag", "partitioned" : false }

>db.runCommand({listshards:1})

******集群升级部分******
2. 登录mognos禁用balance
mongos> sh.stopBalancer()

mongos> sh.isBalancerRunning()
true
mongos> sh.getBalancerState()
true

mongos> sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("5e92898736edafe6acf3f45c")
}
shards:
{ "_id" : "usersale1_tag", "host" : "usersale1Rs/192.168.1.46:28111,192.168.1.47:28111,192.168.1.48:28111", "state" : 1 }
{ "_id" : "usersale2_tag", "host" : "usersale2Rs/192.168.1.49:28112,192.168.1.50:28112,192.168.1.51:28112", "state" : 1 }
active mongoses:
"3.4.24" : 3
autosplit:
Currently enabled: yes
balancer:
Currently enabled: yes
Currently running: yes
NaN
Collections with active migrations:
ycsb.userinfo started at Sun Apr 12 2020 11:55:04 GMT+0800 (CST)
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
32 : Success
databases:
{ "_id" : "ycsb", "primary" : "usersale1_tag", "partitioned" : true }
ycsb.userinfo
shard key: { "_id" : 1 }
unique: false
balancing: true
chunks:
usersale1_tag 309
usersale2_tag 32
too many chunks to print, use verbose if you want to force print

mongos> sh.stopBalancer()
{ "ok" : 1 }
mongos> sh.getBalancerState()
false

3. 升级config server

3.1 从节点先使用3.6版本依次替换3.4 版本,并依次重启
主节点查看 configRS:PRIMARY> rs.status()

3.2 关闭secondary config server
# /usr/local/mongodb-3.4.24/bin/mongod -f /DATA/mongo_user_sale/etc/config.conf --shutdown
killing process with pid: 14654
[root@node10-133-1-48 operation]# /usr/local/mongodb-3.6.17/bin/mongod -f /DATA/mongo_user_sale/etc/config.conf
about to fork child process, waiting until server is ready for connections.
forked process: 25427
child process started successfully, parent exiting

3.3 对于主节点，先切换成从节点
rs.stepDown() 或调整优先级
cfg = rs.conf()
cfg.members[1].priority = 136
rs.reconfig(cfg)

# /usr/local/mongodb-3.4.24/bin/mongod -f /DATA/mongo_user_sale/etc/config.conf --shutdown
killing process with pid: 14654
[root@node10-133-1-48 operation]# /usr/local/mongodb-3.6.17/bin/mongod -f /DATA/mongo_user_sale/etc/config.conf

4. 升级shard
按复制集依次处理
3.1 从节点先使用3.6版本依次替换3.4 版本,并依次重启
3.2 关闭secondary shard1 server
设置为隐藏节点
cfg = rs.conf()
cfg.members[1].priority = 0
cfg.members[1].hidden = true
rs.reconfig(cfg)
使用mongostat查看是否select还有值>1
[root@node10-133-1-46 operation]# /usr/local/mongodb-3.4.24/bin/mongod -f /DATA/mongo_user_sale/etc/mongo_usersale.conf --shutdown
killing process with pid: 48861
[root@node10-133-1-46 operation]#
[root@node10-133-1-46 operation]# /usr/local/mongodb-3.6.17/bin/mongod -f /DATA/mongo_user_sale/etc/mongo_usersale.conf
cfg = rs.conf()
cfg.members[1].priority = 10
cfg.members[1].hidden = false
rs.reconfig(cfg)

3.3
对于主节点，先切换成从节点
先调整优先级，让其他一台secondary成为primaryk
然后将该节点设置为隐藏节点

sharding2升级步骤参考sharding1即可

5. 升级mongos 实例
5.1 使用kill 或者 kill -2 杀掉mognos实例
kill -2 17739
5.2 使用3.6代替3.4 重启
//usr/local/mongodb-3.6.17/bin/mongos -f /DATA/mongo_user_sale/etc/mongos.conf
其他mognos 实例类似

6. 登录任一mongos实例重启启用balancer
/usr/local/mongodb-3.4.18/bin/mongo --port 5080
mongos> sh.startBalancer()

7. 启用后台不兼容3.6 特性
mongos>
mongos> db.adminCommand( { setFeatureCompatibilityVersion: "3.6" } )
{ "ok" : 1 }

mongos执行不输出任何: db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )
可以在sharding上执行和config执行

usersale2Rs:PRIMARY> db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )
{
"featureCompatibilityVersion" : {
"version" : "3.4"
},
"ok" : 1,
"$gleStats" : {
"lastOpTime" : Timestamp(0, 0),
"electionId" : ObjectId("7fffffff0000000000000004")
},
"$configServerState" : {
"opTime" : {
"ts" : Timestamp(1586669968, 1),
"t" : NumberLong(3)
}
}
}
usersale2Rs:PRIMARY> db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )
{
"featureCompatibilityVersion" : {
"version" : "3.6"
},
"ok" : 1,
"operationTime" : Timestamp(1586670057, 521),
"$gleStats" : {
"lastOpTime" : Timestamp(0, 0),
"electionId" : ObjectId("7fffffff0000000000000004")
},
"$configServerState" : {
"opTime" : {
"ts" : Timestamp(1586670058, 589),
"t" : NumberLong(3)
}
},
"$clusterTime" : {
"clusterTime" : Timestamp(1586670060, 2502),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}