【DataSophon】DataSophon1.2.1 ranger usersync整合

一、简介

二、实现步骤

2.1 ranger-usersync包下载编译

2.2 构建压缩包

2.3 编辑元数据文件

2.4 修改源码

三、重新安装

一、简介

如下是DDP1.2.1默认有的rangerAdmin，我们需要将rangerusersync整合进来 ,实现将Linux机器上的用户和组信息同步到RangerAdmin的数据库中进行管理。

二、实现步骤

2.1 ranger-usersync包下载编译

ranger官网tar包下载

https://ranger.apache.org/download.html

自己编译或者网上下载

参考文章：

Apache Ranger - Download Apache Ranger?

Ranger安装和使用-CSDN博客

https://juejin.cn/post/6844904159930482696

https://zhuanlan.zhihu.com/p/562012618

权限管理Ranger详解_大数据权限管理利器 - ranger-CSDN博客

2.2 构建压缩包

将ranger-usersync安装包集成到ranger组件中重新打包，如下是ranger admin包和ranger-usersync包。

# 解压ranger组件原有包
tar -zxvf ranger-2.1.0.tar.gz
cd ranger-2.1.0# 将编译好的的sync安装包解压到当前目录
tar -zxvf ranger-2.1.0-usersync.tar.gz ./
cd ranger-2.1.0-usersync
vim ranger_usersync.sh

#!/bin/bash# 获取脚本当前目录
current_path=$(dirname "$0")# 使用说明
usage="Usage: $0 {start|stop|status|restart}"start(){echo "ranger userSync start"sh "$current_path/ranger-usersync" startif [ $? -eq 0 ]; thenecho "ranger userSync start success"elseecho "ranger userSync start failed"exit 1fi
}stop(){echo "ranger userSync stop"sh "$current_path/ranger-usersync" stopif [ $? -eq 0 ]; thenecho "ranger userSync stop success"elseecho "ranger userSync stop failed"exit 1fi
}status(){process_name="UnixAuthenticationService"# 使用 pgrep 命令检测进程是否存在pgrep -f "$process_name" > /dev/nullif [ $? -eq 0 ]; thenecho "进程 $process_name 存在"exit 0elseecho "进程 $process_name 不存在"exit 1fi
}restart(){echo "ranger userSync restart"sh "$current_path/ranger-usersync" restartif [ $? -eq 0 ]; thenecho "ranger userSync restart success"elseecho "ranger userSync restart failed"exit 1fi
}# 处理参数
startStop=$1case $startStop instart)start;;stop)stop;;status)status;;restart)restart;;*)echo "$usage"exit 1;;
esacecho "End $startStop ranger userSync"

打包

tar -zcvf ranger-2.1.0.tar.gz ranger-2.1.0
md5sum ranger-2.1.0.tar.gz
echo '756fa828e02d8f890ca2165d237ef487' > ranger-2.1.0.tar.gz.md5
cp ranger-2.1.0.tar.gz ranger-2.1.0.tar.gz.md5 /opt/datasophon/DDP/packages/

2.3 编辑元数据文件

ranger安装配置文件

vim /opt/datasophon/DDP/packages/datasophon-manager-1.2.1/conf/meta/DDP-1.2.1/RANGER/service_ddl.json

{"name": "RANGER","label": "Ranger","description": "权限控制框架","version": "2.1.0","sortNum": 10,"dependencies":[],"packageName": "ranger-2.1.0.tar.gz","decompressPackageName": "ranger-2.1.0","roles": [{"name": "RangerAdmin","label": "RangerAdmin","roleType": "master","cardinality": "1","logFile": "/var/log/ranger/admin/ranger-admin-${host}-root.log","jmxPort": 6081,"sortNum": 1,"startRunner": {"timeout": "60","program": "bin/ranger_admin.sh","args": ["start"]},"stopRunner": {"timeout": "600","program": "bin/ranger_admin.sh","args": ["stop"]},"statusRunner": {"timeout": "60","program": "bin/ranger_admin.sh","args": ["status"]},"externalLink": {"name": "RangerAdmin Ui","label": "RangerAdmin Ui","url": "http://${host}:6080"}},{"name": "RangerUsersync","label": "RangerUsersync","roleType": "master","runAs": {"user": "root","group": "root"},"cardinality": "1","logFile": "ranger-2.1.0-usersync/logs/usersync-${host}-ranger.log","jmxPort": "","sortNum": 2,"startRunner": {"timeout": "60","program": "ranger-2.1.0-usersync/ranger_usersync.sh","args": ["start"]},"stopRunner": {"timeout": "600","program": "ranger-2.1.0-usersync/ranger_usersync.sh","args": ["stop"]},"statusRunner": {"timeout": "60","program": "ranger-2.1.0-usersync/ranger_usersync.sh","args": ["status"]},"restartRunner": {"timeout": "60","program": "ranger-2.1.0-usersync/ranger_usersync.sh","args": ["restart"]}}],"configWriter": {"generators": [{"filename": "install.properties","configFormat": "custom","templateName": "ranger-install.ftl","outputDirectory": "","includeParams": ["rootPassword","dbHost","database","rangerUser","rangerPassword","rangerAdminUrl","enableHDFSPlugin","enableHIVEPlugin","enableHBASEPlugin","spnegoPrincipal","spnegoKeytab","adminPrincipal","adminKeytab","hadoopHome","rangerHome"]},{"filename": "install.properties1","configFormat": "custom","templateName": "ranger-usersync-install.ftl","outputDirectory": "ranger-2.1.0-usersync","includeParams": ["rangerAdminUrl","adminPrincipal","adminKeytab","hadoopHome","syncInterval"]}]},"parameters": [{"name": "rootPassword","label": "数据库root用户密码","description": "","required": true,"configType": "map","type": "input","value": "","configurableInWizard": true,"hidden": false,"defaultValue": "123456"},{"name": "dbHost","label": "数据库地址","description": "","required": true,"configType": "map","type": "input","value": "","configurableInWizard": true,"hidden": false,"defaultValue": "${apiHost}"},{"name": "database","label": "数据库名","description": "","required": true,"configType": "map","type": "input","value": "","configurableInWizard": true,"hidden": false,"defaultValue": "ranger"},{"name": "rangerUser","label": "Ranger数据库用户","description": "","required": true,"configType": "map","type": "input","value": "","configurableInWizard": true,"hidden": false,"defaultValue": "ranger"},{"name": "rangerPassword","label": "Ranger数据库密码","description": "","required": true,"configType": "map","type": "input","value": "","configurableInWizard": true,"hidden": false,"defaultValue": "ranger"},{"name": "rangerAdminUrl","label": "Ranger访问地址","description": "","required": true,"configType": "map","type": "input","value": "","configurableInWizard": true,"hidden": false,"defaultValue": "${rangerAdminUrl}"},{"name": "enableHDFSPlugin","label": "启用HDFS Ranger插件","description": "","required": true,"type": "switch","value": false,"configurableInWizard": true,"hidden": false,"defaultValue": false},{"name": "enableHIVEPlugin","label": "启用Hive Ranger插件","description": "","required": true,"type": "switch","value": false,"configurableInWizard": true,"hidden": false,"defaultValue": false},{"name": "enableHBASEPlugin","label": "启用Hbase Ranger插件","description": "","required": true,"type": "switch","value": false,"configurableInWizard": true,"hidden": false,"defaultValue": false},{"name": "enableKerberos","label": "开启Kerberos认证","description": "开启Kerberos认证","required": false,"type": "switch","value": false,"configurableInWizard": true,"hidden": true,"defaultValue": false},{"name": "spnegoPrincipal","label": "Spnego Principal","description": "","configWithKerberos": true,"required": false,"configType": "map","type": "input","value": "HTTP/${host}@HADOOP.COM","configurableInWizard": true,"hidden": true,"defaultValue": "HTTP/${host}@HADOOP.COM"},{"name": "spnegoKeytab","label": "Spnego Keytab","description": "","configWithKerberos": true,"required": false,"configType": "map","type": "input","value": "/etc/security/keytab/spnego.service.keytab","configurableInWizard": true,"hidden": true,"defaultValue": "/etc/security/keytab/spnego.service.keytab"},{"name": "adminPrincipal","label": "Ranger Admin Principal","description": "","configWithKerberos": true,"required": false,"configType": "map","type": "input","value": "rangeradmin/${host}@HADOOP.COM","configurableInWizard": true,"hidden": true,"defaultValue": "rangeradmin/${host}@HADOOP.COM"},{"name": "adminKeytab","label": "Ranger Admin Keytab","description": "","configWithKerberos": true,"required": false,"configType": "map","type": "input","value": "/etc/security/keytab/rangeradmin.keytab","configurableInWizard": true,"hidden": true,"defaultValue": "/etc/security/keytab/rangeradmin.keytab"},{"name": "hadoopHome","label": "HADOOP_HOME","description": "","configWithKerberos": true,"required": true,"configType": "map","type": "input","value": "${HADOOP_HOME}","configurableInWizard": true,"hidden": true,"defaultValue": "${HADOOP_HOME}"},{"name": "rangerHome","label": "RANGER_HOME","description": "","required": true,"configType": "map","type": "input","value": "${RANGER_HOME}","configurableInWizard": true,"hidden": false,"defaultValue": "${RANGER_HOME}"},{"name": "syncInterval","label": "SYNC_INTERVAL","description": "userSync同步间隔时间，单位(分钟)","required": true,"configType": "map","type": "input","value": "1","configurableInWizard": true,"hidden": false,"defaultValue": "1"}]
}

各worker元数据文件，已部署的路径：

vim /opt/datasophon/datasophon-worker/conf/templates/ranger-usersync-install.ftl

ranger配置文件 install.properties ,使用了 SYNC_SOURCE = unix

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.# The base path for the usersync process
ranger_base_dir = /etc/ranger#
# The following URL should be the base URL for connecting to the policy manager web application
# For example:
#
#  POLICY_MGR_URL = http://policymanager.xasecure.net:6080
#
POLICY_MGR_URL = ${rangerAdminUrl}# sync source,  only unix and ldap are supported at present
# defaults to unix
SYNC_SOURCE = unix#
# Minimum Unix User-id to start SYNC.
# This should avoid creating UNIX system-level users in the Policy Manager
#
MIN_UNIX_USER_ID_TO_SYNC = 500# Minimum Unix Group-id to start SYNC.
# This should avoid creating UNIX system-level users in the Policy Manager
#
MIN_UNIX_GROUP_ID_TO_SYNC = 500# sync interval in minutes
# user, groups would be synced again at the end of each sync interval
# defaults to 5   if SYNC_SOURCE is unix
# defaults to 360 if SYNC_SOURCE is ldap
SYNC_INTERVAL = ${syncInterval}#User and group for the usersync process
unix_user=ranger
unix_group=ranger#change password of rangerusersync user. Please note that this password should be as per rangerusersync user in ranger
rangerUsersync_password=admin123#Set to run in kerberos environment
usersync_principal=<#if adminPrincipal??>${adminPrincipal}</#if>
usersync_keytab=<#if adminKeytab??>${adminKeytab}</#if>
hadoop_conf=${hadoopHome}/etc/hadoop/conf
#
# The file where all credential is kept in cryptic format
#
CRED_KEYSTORE_FILENAME=/etc/ranger/usersync/conf/rangerusersync.jceks# SSL Authentication
AUTH_SSL_ENABLED=false
AUTH_SSL_KEYSTORE_FILE=/etc/ranger/usersync/conf/cert/unixauthservice.jks
AUTH_SSL_KEYSTORE_PASSWORD=UnIx529p
AUTH_SSL_TRUSTSTORE_FILE=
AUTH_SSL_TRUSTSTORE_PASSWORD=# ---------------------------------------------------------------
# The following properties are relevant only if SYNC_SOURCE = ldap
# ---------------------------------------------------------------# The below properties ROLE_ASSIGNMENT_LIST_DELIMITER, USERS_GROUPS_ASSIGNMENT_LIST_DELIMITER, USERNAME_GROUPNAME_ASSIGNMENT_LIST_DELIMITER,
#and GROUP_BASED_ROLE_ASSIGNMENT_RULES can be used to assign role to LDAP synced users and groups
#NOTE all the delimiters should have different values and the delimiters should not contain characters that are allowed in userName or GroupName# default value ROLE_ASSIGNMENT_LIST_DELIMITER = &
ROLE_ASSIGNMENT_LIST_DELIMITER = &#default value USERS_GROUPS_ASSIGNMENT_LIST_DELIMITER = :
USERS_GROUPS_ASSIGNMENT_LIST_DELIMITER = :#default value USERNAME_GROUPNAME_ASSIGNMENT_LIST_DELIMITER = ,
USERNAME_GROUPNAME_ASSIGNMENT_LIST_DELIMITER = ,# with above mentioned delimiters a sample value would be ROLE_SYS_ADMIN:u:userName1,userName2&ROLE_SYS_ADMIN:g:groupName1,groupName2&ROLE_KEY_ADMIN:u:userName&ROLE_KEY_ADMIN:g:groupName&ROLE_USER:u:userName3,userName4&ROLE_USER:g:groupName3
#&ROLE_ADMIN_AUDITOR:u:userName&ROLE_KEY_ADMIN_AUDITOR:u:userName&ROLE_KEY_ADMIN_AUDITOR:g:groupName&ROLE_ADMIN_AUDITOR:g:groupName
GROUP_BASED_ROLE_ASSIGNMENT_RULES =# URL of source ldap
# a sample value would be:  ldap://ldap.example.com:389
# Must specify a value if SYNC_SOURCE is ldap
SYNC_LDAP_URL =# ldap bind dn used to connect to ldap and query for users and groups
# a sample value would be cn=admin,ou=users,dc=hadoop,dc=apache,dc=org
# Must specify a value if SYNC_SOURCE is ldap
SYNC_LDAP_BIND_DN =# ldap bind password for the bind dn specified above
# please ensure read access to this file  is limited to root, to protect the password
# Must specify a value if SYNC_SOURCE is ldap
# unless anonymous search is allowed by the directory on users and group
SYNC_LDAP_BIND_PASSWORD =# ldap delta sync flag used to periodically sync users and groups based on the updates in the server
# please customize the value to suit your deployment
# default value is set to true when is SYNC_SOURCE is ldap
SYNC_LDAP_DELTASYNC =# search base for users and groups
# sample value would be dc=hadoop,dc=apache,dc=org
SYNC_LDAP_SEARCH_BASE =# search base for users
# sample value would be ou=users,dc=hadoop,dc=apache,dc=org
# overrides value specified in SYNC_LDAP_SEARCH_BASE
SYNC_LDAP_USER_SEARCH_BASE = # search scope for the users, only base, one and sub are supported values
# please customize the value to suit your deployment
# default value: sub
SYNC_LDAP_USER_SEARCH_SCOPE = sub# objectclass to identify user entries
# please customize the value to suit your deployment
# default value: person
SYNC_LDAP_USER_OBJECT_CLASS = person# optional additional filter constraining the users selected for syncing
# a sample value would be (dept=eng)
# please customize the value to suit your deployment
# default value is empty
SYNC_LDAP_USER_SEARCH_FILTER =# attribute from user entry that would be treated as user name
# please customize the value to suit your deployment
# default value: cn
SYNC_LDAP_USER_NAME_ATTRIBUTE = cn# attribute from user entry whose values would be treated as
# group values to be pushed into Policy Manager database
# You could provide multiple attribute names separated by comma
# default value: memberof, ismemberof
SYNC_LDAP_USER_GROUP_NAME_ATTRIBUTE = memberof,ismemberof
#
# UserSync - Case Conversion Flags
# possible values:  none, lower, upper
SYNC_LDAP_USERNAME_CASE_CONVERSION=lower
SYNC_LDAP_GROUPNAME_CASE_CONVERSION=lower#user sync log path
logdir=logs
#/var/log/ranger/usersync# PID DIR PATH
USERSYNC_PID_DIR_PATH=/var/run/ranger# do we want to do ldapsearch to find groups instead of relying on user entry attributes
# valid values: true, false
# any value other than true would be treated as false
# default value: false
SYNC_GROUP_SEARCH_ENABLED=# do we want to do ldapsearch to find groups instead of relying on user entry attributes and
# sync memberships of those groups
# valid values: true, false
# any value other than true would be treated as false
# default value: false
SYNC_GROUP_USER_MAP_SYNC_ENABLED=# search base for groups
# sample value would be ou=groups,dc=hadoop,dc=apache,dc=org
# overrides value specified in SYNC_LDAP_SEARCH_BASE,  SYNC_LDAP_USER_SEARCH_BASE
# if a value is not specified, takes the value of  SYNC_LDAP_SEARCH_BASE
# if  SYNC_LDAP_SEARCH_BASE is also not specified, takes the value of SYNC_LDAP_USER_SEARCH_BASE
SYNC_GROUP_SEARCH_BASE=# search scope for the groups, only base, one and sub are supported values
# please customize the value to suit your deployment
# default value: sub
SYNC_GROUP_SEARCH_SCOPE=# objectclass to identify group entries
# please customize the value to suit your deployment
# default value: groupofnames
SYNC_GROUP_OBJECT_CLASS=# optional additional filter constraining the groups selected for syncing
# a sample value would be (dept=eng)
# please customize the value to suit your deployment
# default value is empty
SYNC_LDAP_GROUP_SEARCH_FILTER=# attribute from group entry that would be treated as group name
# please customize the value to suit your deployment
# default value: cn
SYNC_GROUP_NAME_ATTRIBUTE=# attribute from group entry that is list of members
# please customize the value to suit your deployment
# default value: member
SYNC_GROUP_MEMBER_ATTRIBUTE_NAME=# do we want to use paged results control during ldapsearch for user entries
# valid values: true, false
# any value other than true would be treated as false
# default value: true
# if the value is false, typical AD would not return more than 1000 entries
SYNC_PAGED_RESULTS_ENABLED=# page size for paged results control
# search results would be returned page by page with the specified number of entries per page
# default value: 500
SYNC_PAGED_RESULTS_SIZE=
#LDAP context referral could be ignore or follow
SYNC_LDAP_REFERRAL = ignore# if you want to enable or disable jvm metrics for usersync process
# valid values: true, false
# any value other than true would be treated as false
# default value: false
# if the value is false, jvm metrics is not created
JVM_METRICS_ENABLED=# filename of jvm metrics created for usersync process
# default value: ranger_usersync_metric.json
JVM_METRICS_FILENAME=#file directory for jvm metrics
# default value : logdir
JVM_METRICS_FILEPATH=#frequency for jvm metrics to be updated
# default value : 10000 milliseconds
JVM_METRICS_FREQUENCY_TIME_IN_MILLIS=

2.4 修改源码

com.datasophon.worker.strategy.RangerAdminHandlerStrategy

/**  Licensed to the Apache Software Foundation (ASF) under one or more*  contributor license agreements.  See the NOTICE file distributed with*  this work for additional information regarding copyright ownership.*  The ASF licenses this file to You under the Apache License, Version 2.0*  (the "License"); you may not use this file except in compliance with*  the License.  You may obtain a copy of the License at**     http://www.apache.org/licenses/LICENSE-2.0**  Unless required by applicable law or agreed to in writing, software*  distributed under the License is distributed on an "AS IS" BASIS,*  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.*  See the License for the specific language governing permissions and*  limitations under the License.*/package com.datasophon.worker.strategy;import cn.hutool.core.io.FileUtil;
import com.datasophon.common.Constants;
import com.datasophon.common.cache.CacheUtils;
import com.datasophon.common.command.ServiceRoleOperateCommand;
import com.datasophon.common.enums.CommandType;
import com.datasophon.common.utils.ExecResult;
import com.datasophon.common.utils.ShellUtils;
import com.datasophon.worker.handler.ServiceHandler;
import com.datasophon.worker.utils.KerberosUtils;import java.util.ArrayList;public class RangerAdminHandlerStrategy extends AbstractHandlerStrategy implements ServiceRoleStrategy {public RangerAdminHandlerStrategy(String serviceName, String serviceRoleName) {super(serviceName, serviceRoleName);}@Overridepublic ExecResult handler(ServiceRoleOperateCommand command) {String workPath = Constants.INSTALL_PATH + Constants.SLASH + command.getDecompressPackageName();ExecResult startResult = new ExecResult();ServiceHandler serviceHandler = new ServiceHandler(command.getServiceName(), command.getServiceRoleName());if (command.getEnableKerberos()) {logger.info("start to get ranger keytab file");String hostname = CacheUtils.getString(Constants.HOSTNAME);KerberosUtils.createKeytabDir();if (!FileUtil.exist("/etc/security/keytab/spnego.service.keytab")) {KerberosUtils.downloadKeytabFromMaster("HTTP/" + hostname, "spnego.service.keytab");}if (!FileUtil.exist("/etc/security/keytab/rangeradmin.keytab")) {KerberosUtils.downloadKeytabFromMaster("rangeradmin/" + hostname, "rangeradmin.keytab");}}if (command.getCommandType().equals(CommandType.INSTALL_SERVICE) && command.getServiceRoleName().equals("RangerUsersync")) {ShellUtils.exceShell("mv " + workPath + "/ranger-2.1.0-usersync/install.properties1 " + workPath + "/ranger-2.1.0-usersync/install.properties");ShellUtils.exceShell("chmod 755 " + workPath + "/ranger-2.1.0-usersync/install.properties");logger.info("setup ranger user sync");ArrayList<String> commands = new ArrayList<>();commands.add("sh");commands.add("./setup.sh");ExecResult execResult = ShellUtils.execWithStatus(workPath + "/ranger-2.1.0-usersync", commands, 300L, logger);if (execResult.getExecResult()) {logger.info("setup ranger user sync success");} else {logger.info("setup ranger user sync failed");return execResult;}ShellUtils.exceShell("sed -i '/<name>ranger\\.usersync\\.enabled<\\/name>/{n;s/<value>false<\\/value>/<value>true<\\/value>/}' "+ workPath +"/ranger-2.1.0-usersync/conf/ranger-ugsync-site.xml");startResult = serviceHandler.start(command.getStartRunner(), command.getStatusRunner(),command.getDecompressPackageName(), command.getRunAs());} else {startResult = serviceHandler.start(command.getStartRunner(), command.getStatusRunner(),command.getDecompressPackageName(), command.getRunAs());}return startResult;}}

com.datasophon.worker.strategy.ServiceRoleStrategyContext

map.put("RangerUsersync", new RangerAdminHandlerStrategy("RANGER", "RangerUsersync"));

datasophon-worker jar包更新

mv datasophon-worker-1.2.1.jar      /opt/datasophon/datasophon-worker/lib/

三、重新安装

添加ranger服务

分配服务Master角色

服务配置

选择"settings"我们可以看到linux 上的用户已同步成功。