搭建Atlas2.2.0 集成CDH6.3.2 生产环境+kerberos

首先确保环境的干净，如果之前有安装过清理掉相关残留

确保安装atlas的服务器有足够的内存（至少16G），有必要的hadoop角色

  • HDFS客户端 — 检索和更新Hadoop使用的用户组信息（UGI）中帐户成员资格的信息。对调试很有用。
  • HBase Client - Atlas 存储其 Janus 数据库，用于初始导入 HBase 内容，因此它需要持续访问 HBase 服务中的两个表。
  • Hive 客户端 - 用于初始导入 Hive 内容。

准备编译环境

mvn3.8.8 必须3.8以上的版本 3.6无法编译

java 1.8.0_181 跟你的CDH环境保持一致

node node-v16.20.2 

下载和解压缩源代码

该项目的网站可以在这里找到 Apache Atlas – Data Governance and Metadata framework for Hadoop

查找并下载 Apache Atlas 

更改pom.xml

在主pom（就是文件夹打开第一个）添加一个包含 maven 工件的 clouder 存储库

<repository><id>cloudera</id><url>https://repository.cloudera.com/artifactory/cloudera-repos</url><releases><enabled>true</enabled></releases><snapshots><enabled>false</enabled></snapshots>
</repository>

然后修改对应的cdh组件版本 

<hadoop.version>3.0.0-cdh6.3.2</hadoop.version>
<hbase.version>2.1.0-cdh6.3.2</hbase.version>
<hive.version>2.1.1-cdh6.3.2</hive.version>
<kafka.scala.binary.version>2.11</kafka.scala.binary.version>
<kafka.version>2.2.1-cdh6.3.2</kafka.version>
<solr-test-framework.version>7.4.0-cdh6.3.2</solr-test-framework.version>
<lucene-solr.version>7.4.0</lucene-solr.version>
<solr.version>7.4.0-cdh6.3.2</solr.version>
<sqoop.version>1.4.7-cdh6.3.2</sqoop.version>
<zookeeper.version>3.4.5-cdh6.3.2</zookeeper.version>

然后修改一些jar包的版本

将“atlas-buildtools”工件的版本从“1.0”更改为“0.8.1”<dependency><groupId>org.apache.atlas</groupId><artifactId>atlas-buildtools</artifactId><version>0.8.1</version></dependency>修改jsr.version为2.0.1<jsr.version>2.0.1</jsr.version>

修改一些次pom 

主目录下 
grep -rn jsr311-apii | grep pom.xmladdons/impala-bridge/pom.xml:332
addons/falcon-bridge/pom.xml:178
addons/hive-bridge/pom.xml:312: 
addons/hbase-bridge/pom.xml:345:
addons/storm-bridge/pom.xml:360:
addons/sqoop-bridge/pom.xml:250:这几个pom中jsr311-api改成javax.ws.rs-api

修改其他文件

 在文件

addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java

中，转到第618行，

注释”String catalogName = hiveDB.getCatalogName（） ！= null ？hiveDB.getCatalogName（）.toLowerCase（） ： null;“

并添加 ”String catalogName = null;“：

public static String getDatabaseName(Database hiveDB) {String dbName      = hiveDB.getName().toLowerCase();//String catalogName = hiveDB.getCatalogName() != null ? hiveDB.getCatalogName().toLowerCase() : null;String catalogName = null;if (StringUtils.isNotEmpty(catalogName) && !StringUtils.equals(catalogName, DEFAULT_METASTORE_CATALOG)) {dbName = catalogName + SEP + dbName;}return dbName;
}

在文件

addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java

中

转到第83行”this.metastoreHandler = （listenerEvent ！= null） ？metastoreEvent.getIHMSHandler（） ： null;“，

注释它并添加”this.metastoreHandler = null;“： 

public AtlasHiveHookContext(HiveHook hook, HiveOperation hiveOperation, HookContext hiveContext, HiveHookObjectNamesCache knownObjects,HiveMetastoreHook metastoreHook, ListenerEvent listenerEvent) throws Exception {this.hook             = hook;this.hiveOperation    = hiveOperation;this.hiveContext      = hiveContext;this.hive             = hiveContext != null ? Hive.get(hiveContext.getConf()) : null;this.knownObjects     = knownObjects;this.metastoreHook    = metastoreHook;this.metastoreEvent   = listenerEvent;//this.metastoreHandler = (listenerEvent != null) ? metastoreEvent.getIHMSHandler() : null;this.metastoreHandler = null;init();
}

 在文件addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java

注释第 293 行提到“MATERIALIZED_VIEW”：

private boolean isDdlOperation(AtlasEntity entity) {return entity != null && !context.isMetastoreHook()&& (context.getHiveOperation().equals(HiveOperation.CREATETABLE_AS_SELECT)|| context.getHiveOperation().equals(HiveOperation.CREATEVIEW)|| context.getHiveOperation().equals(HiveOperation.ALTERVIEW_AS));//|| context.getHiveOperation().equals(HiveOperation.CREATE_MATERIALIZED_VIEW));
}

注意这里要加；号，因为原来的符号被注释了 

 在文件addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java

注释提及“MATERIALIZED_VIEW”的第 212 行和第 217 行

开始构建。基本无坑。有问题多试几次。有时候会因为网络问题下不到包

mvn clean  -DskipTests package -Pdist  -Drat.skip=true

包在distro/target/apache-atlas-2.2.0-bin.tar.gz

不要用官方文档说的server包 那个包没有各种hook文件

解压到安装目录，开始安装

为atlas部署准备 CDH 集群服务

  • Atlas使用HBase来存储他的Janus数据库。
  • Solr 用于存储和搜索审核日志。
  • Kafka被用作从Atlas库（即嵌入Hadoop服务中的钩子）到Atlas本身的消息发送器。

1.1. 在 HBase 中创建必要的表

  1. 在 Atlas 计算机或安装了“HBase 网关”角色的任何其他计算机上，创建必要的表
TABLE1="apache_atlas_entity_audit"
TABLE2="apache_atlas_janus"echo "create '${TABLE1}', 'dt'"  | hbase shell
echo "create '${TABLE2}', 's'" | hbase shell

检查已创建的表

  1. 在 Atlas 计算机或安装了“HBase 网关”角色的任何其他计算机上，执行：

    echo "list" | hbase shell
    复制

    标准输出：

    Took 0.0028 seconds
list
TABLE                        
apache_atlas_entity_audit                             
apache_atlas_janus
2 row(s)
Took 0.6872 seconds                 
["apache_atlas_entity_audit", "apache_atlas_janus"]

添加hbase集群配置文件到conf/hbase下

ln -s /etc/hbase/conf/ /data/apache-atlas-2.2.0/conf/hbase

Apache Kafka

Atlas 使用 Apache Kafka 接收有关 Hadoop 服务中发生的事件的消息。消息是使用嵌入在某些服务中的Atlas的特殊库发送的。目前，Atlas 读取有关 Hbase 和 Hive 中事件的消息，例如创建和删除表、添加列、等等等等......

在 Kafka 中添加必要的topic

  1. Apache Atlas 需要 Apache Kafka 中的三个topic。在安装了 Kafka 的计算机上创建它们：

kafka-topics --zookeeper S0:2181,S1:2181,S2:2181,S3:2181 --create --replication-factor 3 --partitions 3 --topic _HOATLASOK
kafka-topics --zookeeper S0:2181,S1:2181,S2:2181,S3:2181 --create --replication-factor 3 --partitions 3 --topic ATLAS_ENTITIES
kafka-topics --zookeeper S0:2181,S1:2181,S2:2181,S3:2181 --create --replication-factor 3 --partitions 3 --topic ATLAS_HOOK

有kerberos的会麻烦一点 具体看这篇

Kerberos环境下 命令行连接kafka 和zk_启用kerberos后zk_Mumunu-的博客-CSDN博客

 配置atlas的sentry role 以访问kafka topic 

在具有“Kafka 网关”和“sentry网关”角色的机器上，在sentry中创建“kafka4atlas_role”角色：

KROLE="kafka4atlas_role"kafka-sentry -cr -r ${KROLE}
将创建的角色分配给 atlas 组：kafka-sentry -arg -r ${KROLE} -g atlas为消费者分配权限：
TOPIC1="_HOATLASOK"
TOPIC2="ATLAS_ENTITIES"
TOPIC3="ATLAS_HOOK"kafka-sentry -gpr -r ${KROLE} -p "Host=*->CONSUMERGROUP=*->action=read"
kafka-sentry -gpr -r ${KROLE} -p "Host=*->CONSUMERGROUP=*->action=describe"kafka-sentry -gpr -r ${KROLE} -p "HOST=*->TOPIC=${TOPIC1}->action=read"
kafka-sentry -gpr -r ${KROLE} -p "HOST=*->TOPIC=${TOPIC2}->action=read"
kafka-sentry -gpr -r ${KROLE} -p "HOST=*->TOPIC=${TOPIC3}->action=read"
kafka-sentry -gpr -r ${KROLE} -p "HOST=*->TOPIC=${TOPIC1}->action=describe"
kafka-sentry -gpr -r ${KROLE} -p "HOST=*->TOPIC=${TOPIC2}->action=describe"
kafka-sentry -gpr -r ${KROLE} -p "HOST=*->TOPIC=${TOPIC3}->action=describe"
为生产者分配权限
kafka-sentry -gpr -r ${KROLE} -p "HOST=*->TOPIC=${TOPIC1}->action=write"
kafka-sentry -gpr -r ${KROLE} -p "HOST=*->TOPIC=${TOPIC2}->action=write"
kafka-sentry -gpr -r ${KROLE} -p "HOST=*->TOPIC=${TOPIC3}->action=write"
检查sentry设置$ kafka-sentry -lr
....
solradm_role
kafka4atlas_role
显示组及其分配角色的列表：$ kafka-sentry -lg
...
atlas = kafka4atlas_role
test2_solr_admins = solradm_role
显示权限列表：$ kafka-sentry -lp -r kafka4atlas_role
...
HOST=*->TOPIC=_HOATLASOK->action=read
HOST=*->TOPIC=_HOATLASOK->action=describe
HOST=*->TOPIC=ATLAS_HOOK->action=read
HOST=*->TOPIC=ATLAS_ENTITIES->action=describe
HOST=*->TOPIC=ATLAS_HOOK->action=describe
HOST=*->CONSUMERGROUP=*->action=describe
HOST=*->TOPIC=_HOATLASOK->action=write
HOST=*->TOPIC=ATLAS_ENTITIES->action=write
HOST=*->TOPIC=ATLAS_HOOK->action=write
HOST=*->TOPIC=ATLAS_ENTITIES->action=read
HOST=*->CONSUMERGROUP=*->action=read

集成CDH的Solr
①将apache-atlas-2.1.0/conf/solr文件拷贝到solr的安装目录下，即/opt/cloudera/parcels/CDh/lib/solr下，然后更名为atlas-solr

②创建collection

vi /etc/passwd
/sbin/nologin 修改为 /bin/bash
su - solr/opt/cloudera/parcels/CDH/lib/solr/bin/solr create -c  vertex_index -d /opt/cloudera/parcels/CDH/lib/solr/atlas-solr -shards 3 -replicationFactor 2/opt/cloudera/parcels/CDH/lib/solr/bin/solr create -c  edge_index -d /opt/cloudera/parcels/CDH/lib/solr/atlas-solr -shards 3 -replicationFactor 2/opt/cloudera/parcels/CDH/lib/solr/bin/solr create -c  fulltext_index -d /opt/cloudera/parcels/CDH/lib/solr/atlas-solr -shards 3 -replicationFactor 2


③验证创建collection成功
登录 solr web控制台： http://xxxx:8983 验证是否启动成功

创建好相关的kerberos帐号和keytab

修改atlas-application.properties

#########  Graph Database Configs  ########## Graph Database#Configures the graph database to use.  Defaults to JanusGraph
#atlas.graphdb.backend=org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase# Graph Storage
# Set atlas.graph.storage.backend to the correct value for your desired storage
# backend. Possible values:
#
# hbase
# cassandra
# embeddedcassandra - Should only be set by building Atlas with  -Pdist,embedded-cassandra-solr
# berkeleyje
#
# See the configuration documentation for more information about configuring the various  storage backends.
#
atlas.graph.storage.backend=hbase
atlas.graph.storage.hbase.table=apache_atlas_janus#Hbase
#For standalone mode , specify localhost
#for distributed mode, specify zookeeper quorum here
atlas.graph.storage.hostname=S0:2181,S1:2181,S2:2181
atlas.graph.storage.hbase.regions-per-server=1
atlas.graph.stoorage.lock.wait-time=10000#In order to use Cassandra as a backend, comment out the hbase specific properties above, and uncomment the
#the following properties
#atlas.graph.storage.clustername=
#atlas.graph.storage.port=# Gremlin Query Optimizer
#
# Enables rewriting gremlin queries to maximize performance. This flag is provided as
# a possible way to work around any defects that are found in the optimizer until they
# are resolved.
#atlas.query.gremlinOptimizerEnabled=true# Delete handler
#
# This allows the default behavior of doing "soft" deletes to be changed.
#
# Allowed Values:
# org.apache.atlas.repository.store.graph.v1.SoftDeleteHandlerV1 - all deletes are "soft" deletes
# org.apache.atlas.repository.store.graph.v1.HardDeleteHandlerV1 - all deletes are "hard" deletes
#
#atlas.DeleteHandlerV1.impl=org.apache.atlas.repository.store.graph.v1.SoftDeleteHandlerV1# Entity audit repository
#
# This allows the default behavior of logging entity changes to hbase to be changed.
#
# Allowed Values:
# org.apache.atlas.repository.audit.HBaseBasedAuditRepository - log entity changes to hbase
# org.apache.atlas.repository.audit.CassandraBasedAuditRepository - log entity changes to cassandra
# org.apache.atlas.repository.audit.NoopEntityAuditRepository - disable the audit repository
#
atlas.EntityAuditRepository.impl=org.apache.atlas.repository.audit.HBaseBasedAuditRepository# if Cassandra is used as a backend for audit from the above property, uncomment and set the following
# properties appropriately. If using the embedded cassandra profile, these properties can remain
# commented out.
# atlas.EntityAuditRepository.keyspace=atlas_audit
# atlas.EntityAuditRepository.replicationFactor=1# Graph Search Index
atlas.graph.index.search.backend=solr#Solr
#Solr cloud mode properties
atlas.graph.index.search.solr.mode=cloud
atlas.graph.index.search.solr.zookeeper-url=S0:2181/solr,S1:2181/solr,S2:2181/solr
atlas.graph.index.search.solr.zookeeper-connect-timeout=60000
atlas.graph.index.search.solr.zookeeper-session-timeout=60000
atlas.graph.index.search.solr.wait-searcher=true#Solr http mode properties
#atlas.graph.index.search.solr.mode=http
#atlas.graph.index.search.solr.http-urls=http://localhost:8983/solr# ElasticSearch support (Tech Preview)
# Comment out above solr configuration, and uncomment the following two lines. Additionally, make sure the
# hostname field is set to a comma delimited set of elasticsearch master nodes, or an ELB that fronts the masters.
#
# Elasticsearch does not provide authentication out of the box, but does provide an option with the X-Pack product
# https://www.elastic.co/products/x-pack/security
#
# Alternatively, the JanusGraph documentation provides some tips on how to secure Elasticsearch without additional
# plugins: https://docs.janusgraph.org/latest/elasticsearch.html
#atlas.graph.index.search.hostname=localhost
#atlas.graph.index.search.elasticsearch.client-only=true# Solr-specific configuration property
atlas.graph.index.search.max-result-set-size=150#########  Import Configs  #########
#atlas.import.temp.directory=/temp/import#########  Notification Configs  #########
atlas.notification.embedded=false
atlas.kafka.data=${sys:atlas.home}/data/kafka
atlas.kafka.zookeeper.connect=S0:2181,S1:2181,S2:2181
atlas.kafka.bootstrap.servers=S0:9092,S1:9092,S2:9092
atlas.kafka.zookeeper.session.timeout.ms=60000
atlas.kafka.zookeeper.connection.timeout.ms=60000
atlas.kafka.zookeeper.sync.time.ms=20
atlas.kafka.auto.commit.interval.ms=1000
atlas.kafka.hook.group.id=atlasatlas.kafka.enable.auto.commit=false
atlas.kafka.auto.offset.reset=earliest
atlas.kafka.session.timeout.ms=30000
atlas.kafka.offsets.topic.replication.factor=1
atlas.kafka.poll.timeout.ms=1000atlas.notification.create.topics=true
atlas.notification.replicas=1
atlas.notification.topics=ATLAS_HOOK,ATLAS_ENTITIES
atlas.notification.log.failed.messages=true
atlas.notification.consumer.retry.interval=500
atlas.notification.hook.retry.interval=1000
# Enable for Kerberized Kafka clusters
#atlas.notification.kafka.service.principal=kafka/_HOST@EXAMPLE.COM
#atlas.notification.kafka.keytab.location=/etc/security/keytabs/kafka.service.keytab## Server port configuration
atlas.server.http.port=21000
#atlas.server.https.port=21443#########  Security Properties  ########## SSL config
atlas.enableTLS=false#truststore.file=/path/to/truststore.jks
#cert.stores.credential.provider.path=jceks://file/path/to/credentialstore.jceks#following only required for 2-way SSL
#keystore.file=/path/to/keystore.jks# Authentication config
atlas.authentication.method=kerberos
atlas.authentication.keytab=/data/hive.keytab
atlas.authentication.principal=hive@TEST.COMatlas.authentication.method.kerberos=true
atlas.authentication.method.kerberos.principal=hive@TEST.COM
atlas.authentication.method.kerberos.keytab=/data/hive.keytab
atlas.authentication.method.kerberos.name.rules=RULE:[2:$1@$0](hive@TEST.COM)s/.*/hive/
atlas.authentication.method.kerberos.token.validity=3600#atlas.authentication.method.file=true#### ldap.type= LDAP or AD
atlas.authentication.method.ldap.type=none#### user credentials file
atlas.authentication.method.file.filename=${sys:atlas.home}/conf/users-credentials.properties### groups from UGI
#atlas.authentication.method.ldap.ugi-groups=true######## LDAP properties #########
#atlas.authentication.method.ldap.url=ldap://<ldap server url>:389
#atlas.authentication.method.ldap.userDNpattern=uid={0},ou=People,dc=example,dc=com
#atlas.authentication.method.ldap.groupSearchBase=dc=example,dc=com
#atlas.authentication.method.ldap.groupSearchFilter=(member=uid={0},ou=Users,dc=example,dc=com)
#atlas.authentication.method.ldap.groupRoleAttribute=cn
#atlas.authentication.method.ldap.base.dn=dc=example,dc=com
#atlas.authentication.method.ldap.bind.dn=cn=Manager,dc=example,dc=com
#atlas.authentication.method.ldap.bind.password=<password>
#atlas.authentication.method.ldap.referral=ignore
#atlas.authentication.method.ldap.user.searchfilter=(uid={0})
#atlas.authentication.method.ldap.default.role=<default role>######### Active directory properties #######
#atlas.authentication.method.ldap.ad.domain=example.com
#atlas.authentication.method.ldap.ad.url=ldap://<AD server url>:389
#atlas.authentication.method.ldap.ad.base.dn=(sAMAccountName={0})
#atlas.authentication.method.ldap.ad.bind.dn=CN=team,CN=Users,DC=example,DC=com
#atlas.authentication.method.ldap.ad.bind.password=<password>
#atlas.authentication.method.ldap.ad.referral=ignore
#atlas.authentication.method.ldap.ad.user.searchfilter=(sAMAccountName={0})
#atlas.authentication.method.ldap.ad.default.role=<default role>#########  JAAS Configuration ########atlas.jaas.KafkaClient.loginModuleName=com.sun.security.auth.module.Krb5LoginModule
atlas.jaas.KafkaClient.loginModuleControlFlag=required
atlas.jaas.KafkaClient.option.useKeyTab=true
atlas.jaas.KafkaClient.option.storeKey=true
atlas.jaas.KafkaClient.option.serviceName=kafka
atlas.jaas.KafkaClient.option.keyTab=/data/atlas.service.keytab
atlas.jaas.KafkaClient.option.principal=atlas/s1.hadoop.com@TEST.COMatlas.jaas.Client.loginModuleName=com.sun.security.auth.module.Krb5LoginModule
atlas.jaas.Client.loginModuleControlFlag=required
atlas.jaas.Client.option.useKeyTab=true
atlas.jaas.Client.option.storeKey=true
atlas.jaas.Client.option.keyTab=/data/atlas.service.keytab
atlas.jaas.Client.option.principal=atlas/s1.hadoop.com@TEST.COM#########  Server Properties  #########
atlas.rest.address=http://localhost:21000
# If enabled and set to true, this will run setup steps when the server starts
#atlas.server.run.setup.on.start=false#########  Entity Audit Configs  #########
atlas.audit.hbase.tablename=apache_atlas_entity_audit
atlas.audit.zookeeper.session.timeout.ms=1000
atlas.audit.hbase.zookeeper.quorum=S0:2181,S1:2181,S2:2181#########  High Availability Configuration ########
atlas.server.ha.enabled=false
#### Enabled the configs below as per need if HA is enabled #####
#atlas.server.ids=id1
#atlas.server.address.id1=localhost:21000
#atlas.server.ha.zookeeper.connect=localhost:2181
#atlas.server.ha.zookeeper.retry.sleeptime.ms=1000
#atlas.server.ha.zookeeper.num.retries=3
#atlas.server.ha.zookeeper.session.timeout.ms=20000
## if ACLs need to be set on the created nodes, uncomment these lines and set the values ##
#atlas.server.ha.zookeeper.acl=<scheme>:<id>
#atlas.server.ha.zookeeper.auth=<scheme>:<authinfo>######### Atlas Authorization #########
atlas.authorizer.impl=simple
atlas.authorizer.simple.authz.policy.file=atlas-simple-authz-policy.json#########  Type Cache Implementation ########
# A type cache class which implements
# org.apache.atlas.typesystem.types.cache.TypeCache.
# The default implementation is org.apache.atlas.typesystem.types.cache.DefaultTypeCache which is a local in-memory type cache.
#atlas.TypeCache.impl=#########  Performance Configs  #########
#atlas.graph.storage.lock.retries=10
#atlas.graph.storage.cache.db-cache-time=120000#########  CSRF Configs  #########
atlas.rest-csrf.enabled=true
atlas.rest-csrf.browser-useragents-regex=^Mozilla.*,^Opera.*,^Chrome.*
atlas.rest-csrf.methods-to-ignore=GET,OPTIONS,HEAD,TRACE
atlas.rest-csrf.custom-header=X-XSRF-HEADER############ KNOX Configs ################
#atlas.sso.knox.browser.useragent=Mozilla,Chrome,Opera
#atlas.sso.knox.enabled=true
#atlas.sso.knox.providerurl=https://<knox gateway ip>:8443/gateway/knoxsso/api/v1/websso
#atlas.sso.knox.publicKey=############ Atlas Metric/Stats configs ################
# Format: atlas.metric.query.<key>.<name>
atlas.metric.query.cache.ttlInSecs=900
#atlas.metric.query.general.typeCount=
#atlas.metric.query.general.typeUnusedCount=
#atlas.metric.query.general.entityCount=
#atlas.metric.query.general.tagCount=
#atlas.metric.query.general.entityDeleted=
#
#atlas.metric.query.entity.typeEntities=
#atlas.metric.query.entity.entityTagged=
#
#atlas.metric.query.tags.entityTags=#########  Compiled Query Cache Configuration  ########## The size of the compiled query cache.  Older queries will be evicted from the cache
# when we reach the capacity.#atlas.CompiledQueryCache.capacity=1000# Allows notifications when items are evicted from the compiled query
# cache because it has become full.  A warning will be issued when
# the specified number of evictions have occurred.  If the eviction
# warning threshold <= 0, no eviction warnings will be issued.#atlas.CompiledQueryCache.evictionWarningThrottle=0#########  Full Text Search Configuration  ##########Set to false to disable full text search.
#atlas.search.fulltext.enable=true#########  Gremlin Search Configuration  ##########Set to false to disable gremlin search.
atlas.search.gremlin.enable=false########## Add http headers ############atlas.headers.Access-Control-Allow-Origin=*
#atlas.headers.Access-Control-Allow-Methods=GET,OPTIONS,HEAD,PUT,POST
#atlas.headers.<headerName>=<headerValue>#########  UI Configuration ########atlas.ui.default.version=v1

要改的配置很多。。务必仔细核对。很多默认配置都是有问题的,keytab 新建或者复用都可以，担心可能会涉及到权限问题所以我选择了hive的账户。hbase中应该也需要配置相应的权限。没测试过是否需要配置

修改atlas-env.sh

#!/usr/bin/env bash# The java implementation to use. If JAVA_HOME is not found we expect java and jar to be in path
export JAVA_HOME=/usr/java/default
export HBASE_CONF_DIR=/etc/hbase/conf
# any additional java opts you want to set. This will apply to both client and server operations
#export ATLAS_OPTS=# any additional java opts that you want to set for client only
#export ATLAS_CLIENT_OPTS=# java heap size we want to set for the client. Default is 1024MB
#export ATLAS_CLIENT_HEAP=# any additional opts you want to set for atlas service.
#export ATLAS_SERVER_OPTS=# indicative values for large number of metadata entities (equal or more than 10,000s)
export ATLAS_SERVER_OPTS="-server -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+PrintTenuringDistribution -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dumps/atlas_server.hprof -Xloggc:logs/gc-worker.log -verbose:gc -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1m -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps -Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/data/atlas2.2/conf/jaas.conf"# java heap size we want to set for the atlas server. Default is 1024MB
#export ATLAS_SERVER_HEAP=# indicative values for large number of metadata entities (equal or more than 10,000s) for JDK 8
export ATLAS_SERVER_HEAP="-Xms15360m -Xmx15360m -XX:MaxNewSize=5120m -XX:MetaspaceSize=100M -XX:MaxMetaspaceSize=512m"# What is is considered as atlas home dir. Default is the base locaion of the installed software
export ATLAS_HOME_DIR=/opt/atlas2.2# Where log files are stored. Defatult is logs directory under the base install location
#export ATLAS_LOG_DIR=# Where pid files are stored. Defatult is logs directory under the base install location
#export ATLAS_PID_DIR=# where the atlas titan db data is stored. Defatult is logs/data directory under the base install location
#export ATLAS_DATA_DIR=# Where do you want to expand the war file. By Default it is in /server/webapp dir under the base install dir.
#export ATLAS_EXPANDED_WEBAPP_DIR=# indicates whether or not a local instance of HBase should be started for Atlas
export MANAGE_LOCAL_HBASE=false# indicates whether or not a local instance of Solr should be started for Atlas
export MANAGE_LOCAL_SOLR=false# indicates whether or not cassandra is the embedded backend for Atlas
export MANAGE_EMBEDDED_CASSANDRA=false# indicates whether or not a local instance of Elasticsearch should be started for Atlas
export MANAGE_LOCAL_ELASTICSEARCH=false

 env中的jaas.conf 需要增加一个jaas.conf

Client {com.sun.security.auth.module.Krb5LoginModule requireduseKeyTab=trueKeyTab="/data/atlas.service.keytab"storeKey=trueprincipal="atlas/s1.hadoop.com@TEST.COM"debug=false;
};

 集成hive

首先去CDH的hive上添加3处配置

HiveServer2 的 Java 配置选项   {{JAVA_GC_ARGS}} -Datlas.conf=/data/apache-atlas-2.2.0/conf/

 

 hive-site.xml的HiveServer2 高级配置代码段 （安全阀）

名称： hive.exec.post.hooks
值： org.apache.atlas.hive.hook.HiveHook

HiveServer2 环境高级配置片段（安全阀）

HIVE_AUX_JARS_PATH=/data/apache-atlas-2.2.0/hook/hive/

 复制一份atlas-application.properties到/etc/hive/conf下 。注意需要修改

改为false
atlas.authentication.method.kerberos=false
增加
atlas.client.readTimeoutMSecs=90000
atlas.client.connectTimeoutMSecs=90000

最后两个配置的含义是读取连接时间，，默认的太短

然后就可以启动了

bin/atlas-start.pybin/atlas-stop.py

启动过程如下图所示

Atlas2.2.0编译、安装及使用(集成ElasticSearch，导入Hive数据)

该过程会耗时较久，包含index创建、数据的初始化等操作，可能长达数小时，请耐心等待。
此时可以跟一下atlas的启动日志，直到日志不再刷新，再lsof或netstat查一下21000是否已经监听了，如已存在，则打开浏览器输入ip:21000登录atlas页面

千万不要相信他提示的Apache Atlas Server started!!!和jps显示的Atlas进程 ，因为启动脚本超过一定时间后一定会报成功，但此时21000端口还未被监听，服务是不可用的，真正可用还是以21000被成功监听，可以进到Atlas登录页面为准

然后开始正式使用

导入hive数据

记得kinit 
bin/import-hive.sh 
也可以单独导入某个库
bin/import-hive.sh -d default

 过程中会提示输入atlas用户名和密码，都输入admin即可
成功后会提示

Atlas2.2.0编译、安装及使用(集成ElasticSearch，导入Hive数据)

该过程时间视hive现有数据量大小而定

登录后如下图

Atlas2.2.0编译、安装及使用(集成ElasticSearch，导入Hive数据)

此时可以点击右上角小图标

Atlas2.2.0编译、安装及使用(集成ElasticSearch，导入Hive数据)

查看总体数据情况
 

Atlas2.2.0编译、安装及使用(集成ElasticSearch，导入Hive数据)


 

查看所有hive表
 

Atlas2.2.0编译、安装及使用(集成ElasticSearch，导入Hive数据)


 

随便点击一个表查看详情

Atlas2.2.0编译、安装及使用(集成ElasticSearch，导入Hive数据)

Atlas2.2.0编译、安装及使用(集成ElasticSearch，导入Hive数据)

Atlas2.2.0编译、安装及使用(集成ElasticSearch，导入Hive数据)

可以清楚地看到这个表的各项信息、字段及血缘图等
 
我们也可以通过左侧搜索栏检索过滤想要查找的项

Atlas2.2.0编译、安装及使用(集成ElasticSearch，导入Hive数据)

本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若转载，请注明出处：http://www.mzph.cn/news/108642.shtml

如若内容造成侵权/违法违规/事实不符，请联系多彩编程网进行投诉反馈email:809451989@qq.com，一经查实，立即删除！

相关文章

Day2力扣打卡

Day2力扣打卡

打卡记录 无限数组的最短子数组&#xff08;滑动窗口&#xff09; 链接 思路&#xff1a;先求单个数组的总和&#xff0c;再对两个重复数组所组成的新数组上使用 不定长的滑动窗口 来求得满足目标的最小长度。 class Solution { public:int minSizeSubarray(vector<int>…
阅读更多...
项目经理每天，每周，每月的工作清单

项目经理每天，每周，每月的工作清单

很多不懂项目管理的伙伴问&#xff0c;项目经理每天每周每个月的工作是什么呢&#xff1f; 仿佛他们什么都管&#xff0c;但是又没有具体的产出&#xff0c;但是每天看他们比谁都忙&#xff0c;其实很简单&#xff0c;项目中的每个环节负责具体的事情&#xff0c;但是每个环节…
阅读更多...
Python---if选择判断结构、嵌套结构（if elif else）

Python---if选择判断结构、嵌套结构（if elif else）

1、if选择判断结构作用 if 英 /ɪf/ conj. &#xff08;表条件&#xff09;如果&#xff1b;&#xff08;表假设&#xff09;要是&#xff0c;假如&#xff1b;无论何时&#xff1b;虽然&#xff0c;即使&#xff1b;&#xff08;用于间接疑问&#xff09;是否&#xff1b…
阅读更多...
uniapp-vue3-微信小程序-标签选择器wo-tag

uniapp-vue3-微信小程序-标签选择器wo-tag

采用uniapp-vue3实现, 是一款支持高度自定义的标签选择器组件&#xff0c;支持H5、微信小程序&#xff08;其他小程序未测试过&#xff0c;可自行尝试&#xff09; 可到插件市场下载尝试&#xff1a; https://ext.dcloud.net.cn/plugin?id14960 使用示例 <template>&…
阅读更多...
System.exit()方法参数

System.exit()方法参数

说明文档&#xff1a;System (Java Platform SE 8 ) 终止当前正在运行的Java虚拟机。该参数用作状态代码&#xff1b;按照惯例&#xff0c;非零状态码表示异常终止。 此方法调用类Runtime中的exit方法。此方法从不正常返回。 调用System.exit&#xff08;n&#xff09;实际上等…
阅读更多...
Android MediaCodec将h264实时视频流数据解码为yuv，并转换yuv的颜色格式为nv21

Android MediaCodec将h264实时视频流数据解码为yuv，并转换yuv的颜色格式为nv21

初始化mediacodec //宽高根据摄像头分辨率设置private int Width 1280;private int Height 720;private MediaCodec mediaCodec;private ByteBuffer[] inputBuffers;private void initMediaCodec(Surface surface) {try {Log.d(TAG, "onGetNetVideoData: ");//创建…
阅读更多...
牛客：FZ12 牛牛的顺时针遍历

牛客：FZ12 牛牛的顺时针遍历

FZ12 牛牛的顺时针遍历 文章目录 FZ12 牛牛的顺时针遍历题目描述题解思路题解代码 题目描述 题解思路 通过一个变量来记录当前方向&#xff0c;遍历矩阵&#xff0c;每次遍历一条边&#xff0c;将该边的信息加入到结果中 题解代码 func spiralOrder(matrix [][]int) []int {…
阅读更多...
Docker安装ES7.14和Kibana7.14(无账号密码)

Docker安装ES7.14和Kibana7.14(无账号密码)

一、Docker安装ES7.14.0 1、下载镜像 docker pull elasticsearch:7.14.0 2、docker安装7.14.0 mkdir -p /usr/local/elasticsearch/config mkdir -p /usr/local/elasticsearch/data chmod 777 -R /usr/local/elasticsearch/ echo "http.host: 0.0.0.0" >> /u…
阅读更多...
IDEA中点击New没有Java Class

IDEA中点击New没有Java Class

解决办法&#xff1a;右键src&#xff0c;也可以是其他文件名&#xff0c;点击Mark Directory as 点击Sources Root即可
阅读更多...
【Java 进阶篇】JavaScript DOM Element 对象详解

【Java 进阶篇】JavaScript DOM Element 对象详解

JavaScript是一门广泛用于网页开发的脚本语言&#xff0c;而DOM&#xff08;文档对象模型&#xff09;是JavaScript在网页中操作HTML和XML文档的核心。DOM以树状结构表示文档&#xff0c;允许开发者以编程方式访问、操作和修改文档的内容和结构。在DOM中&#xff0c;Element对象…
阅读更多...
海洋CMS仿爱美剧影视电影视频网站模版源码/自适应手机端

海洋CMS仿爱美剧影视电影视频网站模版源码/自适应手机端

海洋CMS仿爱美剧网站模板&#xff0c;自适应手机端&#xff0c;内含视频、资讯、留言模块。 下载地址&#xff1a;https://bbs.csdn.net/topics/617419787
阅读更多...
Postman简单使用

Postman简单使用

文章目录 一.接口测试流程二、Postman接口测试工具三、接口关联四、全局变量和环境变量 一.接口测试流程 拿到API接口文档&#xff08;从开发拿或者抓包获取&#xff09;&#xff0c;熟悉接口业务&#xff0c;接口地址&#xff0c;错误码等等 编写接口的测试用例以及评审 编写…
阅读更多...
docker中使用GPU+rocksdb

docker中使用GPU+rocksdb

配置环境 delldell-Precision-3630-Tower  ~  lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.6 LTS Release: 20.04 Codename: focaldelldell-Precision-3630-Tower  ~  nvcc --version nvcc: NVIDIA (R) Cuda comp…
阅读更多...
【数据结构】排序--快速排序

【数据结构】排序--快速排序

目录 一 概念 二 快速排序的实现 1. hoare版本 (1)代码实现 (2)单趟排序图解 (3) 递归实现图解 (4)细节控制 (5)时间复杂度 (6)三数取中优化 2 挖坑法 (1)代码实现 (2)单趟图解 3 前后指针法 (1) 代码实现 (2) 单趟图解 ​编辑4 优化子区间 5 非递归快速排…
阅读更多...
FPGA project : flash_write

FPGA project : flash_write

本实验重点学习了&#xff1a; flash的页编程指令pp。 在写之前要先进行擦除&#xff08;全擦除和页擦除&#xff09;&#xff1b; 本实验&#xff1a;先传写指令&#xff0c;然后进入写锁存周期&#xff0c;然后传页编程指令&#xff0c;3个地址&#xff1b; 然后传数据&a…
阅读更多...
【Java基础面试十五】、 说一说你对多态的理解

【Java基础面试十五】、 说一说你对多态的理解

文章底部有个人公众号&#xff1a;热爱技术的小郑。主要分享开发知识、学习资料、毕业设计指导等。有兴趣的可以关注一下。为何分享&#xff1f; 踩过的坑没必要让别人在再踩&#xff0c;自己复盘也能加深记忆。利己利人、所谓双赢。 面试官&#xff1a;说一说你对多态的理解 …
阅读更多...
常见6种易被忽略的软件隐藏缺陷

常见6种易被忽略的软件隐藏缺陷

软件隐藏缺陷常常会被测试人员忽略或遗漏&#xff0c;其往往会对项目的成功和用户体验产生不可忽视的负面影响&#xff0c;易造成软件数据泄露、系统崩溃或安全问题等&#xff0c;直接影响系统稳定性和用户满意度。 因此我们需要高度重视软件的隐藏缺陷&#xff0c;重视全面的软…
阅读更多...
开源项目汇总

开源项目汇总

element-plus 人人开源 人人开源 多租户 若依 jeecg https://gitee.com/jeecg/jeecg?_fromgitee_search#https://gitee.com/link?targethttp%3A%2F%2Fidoc.jeecg.com jeeplus JeePlus快速开发平台 j2eefast Sa-Plus
阅读更多...
“Flex弹性布局、轮播图mock遍历数据和首页布局解析与实践“

“Flex弹性布局、轮播图mock遍历数据和首页布局解析与实践“

目录 引言1. Flex弹性布局介绍及使用什么是Flex弹性布局&#xff1f;Flex容器与Flex项目Flex属性详解 2. 轮播图mock遍历数据简述轮播图的作用和意义处理mock数据的重要性使用Mock模拟数据遍历 3. 首页布局总结 引言 在现代网页开发中&#xff0c;灵活性和响应式布局是至关重要…
阅读更多...
【C++】--遇到抛异常没有及时释放的空间该怎么办？？---智能指针来帮你解决（以及定制删除器）

【C++】--遇到抛异常没有及时释放的空间该怎么办？？---智能指针来帮你解决（以及定制删除器）

&#x1f496;作者&#xff1a;小树苗渴望变成参天大树&#x1f388; &#x1f389;作者宣言&#xff1a;认真写好每一篇博客&#x1f4a4; &#x1f38a;作者gitee:gitee✨ &#x1f49e;作者专栏&#xff1a;C语言,数据结构初阶,Linux,C 动态规划算法&#x1f384; 如 果 你 …
阅读更多...
推荐文章
最新文章

Copyright @ 2022~2023