kafka-ranger

ranger-1.0.0 kafka-1.0.0(confluent-4.0.0)

安装ranger-1.0.0-kafka-plugin

下面是安装过程中遇到的一些问题

下载并解压 ranger-1.0.0-kafka-plugin.tar.gz

修改配置文件install.properties

COMPONENT_INSTALL_DIR_NAME=/usr/local/confluent/

POLICY_MGR_URL=http://192.168.206.144:6080

REPOSITORY_NAME=kafkadev

CUSTOM_USER=kafka

CUSTOM_GROUP=hadoop

增加kafka的configs和libs的软连接

ln -s /usr/local/confluent/etc/kafka /usr/local/confluent/config

ln -s /usr/local/confluent/share/java/kafka /usr/local/confluent/libs

把kafka的配置文件目录加到CLASSPATH

reason: the program will only load server.properties when kafka starting, so we need to do this that program could find the configuration files of ranger-kafka.

export CLASSPATH=/usr/local/confluent/etc/kafka

ERROR: Server not found in Kerberos database

one reason: the kafka-host must be in advertised.listeners

Server not found in Kerberos database

[2018-07-05 15:48:03,763] DEBUG Accepted connection from /172.17.0.15:38950 on /172.17.0.15:9093 and assigned it to processor 0, sendBu

fferSize [actual|requested]: [102400|102400] recvBufferSize [actual|requested]: [102400|102400] (kafka.network.Acceptor)

[2018-07-05 15:48:03,770] DEBUG Processor 0 listening to new connection from /172.17.0.15:38950 (kafka.network.Processor)

[2018-07-05 15:48:03,771] DEBUG Set SASL client state to SEND_APIVERSIONS_REQUEST (org.apache.kafka.common.security.authenticator.SaslC

lientAuthenticator)

[2018-07-05 15:48:03,774] DEBUG Creating SaslClient: client=kafka/master.mesos@LINKTIME.CLOUD;service=kafka;serviceHostname=e318e3a9e22

c;mechs=[GSSAPI] (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)

[2018-07-05 15:48:03,783] DEBUG [Controller id=2, targetBrokerId=2] Created socket with SO_RCVBUF = 530904, SO_SNDBUF = 1313280, SO_TIM

EOUT = 0 to node 2 (org.apache.kafka.common.network.Selector)

[2018-07-05 15:48:03,796] DEBUG Set SASL client state to RECEIVE_APIVERSIONS_RESPONSE (org.apache.kafka.common.security.authenticator.S

aslClientAuthenticator)

[2018-07-05 15:48:03,798] DEBUG [Controller id=2, targetBrokerId=2] Completed connection to node 2. Ready. (org.apache.kafka.clients.Ne

tworkClient)

[2018-07-05 15:48:03,803] DEBUG Set SASL server state to HANDSHAKE_OR_VERSIONS_REQUEST (org.apache.kafka.common.security.authenticator.

SaslServerAuthenticator)

[2018-07-05 15:48:03,803] DEBUG Handling Kafka request API_VERSIONS (org.apache.kafka.common.security.authenticator.SaslServerAuthentic

ator)

[2018-07-05 15:48:03,816] DEBUG Set SASL server state to HANDSHAKE_REQUEST (org.apache.kafka.common.security.authenticator.SaslServerAu

thenticator)

[2018-07-05 15:48:03,827] DEBUG Set SASL client state to SEND_HANDSHAKE_REQUEST (org.apache.kafka.common.security.authenticator.SaslCli

entAuthenticator)

[2018-07-05 15:48:03,829] DEBUG Set SASL client state to RECEIVE_HANDSHAKE_RESPONSE (org.apache.kafka.common.security.authenticator.Sas

lClientAuthenticator)

[2018-07-05 15:48:03,829] DEBUG Handling Kafka request SASL_HANDSHAKE (org.apache.kafka.common.security.authenticator.SaslServerAuthent

icator)

[2018-07-05 15:48:03,830] DEBUG Using SASL mechanism 'GSSAPI' provided by client (org.apache.kafka.common.security.authenticator.SaslSe

rverAuthenticator)

[2018-07-05 15:48:03,831] DEBUG Set SASL client state to INITIAL (org.apache.kafka.common.security.authenticator.SaslClientAuthenticato

r)

[2018-07-05 15:48:03,835] DEBUG Creating SaslServer for kafka/master.mesos@LINKTIME.CLOUD with mechanism GSSAPI (org.apache.kafka.commo

n.security.authenticator.SaslServerAuthenticator)

[2018-07-05 15:48:03,847] DEBUG Set SASL server state to AUTHENTICATE (org.apache.kafka.common.security.authenticator.SaslServerAuthent

icator)

[2018-07-05 15:48:03,869] DEBUG [Controller id=2, targetBrokerId=2] Connection with e318e3a9e22c/172.17.0.15 disconnected due to authen

tication exception (org.apache.kafka.common.network.Selector)

org.apache.kafka.common.errors.SaslAuthenticationException: An error: (java.security.PrivilegedActionException: javax.security.sasl.Sas

lException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos d

atabase (7) - LOOKING_UP_SERVER)]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.

Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]

schema-registry restart fail

[2018-07-06 04:01:58,149] INFO Shutting down schema registry (io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry:719)

[2018-07-06 04:01:58,152] ERROR Server died unexpectedly: (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain:51)

java.lang.NullPointerException

at io.confluent.kafka.schemaregistry.storage.KafkaStore.close(KafkaStore.java:366)

at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.close(KafkaSchemaRegistry.java:720)

at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.onShutdown(SchemaRegistryRestApplication.java:111)

at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:66)

at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:42)

at io.confluent.rest.Application.createServer(Application.java:157)

at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:43)

kafka error log:

...

[2018-07-06 04:01:58,070] ERROR Unsupported access type. operation=DescribeConfigs (org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer)

[2018-07-06 04:01:58,070] FATAL Unsupported access type. session=Session(User:schemaRegistry,/172.17.0.1), operation=DescribeConfigs, resource=Topic:__schemas (org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer)

[2018-07-06 04:01:58,070] ERROR Unsupported access type. operation=DescribeConfigs, request=RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={null} elements={topic=__schemas; } }} accessType={_any} user={schemaRegistry} userGroups={} accessTime={Fri Jul 06 04:01:58 CST 2018} clientIPAddress={172.17.0.1} forwardedAddresses={} remoteIPAddress={null} clientType={null} action={null} requestData={__schemas} sessionId={null} resourceMatchingScope={SELF} clusterName={} context={} } (org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer)

...

notice

Must add all permissions of topics(*) for user kafka that is the same as sasl.kerberos.service.name.