异常:在hue上配置的一些定时任务突然停止执行。
1.打开页面HUE->WorkFlow 发现页面异常,无法进入WorkFlow,如下图
2.查看HUE日志
查看到WebHdfsException异常,访问HDFS文件浏览器报错,
[26/Jun/2019 09:29:55 +0800] exception ERROR Internal Server Error: /oozie/editor/workflow/new/
Traceback (most recent call last):File "/opt/cloudera/parcels/CDH-6.0.0-1.cdh6.0.0.p0.537114/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11-py2.7.egg/django/core/handlers/exception.py", line 41, in innerresponse = get_response(request)File "/opt/cloudera/parcels/CDH-6.0.0-1.cdh6.0.0.p0.537114/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11-py2.7.egg/django/core/handlers/base.py", line 249, in _legacy_get_responseresponse = self._get_response(request)File "/opt/cloudera/parcels/CDH-6.0.0-1.cdh6.0.0.p0.537114/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11-py2.7.egg/django/core/handlers/base.py", line 187, in _get_responseresponse = self.process_exception_by_middleware(e, request)File "/opt/cloudera/parcels/CDH-6.0.0-1.cdh6.0.0.p0.537114/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11-py2.7.egg/django/core/handlers/base.py", line 185, in _get_responseresponse = wrapped_callback(request, *callback_args, **callback_kwargs)File "/opt/cloudera/parcels/CDH-6.0.0-1.cdh6.0.0.p0.537114/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11-py2.7.egg/django/utils/decorators.py", line 185, in innerreturn func(*args, **kwargs)File "/opt/cloudera/parcels/CDH-6.0.0-1.cdh6.0.0.p0.537114/lib/hue/apps/oozie/src/oozie/decorators.py", line 113, in decoratereturn view_func(request, *args, **kwargs)File "/opt/cloudera/parcels/CDH-6.0.0-1.cdh6.0.0.p0.537114/lib/hue/apps/oozie/src/oozie/views/editor2.py", line 130, in new_workflowworkflow.check_workspace(request.fs, request.user)File "/opt/cloudera/parcels/CDH-6.0.0-1.cdh6.0.0.p0.537114/lib/hue/apps/oozie/src/oozie/models2.py", line 94, in check_workspacecreate_directories(fs, [REMOTE_SAMPLE_DIR.get()])File "/opt/cloudera/parcels/CDH-6.0.0-1.cdh6.0.0.p0.537114/lib/hue/desktop/libs/liboozie/src/liboozie/submission2.py", line 520, in create_directoriesif not fs.do_as_user(fs.DEFAULT_USER, fs.exists, directory):File "/opt/cloudera/parcels/CDH-6.0.0-1.cdh6.0.0.p0.537114/lib/hue/desktop/core/src/desktop/lib/fs/proxyfs.py", line 102, in do_as_userreturn fn(*args, **kwargs)File "/opt/cloudera/parcels/CDH-6.0.0-1.cdh6.0.0.p0.537114/lib/hue/desktop/core/src/desktop/lib/fs/proxyfs.py", line 136, in existsreturn self._get_fs(path).exists(path)File "/opt/cloudera/parcels/CDH-6.0.0-1.cdh6.0.0.p0.537114/lib/hue/desktop/libs/hadoop/src/hadoop/fs/webhdfs.py", line 297, in existsreturn self._stats(path) is not NoneFile "/opt/cloudera/parcels/CDH-6.0.0-1.cdh6.0.0.p0.537114/lib/hue/desktop/libs/hadoop/src/hadoop/fs/webhdfs.py", line 285, in _statsraise ex
WebHdfsException: 403 Client Error: Forbidden for url: http://host009:9870/webhdfs/v1/user/hue/oozie/deployments/_%24USER_-oozie-%24JOBID-%24TIME?op=GETFILESTATUS&user.name=hue&doas=hue
{"RemoteException":{"exception":"StandbyException","javaClassName":"org.apache.hadoop.ipc.StandbyException","message":"Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error"}} (error 403)
[26/Jun/2019 09:29:55 +0800] access ERROR 10.248.111.1 sss - "POST /desktop/log_js_error HTTP/1.1"-- JS ERROR: {"msg":"TypeError: _super is undefined","url":"http://10.248.111.111:8888/static/desktop/js/popover-extra-placements.b89dd43e6070.js","line":14,"column":3,"stack":"@http://10.248.111.111:8888/static/desktop/js/popover-extra-placements.b89dd43e6070.js:14:3\n@http://10.248.111.111:8888/static/desktop/js/popover-extra-placements.b89dd43e6070.js:113:3\n"}
发现到WebHdfsException异常,访问HDFS文件浏览器报错,异常中 host009:9870 就是我备用NameNode的 Web UI端口
3.异常解决方案1:
我有两个NameNode,可能是之前一个NameNode异常了,HUE切换到另外一个NameNode进行访问,但是拒绝访问,重启NameNode后再次访问HUE->WorkFlow恢复正常。
3.异常解决方案2:
在HUE中修改连接的NameNode,选择现在活动的那个Namenode