526互联

Flink1.13.6 部署踩坑记录

发布时间 2023-06-15 18:14:13作者: 超级无敌小剑

环境

   Hadoop集群是Ambari2.7.5的版本 

   Flink是1.13.6_2.12的版本

问题记录

    1.缺少jar包

报错：ERROR org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Error while running the Flink session.
java.lang.NoClassDefFoundError: com/sun/jersey/core/util/FeaturesAndProperties
        at java.lang.ClassLoader.defineClass1(Native Method) ~[?:1.8.0_101]
        at java.lang.ClassLoader.defineClass(ClassLoader.java:763) ~[?:1.8.0_101]
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) ~[?:1.8.0_101]
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) ~[?:1.8.0_101]

缺少 jersey-server-1.9.jar
     jersey-core-1.9.jar
     jersey-server-1.9.jar
这三个jar包,下载这个包放到Flink/lib/目录下,其他节点也需要

  2. java.lang.NoClassDefFoundError: javax/ws/rs/ext/MessageBodyReader
      Caused by: java.lang.ClassNotFoundException: javax.ws.rs.ext.MessageBodyReader
     缺少 javax.ws.rs-api-2.0.jar 这个jar包 下载这个包放到Flink/lib/目录下,其他节点也需要

 3.配置项问题

Exception in thread "Thread-5" java.lang.IllegalStateException: Trying to access closed classloader. Please check if you store classloaders directly or indirectly in static fields. If the stacktrace suggests that the leak occurs in a third party library and cannot be fixed immediately, you can disable this check with the configuration 'classloader.check-leaked-classloader'.

解决办法：
在flink-conf.yaml中添加
classloader.check-leaked-classloader: false

 4. 内存问题

org.apache.flink.configuration.IllegalConfigurationException: TaskManager memory configuration failed: Derived JVM Overhead size (1.750gb (1879048192 bytes)) is not in configured JVM Overhead range [192.000mb (201326592 bytes), 1024.000mb (1073741824 bytes)].
at org.apache.flink.runtime.clusterframework.TaskExecutorProcessUtils.processSpecFromConfig(TaskExecutorProcessUtils.java:163)
或者	
The configured Total Process Memory size (2.000gb (2147483648 bytes)) is less than the sum of the derived Total Flink Memory size (8.000gb (8589934592 bytes)) and the configured or default JVM Metaspace size  (256.000mb (268435456 bytes)).	
这种错误。

解决办法：检查自己的flink-conf.yaml这个配置文件中
jobmanager.memory.process.size
taskmanager.memory.flink.size
这两个参数和自己启动yarn-session的-jm -tm这两个参数的大小关系,配置文件中的总内存大小需要大于启动参数的 -tm大小+256M

注：
配置项  	  TaskManager 配置参数  	  JobManager 配置参数  
Flink 总内存 taskmanager.memory.flink.size	jobmanager.memory.flink.size
进程总内存	 taskmanager.memory.process.size	jobmanager.memory.process.size

这两个内存配置存在冲突,配置文件建议不要一起配置