526互联

flume常见问题

发布时间 2023-12-19 21:52:39作者: DogLeftover
  • 报错1：执行到如下就不执行了，没有显示Successfully registered new MBean.
[root@slave1 bin]# /usr/local/software/flume-1.9.0/bin/flume-ng agent -n a1 -c /usr/local/software/flume-1.9.0/conf/ -f file-flume-kafka.conf -Dflume.root.logger=INFO，console
Info: Including Hadoop libraries found via (/usr/local/software/hadoop-2.9.2/bin/hadoop) for HDFS access
Info: Including HBASE libraries found via (/usr/local/software/hbase-2.1.0/bin/hbase) for HBASE access
Info: Including Hive libraries found via (/usr/local/software/hive-2.3.9) for Hive access
+ exec /usr/local/software/jdk1.8.0_181/bin/java -Xmx20m -Dflume.root.logger=INFO，console -cp '/usr/local/software/flume-2.9.2/contrib/capacity-scheduler/*.jar:/usr/local/software/hbase-2.1.0/conf:/usr/local/software/hive-2.3.9/lib/*' -Djava.library.path=:/usr/local/software/hadoop-2.9.2/lib/native:/usr/local/software/hadoop-2.9.2/lib/native org.apache.flume.node.Application -n a1 -f file-flume-kafka.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/software/flume-1.9.0/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/software/hadoop-2.9.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/software/hive-2.3.9/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
log4j:WARN No appenders could be found for logger (org.apache.flume.util.SSLUtil).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
  • 解决方案
# 执行的命令中包含中文逗号，改为英文逗号后尝试
/usr/local/software/flume-1.9.0/bin/flume-ng agent -n a1 -c /usr/local/software/flume-1.9.0/conf/ -f file-flume-kafka.conf -Dflume.root.logger=INFO,console

# 成功，显示如下
2023-12-20 05:20:23,694 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: SOURCE, name: r1: Successfully registered new MBean.
2023-12-20 05:20:23,694 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: SOURCE, name: r1 started
  • 报错2：Unable to load channel type: org.apache.flume.channel.kafka.Kafkachannel
org.apache.flume.FlumeException: Unable to load channel type: org.apache.flume.channel.kafka.Kafkachannel, class: org.apache.flume.channel.kafka.Kafkachannel
	at org.apache.flume.channel.DefaultChannelFactory.getClass(DefaultChannelFactory.java:68)
	at org.apache.flume.node.AbstractConfigurationProvider.getOrCreateChannel(AbstractConfigurationProvider.java:236)
	at org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:200)
	at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:104)
	at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:145)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.flume.channel.kafka.Kafkachannel
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at org.apache.flume.channel.DefaultChannelFactory.getClass(DefaultChannelFactory.java:66)
	... 11 more
  • 解决方案
# 没有找到
a1.channels.c1.type = org.apache.flume.channel.kafka.Kafkachannel
# 仔细对比后，发现应该改为如下
a1.channels.c1.type = org.apache.flume.channel.kafka.KafkaChannel
  • 错误3：Interceptor.Type not specified for i1
2023-12-20 05:15:14,436 (conf-file-poller-0) [ERROR - org.apache.flume.channel.ChannelProcessor.configureInterceptors(ChannelProcessor.java:106)] Type not specified for interceptor i1
2023-12-20 05:15:14,436 (conf-file-poller-0) [ERROR - org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:355)] Source r1 has been removed due to an error during configuration
org.apache.flume.FlumeException: Interceptor.Type not specified for i1
	at org.apache.flume.channel.ChannelProcessor.configureInterceptors(ChannelProcessor.java:107)
	at org.apache.flume.channel.ChannelProcessor.configure(ChannelProcessor.java:82)
	at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
	at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:342)
	at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:105)
	at
  • 解决方案：这2行要么同时放开，要么同时注释
#a1.sources.r1.interceptors = i1
#a1.sources.r1.interceptors.i1.type = com.ychen.flume.interceptors.TimestampAndTableNameInterceptor$Builder