526互联

Hadoop启动报错：org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "hdfs"

发布时间 2023-07-06 17:28:08作者: 等不到的口琴

报错信息

org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "hdfs"

	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3281)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3301)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227)
	at com.courage.hadoop.TestHdfs.conn(TestHdfs.java:22)
	
java.lang.NullPointerException
	at com.courage.hadoop.TestHdfs.close(TestHdfs.java:32)

原因

jar包或打包问题

Different JARs (hadoop-commons for LocalFileSystem, hadoop-hdfs for DistributedFileSystem) each contain a different file called org.apache.hadoop.fs.FileSystem in their META-INFO/servicesdirectory. This file lists the canonical classnames of the filesystem implementations they want to declare (This is called a Service Provider Interface implemented via java.util.ServiceLoader, see org.apache.hadoop.FileSystem line 2622).
When we use maven-assembly-plugin, it merges all our JARs into one, and all META-INFO/services/org.apache.hadoop.fs.FileSystem overwrite each-other. Only one of these files remains (the last one that was added). In this case, the FileSystem list from hadoop-commonsoverwrites the list from hadoop-hdfs, so DistributedFileSystem was no longer declared.

解决办法

1.通过配置方式：

Java代码

这个解决hdfs问题
hadoopConf.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
这个解决本地file问题
hadoopConf.set("fs.file.impl", org.apache.hadoop.fs.LocalFileSystem.class.getName());

2.通过打包插件方式

Xml代码

<plugin>  
    <groupId>org.apache.maven.plugins</groupId>  
    <artifactId>maven-shade-plugin</artifactId>  
    <version>2.3</version>  
    <executions>  
      <execution>  
        <phase>package</phase>  
        <goals>  
          <goal>shade</goal>  
        </goals>  
        <configuration>  
          <transformers>  
            <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>  
          </transformers>  
        </configuration>  
      </execution>  
    </executions>  
  </plugin>