PostgreSQL
 sql >> Cơ Sở Dữ Liệu >  >> RDS >> PostgreSQL

Sqoop trên Hadoop:NoSuchMethodError:com.google.common.base.Stopwatch.createStarted ()

Sau một số nghiên cứu sâu hơn, tôi thấy rằng Hadoop 2.x ghi đè classpath, vì vậy giải pháp là tạo một uberjar và chuyển nó cho hadoop.

Tôi cũng đã thay đổi để sử dụng jar sqoop cụ thể cho hadoop260.

Vì vậy, tôi đã tạo một pom.xml tệp, đã chạy maven package trên đó để tạo uberjar:

<?xml version="1.0" encoding="UTF-8"?>
<!-- SEE: https://cloud.google.com/blog/products/data-analytics/managing-java-dependencies-apache-spark-applications-cloud-dataproc -->
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
    </properties>
    <groupId>org.example.exporter</groupId>
    <artifactId>example-exporter-postgresql</artifactId>
    <version>0.0.1</version>
    <!-- YOUR_DEPENDENCIES -->
    <dependencies>
        <!-- https://mvnrepository.com/artifact/org.apache.sqoop/sqoop -->
        <dependency>
            <groupId>org.apache.sqoop</groupId>
            <artifactId>sqoop</artifactId>
            <version>1.4.7</version>
            <classifier>hadoop260</classifier>
        </dependency>
        <!-- https://mvnrepository.com/artifact/postgresql/postgresql -->
        <dependency>
            <groupId>org.postgresql</groupId>
            <artifactId>postgresql</artifactId>
            <version>42.2.9</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/com.google.cloud.sql/postgres-socket-factory -->
        <dependency>
            <groupId>com.google.cloud.sql</groupId>
            <artifactId>postgres-socket-factory</artifactId>
            <version>1.0.15</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.avro/avro-tools -->
        <dependency>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro-tools</artifactId>
            <version>1.9.1</version>
        </dependency>

    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <transformers>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <mainClass>org.apache.sqoop.Sqoop</mainClass>
                                </transformer>
                            </transformers>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/maven/**</exclude>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                            <relocations>
                                <relocation>
                                    <pattern>com</pattern>
                                    <shadedPattern>repackaged.com.google.common</shadedPattern>
                                    <includes>
                                        <include>com.google.common.**</include>
                                    </includes>
                                </relocation>
                            </relocations>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>



  1. Database
  2.   
  3. Mysql
  4.   
  5. Oracle
  6.   
  7. Sqlserver
  8.   
  9. PostgreSQL
  10.   
  11. Access
  12.   
  13. SQLite
  14.   
  15. MariaDB
  1. Postgres:Ràng buộc chỉ bằng kiểm tra CHÈN

  2. ImportError khi chạy cgi-bin trên localhost - ký hiệu không xác định:lo_truncate64

  3. PostgreSQL lưu trữ các tệp cấu hình / conf ở đâu?

  4. Lỗi trong diễn biến trên Heroku

  5. Hàm tổng hợp JSONB sqlalchemy