Accessing Cloud Data
Also available as:
PDF
loading table of contents...

Working with Third-party S3-compatible Object Stores

The S3A Connector can work with third-party object stores; some vendors test the connector against their stores —and even actively collaborate in developing the connector in the open source community.

OptionChange to
fs.s3a.endpoint(the hostname of the S3 Store)
fs.s3a.path.style.accesstrue
fs.s3a.signing-algorithmIf the default signing mechanism is rejected, another mechanism may work from: "QueryStringSignerType", "S3SignerType", "AWS3SignerType", "AWS4SignerType", "AWS4UnsignedPayloadSignerType" and "NoOpSignerType".
fs.s3a.connection.ssl.enabledSet to "false" if HTTP is used instead of HTTPS
fs.s3a.multiobjectdelete.enableSet to "false" if bulk delete operations are not supported.
fs.s3a.list.versionSet to "1" if the list directories with the default "2" option fails with an error.

Third-party object stores are generally consistent; there is no need for S3Guard. The S3A Committers will still offer better performance, and should be used for MapReduce and Spark.

Encryption may or may not be supported: consulte the documentation.

Security permissions are likely to be implemented diffrently from the IAM role mode —again, consult the documentation to see what is available.