Preparing the Environment
HDP supports hardware acceleration with Advanced Encryption Standard New Instructions (AES-NI). Compared with the software implementation of AES, hardware acceleration offers an order of magnitude faster encryption/decryption.
CPU Support for AES NI optimizationTo use AES-NI optimization you need CPU and library support, described in the following subsections.
AES-NI optimization requires an extended CPU instruction set for AES hardware acceleration.
There are several ways to check for this; for example:
$ cat /proc/cpuinfo | grep aes
Look for output with flags and 'aes'.
Library Support for AES NI optimization
You will need a version of the libcrypto.so library that supports hardware acceleration, such as OpenSSL 1.0.1e. (Many OS versions have an older version of the library that does not support AES-NI.)
A version of the
libcrypto.so library with AES-NI support must be
installed on HDFS cluster nodes and MapReduce client hosts -- that is, any host from
which you issue HDFS or MapReduce requests. The following instructions describe how to
install and configure the
RHEL/CentOS 6.5 or later
On HDP cluster nodes, the installed version of
AES-NI, but you will need to make sure that the symbolic link exists:
$ sudo ln -s /usr/lib64/libcrypto.so.1.0.1e /usr/lib64/libcrypto.so
On MapReduce client hosts, install the
$ sudo yum install openssl-devel
Verifying AES NI Support
To verify that a client host is ready to use the AES-NI instruction set optimization for HDFS encryption, use the following command:
You should see a response similar to the following:
15/08/12 13:48:39 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native 14/12/12 13:48:39 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library Native library checking: hadoop: true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0 zlib: true /lib64/libz.so.1 snappy: true /usr/lib64/libsnappy.so.1 lz4: true revision:99 bzip2: true /lib64/libbz2.so.1 openssl: true /usr/lib64/libcrypto.so
If you see
true in the
openssl row, Hadoop has
detected the right version of
libcrypto.so and optimization will
If you see
false in this row, you do not have the correct version.