Security
Also available as:
PDF
loading table of contents...

Contents

1. HDP Security Overview
What's New in This Release
Understanding Data Lake Security
HDP Security Features
Administration
Authentication and Perimeter Security
Authorization
Audit
Data Protection
2. Authentication
Enabling Kerberos Authentication Using Ambari
Kerberos Overview
Kerberos Principals
Installing and Configuring the KDC
Enabling Kerberos Security
Kerberos Client Packages
Disabling Kerberos Security
Customizing the Attribute Template
Managing Admin Credentials
Configuring HDP Components for Kerberos Using Ambari
Configuring Kafka for Kerberos Using Ambari
Configuring Storm for Kerberos Using Ambari
Configuring Ambari Authentication with LDAP or AD
Configuring Ambari for LDAP or Active Directory Authentication
Configuring Ranger Authentication with UNIX, LDAP, or AD
Encrypting Database and LDAP Passwords in Ambari
Configuring LDAP Authentication in Hue
Enabling the LDAP Backend
Enabling User Authentication with Search Bind
Setting the Search Base to Find Users and Groups
Specifying the URL of the LDAP Server
Specifying LDAPS and StartTLS Support
Specifying Bind Credentials for LDAP Searches
Synchronizing Users and Groups
Setting Search Bind Authentication and Importing Users and Groups
Setting LDAP Users' Filter
Setting an LDAP Groups Filter
Setting Multiple LDAP Servers
Advanced Security Options for Ambari
Configuring Ambari for Non-Root
Optional: Ambari Web Inactivity Timeout
Optional: Set Up Kerberos for Ambari Server
Optional: Set Up Two-Way SSL Between Ambari Server and Ambari Agents
Optional: Configure Ciphers and Protocols for Ambari Server
Optional: HTTP Cookie Persistence
Enabling SPNEGO Authentication for Hadoop
Configure Ambari Server for Authenticated HTTP
Configuring HTTP Authentication for HDFS, YARN, MapReduce2, HBase, Oozie, Falcon and Storm
Enabling Browser Access to a SPNEGO-enabled Web UI
Setting Up Kerberos Authentication for Non-Ambari Clusters
Preparing Kerberos
Configuring HDP for Kerberos
Setting up One-Way Trust with Active Directory
Configuring Proxy Users
Perimeter Security with Apache Knox
Apache Knox Gateway Overview
Configuring the Knox Gateway
Defining Cluster Topologies
Configuring a Hadoop Server for Knox
Mapping the Internal Nodes to External URLs
Configuring Authentication
Configuring Identity Assertion
Configuring Service Level Authorization
Audit Gateway Activity
Gateway Security
Setting Up Knox Services for HA
Knox CLI Testing Tools
Knox SSO
Identity Providers (IdP)
Setting up Knox SSO for Ambari
Setting up Knox SSO for Ranger
Setting up Knox SSO for Apache Atlas
3. Configuring Authorization in Hadoop
Installing Ranger Using Ambari
Overview
Installation Prerequisites
Ranger Installation
Enabling Ranger Plugins
Ranger Plugins - Kerberos Overview
Using Ranger to Provide Authorization in Hadoop
About Ranger Policies
Using the Ranger Console
Configuring Resource-Based Services
Resource-Based Policy Management
Row-level Filtering and Column Masking in Hive
Adding Tag-based Service
Tag-Based Policy Management
Users/Groups and Permissions Administration
Reports Administration
Special Requirements for High Availability Environments
Adding a New Component to Apache Ranger
Developing a Custom Authorization Module
Apache Ranger Public REST API
4. Data Protection: Wire Encryption
Enabling RPC Encryption
Enabling Data Transfer Protocol
Enabling SSL: Understanding the Hadoop SSL Keystore Factory
Creating and Managing SSL Certificates
Obtain a Certificate from a Trusted Third-Party Certification Authority (CA)
Create and Set Up an Internal CA (OpenSSL)
Installing Certificates in the Hadoop SSL Keystore Factory (HDFS, MapReduce, and YARN)
Using a CA-Signed Certificate
Enabling SSL for HDP Components
Enable SSL for WebHDFS, MapReduce Shuffle, Tez, and YARN
Enable SSL for HttpFS
Enable SSL on Oozie
Configure the Oozie Client to Connect Using SSL
Connect to the Oozie Web UI Using SSL
Configure Oozie HCatalogJob Properties
Enable SSL on the HBase REST Server
Enable SSL on the HBase Web UI
Enable SSL on HiveServer2
Setting up SSL with self-signed certificates
Selectively disabling SSL protocol versions
Enable SSL for Kafka Clients
Configuring the Kafka Broker
Configuring Kafka Producer and Kafka Consumer
Enable SSL for Accumulo
Generate a Certificate Authority
Generate a Certificate/Keystore Per Host
Configure Accumulo Servers
Configure Accumulo Clients
Enable SSL for Apache Atlas
Configuring Apache Atlas SSL
Credential Provider Utility Script
SPNEGO setup for WebHCat
Configure SSL for Hue
Enabling SSL on Hue by Using a Private Key
Enabling SSL on Hue Without Using a Private Key
Configure SSL for Knox
Self-Signed Certificate with Specific Hostname for Evaluations
CA-Signed Certificates for Production
Setting Up Trust for the Knox Gateway Clients
Securing Phoenix
Set Up SSL for Ambari
Set Up Truststore for Ambari Server
Configure Ambari Ranger SSL
Configuring Ambari Ranger SSL Using Public CA Certificates
Configuring Ambari Ranger SSL Using a Self-Signed Certificate
Configure Ranger Admin Database for SSL-Enabled MySQL
Configure Non-Ambari Ranger SSL
Configuring Non-Ambari Ranger SSL Using Public CA Certificates
Configuring Non-Ambari Ranger SSL Using a Self Signed Certificate
Connecting to SSL-Enabled Components
Connect to SSL Enabled HiveServer2 using JDBC
Connect to SSL Enabled Oozie Server
5. Auditing in Hadoop
Using Apache Solr for Ranger Audits
Prerequisites
Installing Externally Managed SolrCloud
Configuring Externally Managed SolrCloud
Configuring Externally Managed Solr Standalone
Configuring SolrCloud for Kerberos
Migrating Audit Logs from DB to Solr in Ambari Clusters
Manually Enabling Audit Settings in Ambari Clusters
Manually Updating Ambari Solr Audit Settings
Manually Updating HDFS Audit Settings (for Ambari installs)
Enabling Audit Logging in Non-Ambari Clusters
Managing Auditing in Ranger
View Operation Details
Differentiate Events from Multiple Clusters
Access
Admin
Login Sessions
Plugins
Plugin Status
6. ACLs on HDFS
Configuring ACLs on HDFS
Using CLI Commands to Create and List ACLs
ACL Examples
ACLS on HDFS Features
Use Cases for ACLs on HDFS
7. Data Protection: HDFS Encryption
Ranger KMS Administration
Installing the Ranger Key Management Service
Store Master Key in a Hardware Security Module (HSM)
Enable Ranger KMS Audit
Enabling SSL for Ranger KMS
Install Multiple Ranger KMS
Using the Ranger Key Management Service
Ranger KMS Properties
Troubleshooting Ranger KMS
HDFS "Data at Rest" Encryption
HDFS Encryption Overview
Configuring and Starting the Ranger Key Management Service (Ranger KMS)
Configuring and Using HDFS Data at Rest Encryption
Configuring HDP Services for HDFS Encryption
Appendix: Creating an HDFS Admin User
8. Running DataNodes as Non-Root
Introduction
Configuring DataNode SASL
9. Addendum
ZooKeeper ACLs Best Practices
Accumulo
Ambari Solr
Atlas
HBase
HDFS/WebHDFS
Hive/HCatalog
Kafka
Oozie
Ranger
Ranger KMS/Hadoop KMS
Slider
Storm
WebHCat
YARN
YARN Registry
ZooKeeper
Ranger AD Integration
Ranger Architecture
Ranger AD Integration
Issue Ranger Group Mapping

List of Tables

2.1. Browser Settings for Storm UI
2.2. UNIX Authentication Settings
2.3. Active Directory Authentication Settings
2.4. Active Directory Custom ranger-admin-site Settings
2.5. LDAP Authentication Settings
2.6. LDAP Custom ranger-admin-site Settings
2.7. Active Directory Authentication Settings
2.8. Service Principals
2.9. Service Keytab File Names
2.10. General core-site.xml, Knox, and Hue
2.11. core-site.xml Master Node Settings -- Knox Gateway
2.12. core-site.xml Master Node Settings -- Hue
2.13. hdfs-site.xml File Property Settings
2.14. yarn-site.xml Property Settings
2.15. mapred-site.xml Property Settings
2.16. hbase-site.xml Property Settings for HBase Server and Phoenix Query Server
2.17. hive-site.xml Property Settings
2.18. oozie-site.xml Property Settings
2.19. webhcat-site.xml Property Settings
2.20. Supported Hadoop Services
2.21. Supported Hadoop UIs
2.22. Apache Service Gateway Directories
2.23. Cluster Topology Provider and Service Roles
2.24. gateway-site.xml Configuration Elements
2.25. Identity Assertion Providers
2.26. LDAP Authentication and Authorization Arguments
3.1. Ranger DB Host
3.2. Driver Class Name
3.3. Ranger DB Username Settings
3.4. JDBC Connect String
3.5. DBA Credential Settings
3.6. UNIX User Sync Properties
3.7. LDAP/AD Common Configs
3.8. LDAP/AD User Configs
3.9. LDAP/AD Group Configs
3.10. Atlas Tag Source Properties
3.11. AtlasREST Source Properties
3.12. File Tag Source Properties
3.13. UNIX Authentication Settings
3.14. LDAP Authentication Settings
3.15. AD Settings
3.16. LDAP Advanced ranger-ugsync-site Settings
3.17. AD Advanced ranger-ugsync-site Settings
3.18. Advanced ranger-ugsync-site Settings for LDAP and AD
3.19. HDFS Plugin Properties
3.20. Hive Plugin Properties
3.21. HBase Plugin Properties
3.22. Knox Plugin Properties
3.23. Knox Configuration Properties
3.24. Service Details
3.25. Config Properties
3.26. Service Details
3.27. Config Properties
3.28. Service Details
3.29. Config Properties
3.30. Service Details
3.31. Config Properties
3.32. Service Details
3.33. Config Properties
3.34. Service Details
3.35. Config Properties
3.36. Service Details
3.37. Config Properties
3.38. Service Details
3.39. Config Properties
3.40. Service Details
3.41. Config Properties
3.42. Policy Details
3.43. Allow Conditions
3.44. Policy Details
3.45. Allow Conditions
3.46. Policy Details
3.47. Allow Conditions
3.48. Policy Details
3.49. Allow Conditions
3.50. Policy Details
3.51. Allow Conditions
3.52. Policy Details
3.53. Allow Conditions
3.54. Policy Details
3.55. Allow Conditions
3.56. Storm User and Group Permissions
3.57. Policy Details
3.58. Allow Conditions
3.59. Policy Details
3.60. Allow Conditions
3.61. Export Policy Options
3.62. Policy Details
3.63. Row Filter Conditions
3.64. Policy Details
3.65. Mask Conditions
3.66. Policy Details
3.67. Mask Conditions
3.68. Policy Details
3.69. Allow, Exclude from Allow, Deny, and Exclude from Deny Conditions
3.70. Policy Details
3.71. Allow Conditions
3.72. Deny Conditions
3.73. Exclude from Allow Conditions
3.74. Export Policy Options
4.1. Components that Support SSL
4.2. Configure SSL Data Protection for HDP Components
4.3. Configuration Properties in ssl-server.xml
4.4. Atlas Advanced application-properties
4.5. Atlas Custom application-properties
5.1. Solr install.properties Values for setup.sh script
5.2. Solr install.properties Values
5.3. Solr install.properties Values
5.4. JDBC Audit String
5.5. Search Criteria
5.6. Search Criteria
5.7. Search Criteria
5.8. Agents Search Criteria
5.9. Plugin Status Search Criteria
6.1. ACL Options
6.2. getfacl Options
7.1. Properties in Advanced dbks-site Menu (dbks-site.xml)
7.2. Properties in Advanced kms-env
7.3. Properties in Advanced kms-properties (install.properties)
7.4. Properties in Advanced kms-site (kms-site.xml)
7.5. Properties in Advanced ranger-kms-audit (ranger-kms-audit.xml)
7.6. Properties in Advanced ranger-kms-policymgr-ssl
7.7. Properties in Advanced ranger-kms-security
7.8. Troubleshooting Suggestions