BUG-51254

When using RHEL/CentOS 7.2 or above, systemd does not work on ambari-server.

The following message appears when systemctl on ambari-server is used on RHEL/CentOS 7.2 or above:

[root@c7201 ~]# systemctl status ambari-server ambari-server.service Loaded: not-found (Reason: No such file or directory) Active: inactive (dead)

To correct the problem, run the following on the ambari-server host:

unlink /etc/rc.d/init.d/ambari-server && cp -a /usr/sbin/ambari-server /etc/rc.d/init.d/ambari-server && systemctl daemon-reload

BUG-50231

After upgrading to Ambari 2.2, the Oozie configuration is not properly set, which can cause issues running Oozie and with HDP upgrade.

After upgrading to Ambari 2.2, if your cluster includes Oozie, modify the oozie.service.HadoopAccessorService.hadoop.configurations property.

In Ambari Web, browse to Services > Oozie > Configs and modify the property:

*=/etc/hadoop/conf

to

*={{hadoop_conf_dir}} 

Once complete, restart Oozie.

BUG-50160

If your cluster includes more than one HiveServer2 component, when you perform an Express Upgrade, restart of HiveServer2 might fail during upload of JARs.

If your cluster includes more than one HiveServer2 component, when you perform an Express Upgrade, restart of HiveServer2 might fail during upload of JARs with the following error:

resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X PUT -T /usr/hdp/2.3.4.0-3485/sqoop/sqoop.tar.gz 
--negotiate -u : 'http://os-r7-ueajas-daltom20sechanrmlevel-1.novalocal:50070/webhdfs/v1/hdp/apps/2.3.4.0-3485/sqoop
/sqoop.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444' 1>/tmp/tmppXGRGK 2>/tmp/tmpfojEmU' returned 55. curl: (55) 
Send failure: Connection reset by peer
201 

This is due to the fact that during Express Upgrade, Ambari attempts to upload the necessary JARs for each instance of the HiveServer2 component in parallel. Even if this error occurs, the JARs will be uploaded successfully. You can click Retry to execute the start another time, which should succeed.

BUG-50158

If your cluster includes more than one Hive Metastore component, when you perform an Express Upgrade, schema upgrade might fail when starting Hive Metastore.

If your cluster includes more than one Hive Metastore component, when you perform an Express Upgrade, schema upgrade might fail when starting with the following error:

resource_management.core.exceptions.Fail: Execution of '/usr/hdp/2.3.4.0-3485/hive/bin/schematool -dbType postgres -upgradeSchema'
returned 1. WARNING: Use "yarn jar" to launch YARN applications.
Metastore connection URL:	 jdbc:postgresql://172.22.76.197:5432/hivedb
Metastore Connection Driver :	 org.postgresql.Driver
Metastore connection User:	 hiveuser
Starting upgrade metastore schema from version 0.14.0 to 1.2.0
Upgrade script upgrade-0.14.0-to-1.1.0.postgres.sql
Error: ERROR: relation "NOTIFICATION_LOG" already exists (state=42P07,code=0)
org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore state would be inconsistent !!
*** schemaTool failed *** 

This is due to the fact that during Express Upgrade, Ambari uses the schematool to upgrade the Hive Metastore schema against each instance of the component in parallel. Even with the error, the schema will be upgraded successfully. You can click Retry to execute the start another time, which should succeed.

BUG-50125

When upgrading to HDP 2.3.2 on Ubuntu or Debian, you are using custom service accounts, the upgrade (Rolling or Express) fails while stopping HBase RegionServers.

If you perform an upgrade (Rolling or Express) to HDP 2.3.2, the HBase RegionServers will fail when stopping if you have custom service accounts configured. This occurs because when the HDP 2.3.2 packages are installed, the ownership of the pid directory for HBase is changed. To correct this situation, on the host, change the pid directory ownership for your custom service accounts. For example: if you custom service account for HBase is "cstm-hbase", change the ownership as described below and proceed with the upgrade.

chown cstm-hbase:hbase -R /var/run/hbase/  

BUG-50106

In an environment with umask 027 and after an Express Upgrade, enable Kerberos fails when trying to start services because of "slider-client.xml" permissions.

This issue occurs when Ambari has created the "slider-client.xml" file without enforcing any special permissions. Change file mode for the file after applying the configs:

chmod o+r slider-client.xml

BUG-50044

When using HDP 2.3 and adding Atlas Service to a cluster that includes Hive, the Atlas Metadata Server fails to start.

When adding Atlas to a cluster that includes Hive, the Atlas Metadata Server fails to start with the following error:

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py", line 132, in <module>
    MetadataServer().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py", line 53, in start
    self.configure(env)
  ...
  File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 51, in __call__
    return self.get_content()
  File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 75, in get_content
    raise Fail("{0} Source file {1} is not found".format(repr(self), path))
resource_management.core.exceptions.Fail: StaticFile('/usr/hdp/current/atlas-server/server/webapp/atlas.war') 
Source file /usr/hdp/current/atlas-server/server/webapp/atlas.war is not found 

This situation occurs because Atlas was not installed on the host although Ambari thought it was successful (and therefore, failed to start).

The workaround is to manually install the following package on the host. Once installed, you can start Atlas from Ambari Web.

yum install atlas-metadata_2_3_*

BUG-49992

If you have performed a cluster install via Blueprints, adding a ZooKeeper Server via Ambari Web fails to install with the following error:

Traceback (most recent call last):\n  
File \"/var/lib/ambari-agent/cache/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py\",
line 179, in <module>\n ZooKeeperServer().execute()\n  File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\",
line 219, in execute\n method(env)\n  File \"/var/lib/ambari-agent/cache/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py\", 
line 70, in install\n self.configure(env)\n  File \"/var/lib/ambari-agent/cache/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py\", 
line 49, in configure\n zookeeper(type='server', upgrade_type=upgrade_type)\n  File \"/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py\", 
line 89, in thunk\n return fn(*args, **kwargs)\n  File \"/var/lib/ambari-agent/cache/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py\", 
line 82, in zookeeper\n myid = str(sorted(params.zookeeper_hosts).index(params.hostname) + 1)\nValueError: list.index(x): x not in list 

Please contact Hortonworks support and reference BUG-49992 for a workaround.

BUG-49740

When performing a Blueprint cluster install without Tez, YARN ATS shows an alert.

In your Blueprint, be sure to set the yarn-site configuration property yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes to empty to remove the default reference to Tez.

AMBARI-14352

BUG-49296

During a Rolling or Express Upgrade, intermittent failure when clicking Pause Cluster.

When an operation fails during a Rolling Upgrade or Express Upgrade, the user is provided with an option to Pause Upgrade. On rare occasions, choosing this option does not actually pause the upgrade. When "Pause Upgrade" fails, the UI shows "Upgrade: Action Required", rather than "Upgrade: Paused" at the top of the page. To workaround this, and click on the "Upgrade: Action Required" link and click the "Pause Upgrade" button again. If this still does not work, log out and log back in to Ambari Web , and try again."

BUG-49945

When adding a host to a cluster and re-using a hostname without cleaning-up the previous HDP versions on the host, that host might end-up not running the correct HDP version.

In a case where you re-use a host that had previous versions of HDP installed, and re-use that same hostname with Ambari, the host will start HDP based on the last known running HDP version on that host. You should avoid doing this by cleaning-up the host and remove any previous HDP versions prior to re-adding it to the cluster. If you fail to clean-up the host prior to re-adding, you must manually go onto that host to set the version:

1) From Ambari Web, stop all components on the host.

2) On the host, use the hdp-select utility to point to the currently running HDP version and restart all

hdp-select set all <version>

3) From Ambari Web, restart all components on the host.

You should see the correct HDP version for that host now current in Ambari.

SNMPv3 is not supported.

The SNMPv3 notification method is not supported.

If using HDP 2.1 and Ambari 2.2, then the user may not add services after registering a repo for HDP 2.3.

If the cluster is running HDP 2.1 and you register a repo for HDP 2.3, when attempting to add a service will use the new 2.3 bits instead of the expected 2.1 bits. Only register an HDP 2.3 repo immediately before you are ready to start Express Upgrade from HDP 2.1 to 2.3.

When adding a ZooKeeper service, Kafka property is not updated.

If you are running Kafka and add an additional ZooKeeper server to your cluster, the zookeeper.connect property is not automatically updated to include the newly added ZooKeeper server.

You must manually add the ZooKeeper server to the zookeeper.connect property.

After upgrading from HDP 2.1 and restarting Ambari, the Admin > Stack and Versions > Versions tab does not show in Ambari Web.

After performing an upgrade from HDP 2.1 and restarting Ambari Server and the Agents, if you browse to Admin > Stack and Versions in Ambari Web, the Versions tab does not display. Give all the Agent hosts in the cluster a chance connect to Ambari Server by wait for Ambari to show the Agent heartbeats as green and then refresh your browser.

After adding Falcon to your cluster, the Oozie configuration is not properly updated.

After adding Falcon to your cluster using "Add Service", the Oozie configuration is not properly updated. After completion of Add Service wizard, add properties on Services > Oozie > Configs > Advanced > Custom oozie-site. The list of properties can be found here: https://github.com/apache/ambari/blob/branch-2.1/ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/configuration/oozie-site.xml. Once added, Restart Oozie and execute this command on oozie server host:

su oozie -c '/usr/hdp/current/oozie-server/bin/oozie-setup.sh prepare-war'

Start Oozie.

Storm has no metrics if service is installed via a Blueprint.

The following properties need to be added to storm-site. Browse to Services > Storm > Configs and add the following properties. Restart the Storm service.

topology.metrics.consumer.register=
[{'class': 'org.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsSink'
, 'parallelism.hint': 1}]
metrics.reporter.register=
org.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsReporter

When enabling Kerberos, Kafka security configuration is set and all the ZooKeeper nodes in Kafka will have ACLs set so that only Kafka brokers can modify entries in ZooKeeper. If you disable Kerberos, you user must set all the Kafka ZooKeeper entries to world readable/writable prior to disabling Kerberos. Before disabling kerberos for Kafka. Log-in as user "kafka" on one of Kafka nodes:

kinit -k -t /etc/security/keytabs/kafka.service.keytab kafka/_HOST

where _HOST should be replaced by the hostname of that node. Run the following command to open zookeeper shell:

/usr/hdp/current/kafka-broker/bin/zookeeper-shell.sh hostname:2181

where hostname here should be replaced by one of the zookeeper nodes

setAcl /brokers world:anyone:crdwa 
setAcl /config world:anyone:crdwa 
setAcl /controller world:anyone:crdwa 
setAcl /admin world:anyone:crdwa

If the above commands do not run prior to disabling Kerberos, the only option is to set "zookeeper.connect property" to a new ZooKeeper root. This can be done by appending "/newroot" to "zookeeper.connect.property" string. For example "host1:port1,host2:port2,host3:port3/newroot"

Falcon Service Check may fail when performing Rolling Upgrade, with the following error:

2015-06-25 18:09:07,235 ERROR - [main:]
 ~ Failed to start ActiveMQ JMS Message Broker.
 Reason: java.io.IOException: Invalid location: 1:6763311, :
 java.lang.NegativeArraySizeException (BrokerService:528) 
 java.io.IOException: Invalid location: 1:6763311, :
 java.lang.NegativeArraySizeException
 at
 org.apache.kahadb.journal.DataFileAccessor.readRecord(DataFileAccessor.java:94)

This condition is rare.

When performing a Rolling Upgrade from HDP 2.2 to HDP 2.3 and Falcon Service Check fails with the above error, browse to the Falcon ActiveMQ data dir (specified in falcon properties file), remove the corrupted queues, and stop and start the Falcon Server.

cd <ACTIVEMQ_DATA_DIR>
rm -rf ./localhost
cd /usr/hdp/current/falcon-server 
su -l <FALCON_USER> 
./bin/falcon-stop
./bin/falcon-start

After switching RegionServer ports, Ambari will report RegionServers are live and dead.

HBase maintains the list of dead servers and live servers according to it's semantics. Normally a new server coming up again with the same port will cause the old server to be removed from the dead server list. But due to port change, it will stay in that list for ~2 hours. If the server does not come at all, it will still be removed from the list after 2 hours. Ambari will alert, based on that list until the RegionServers are removed from the list by HBase.

After adding or deleting ZooKeeper Servers to an existing cluster, Service Check fails.

After adding or deleting ZooKeeper Servers to an existing cluster, Service Check fails due to conflicting zk ids. Restart ZooKeeper service to clear the ids.

If you attempt to rename a cluster to a string > 100 chars, Ambari Server will hang. Restart Ambari Server to clear the hang.