6. Behavioral Changes

Behavioral changes denote a marked change in behavior from the previously released version to this version of software. In HDP 2.3.4, behavioral changes affect the following Hadoop components.

Table 1.5. Behavioral Changes

Hortonworks Bug ID Component Apache JIRA Summary Details

Hortonworks Bug ID	Component	Apache JIRA	Summary	Details
BUG-45739 BUG-44184	HDFS		When HDFS metadata is replaced, namenode enters safe mode.	Scenario: A user loads an old Fsimage and starts Namenode. This Fsimage is missing the latest transactions that recently occurred on the cluster. Previous Behavior: When a user starts the Namenode with this old image, the Namenode will come out of safemode as long as the amount of blocks reported reaches the set threshold (0.999 is default for `dfs.namenode.safemode.threshold-pct`). When the Namenode learns about a block without a filename, it instructs the Datanode to delete those blocks, resulting in lost data. New Behavior: The new behavior is that when the Namenode detects such a scenario where there are blocks in the cluster which are tagged with future generation stamps, the Namenode will remain in the safemode state to prevent the blocks from being deleted. If the user manually executes `hdfs dfsadmin –safemode leave`, the Namenode will refuse to leave safemode. At this point, the user has an opportunity to shut down the Namenode and load the most current FSimage. If the user for some reason is not able to find the latest Fsimage, the last resort to get the cluster up and running with minimal data loss is to force the Namenode out of safemode by running the command `hdfs dfsadmin –safemode forceExit` after the Namenode has been started.
BUG-46334	HDFS	HDFS-9184	The feature introduces a new setting `hadoop.caller.context.enabled`. When set to additional fields are written into namenode audit log records to help identify the job or query that introduced each NameNode operation. This feature is enabled by default starting with this release of HDP.	New Behavior: This feature brings a new key-value pair at the end of each audit log record. The newly added key at is `callerContext`, value `context:signature`. The overall format would be callerContext=context:signature. If the signature is null or empty, the value will be context only, in the format of `callerContext=context`. If the `hadoop.caller.context.enabled` config key is false, the key-value pair will not be showing. The audit log format is not changed in this case. It is also possible to limit the maximum length of context and signature. Consider the `hadoop.caller.context.max.size` config key (default 128 bytes) and `hadoop.caller.context.signature.max.size` (default 40 bytes) config key respectively. There is a chance that the new information in the audit log may break existing scripts/automation that was being used to analyze the audit log. In this case the scripts may need to be fixed. We do not recommend disabling this feature as it can be a useful troubleshooting aid.

BUG-45739

BUG-44184

HDFS

When HDFS metadata is replaced, namenode enters safe mode.

Scenario: A user loads an old Fsimage and starts Namenode. This Fsimage is missing the latest transactions that recently occurred on the cluster.

Previous Behavior: When a user starts the Namenode with this old image, the Namenode will come out of safemode as long as the amount of blocks reported reaches the set threshold (0.999 is default for dfs.namenode.safemode.threshold-pct). When the Namenode learns about a block without a filename, it instructs the Datanode to delete those blocks, resulting in lost data.

New Behavior: The new behavior is that when the Namenode detects such a scenario where there are blocks in the cluster which are tagged with future generation stamps, the Namenode will remain in the safemode state to prevent the blocks from being deleted. If the user manually executes hdfs dfsadmin –safemode leave, the Namenode will refuse to leave safemode. At this point, the user has an opportunity to shut down the Namenode and load the most current FSimage. If the user for some reason is not able to find the latest Fsimage, the last resort to get the cluster up and running with minimal data loss is to force the Namenode out of safemode by running the command hdfs dfsadmin –safemode forceExit after the Namenode has been started.

BUG-46334

HDFS

HDFS-9184

The feature introduces a new setting hadoop.caller.context.enabled. When set to additional fields are written into namenode audit log records to help identify the job or query that introduced each NameNode operation. This feature is enabled by default starting with this release of HDP.

New Behavior: This feature brings a new key-value pair at the end of each audit log record. The newly added key at is callerContext, value context:signature. The overall format would be callerContext=context:signature. If the signature is null or empty, the value will be context only, in the format of callerContext=context. If the hadoop.caller.context.enabled config key is false, the key-value pair will not be showing. The audit log format is not changed in this case. It is also possible to limit the maximum length of context and signature. Consider the hadoop.caller.context.max.size config key (default 128 bytes) and hadoop.caller.context.signature.max.size (default 40 bytes) config key respectively.

There is a chance that the new information in the audit log may break existing scripts/automation that was being used to analyze the audit log. In this case the scripts may need to be fixed. We do not recommend disabling this feature as it can be a useful troubleshooting aid.

Legal notices