Installing HDF Services on an Existing HDP Cluster
Also available as:
PDF

Upgrading Log Rotation Configuration

Ambari 2.5.0 provides s a simplified log rotation configuration. These changes will be made automatically during your next stack upgrade, but are not automatically made during the Ambari upgrade. After upgrading Ambari from version 2.x to 2.5.0, if you want to utilize the simplified log rotation configuration, you must update configurations for all services in your cluster, using the following steps:

Steps

  1. ZooKeeper

    1. In Ambari Web, browse to ZooKeeper > Configs.

    2. Scroll down to Custom zookeeper-log4j.

    3. In Custom zookeeper-log4j, click Add Property.

    4. In Add Property, type the following properties and values:

      zookeeper_log_max_backup_size=10

      zookeeper_log_number_of_backup_files=10

      For example:

    5. Click Add.

    6. Browse to Advanced zookeeper-log4j.

    7. In Advanced zookeeper-log4j content section, find and replace the following properties and values:

      Find: log4j.appender.ROLLINGFILE.MaxFileSize=<value>

      Replace: log4j.appender.ROLLINGFILE.MaxFileSize={{zookeeper_log_number_of_backup_files}}MB

      Find: #log4j.appender.ROLLINGFILE.MaxBackupIndex=<value>MB

      Replace: #log4j.appender.ROLLINGFILE.MaxBackupIndex={{zookeeper_log_number_of_backup_files}}

      For example:

    8. In Configs, click Save.

      For example:

    9. Restart ZooKeeper, as prompted.

  2. Kafka

    1. In Ambari Web, browse to Kafka > Configs.

    2. Scroll down to Custom Kafka-log4j.

    3. In Custom Kafka-log4j, click Add Property.

    4. In Add Property, type the following properties and values:

      kafka_log_maxfilesize=256

      kafka_log_maxbackupindex=20

      controller_log_maxfilesize=256

      controller_log_maxbackupindex=20

    5. Click Add.

    6. Browse to Advanced kafka-log4j.

    7. In Advanced kafka-log4j content section, find and replace the following properties and values:

      Find: log4j.appender.kafkaAppender=org.apache.log4j.DailyRollingFileAppender

      Add: log4j.appender.kafkaAppender.MaxFileSize = {{kafka_log_maxfilesize}}MB

      Add: log4j.appender.kafkaAppender.MaxBackupIndex = {{kafka_log_maxbackupindex}}MB

      Find: log4j.appender.controllerAppender=org.apache.log4j.DailyRollingFileAppender

      Add: log4j.appender.controllerAppender.MaxFileSize = {{controller_log_maxfilesize}}MB

      Add: log4j.appender.controllerAppender.MaxBackupIndex = {{controller_log_maxbackupindex}}

    8. In Configs, click Save.

    9. Restart Kafka, as prompted.

  3. Ranger

    1. In Ambari Web, browse to Ranger > Configs > Advanced.

    2. Scroll down to Custom admin-log4j.

    3. In Custom admin-log4j, click Add Property.

    4. In Add Property, type the following properties and values:

      ranger_xa_log_maxfilesize=256

      ranger_xa_log_maxbackupindex=20

    5. Click Add.

    6. Browse to Advanced admin-log4j.

    7. In Advanced admin-log4j content section, find and replace the following properties and values:

      Find: log4j.appender.xa_log_appender=org.apache.log4j.DailyRollingFileAppender

      Add: log4j.appender.xa_log_appender.MaxFileSize={{ranger_xa_log_maxfilesize}}MB

      Add: log4j.appender.xa_log_appender.MaxBackupIndex={{ranger_xa_log_maxbackupindex}}

    8. Scroll down to Custom usersync-log4j.

    9. In Custom usersync-log4j, click Add Property.

    10. In Add Property, type the following properties and values:

      ranger_usersync_log_maxfilesize=256

      ranger_usersync_log_number_of_backup_files=20

    11. Click Add.

    12. Browse to Advanced usersync-log4j.

    13. In Advanced usersync-log4j content section, find and replace the following properties and values:

      Find: log4j.appender.logFile=org.apache.log4j.DailyRollingFileAppender

      Add: log4j.appender.logFile.MaxFileSize = {{ranger_usersync_log_maxfilesize}}MB

      Add: log4j.appender.logFile.MaxBackupIndex = {{ranger_usersync_log_number_of_backup_files}}

    14. Scroll down to Custom tagsync-log4j.

    15. In Custom tagsync-log4j, click Add Property.

    16. In Add Property, type the following properties and values:

      ranger_tagsync_log_maxfilesize=256

      ranger_tagsync_log_number_of_backup_files=20

    17. Click Add.

    18. Browse to Advanced tagsync-log4j.

    19. In Advanced tagsync-log4j content section, find and replace the following properties and values:

      Find: log4j.appender.logFile=org.apache.log4j.DailyRollingFileAppender

      Add: log4j.appender.logFile.MaxFileSize = {{ranger_tagsync_log_maxfilesize}}MB

      Add: log4j.appender.logFile.MaxBackupIndex = {{ranger_tagsync_log_number_of_backup_files}}

    20. In Configs, click Save.

    21. Restart Ranger, as prompted.

  4. Ranger-KMS

    1. In Ambari Web, browse to Ranger-KMS > Configs > Advanced.

    2. Scroll down to Custom kms-log4j.

    3. In Custom kms-log4j, click Add Property.

    4. In Add Property, type the following properties and values:

      ranger_kms_log_maxfilesize=256

      ranger_kms_log_maxbackupindex=20

      ranger_kms_audit_log_maxfilesize=256

      ranger_kms_audit_log_maxbackupindex=20

    5. Click Add.

    6. Browse to Advanced kms-log4j .

    7. In Advanced kms-log4j content section, find and replace the following properties and values:

      Find: log4j.appender.kms=org.apache.log4j.DailyRollingFileAppender

      Add: log4j.appender.kms.MaxFileSize = {{ranger_kms_log_maxfilesize}}MB

      Add: log4j.appender.kms.MaxBackupIndex = {{ranger_kms_log_maxbackupindex}}

      Find: log4j.appender.kms-audit=org.apache.log4j.DailyRollingFileAppender

      Add: log4j.appender.kms-audit.MaxFileSize={{ranger_kms_audit_log_maxfilesize}}MB

      Add: log4j.appender.kms-audit.MaxBackupIndex = {{ranger_kms_audit_log_maxbackupindex}}

    8. In Configs, click Save.

    9. Restart Ranger-KMS.

  5. Storm

    1. In Ambari Web, browse to Storm > Configs.

    2. Scroll down to Custom cluster-log4j property.

    3. In Custom cluster-log4j property, click Add Property.

    4. In Add Property, type the following properties and values:

      storm_a1_maxfilesize=100

      storm_a1_maxbackupindex=9

    5. Click Add.

    6. Browse to Advanced storm-cluster-log4j .

    7. In Advanced storm-cluster-log4j content section, find and replace the following properties and values:

      Find: In RollingFile="A1"<SizeBasedTriggeringPolicy size=”<value>MB"/>

      Replace: <SizeBasedTriggeringPolicy size="{{storm_a1_maxfilesize}}MB"/>

      Find: In RollingFile="A1"<DefaultRolloverStrategy max="<value>"/>

      Replace: <DefaultRolloverStrategy max="{{storm_a1_maxbackupindex}}"/>

    8. Scroll down to Custom worker-log4j property.

    9. In Custom worker-log4j property, click Add Property.

    10. In Add Property, type the following properties and values:

      storm_wrkr_a1_maxfilesize=100

      storm_wrkr_a1_maxbackupindex=9

      storm_wrkr_out_maxfilesize=100

      storm_wrkr_out_maxbackupindex=4

      storm_wrkr_err_maxfilesize=100

      storm_wrkr_err_maxbackupindex=4

    11. Click Add.

    12. Browse to Advanced storm-worker-log4j .

    13. In Advanced storm-worker-log4j content section, find and replace the following properties and values:

      Find: In RollingFile="A1"<SizeBasedTriggeringPolicy size="<value> MB"/>

      Replace: <SizeBasedTriggeringPolicy size="{{storm_wrkr_a1_maxfilesize}} MB"/>

      Find: In RollingFile="A1"<DefaultRolloverStrategy max="<value>"/>

      Replace: <DefaultRolloverStrategy max="{{storm_wrkr_a1_maxbackupindex}}"/>

      Find: In RollingFile="STDOUT"<SizeBasedTriggeringPolicy size="<value>" MB/>

      Replace: <SizeBasedTriggeringPolicy size="{{storm_wrkr_out_maxfilesize}} MB"/>

      Find: In RollingFile="STDOUT"<DefaultRolloverStrategy max="<value>"/>

      Replace: <DefaultRolloverStrategy max="{{storm_wrkr_out_maxbackupindex}}"/>

      Find: In RollingFile=”STDERR”<SizeBasedTriggeringPolicy size=”<value>” MB/>

      Replace: <SizeBasedTriggeringPolicy size="{{storm_wrkr_err_maxfilesize}} MB"/>

      Find: In RollingFile="STDOUT"<DefaultRolloverStrategy max="<value>"/>

      Replace: <DefaultRolloverStrategy max="{{storm_wrkr_err_maxbackupindex}}"/>

    14. In Configs, click Save.

    15. Restart Storm, as prompted.