pentaho-mr执行记录

1、kettle日志记录

总体mapper执行非常慢，reducer执行快。

2019/05/13 02:08:53 - cfgbuilder - Warning: The configuration parameter [org] is not supported by the default configuration builder for scheme: sftp

2019/05/13 02:09:37 - Spoon - 正在开始任务...

2019/05/13 02:09:42 - aggregate_01 - 开始执行任务

2019/05/13 02:09:42 - aggregate_01 - exec(0, 0, START.0)

2019/05/13 02:09:42 - START - Starting job entry

2019/05/13 02:09:42 - aggregate_01 - 开始项[Pentaho MapReduce]

2019/05/13 02:09:42 - aggregate_01 - exec(1, 0, Pentaho MapReduce.0)

2019/05/13 02:09:42 - Pentaho MapReduce - Starting job entry

2019/05/13 02:09:42 - cfgbuilder - Warning: The configuration parameter [org] is not supported by the default configuration builder for scheme: sftp

2019/05/13 02:09:42 - Pentaho MapReduce - Using org.apache.hadoop.io.Text for the map output key

2019/05/13 02:09:42 - Pentaho MapReduce - Using org.apache.hadoop.io.LongWritable for the map output value

2019/05/13 02:09:42 - Pentaho MapReduce - Using org.apache.hadoop.io.Text for the output key

2019/05/13 02:09:42 - Pentaho MapReduce - Using org.apache.hadoop.io.DoubleWritable for the output value

2019/05/13 02:09:46 - Pentaho MapReduce - Using Kettle installation from /opt/pentaho/mapreduce/6.1.0.1-196-6.1.0.1-196-hdp23

2019/05/13 02:09:46 - Pentaho MapReduce - Configuring Pentaho MapReduce job to use Kettle installation from /opt/pentaho/mapreduce/6.1.0.1-196-6.1.0.1-196-hdp23

2019/05/13 02:09:47 - Pentaho MapReduce - mapreduce.application.classpath: classes/,$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/2.6.3.0-235/hadoop/lib/hadoop-lzo-0.6.0.2.6.3.0-235.jar:/etc/hadoop/conf/secure

2019/05/13 02:09:47 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:09:52 - Pentaho MapReduce - Setup Complete: 0.0 Mapper Completion: 0.0 Reducer Completion: 0.0

2019/05/13 02:09:57 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:09:57 - Pentaho MapReduce - Setup Complete: 0.0 Mapper Completion: 0.0 Reducer Completion: 0.0

2019/05/13 02:10:02 - Pentaho MapReduce - Setup Complete: 0.0 Mapper Completion: 0.0 Reducer Completion: 0.0

2019/05/13 02:10:07 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:10:07 - Pentaho MapReduce - Setup Complete: 0.0 Mapper Completion: 0.0 Reducer Completion: 0.0

2019/05/13 02:10:12 - Pentaho MapReduce - Setup Complete: 0.0 Mapper Completion: 0.0 Reducer Completion: 0.0

2019/05/13 02:10:17 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:10:18 - Pentaho MapReduce - Setup Complete: 0.0 Mapper Completion: 0.0 Reducer Completion: 0.0

2019/05/13 02:10:25 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 0.0 Reducer Completion: 0.0

2019/05/13 02:10:27 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:10:30 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 0.0 Reducer Completion: 0.0

2019/05/13 02:10:35 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 0.0 Reducer Completion: 0.0

2019/05/13 02:10:37 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:10:40 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 0.0 Reducer Completion: 0.0

2019/05/13 02:10:45 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 0.0 Reducer Completion: 0.0

2019/05/13 02:10:47 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:10:50 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 0.0 Reducer Completion: 0.0

2019/05/13 02:10:55 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 0.0 Reducer Completion: 0.0

2019/05/13 02:10:57 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:11:00 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 0.0 Reducer Completion: 0.0

2019/05/13 02:11:06 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 0.0 Reducer Completion: 0.0

2019/05/13 02:11:07 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:11:11 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 0.0 Reducer Completion: 0.0

2019/05/13 02:11:16 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 0.0 Reducer Completion: 0.0

2019/05/13 02:11:17 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:11:21 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 0.0 Reducer Completion: 0.0

2019/05/13 02:11:26 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 10.000000149011612 Reducer Completion: 0.0

2019/05/13 02:11:26 - Pentaho MapReduce - [SUCCEEDED] -- Task: attempt_1557683339547_0003_m_000000_0 Attempt: attempt_1557683339547_0003_m_000000_0 Event: 0

2019/05/13 02:11:26 - Pentaho MapReduce - Container killed by the ApplicationMaster.

Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143.

2019/05/13 02:11:27 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:11:31 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 10.000000149011612 Reducer Completion: 0.0

2019/05/13 02:11:36 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 10.000000149011612 Reducer Completion: 0.0

2019/05/13 02:11:37 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:11:41 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 10.000000149011612 Reducer Completion: 0.0

2019/05/13 02:11:46 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 10.000000149011612 Reducer Completion: 0.0

2019/05/13 02:11:47 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:11:51 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 10.000000149011612 Reducer Completion: 0.0

2019/05/13 02:11:56 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 20.000000298023224 Reducer Completion: 0.0

2019/05/13 02:11:56 - Pentaho MapReduce - [SUCCEEDED] -- Task: attempt_1557683339547_0003_m_000001_0 Attempt: attempt_1557683339547_0003_m_000001_0 Event: 1

2019/05/13 02:11:56 - Pentaho MapReduce - Container killed by the ApplicationMaster.

Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143.

2019/05/13 02:11:57 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:12:01 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 20.000000298023224 Reducer Completion: 0.0

2019/05/13 02:12:06 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 20.000000298023224 Reducer Completion: 0.0

2019/05/13 02:12:07 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:12:11 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 20.000000298023224 Reducer Completion: 0.0

2019/05/13 02:12:16 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 20.000000298023224 Reducer Completion: 0.0

2019/05/13 02:12:17 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:12:21 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 20.000000298023224 Reducer Completion: 0.0

2019/05/13 02:12:26 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 20.000000298023224 Reducer Completion: 0.0

2019/05/13 02:12:27 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:12:31 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 20.000000298023224 Reducer Completion: 0.0

2019/05/13 02:12:36 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 20.000000298023224 Reducer Completion: 0.0

2019/05/13 02:12:37 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:12:41 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 20.000000298023224 Reducer Completion: 0.0

2019/05/13 02:12:46 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 20.000000298023224 Reducer Completion: 0.0

2019/05/13 02:12:47 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:12:51 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 20.000000298023224 Reducer Completion: 0.0

2019/05/13 02:12:56 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 20.000000298023224 Reducer Completion: 0.0

2019/05/13 02:12:57 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:13:01 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 20.000000298023224 Reducer Completion: 0.0

2019/05/13 02:13:06 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 20.000000298023224 Reducer Completion: 0.0

2019/05/13 02:13:07 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:13:11 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 20.000000298023224 Reducer Completion: 0.0

2019/05/13 02:13:16 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 30.000001192092896 Reducer Completion: 0.0

2019/05/13 02:13:16 - Pentaho MapReduce - [SUCCEEDED] -- Task: attempt_1557683339547_0003_m_000002_0 Attempt: attempt_1557683339547_0003_m_000002_0 Event: 2

2019/05/13 02:13:16 - Pentaho MapReduce - Container killed by the ApplicationMaster.

Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143.

2019/05/13 02:13:17 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:13:21 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 30.000001192092896 Reducer Completion: 0.0

2019/05/13 02:13:27 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:13:30 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 30.000001192092896 Reducer Completion: 0.0

2019/05/13 02:13:35 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 30.000001192092896 Reducer Completion: 0.0

2019/05/13 02:13:37 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:13:40 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 30.000001192092896 Reducer Completion: 0.0

2019/05/13 02:13:45 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 30.000001192092896 Reducer Completion: 0.0

2019/05/13 02:13:47 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:13:50 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 30.000001192092896 Reducer Completion: 0.0

2019/05/13 02:13:55 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 30.000001192092896 Reducer Completion: 0.0

2019/05/13 02:13:57 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:14:00 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 30.000001192092896 Reducer Completion: 0.0

2019/05/13 02:14:05 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 30.000001192092896 Reducer Completion: 0.0

2019/05/13 02:14:07 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:14:10 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 40.00000059604645 Reducer Completion: 0.0

2019/05/13 02:14:10 - Pentaho MapReduce - [SUCCEEDED] -- Task: attempt_1557683339547_0003_m_000003_0 Attempt: attempt_1557683339547_0003_m_000003_0 Event: 3

2019/05/13 02:14:10 - Pentaho MapReduce - Container killed by the ApplicationMaster.

Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143.

2019/05/13 02:14:15 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 40.00000059604645 Reducer Completion: 0.0

2019/05/13 02:14:17 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:14:20 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 40.00000059604645 Reducer Completion: 0.0

2019/05/13 02:14:25 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 40.00000059604645 Reducer Completion: 0.0

2019/05/13 02:14:27 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:14:30 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 40.00000059604645 Reducer Completion: 0.0

2019/05/13 02:14:35 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 40.00000059604645 Reducer Completion: 0.0

2019/05/13 02:14:37 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:14:40 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 40.00000059604645 Reducer Completion: 0.0

2019/05/13 02:14:45 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 40.00000059604645 Reducer Completion: 0.0

2019/05/13 02:14:47 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:14:50 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 40.00000059604645 Reducer Completion: 0.0

2019/05/13 02:14:56 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 40.00000059604645 Reducer Completion: 0.0

2019/05/13 02:14:57 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:15:01 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 50.0 Reducer Completion: 0.0

2019/05/13 02:15:01 - Pentaho MapReduce - [SUCCEEDED] -- Task: attempt_1557683339547_0003_m_000004_0 Attempt: attempt_1557683339547_0003_m_000004_0 Event: 4

2019/05/13 02:15:01 - Pentaho MapReduce - Container killed by the ApplicationMaster.

Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143.

2019/05/13 02:15:06 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 50.0 Reducer Completion: 0.0

2019/05/13 02:15:07 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:15:11 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 50.0 Reducer Completion: 0.0

2019/05/13 02:15:16 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 50.0 Reducer Completion: 0.0

2019/05/13 02:15:17 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:15:21 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 50.0 Reducer Completion: 0.0

2019/05/13 02:15:26 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 50.0 Reducer Completion: 0.0

2019/05/13 02:15:27 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:15:31 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 50.0 Reducer Completion: 0.0

2019/05/13 02:15:36 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 50.0 Reducer Completion: 0.0

2019/05/13 02:15:37 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:15:41 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 50.0 Reducer Completion: 0.0

2019/05/13 02:15:46 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 50.0 Reducer Completion: 0.0

2019/05/13 02:15:47 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:15:51 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 50.0 Reducer Completion: 0.0

2019/05/13 02:15:56 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 50.0 Reducer Completion: 0.0

2019/05/13 02:15:57 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:16:01 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 50.0 Reducer Completion: 0.0

2019/05/13 02:16:06 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 50.0 Reducer Completion: 0.0

2019/05/13 02:16:07 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:16:11 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 60.00000238418579 Reducer Completion: 0.0

2019/05/13 02:16:11 - Pentaho MapReduce - [SUCCEEDED] -- Task: attempt_1557683339547_0003_m_000005_0 Attempt: attempt_1557683339547_0003_m_000005_0 Event: 5

2019/05/13 02:16:16 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 60.00000238418579 Reducer Completion: 0.0

2019/05/13 02:16:17 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:16:21 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 60.00000238418579 Reducer Completion: 0.0

2019/05/13 02:16:26 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 60.00000238418579 Reducer Completion: 0.0

2019/05/13 02:16:27 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:16:31 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 60.00000238418579 Reducer Completion: 0.0

2019/05/13 02:16:36 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 60.00000238418579 Reducer Completion: 0.0

2019/05/13 02:16:37 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:16:41 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 60.00000238418579 Reducer Completion: 0.0

2019/05/13 02:16:46 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 60.00000238418579 Reducer Completion: 0.0

2019/05/13 02:16:47 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:16:51 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 60.00000238418579 Reducer Completion: 0.0

2019/05/13 02:16:56 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 60.00000238418579 Reducer Completion: 0.0

2019/05/13 02:16:57 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:17:01 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 60.00000238418579 Reducer Completion: 0.0

2019/05/13 02:17:06 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 69.9999988079071 Reducer Completion: 0.0

2019/05/13 02:17:06 - Pentaho MapReduce - [SUCCEEDED] -- Task: attempt_1557683339547_0003_m_000006_0 Attempt: attempt_1557683339547_0003_m_000006_0 Event: 6

2019/05/13 02:17:07 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:17:11 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 69.9999988079071 Reducer Completion: 0.0

2019/05/13 02:17:16 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 69.9999988079071 Reducer Completion: 0.0

2019/05/13 02:17:17 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:17:21 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 69.9999988079071 Reducer Completion: 0.0

2019/05/13 02:17:26 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 69.9999988079071 Reducer Completion: 0.0

2019/05/13 02:17:27 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:17:31 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 69.9999988079071 Reducer Completion: 0.0

2019/05/13 02:17:36 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 80.0000011920929 Reducer Completion: 0.0

2019/05/13 02:17:36 - Pentaho MapReduce - [SUCCEEDED] -- Task: attempt_1557683339547_0003_m_000007_0 Attempt: attempt_1557683339547_0003_m_000007_0 Event: 7

2019/05/13 02:17:36 - Pentaho MapReduce - Container killed by the ApplicationMaster.

Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143.

2019/05/13 02:17:37 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:17:41 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 80.0000011920929 Reducer Completion: 0.0

2019/05/13 02:17:46 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 80.0000011920929 Reducer Completion: 0.0

2019/05/13 02:17:47 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:17:51 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 80.0000011920929 Reducer Completion: 0.0

2019/05/13 02:17:56 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 80.0000011920929 Reducer Completion: 0.0

2019/05/13 02:17:57 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:18:01 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 80.0000011920929 Reducer Completion: 0.0

2019/05/13 02:18:06 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 80.0000011920929 Reducer Completion: 0.0

2019/05/13 02:18:07 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:18:11 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 89.99999761581421 Reducer Completion: 0.0

2019/05/13 02:18:11 - Pentaho MapReduce - [SUCCEEDED] -- Task: attempt_1557683339547_0003_m_000008_0 Attempt: attempt_1557683339547_0003_m_000008_0 Event: 8

2019/05/13 02:18:11 - Pentaho MapReduce - Container killed by the ApplicationMaster.

Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143.

2019/05/13 02:18:16 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 89.99999761581421 Reducer Completion: 0.0

2019/05/13 02:18:17 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:18:21 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 89.99999761581421 Reducer Completion: 0.0

2019/05/13 02:18:26 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 89.99999761581421 Reducer Completion: 0.0

2019/05/13 02:18:27 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:18:31 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 89.99999761581421 Reducer Completion: 0.0

2019/05/13 02:18:36 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 89.99999761581421 Reducer Completion: 0.0

2019/05/13 02:18:37 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:18:41 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 89.99999761581421 Reducer Completion: 0.0

2019/05/13 02:18:46 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 89.99999761581421 Reducer Completion: 0.0

2019/05/13 02:18:47 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:18:51 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 89.99999761581421 Reducer Completion: 0.0

2019/05/13 02:18:56 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 100.0 Reducer Completion: 0.0

2019/05/13 02:18:56 - Pentaho MapReduce - [SUCCEEDED] -- Task: attempt_1557683339547_0003_m_000009_0 Attempt: attempt_1557683339547_0003_m_000009_0 Event: 9

2019/05/13 02:18:56 - Pentaho MapReduce - Container killed by the ApplicationMaster.

Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143.

2019/05/13 02:18:57 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:19:01 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 100.0 Reducer Completion: 0.0

2019/05/13 02:19:06 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 100.0 Reducer Completion: 0.0

2019/05/13 02:19:07 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:19:11 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 100.0 Reducer Completion: 0.0

2019/05/13 02:19:16 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 100.0 Reducer Completion: 0.0

2019/05/13 02:19:17 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:19:21 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 100.0 Reducer Completion: 66.66666865348816

2019/05/13 02:19:26 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 100.0 Reducer Completion: 66.66666865348816

2019/05/13 02:19:27 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:19:31 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 100.0 Reducer Completion: 66.66666865348816

2019/05/13 02:19:36 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 100.0 Reducer Completion: 66.66666865348816

2019/05/13 02:19:37 - aggregate_01 - Triggering heartbeat signal for aggregate_01 at every 10 seconds

2019/05/13 02:19:41 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 100.0 Reducer Completion: 66.66666865348816

2019/05/13 02:19:46 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 100.0 Reducer Completion: 100.0

2019/05/13 02:19:46 - Pentaho MapReduce - [SUCCEEDED] -- Task: attempt_1557683339547_0003_r_000000_0 Attempt: attempt_1557683339547_0003_r_000000_0 Event: 10

2019/05/13 02:19:46 - aggregate_01 - 完成作业项[Pentaho MapReduce] (结果=[true])

2019/05/13 02:19:46 - aggregate_01 - 任务执行完毕

2019/05/13 02:19:46 - Spoon - 任务已经结束.

2、hadoop集群日志

http://slave01.hadoop:8088/cluster/app/application_1557683339547_0003

3、mr结果查询验证

4、问题总结

1）从集群拷贝yarn-site.xml到kettle plugin目录，mr通过yarn调度执行，作业不直接对mr

2）内存不足，修改yarn内存参数，从510MB改为1536MB，同时，加大虚拟机内存从1536mb调至2048mb

yarn.scheduler.maximum-allocation-mb

yarn.nodemanager.resource.memory-mb

参考：http://blog.sina.com.cn/s/blog_e699b42b0102xg2k.html

https://blog.csdn.net/mlljava1111/article/details/51867883

3）在本地mapred-site.xml修改${hdp.version}，引用改为实际值2.6.3.0-235

4）mr输出output path不能存在，前几次执行失败后，需先删新建目录

hadoop fs -rm -R /user/zhengyuan/aggregate_mr/

5）环境变量添加HADOOP_USER_NAME，使用hdfs，伪装本地用户

也有说：客户端的pdi连接到集群的hadoop，要伪装成集群的用户，在spoon.bat增加一行。set HADOOP_USER_NAME=root 参考：https://blog.csdn.net/weixin_33744854/article/details/87469805

最后编辑于：2019.05.13 02:45:10

人面猴
序言：七十年代末，一起剥皮案震惊了整个滨河市，随后出现的几起案子，更是在滨河造成了极大的恐慌，老刑警刘岩，带你破解...
沈念sama阅读 215,634评论 6赞 497
死咒
序言：滨河连续发生了三起死亡事件，死亡现场离奇诡异，居然都是意外死亡，警方通过查阅死者的电脑和手机，发现死者居然都...
沈念sama阅读 91,951评论 3赞 391
救了他两次的神仙让他今天三更去死
文/潘晓璐我一进店门，熙熙楼的掌柜王于贵愁眉苦脸地迎上来，“玉大人，你说我怎么就摊上这事。” “怎么了？”我有些...
开封第一讲书人阅读 161,427评论 0赞 351
道士缉凶录：失踪的卖姜人
文/不坏的土叔我叫张陵，是天一观的道长。经常有香客问我，道长，这世上最难降的妖魔是什么？我笑而不...
开封第一讲书人阅读 57,770评论 1赞 290
港岛之恋（遗憾婚礼）
正文为了忘掉前任，我火速办了婚礼，结果婚礼上，老公的妹妹穿的比我还像新娘。我一直安慰自己，他们只是感情好，可当我...
茶点故事阅读 66,835评论 6赞 388
恶毒庶女顶嫁案：这布局不是一般人想出来的
文/花漫我一把揭开白布。她就那样静静地躺着，像睡着了一般。火红的嫁衣衬着肌肤如雪。梳的纹丝不乱的头发上，一...
开封第一讲书人阅读 50,799评论 1赞 294
城市分裂传说
那天，我揣着相机与录音，去河边找鬼。笑死，一个胖子当着我的面吹牛，可吹牛的内容都是我干的。我是一名探鬼主播，决...
沈念sama阅读 39,768评论 3赞 416
双鸳鸯连环套：你想象不到人心有多黑
文/苍兰香墨我猛地睁开眼，长吁一口气：“原来是场噩梦啊……” “哼！你这毒妇竟也来了？” 一声冷哼从身侧响起，我...
开封第一讲书人阅读 38,544评论 0赞 271
万荣杀人案实录
序言：老挝万荣一对情侣失踪，失踪者是张志新（化名）和其女友刘颖，没想到半个月后，有当地人在树林里发现了一具尸体，经...
沈念sama阅读 44,979评论 1赞 308
护林员之死
正文独居荒郊野岭守林人离奇死亡，尸身上长有42处带血的脓包…… 初始之章·张勋以下内容为张勋视角年9月15日...
茶点故事阅读 37,271评论 2赞 331
白月光启示录
正文我和宋清朗相恋三年，在试婚纱的时候发现自己被绿了。大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
茶点故事阅读 39,427评论 1赞 345
活死人
序言：一个原本活蹦乱跳的男人离奇死亡，死状恐怖，灵堂内的尸体忽然破棺而出，到底是诈尸还是另有隐情，我是刑警宁泽，带...
沈念sama阅读 35,121评论 5赞 340
日本核电站爆炸内幕
正文年R本政府宣布，位于F岛的核电站，受9级特大地震影响，放射性物质发生泄漏。R本人自食恶果不足惜，却给世界环境...
茶点故事阅读 40,756评论 3赞 324
男人毒药：我在死后第九天来索命
文/蒙蒙一、第九天我趴在偏房一处隐蔽的房顶上张望。院中可真热闹，春花似锦、人声如沸。这庄子的主人今日做“春日...
开封第一讲书人阅读 31,375评论 0赞 21
一桩弑父案，背后竟有这般阴谋
文/苍兰香墨我抬头看了看天上的太阳。三九已至，却和暖如春，着一层夹袄步出监牢的瞬间，已是汗流浃背。一阵脚步声响...
开封第一讲书人阅读 32,579评论 1赞 268
情欲美人皮
我被黑心中介骗来泰国打工，没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留，地道东北人。一个月前我还...
沈念sama阅读 47,410评论 2赞 368
代替公主和亲
正文我出身青楼，却偏偏与公主长得像，于是被迫代替她去往敌国和亲。传闻我的和亲对象是个残疾皇子，可洞房花烛夜当晚...
茶点故事阅读 44,315评论 2赞 352

pentaho-mr执行记录

推荐阅读更多精彩内容