tensorflow报错：InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [12] rhs shape= [6]
我是在执行tensorflow/examples/speech_commands/freeze.py时报的错，具体错误如下：
(tensorflow) ubuntu@VM-0-13-ubuntu:~/tf3$ python tensorflow/examples/speech_commands/freeze.py \
> --start_checkpoint=/tmp/speech_commands_train/conv.ckpt-18000 \
> --output_file=/tmp/my_frozen_graph.pb
2018-09-13 14:34:38.589857: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-09-13 14:34:38.699275: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:897] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-09-13 14:34:38.699780: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties:
name: Tesla P40 major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:00:06.0
totalMemory: 22.38GiB freeMemory: 22.21GiB
2018-09-13 14:34:38.699807: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0
2018-09-13 14:34:39.003121: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-09-13 14:34:39.003185: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0
2018-09-13 14:34:39.003194: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0: N
2018-09-13 14:34:39.003625: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 21544 MB memory) -> physical GPU (device: 0, name: Tesla P40, pci bus id: 0000:00:06.0, compute capability: 6.1)
Traceback (most recent call last):
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1278, in _do_call
return fn(*args)
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1263, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1350, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [12] rhs shape= [6]
[[Node: save/Assign = Assign[T=DT_FLOAT, _class=["loc:@final_fc_bias"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](final_fc_bias, save/RestoreV2/_1)]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1725, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 877, in run
run_metadata_ptr)
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1100, in _run
feed_dict_tensor, options, run_metadata)
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1272, in _do_run
run_metadata)
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1291, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [12] rhs shape= [6]
[[Node: save/Assign = Assign[T=DT_FLOAT, _class=["loc:@final_fc_bias"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](final_fc_bias, save/RestoreV2/_1)]]
Caused by op 'save/Assign', defined at:
File "tensorflow/examples/speech_commands/freeze.py", line 208, in <module>
tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "tensorflow/examples/speech_commands/freeze.py", line 134, in main
models.load_variables_from_checkpoint(sess, FLAGS.start_checkpoint)
File "/home/ubuntu/tf3/tensorflow/examples/speech_commands/models.py", line 154, in load_variables_from_checkpoint
saver = tf.train.Saver(tf.global_variables())
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1281, in __init__
self.build()
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1293, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1330, in _build
build_save=build_save, build_restore=build_restore)
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 778, in _build_internal
restore_sequentially, reshape)
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 419, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 112, in restore
self.op.get_shape().is_fully_defined())
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/state_ops.py", line 216, in assign
validate_shape=validate_shape)
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/gen_state_ops.py", line 60, in assign
use_locking=use_locking, name=name)
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 454, in new_func
return func(*args, **kwargs)
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3155, in create_op
op_def=op_def)
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1717, in __init__
self._traceback = tf_stack.extract_stack()
InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [12] rhs shape= [6]
[[Node: save/Assign = Assign[T=DT_FLOAT, _class=["loc:@final_fc_bias"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](final_fc_bias, save/RestoreV2/_1)]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "tensorflow/examples/speech_commands/freeze.py", line 208, in <module>
tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "tensorflow/examples/speech_commands/freeze.py", line 134, in main
models.load_variables_from_checkpoint(sess, FLAGS.start_checkpoint)
File "/home/ubuntu/tf3/tensorflow/examples/speech_commands/models.py", line 155, in load_variables_from_checkpoint
saver.restore(sess, start_checkpoint)
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1759, in restore
err, "a mismatch between the current graph and the graph")
tensorflow.python.framework.errors_impl.InvalidArgumentError: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:
Assign requires shapes of both tensors to match. lhs shape= [12] rhs shape= [6]
[[Node: save/Assign = Assign[T=DT_FLOAT, _class=["loc:@final_fc_bias"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](final_fc_bias, save/RestoreV2/_1)]]
Caused by op 'save/Assign', defined at:
File "tensorflow/examples/speech_commands/freeze.py", line 208, in <module>
tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "tensorflow/examples/speech_commands/freeze.py", line 134, in main
models.load_variables_from_checkpoint(sess, FLAGS.start_checkpoint)
File "/home/ubuntu/tf3/tensorflow/examples/speech_commands/models.py", line 154, in load_variables_from_checkpoint
saver = tf.train.Saver(tf.global_variables())
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1281, in __init__
self.build()
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1293, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1330, in _build
build_save=build_save, build_restore=build_restore)
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 778, in _build_internal
restore_sequentially, reshape)
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 419, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 112, in restore
self.op.get_shape().is_fully_defined())
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/state_ops.py", line 216, in assign
validate_shape=validate_shape)
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/gen_state_ops.py", line 60, in assign
use_locking=use_locking, name=name)
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 454, in new_func
return func(*args, **kwargs)
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3155, in create_op
op_def=op_def)
File "/home/ubuntu/tensorflow/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1717, in __init__
self._traceback = tf_stack.extract_stack()
InvalidArgumentError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:
Assign requires shapes of both tensors to match. lhs shape= [12] rhs shape= [6]
[[Node: save/Assign = Assign[T=DT_FLOAT, _class=["loc:@final_fc_bias"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](final_fc_bias, save/RestoreV2/_1)]]
(tensorflow) ubuntu@VM-0-13-ubuntu:~/tf3$
具体描述一下我遇到此错误的情形，我是在traning自己的数据时，traning时执行的这个命令：
python tensorflow/examples/speech_commands/train.py \
--data_url= \
--data_dir=../traning \
--wanted_words=jia,yu,wo,yi
导出时执行的这个命令：
python tensorflow/examples/speech_commands/freeze.py \
--start_checkpoint=/tmp/speech_commands_train/conv.ckpt-18000 \
--output_file=/tmp/my_frozen_graph.pb
然后就报错了，网上搜索了好久，有说需要把之前traning之后的数据删除，重新traning，我照做后再次traning（对于Tesla P40 GPU来讲，每次traning要花费100分钟左右）还是这个错误。最后认真看了一下，似乎是数量对应不上而已。我改了一下导出命令，就成功了。
python tensorflow/examples/speech_commands/freeze.py \
--start_checkpoint=/tmp/speech_commands_train/conv.ckpt-18000 \
--output_file=/tmp/my_frozen_graph.pb \
--wanted_words=jia,yu,wo,yi
