HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint o...

Traceback (most recent call last):
  File "task2_cakir.py", line 626, in <module>                                                                                       | 0/200 [00:00<?, ?it/s]
    sys.exit(main(sys.argv))
  File "task2_cakir.py", line 560, in main
    app.system_training()
  File "/home/fanyiyi_env/DCASE2017-baseline-system-master-gpu/dcase_framework/decorators.py", line 38, in function_wrapper
    to_return = func(*args, **kwargs)
  File "/home/fanyiyi_env/DCASE2017-baseline-system-master-gpu/dcase_framework/application_core.py", line 3222, in system_training
    validation_files=validation_files
  File "/home/fanyiyi_env/DCASE2017-baseline-system-master-gpu/dcase_framework/learners.py", line 2890, in learn
    class_weight=class_weight
  File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/keras/legacy/interfaces.py", line 87, in wrapper
    return func(*args, **kwargs)
  File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/keras/engine/training.py", line 2147, in fit_generator
    class_weight=class_weight)
  File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/keras/engine/training.py", line 1839, in train_on_batch
    outputs = self.train_function(ins)
  File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/keras/backend/theano_backend.py", line 1224, in __call__
    return self.function(*inputs)
  File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.py", line 917, in __call__
    storage_map=getattr(self.fn, 'storage_map', None))
  File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/theano/gof/link.py", line 325, in raise_with_op
    reraise(exc_type, exc_value, exc_trace)
  File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.py", line 903, in __call__
    self.fn() if output_subset is None else\
  File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/theano/scan_module/scan_op.py", line 963, in rval
    r = p(n, [x[0] for x in i], o)
  File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/theano/scan_module/scan_op.py", line 952, in p
    self, node)
  File "theano/scan_module/scan_perform.pyx", line 259, in theano.scan_module.scan_perform.perform (/home/fanyiyi_env/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-centos-7.2.1511-Core-x86_64-2.7.15-64/scan_perform/mod.cpp:3136)
  File "pygpu/gpuarray.pyx", line 1777, in pygpu.gpuarray.GpuArray.copy
  File "pygpu/gpuarray.pyx", line 718, in pygpu.gpuarray.pygpu_copy
  File "pygpu/gpuarray.pyx", line 406, in pygpu.gpuarray.array_copy
pygpu.gpuarray.GpuArrayException: cuMemAlloc: CUDA_ERROR_OUT_OF_MEMORY: out of memory
Apply node that caused the error: for{gpu,scan_fn}(Subtensor{int64}.0, GpuSubtensor{:int64:}.0, GpuIncSubtensor{Set;:int64:}.0, GpuIncSubtensor{Set;:int64:}.0, GpuIncSubtensor{Set;:int64:}.0, GpuIncSubtensor{Set;:int64:}.0, GpuIncSubtensor{Set;:int64:}.0, GpuIncSubtensor{Set;:int64:}.0, Subtensor{int64}.0, conv2d_1/kernel, conv2d_1/bias, batch_normalization_1/gamma, batch_normalization_1/beta, batch_normalization_1/moving_mean, batch_normalization_1/moving_variance, conv2d_2/kernel, conv2d_2/bias, batch_normalization_2/gamma, batch_normalization_2/beta, batch_normalization_2/moving_mean, batch_normalization_2/moving_variance, conv2d_3/kernel, conv2d_3/bias, batch_normalization_3/gamma, batch_normalization_3/beta, batch_normalization_3/moving_mean, batch_normalization_3/moving_variance, gru_1/kernel, gru_1/bias, gru_1/recurrent_kernel, GpuFromHost<None>.0, GpuFromHost<None>.0)
Toposort index: 363
Inputs types: [TensorType(int64, scalar), GpuArrayType<None>(float32, 3D), GpuArrayType<None>(float32, 3D), GpuArrayType<None>(int32, 3D), GpuArrayType<None>(int32, 3D), GpuArrayType<None>(int32, 3D), GpuArrayType<None>(int32, 3D), GpuArrayType<None>(int32, 3D), TensorType(int64, scalar), GpuArrayType<None>(float32, 4D), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, 4D), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, 4D), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, matrix), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, matrix), GpuArrayType<None>(uint8, scalar), GpuArrayType<None>(float32, 4D)]
Inputs shapes: [(), (1501, 32, 96), (1502, 32, 96), (1502, 15360, 6), (1502, 15360, 6), (1502, 15360, 6), (1502, 15360, 6), (1502, 15360, 6), (), (5, 5, 1, 96), (96,), (96,), (96,), (96,), (96,), (5, 5, 96, 96), (96,), (96,), (96,), (96,), (96,), (5, 5, 96, 96), (96,), (96,), (96,), (96,), (96,), (96, 288), (288,), (96, 288), (), (32, 1, 1501, 40)]
Inputs strides: [(), (4, 576384, 6004), (12288, 384, 4), (368640, 24, 4), (368640, 24, 4), (368640, 24, 4), (368640, 24, 4), (368640, 24, 4), (), (1920, 384, 384, 4), (4,), (4,), (4,), (4,), (4,), (184320, 36864, 384, 4), (4,), (4,), (4,), (4,), (4,), (184320, 36864, 384, 4), (4,), (4,), (4,), (4,), (4,), (1152, 4), (4,), (1152, 4), (), (240160, 240160, 160, 4)]
Inputs values: [array(1501), 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', array(1501), 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', gpuarray.array(1, dtype=uint8), 'not shown']
Outputs clients: [[Shape(for{gpu,scan_fn}.0), GpuSubtensor{::int64}(for{gpu,scan_fn}.0, Constant{-1}), GpuSubtensor{:int64:}(for{gpu,scan_fn}.0, Constant{-1})], [Shape(for{gpu,scan_fn}.1), GpuSubtensor{::int64}(for{gpu,scan_fn}.1, Constant{-1}), GpuSubtensor{:int64:}(for{gpu,scan_fn}.1, Constant{-1})], [Shape(for{gpu,scan_fn}.2), GpuSubtensor{::int64}(for{gpu,scan_fn}.2, Constant{-1}), GpuSubtensor{:int64:}(for{gpu,scan_fn}.2, Constant{-1})], [Shape(for{gpu,scan_fn}.3), GpuSubtensor{::int64}(for{gpu,scan_fn}.3, Constant{-1}), GpuSubtensor{:int64:}(for{gpu,scan_fn}.3, Constant{-1})], [Shape(for{gpu,scan_fn}.4), GpuSubtensor{::int64}(for{gpu,scan_fn}.4, Constant{-1}), GpuSubtensor{:int64:}(for{gpu,scan_fn}.4, Constant{-1})], [Shape(for{gpu,scan_fn}.5), GpuSubtensor{::int64}(for{gpu,scan_fn}.5, Constant{-1}), GpuSubtensor{:int64:}(for{gpu,scan_fn}.5, Constant{-1})], [InplaceGpuDimShuffle{0,1,2}(for{gpu,scan_fn}.6), Shape(for{gpu,scan_fn}.6), GpuSubtensor{::int64}(for{gpu,scan_fn}.6, Constant{-1})]]

Backtrace when the node is created(use Theano flag traceback.limit=N to make it longer):
  File "/home/fanyiyi_env/DCASE2017-baseline-system-master-gpu/dcase_framework/application_core.py", line 3222, in system_training
    validation_files=validation_files
  File "/home/fanyiyi_env/DCASE2017-baseline-system-master-gpu/dcase_framework/learners.py", line 2718, in learn
    self.create_model(input_shape=input_shape)
  File "task2_cakir.py", line 252, in create_model
    dropout=rnn_dropout_W, recurrent_dropout=rnn_dropout_U, implementation=2)(model)
  File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/keras/layers/recurrent.py", line 482, in __call__
    return super(RNN, self).__call__(inputs, **kwargs)
  File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/keras/engine/topology.py", line 603, in __call__
    output = self.call(inputs, **kwargs)
  File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/keras/layers/recurrent.py", line 1515, in call
    initial_state=initial_state)
  File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/keras/layers/recurrent.py", line 589, in call
    input_length=timesteps)
  File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/keras/backend/theano_backend.py", line 1424, in rnn
    go_backwards=go_backwards)

HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.
Exception KeyError: KeyError(<weakref at 0x7f3c8b47ae68; to 'tqdm' at 0x7f3c6c938d90>,) in <bound method tqdm.__del__ of      Learn:   0%|                                                                                                                    | 0/200 [00:00<?, ?it/s]> ignored
解决方法：将keras2.1.2升级为2.2.0即可解决
后续：
之后，增加了各层中的隐藏单元个数，由原来的96——>160,结果又出现了上面的报错，将隐藏单元的数值降低，以后，报错消失。