Skip to content

the error on my personal dataset #34

@kangyang94

Description

@kangyang94

I try to train the model on my own dataset which only has two types.But I got the results as below, do you know how to fix it?
2018-12-11 12:16:09.673189: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: assertion failed: [labels out of bound] [Condition x < y did not hold element-wise:x (mean_iou/confusion_matrix/control_dependency:0) = ] [0 0 0...] [y (mean_iou/ToInt64_2:0) = ] [2]
[[Node: mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert = Assert[T=[DT_STRING, DT_STRING, DT_INT64, DT_STRING, DT_INT64], summarize=3, _device="/job:localhost/replica:0/task:0/device:CPU:0"](mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/Switch/_5603, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/data_0, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/data_1, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/Switch_1/_5605, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/data_3, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/Switch_2/_5607)]]
2018-12-11 12:16:09.855569: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: assertion failed: [labels out of bound] [Condition x < y did not hold element-wise:x (mean_iou/confusion_matrix/control_dependency:0) = ] [0 0 0...] [y (mean_iou/ToInt64_2:0) = ] [2]
[[Node: mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert = Assert[T=[DT_STRING, DT_STRING, DT_INT64, DT_STRING, DT_INT64], summarize=3, _device="/job:localhost/replica:0/task:0/device:CPU:0"](mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/Switch/_5603, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/data_0, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/data_1, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/Switch_1/_5605, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/data_3, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/Switch_2/_5607)]]
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>, assertion failed: [labels out of bound] [Condition x < y did not hold element-wise:x (mean_iou/confusion_matrix/control_dependency:0) = ] [0 0 0...] [y (mean_iou/ToInt64_2:0) = ] [2]
[[Node: mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert = Assert[T=[DT_STRING, DT_STRING, DT_INT64, DT_STRING, DT_INT64], summarize=3, _device="/job:localhost/replica:0/task:0/device:CPU:0"](mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/Switch/_5603, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/data_0, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/data_1, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/Switch_1/_5605, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/data_3, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/Switch_2/_5607)]]

Caused by op u'mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert', defined at:
File "train_enet.py", line 337, in
run()
File "train_enet.py", line 192, in run
mean_IOU, mean_IOU_update = tf.contrib.metrics.streaming_mean_iou(predictions=predictions, labels=annotations, num_classes=num_classes)
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/contrib/metrics/python/ops/metric_ops.py", line 2246, in streaming_mean_iou
updates_collections=updates_collections, name=name)
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/ops/metrics_impl.py", line 944, in mean_iou
num_classes, weights)
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/ops/metrics_impl.py", line 284, in _streaming_confusion_matrix
labels, predictions, num_classes, weights=weights, dtype=dtypes.float64)
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/ops/confusion_matrix.py", line 176, in confusion_matrix
labels, num_classes_int64, message='labels out of bound')],
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/ops/check_ops.py", line 403, in assert_less
return control_flow_ops.Assert(condition, data, summarize=summarize)
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/util/tf_should_use.py", line 107, in wrapped
return _add_should_use_warning(fn(*args, **kwargs))
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 134, in Assert
condition, no_op, true_assert, name="AssertGuard")
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 316, in new_func
return func(*args, **kwargs)
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1864, in cond
orig_res_f, res_f = context_f.BuildCondBranch(false_fn)
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1725, in BuildCondBranch
original_result = fn()
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 132, in true_assert
condition, data, summarize, name="Assert")
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/ops/gen_logging_ops.py", line 47, in _assert
name=name)
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
op_def=op_def)
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1470, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): assertion failed: [labels out of bound] [Condition x < y did not hold element-wise:x (mean_iou/confusion_matrix/control_dependency:0) = ] [0 0 0...] [y (mean_iou/ToInt64_2:0) = ] [2]
[[Node: mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert = Assert[T=[DT_STRING, DT_STRING, DT_INT64, DT_STRING, DT_INT64], summarize=3, _device="/job:localhost/replica:0/task:0/device:CPU:0"](mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/Switch/_5603, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/data_0, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/data_1, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/Switch_1/_5605, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/data_3, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/Switch_2/_5607)]]

Traceback (most recent call last):
File "train_enet.py", line 337, in
run()
File "train_enet.py", line 293, in run
loss, training_accuracy, training_mean_IOU = train_step(sess, train_op, sv.global_step, metrics_op=metrics_op)
File "train_enet.py", line 202, in train_step
total_loss, global_step_count, accuracy_val, mean_IOU_val, _ = sess.run([train_op, global_step, accuracy, mean_IOU, metrics_op])
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 889, in run
run_metadata_ptr)
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1120, in _run
feed_dict_tensor, options, run_metadata)
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1317, in _do_run
options, run_metadata)
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1336, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [labels out of bound] [Condition x < y did not hold element-wise:x (mean_iou/confusion_matrix/control_dependency:0) = ] [0 0 0...] [y (mean_iou/ToInt64_2:0) = ] [2]
[[Node: mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert = Assert[T=[DT_STRING, DT_STRING, DT_INT64, DT_STRING, DT_INT64], summarize=3, _device="/job:localhost/replica:0/task:0/device:CPU:0"](mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/Switch/_5603, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/data_0, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/data_1, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/Switch_1/_5605, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/data_3, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/Switch_2/_5607)]]

Caused by op u'mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert', defined at:
File "train_enet.py", line 337, in
run()
File "train_enet.py", line 192, in run
mean_IOU, mean_IOU_update = tf.contrib.metrics.streaming_mean_iou(predictions=predictions, labels=annotations, num_classes=num_classes)
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/contrib/metrics/python/ops/metric_ops.py", line 2246, in streaming_mean_iou
updates_collections=updates_collections, name=name)
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/ops/metrics_impl.py", line 944, in mean_iou
num_classes, weights)
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/ops/metrics_impl.py", line 284, in _streaming_confusion_matrix
labels, predictions, num_classes, weights=weights, dtype=dtypes.float64)
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/ops/confusion_matrix.py", line 176, in confusion_matrix
labels, num_classes_int64, message='labels out of bound')],
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/ops/check_ops.py", line 403, in assert_less
return control_flow_ops.Assert(condition, data, summarize=summarize)
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/util/tf_should_use.py", line 107, in wrapped
return _add_should_use_warning(fn(*args, **kwargs))
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 134, in Assert
condition, no_op, true_assert, name="AssertGuard")
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 316, in new_func
return func(*args, **kwargs)
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1864, in cond
orig_res_f, res_f = context_f.BuildCondBranch(false_fn)
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1725, in BuildCondBranch
original_result = fn()
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 132, in true_assert
condition, data, summarize, name="Assert")
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/ops/gen_logging_ops.py", line 47, in _assert
name=name)
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
op_def=op_def)
File "/home/ky/miniconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1470, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): assertion failed: [labels out of bound] [Condition x < y did not hold element-wise:x (mean_iou/confusion_matrix/control_dependency:0) = ] [0 0 0...] [y (mean_iou/ToInt64_2:0) = ] [2]
[[Node: mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert = Assert[T=[DT_STRING, DT_STRING, DT_INT64, DT_STRING, DT_INT64], summarize=3, _device="/job:localhost/replica:0/task:0/device:CPU:0"](mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/Switch/_5603, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/data_0, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/data_1, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/Switch_1/_5605, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/data_3, mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert/Switch_2/_5607)]]

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions