Using MovementBasedSegmenter with custom neural network


#1

Hi all

We are currently developing an application where we need to track multiple rectangles with different colors. The MovementBasedSegmenter seems to fit this use case with the option of filtering object contours and responding with the object location and id.
How do we go about creating our custom neural network? Is the structure in terms of entry points and respond methods the same as a custom watcher as described in the QRCodeDetector here?

Additionally, is there already an application with which we can send screenshots from lampix to a different device in order to train/test the network?


#2

Hi @Yves,

The good news is that we are currently working on a trainer app and server, which you will be able to use to automatically train a new custom neural network and also deploy it to your Lampix. Please bare with us, as we are currently testing this system.

In the meantime, you could try taking pictures with a smartphone, train your custom neural network (we are using Keras) and deploy it using the ‘:8888/apps’ endpoint. Once this is done and the neural network shows up in Control Panel, you could develop your application using the MovementBasedSegmenter and specify the name of the neural network.

Let us know if we can further help you. :slight_smile:


#3

We have created, trained and added the custom neural network ‘color-classifier’ to our device. However, we get the following error:
IndexError: tuple index out of range
custom_nn_error

Any ideas on how to fix this? Also, is there any documentation on how the custom model is loaded and used?


#4

@Yves, Could you please provide the ‘.json’ file of your trained network? It seems that the architecture might not be compatible.


#5

@vfintinari sure, here you go. Is there any documentation on how the architecture should be or how the NeuralNetworkClassifier or MovementBasedSegmenter use the provided custom network?

{“class_name”: “Sequential”, “keras_version”: “2.2.2”, “config”: [{“class_name”: “Conv2D”, “config”: {“kernel_initializer”: {“class_name”: “VarianceScaling”, “config”: {“distribution”: “uniform”, “scale”: 1.0, “seed”: null, “mode”: “fan_avg”}}, “name”: “conv2d_1”, “kernel_constraint”: null, “bias_regularizer”: null, “bias_constraint”: null, “dtype”: “float32”, “activation”: “linear”, “trainable”: true, “data_format”: “channels_last”, “filters”: 32, “padding”: “same”, “strides”: [1, 1], “dilation_rate”: [1, 1], “kernel_regularizer”: null, “bias_initializer”: {“class_name”: “Zeros”, “config”: {}}, “batch_input_shape”: [null, 64, 64, 3], “use_bias”: true, “activity_regularizer”: null, “kernel_size”: [3, 3]}}, {“class_name”: “Activation”, “config”: {“activation”: “relu”, “trainable”: true, “name”: “activation_1”}}, {“class_name”: “BatchNormalization”, “config”: {“beta_constraint”: null, “gamma_initializer”: {“class_name”: “Ones”, “config”: {}}, “moving_mean_initializer”: {“class_name”: “Zeros”, “config”: {}}, “name”: “batch_normalization_1”, “epsilon”: 0.001, “trainable”: true, “moving_variance_initializer”: {“class_name”: “Ones”, “config”: {}}, “beta_initializer”: {“class_name”: “Zeros”, “config”: {}}, “scale”: true, “axis”: -1, “gamma_constraint”: null, “gamma_regularizer”: null, “beta_regularizer”: null, “momentum”: 0.99, “center”: true}}, {“class_name”: “MaxPooling2D”, “config”: {“name”: “max_pooling2d_1”, “trainable”: true, “data_format”: “channels_last”, “pool_size”: [2, 2], “padding”: “valid”, “strides”: [2, 2]}}, {“class_name”: “Dropout”, “config”: {“rate”: 0.25, “noise_shape”: null, “trainable”: true, “seed”: null, “name”: “dropout_1”}}, {“class_name”: “Conv2D”, “config”: {“kernel_constraint”: null, “kernel_initializer”: {“class_name”: “VarianceScaling”, “config”: {“distribution”: “uniform”, “scale”: 1.0, “seed”: null, “mode”: “fan_avg”}}, “name”: “conv2d_2”, “bias_regularizer”: null, “bias_constraint”: null, “activation”: “linear”, “trainable”: true, “data_format”: “channels_last”, “padding”: “same”, “strides”: [1, 1], “dilation_rate”: [1, 1], “kernel_regularizer”: null, “filters”: 64, “bias_initializer”: {“class_name”: “Zeros”, “config”: {}}, “use_bias”: true, “activity_regularizer”: null, “kernel_size”: [3, 3]}}, {“class_name”: “Activation”, “config”: {“activation”: “relu”, “trainable”: true, “name”: “activation_2”}}, {“class_name”: “BatchNormalization”, “config”: {“beta_constraint”: null, “gamma_initializer”: {“class_name”: “Ones”, “config”: {}}, “moving_mean_initializer”: {“class_name”: “Zeros”, “config”: {}}, “name”: “batch_normalization_2”, “epsilon”: 0.001, “trainable”: true, “moving_variance_initializer”: {“class_name”: “Ones”, “config”: {}}, “beta_initializer”: {“class_name”: “Zeros”, “config”: {}}, “scale”: true, “axis”: -1, “gamma_constraint”: null, “gamma_regularizer”: null, “beta_regularizer”: null, “momentum”: 0.99, “center”: true}}, {“class_name”: “Conv2D”, “config”: {“kernel_constraint”: null, “kernel_initializer”: {“class_name”: “VarianceScaling”, “config”: {“distribution”: “uniform”, “scale”: 1.0, “seed”: null, “mode”: “fan_avg”}}, “name”: “conv2d_3”, “bias_regularizer”: null, “bias_constraint”: null, “activation”: “linear”, “trainable”: true, “data_format”: “channels_last”, “padding”: “same”, “strides”: [1, 1], “dilation_rate”: [1, 1], “kernel_regularizer”: null, “filters”: 64, “bias_initializer”: {“class_name”: “Zeros”, “config”: {}}, “use_bias”: true, “activity_regularizer”: null, “kernel_size”: [3, 3]}}, {“class_name”: “Activation”, “config”: {“activation”: “relu”, “trainable”: true, “name”: “activation_3”}}, {“class_name”: “BatchNormalization”, “config”: {“beta_constraint”: null, “gamma_initializer”: {“class_name”: “Ones”, “config”: {}}, “moving_mean_initializer”: {“class_name”: “Zeros”, “config”: {}}, “name”: “batch_normalization_3”, “epsilon”: 0.001, “trainable”: true, “moving_variance_initializer”: {“class_name”: “Ones”, “config”: {}}, “beta_initializer”: {“class_name”: “Zeros”, “config”: {}}, “scale”: true, “axis”: -1, “gamma_constraint”: null, “gamma_regularizer”: null, “beta_regularizer”: null, “momentum”: 0.99, “center”: true}}, {“class_name”: “MaxPooling2D”, “config”: {“name”: “max_pooling2d_2”, “trainable”: true, “data_format”: “channels_last”, “pool_size”: [2, 2], “padding”: “valid”, “strides”: [2, 2]}}, {“class_name”: “Dropout”, “config”: {“rate”: 0.25, “noise_shape”: null, “trainable”: true, “seed”: null, “name”: “dropout_2”}}, {“class_name”: “Conv2D”, “config”: {“kernel_constraint”: null, “kernel_initializer”: {“class_name”: “VarianceScaling”, “config”: {“distribution”: “uniform”, “scale”: 1.0, “seed”: null, “mode”: “fan_avg”}}, “name”: “conv2d_4”, “bias_regularizer”: null, “bias_constraint”: null, “activation”: “linear”, “trainable”: true, “data_format”: “channels_last”, “padding”: “same”, “strides”: [1, 1], “dilation_rate”: [1, 1], “kernel_regularizer”: null, “filters”: 128, “bias_initializer”: {“class_name”: “Zeros”, “config”: {}}, “use_bias”: true, “activity_regularizer”: null, “kernel_size”: [3, 3]}}, {“class_name”: “Activation”, “config”: {“activation”: “relu”, “trainable”: true, “name”: “activation_4”}}, {“class_name”: “BatchNormalization”, “config”: {“beta_constraint”: null, “gamma_initializer”: {“class_name”: “Ones”, “config”: {}}, “moving_mean_initializer”: {“class_name”: “Zeros”, “config”: {}}, “name”: “batch_normalization_4”, “epsilon”: 0.001, “trainable”: true, “moving_variance_initializer”: {“class_name”: “Ones”, “config”: {}}, “beta_initializer”: {“class_name”: “Zeros”, “config”: {}}, “scale”: true, “axis”: -1, “gamma_constraint”: null, “gamma_regularizer”: null, “beta_regularizer”: null, “momentum”: 0.99, “center”: true}}, {“class_name”: “Conv2D”, “config”: {“kernel_constraint”: null, “kernel_initializer”: {“class_name”: “VarianceScaling”, “config”: {“distribution”: “uniform”, “scale”: 1.0, “seed”: null, “mode”: “fan_avg”}}, “name”: “conv2d_5”, “bias_regularizer”: null, “bias_constraint”: null, “activation”: “linear”, “trainable”: true, “data_format”: “channels_last”, “padding”: “same”, “strides”: [1, 1], “dilation_rate”: [1, 1], “kernel_regularizer”: null, “filters”: 128, “bias_initializer”: {“class_name”: “Zeros”, “config”: {}}, “use_bias”: true, “activity_regularizer”: null, “kernel_size”: [3, 3]}}, {“class_name”: “Activation”, “config”: {“activation”: “relu”, “trainable”: true, “name”: “activation_5”}}, {“class_name”: “BatchNormalization”, “config”: {“beta_constraint”: null, “gamma_initializer”: {“class_name”: “Ones”, “config”: {}}, “moving_mean_initializer”: {“class_name”: “Zeros”, “config”: {}}, “name”: “batch_normalization_5”, “epsilon”: 0.001, “trainable”: true, “moving_variance_initializer”: {“class_name”: “Ones”, “config”: {}}, “beta_initializer”: {“class_name”: “Zeros”, “config”: {}}, “scale”: true, “axis”: -1, “gamma_constraint”: null, “gamma_regularizer”: null, “beta_regularizer”: null, “momentum”: 0.99, “center”: true}}, {“class_name”: “Conv2D”, “config”: {“kernel_constraint”: null, “kernel_initializer”: {“class_name”: “VarianceScaling”, “config”: {“distribution”: “uniform”, “scale”: 1.0, “seed”: null, “mode”: “fan_avg”}}, “name”: “conv2d_6”, “bias_regularizer”: null, “bias_constraint”: null, “activation”: “linear”, “trainable”: true, “data_format”: “channels_last”, “padding”: “same”, “strides”: [1, 1], “dilation_rate”: [1, 1], “kernel_regularizer”: null, “filters”: 128, “bias_initializer”: {“class_name”: “Zeros”, “config”: {}}, “use_bias”: true, “activity_regularizer”: null, “kernel_size”: [3, 3]}}, {“class_name”: “Activation”, “config”: {“activation”: “relu”, “trainable”: true, “name”: “activation_6”}}, {“class_name”: “BatchNormalization”, “config”: {“beta_constraint”: null, “gamma_initializer”: {“class_name”: “Ones”, “config”: {}}, “moving_mean_initializer”: {“class_name”: “Zeros”, “config”: {}}, “name”: “batch_normalization_6”, “epsilon”: 0.001, “trainable”: true, “moving_variance_initializer”: {“class_name”: “Ones”, “config”: {}}, “beta_initializer”: {“class_name”: “Zeros”, “config”: {}}, “scale”: true, “axis”: -1, “gamma_constraint”: null, “gamma_regularizer”: null, “beta_regularizer”: null, “momentum”: 0.99, “center”: true}}, {“class_name”: “MaxPooling2D”, “config”: {“name”: “max_pooling2d_3”, “trainable”: true, “data_format”: “channels_last”, “pool_size”: [2, 2], “padding”: “valid”, “strides”: [2, 2]}}, {“class_name”: “Dropout”, “config”: {“rate”: 0.25, “noise_shape”: null, “trainable”: true, “seed”: null, “name”: “dropout_3”}}, {“class_name”: “Flatten”, “config”: {“trainable”: true, “name”: “flatten_1”, “data_format”: “channels_last”}}, {“class_name”: “Dense”, “config”: {“kernel_initializer”: {“class_name”: “VarianceScaling”, “config”: {“distribution”: “uniform”, “scale”: 1.0, “seed”: null, “mode”: “fan_avg”}}, “name”: “dense_1”, “kernel_constraint”: null, “bias_regularizer”: null, “bias_constraint”: null, “activation”: “linear”, “trainable”: true, “kernel_regularizer”: null, “bias_initializer”: {“class_name”: “Zeros”, “config”: {}}, “units”: 512, “use_bias”: true, “activity_regularizer”: null}}, {“class_name”: “Activation”, “config”: {“activation”: “relu”, “trainable”: true, “name”: “activation_7”}}, {“class_name”: “BatchNormalization”, “config”: {“beta_constraint”: null, “gamma_initializer”: {“class_name”: “Ones”, “config”: {}}, “moving_mean_initializer”: {“class_name”: “Zeros”, “config”: {}}, “name”: “batch_normalization_7”, “epsilon”: 0.001, “trainable”: true, “moving_variance_initializer”: {“class_name”: “Ones”, “config”: {}}, “beta_initializer”: {“class_name”: “Zeros”, “config”: {}}, “scale”: true, “axis”: -1, “gamma_constraint”: null, “gamma_regularizer”: null, “beta_regularizer”: null, “momentum”: 0.99, “center”: true}}, {“class_name”: “Dropout”, “config”: {“rate”: 0.5, “noise_shape”: null, “trainable”: true, “seed”: null, “name”: “dropout_4”}}, {“class_name”: “Dense”, “config”: {“kernel_initializer”: {“class_name”: “VarianceScaling”, “config”: {“distribution”: “uniform”, “scale”: 1.0, “seed”: null, “mode”: “fan_avg”}}, “name”: “dense_2”, “kernel_constraint”: null, “bias_regularizer”: null, “bias_constraint”: null, “activation”: “linear”, “trainable”: true, “kernel_regularizer”: null, “bias_initializer”: {“class_name”: “Zeros”, “config”: {}}, “units”: 9, “use_bias”: true, “activity_regularizer”: null}}, {“class_name”: “Activation”, “config”: {“activation”: “softmax”, “trainable”: true, “name”: “activation_8”}}], “backend”: “tensorflow”}


#6

@vfintinari curiously after redeploying the app and retrying today, it seems to work! Thanks for the help.


#7

Additionally, @vfintinari, there seems to be a typo on the param filter_thresh on MovementBasedSegmenter

What image is handed to the model by the MovementBasedSegmenter? Is it the whole watcher area or is it a cutout of the contours it finds?