Using MovementBasedSegmenter with custom neural network


#1

Hi all

We are currently developing an application where we need to track multiple rectangles with different colors. The MovementBasedSegmenter seems to fit this use case with the option of filtering object contours and responding with the object location and id.
How do we go about creating our custom neural network? Is the structure in terms of entry points and respond methods the same as a custom watcher as described in the QRCodeDetector here?

Additionally, is there already an application with which we can send screenshots from lampix to a different device in order to train/test the network?


#2

Hi @Yves,

The good news is that we are currently working on a trainer app and server, which you will be able to use to automatically train a new custom neural network and also deploy it to your Lampix. Please bare with us, as we are currently testing this system.

In the meantime, you could try taking pictures with a smartphone, train your custom neural network (we are using Keras) and deploy it using the ‘:8888/apps’ endpoint. Once this is done and the neural network shows up in Control Panel, you could develop your application using the MovementBasedSegmenter and specify the name of the neural network.

Let us know if we can further help you. :slight_smile:


#3

We have created, trained and added the custom neural network ‘color-classifier’ to our device. However, we get the following error:
IndexError: tuple index out of range

Any ideas on how to fix this? Also, is there any documentation on how the custom model is loaded and used?


#4

@Yves, Could you please provide the ‘.json’ file of your trained network? It seems that the architecture might not be compatible.


#5

@vfintinari sure, here you go. Is there any documentation on how the architecture should be or how the NeuralNetworkClassifier or MovementBasedSegmenter use the provided custom network?

{“class_name”: “Sequential”, “keras_version”: “2.2.2”, “config”: [{“class_name”: “Conv2D”, “config”: {“kernel_initializer”: {“class_name”: “VarianceScaling”, “config”: {“distribution”: “uniform”, “scale”: 1.0, “seed”: null, “mode”: “fan_avg”}}, “name”: “conv2d_1”, “kernel_constraint”: null, “bias_regularizer”: null, “bias_constraint”: null, “dtype”: “float32”, “activation”: “linear”, “trainable”: true, “data_format”: “channels_last”, “filters”: 32, “padding”: “same”, “strides”: [1, 1], “dilation_rate”: [1, 1], “kernel_regularizer”: null, “bias_initializer”: {“class_name”: “Zeros”, “config”: {}}, “batch_input_shape”: [null, 64, 64, 3], “use_bias”: true, “activity_regularizer”: null, “kernel_size”: [3, 3]}}, {“class_name”: “Activation”, “config”: {“activation”: “relu”, “trainable”: true, “name”: “activation_1”}}, {“class_name”: “BatchNormalization”, “config”: {“beta_constraint”: null, “gamma_initializer”: {“class_name”: “Ones”, “config”: {}}, “moving_mean_initializer”: {“class_name”: “Zeros”, “config”: {}}, “name”: “batch_normalization_1”, “epsilon”: 0.001, “trainable”: true, “moving_variance_initializer”: {“class_name”: “Ones”, “config”: {}}, “beta_initializer”: {“class_name”: “Zeros”, “config”: {}}, “scale”: true, “axis”: -1, “gamma_constraint”: null, “gamma_regularizer”: null, “beta_regularizer”: null, “momentum”: 0.99, “center”: true}}, {“class_name”: “MaxPooling2D”, “config”: {“name”: “max_pooling2d_1”, “trainable”: true, “data_format”: “channels_last”, “pool_size”: [2, 2], “padding”: “valid”, “strides”: [2, 2]}}, {“class_name”: “Dropout”, “config”: {“rate”: 0.25, “noise_shape”: null, “trainable”: true, “seed”: null, “name”: “dropout_1”}}, {“class_name”: “Conv2D”, “config”: {“kernel_constraint”: null, “kernel_initializer”: {“class_name”: “VarianceScaling”, “config”: {“distribution”: “uniform”, “scale”: 1.0, “seed”: null, “mode”: “fan_avg”}}, “name”: “conv2d_2”, “bias_regularizer”: null, “bias_constraint”: null, “activation”: “linear”, “trainable”: true, “data_format”: “channels_last”, “padding”: “same”, “strides”: [1, 1], “dilation_rate”: [1, 1], “kernel_regularizer”: null, “filters”: 64, “bias_initializer”: {“class_name”: “Zeros”, “config”: {}}, “use_bias”: true, “activity_regularizer”: null, “kernel_size”: [3, 3]}}, {“class_name”: “Activation”, “config”: {“activation”: “relu”, “trainable”: true, “name”: “activation_2”}}, {“class_name”: “BatchNormalization”, “config”: {“beta_constraint”: null, “gamma_initializer”: {“class_name”: “Ones”, “config”: {}}, “moving_mean_initializer”: {“class_name”: “Zeros”, “config”: {}}, “name”: “batch_normalization_2”, “epsilon”: 0.001, “trainable”: true, “moving_variance_initializer”: {“class_name”: “Ones”, “config”: {}}, “beta_initializer”: {“class_name”: “Zeros”, “config”: {}}, “scale”: true, “axis”: -1, “gamma_constraint”: null, “gamma_regularizer”: null, “beta_regularizer”: null, “momentum”: 0.99, “center”: true}}, {“class_name”: “Conv2D”, “config”: {“kernel_constraint”: null, “kernel_initializer”: {“class_name”: “VarianceScaling”, “config”: {“distribution”: “uniform”, “scale”: 1.0, “seed”: null, “mode”: “fan_avg”}}, “name”: “conv2d_3”, “bias_regularizer”: null, “bias_constraint”: null, “activation”: “linear”, “trainable”: true, “data_format”: “channels_last”, “padding”: “same”, “strides”: [1, 1], “dilation_rate”: [1, 1], “kernel_regularizer”: null, “filters”: 64, “bias_initializer”: {“class_name”: “Zeros”, “config”: {}}, “use_bias”: true, “activity_regularizer”: null, “kernel_size”: [3, 3]}}, {“class_name”: “Activation”, “config”: {“activation”: “relu”, “trainable”: true, “name”: “activation_3”}}, {“class_name”: “BatchNormalization”, “config”: {“beta_constraint”: null, “gamma_initializer”: {“class_name”: “Ones”, “config”: {}}, “moving_mean_initializer”: {“class_name”: “Zeros”, “config”: {}}, “name”: “batch_normalization_3”, “epsilon”: 0.001, “trainable”: true, “moving_variance_initializer”: {“class_name”: “Ones”, “config”: {}}, “beta_initializer”: {“class_name”: “Zeros”, “config”: {}}, “scale”: true, “axis”: -1, “gamma_constraint”: null, “gamma_regularizer”: null, “beta_regularizer”: null, “momentum”: 0.99, “center”: true}}, {“class_name”: “MaxPooling2D”, “config”: {“name”: “max_pooling2d_2”, “trainable”: true, “data_format”: “channels_last”, “pool_size”: [2, 2], “padding”: “valid”, “strides”: [2, 2]}}, {“class_name”: “Dropout”, “config”: {“rate”: 0.25, “noise_shape”: null, “trainable”: true, “seed”: null, “name”: “dropout_2”}}, {“class_name”: “Conv2D”, “config”: {“kernel_constraint”: null, “kernel_initializer”: {“class_name”: “VarianceScaling”, “config”: {“distribution”: “uniform”, “scale”: 1.0, “seed”: null, “mode”: “fan_avg”}}, “name”: “conv2d_4”, “bias_regularizer”: null, “bias_constraint”: null, “activation”: “linear”, “trainable”: true, “data_format”: “channels_last”, “padding”: “same”, “strides”: [1, 1], “dilation_rate”: [1, 1], “kernel_regularizer”: null, “filters”: 128, “bias_initializer”: {“class_name”: “Zeros”, “config”: {}}, “use_bias”: true, “activity_regularizer”: null, “kernel_size”: [3, 3]}}, {“class_name”: “Activation”, “config”: {“activation”: “relu”, “trainable”: true, “name”: “activation_4”}}, {“class_name”: “BatchNormalization”, “config”: {“beta_constraint”: null, “gamma_initializer”: {“class_name”: “Ones”, “config”: {}}, “moving_mean_initializer”: {“class_name”: “Zeros”, “config”: {}}, “name”: “batch_normalization_4”, “epsilon”: 0.001, “trainable”: true, “moving_variance_initializer”: {“class_name”: “Ones”, “config”: {}}, “beta_initializer”: {“class_name”: “Zeros”, “config”: {}}, “scale”: true, “axis”: -1, “gamma_constraint”: null, “gamma_regularizer”: null, “beta_regularizer”: null, “momentum”: 0.99, “center”: true}}, {“class_name”: “Conv2D”, “config”: {“kernel_constraint”: null, “kernel_initializer”: {“class_name”: “VarianceScaling”, “config”: {“distribution”: “uniform”, “scale”: 1.0, “seed”: null, “mode”: “fan_avg”}}, “name”: “conv2d_5”, “bias_regularizer”: null, “bias_constraint”: null, “activation”: “linear”, “trainable”: true, “data_format”: “channels_last”, “padding”: “same”, “strides”: [1, 1], “dilation_rate”: [1, 1], “kernel_regularizer”: null, “filters”: 128, “bias_initializer”: {“class_name”: “Zeros”, “config”: {}}, “use_bias”: true, “activity_regularizer”: null, “kernel_size”: [3, 3]}}, {“class_name”: “Activation”, “config”: {“activation”: “relu”, “trainable”: true, “name”: “activation_5”}}, {“class_name”: “BatchNormalization”, “config”: {“beta_constraint”: null, “gamma_initializer”: {“class_name”: “Ones”, “config”: {}}, “moving_mean_initializer”: {“class_name”: “Zeros”, “config”: {}}, “name”: “batch_normalization_5”, “epsilon”: 0.001, “trainable”: true, “moving_variance_initializer”: {“class_name”: “Ones”, “config”: {}}, “beta_initializer”: {“class_name”: “Zeros”, “config”: {}}, “scale”: true, “axis”: -1, “gamma_constraint”: null, “gamma_regularizer”: null, “beta_regularizer”: null, “momentum”: 0.99, “center”: true}}, {“class_name”: “Conv2D”, “config”: {“kernel_constraint”: null, “kernel_initializer”: {“class_name”: “VarianceScaling”, “config”: {“distribution”: “uniform”, “scale”: 1.0, “seed”: null, “mode”: “fan_avg”}}, “name”: “conv2d_6”, “bias_regularizer”: null, “bias_constraint”: null, “activation”: “linear”, “trainable”: true, “data_format”: “channels_last”, “padding”: “same”, “strides”: [1, 1], “dilation_rate”: [1, 1], “kernel_regularizer”: null, “filters”: 128, “bias_initializer”: {“class_name”: “Zeros”, “config”: {}}, “use_bias”: true, “activity_regularizer”: null, “kernel_size”: [3, 3]}}, {“class_name”: “Activation”, “config”: {“activation”: “relu”, “trainable”: true, “name”: “activation_6”}}, {“class_name”: “BatchNormalization”, “config”: {“beta_constraint”: null, “gamma_initializer”: {“class_name”: “Ones”, “config”: {}}, “moving_mean_initializer”: {“class_name”: “Zeros”, “config”: {}}, “name”: “batch_normalization_6”, “epsilon”: 0.001, “trainable”: true, “moving_variance_initializer”: {“class_name”: “Ones”, “config”: {}}, “beta_initializer”: {“class_name”: “Zeros”, “config”: {}}, “scale”: true, “axis”: -1, “gamma_constraint”: null, “gamma_regularizer”: null, “beta_regularizer”: null, “momentum”: 0.99, “center”: true}}, {“class_name”: “MaxPooling2D”, “config”: {“name”: “max_pooling2d_3”, “trainable”: true, “data_format”: “channels_last”, “pool_size”: [2, 2], “padding”: “valid”, “strides”: [2, 2]}}, {“class_name”: “Dropout”, “config”: {“rate”: 0.25, “noise_shape”: null, “trainable”: true, “seed”: null, “name”: “dropout_3”}}, {“class_name”: “Flatten”, “config”: {“trainable”: true, “name”: “flatten_1”, “data_format”: “channels_last”}}, {“class_name”: “Dense”, “config”: {“kernel_initializer”: {“class_name”: “VarianceScaling”, “config”: {“distribution”: “uniform”, “scale”: 1.0, “seed”: null, “mode”: “fan_avg”}}, “name”: “dense_1”, “kernel_constraint”: null, “bias_regularizer”: null, “bias_constraint”: null, “activation”: “linear”, “trainable”: true, “kernel_regularizer”: null, “bias_initializer”: {“class_name”: “Zeros”, “config”: {}}, “units”: 512, “use_bias”: true, “activity_regularizer”: null}}, {“class_name”: “Activation”, “config”: {“activation”: “relu”, “trainable”: true, “name”: “activation_7”}}, {“class_name”: “BatchNormalization”, “config”: {“beta_constraint”: null, “gamma_initializer”: {“class_name”: “Ones”, “config”: {}}, “moving_mean_initializer”: {“class_name”: “Zeros”, “config”: {}}, “name”: “batch_normalization_7”, “epsilon”: 0.001, “trainable”: true, “moving_variance_initializer”: {“class_name”: “Ones”, “config”: {}}, “beta_initializer”: {“class_name”: “Zeros”, “config”: {}}, “scale”: true, “axis”: -1, “gamma_constraint”: null, “gamma_regularizer”: null, “beta_regularizer”: null, “momentum”: 0.99, “center”: true}}, {“class_name”: “Dropout”, “config”: {“rate”: 0.5, “noise_shape”: null, “trainable”: true, “seed”: null, “name”: “dropout_4”}}, {“class_name”: “Dense”, “config”: {“kernel_initializer”: {“class_name”: “VarianceScaling”, “config”: {“distribution”: “uniform”, “scale”: 1.0, “seed”: null, “mode”: “fan_avg”}}, “name”: “dense_2”, “kernel_constraint”: null, “bias_regularizer”: null, “bias_constraint”: null, “activation”: “linear”, “trainable”: true, “kernel_regularizer”: null, “bias_initializer”: {“class_name”: “Zeros”, “config”: {}}, “units”: 9, “use_bias”: true, “activity_regularizer”: null}}, {“class_name”: “Activation”, “config”: {“activation”: “softmax”, “trainable”: true, “name”: “activation_8”}}], “backend”: “tensorflow”}


#6

@vfintinari curiously after redeploying the app and retrying today, it seems to work! Thanks for the help.


#7

Additionally, @vfintinari, there seems to be a typo on the param filter_thresh on MovementBasedSegmenter

What image is handed to the model by the MovementBasedSegmenter? Is it the whole watcher area or is it a cutout of the contours it finds?


#8

Hi @Yves,
Glad it worked!

  1. It really is a typo. I will make sure that it is fixed. Thank you for letting us know.

  2. The image of the whole watcher area is forwarded to the model.


#9

The image of the whole watcher area is forwarded to the model.

So basically if I have a already classified square on the table, then add a new one, the MBS first detects the new square and computes its boundaries, verifies that it is indeed a new object and then calls the custom model with the image of the whole watcher area. I would then have to do the same process in my custom network to get the boundaries of the new object and then classify it.

Is this assumption correct? And if so, wouldn’t it make more sense to just call the custom model with just a cutout of the new object?


#10

Hi @Yves,

Here are some points to clarify the way MBS works:

  1. It detects the contour.

  2. It then sends to the neural network only a crop of the detected contour. This could be done in two ways:

    a) based on the size of the input of your neural network, cutting from the center of the contour half the size of your nn input size;

    b) based on a desired region of interest size. This could be done by providing a file named <name_of_you_nn>.config.json when you load your neural network, which contains the parameter region_of_interest_size. In this case the value of region_of_interest_size will be taken into account when the crop is being made and then resized to the size of your nn.

    An example of this <name_of_you_nn>.config.json:

    {
      "architecture": "",
      "region_of_interest_size": {
      "width": 175,
      "height": 175
      },
      "filters": {}
    }
    

    This will result in a crop that has a width and height of 175 pixels.

    If you wish to take into consideration the 60 cm Lampix configuration and the 90 cm configuration you can provide parameters height_1 and height_2, respectively. An example:

    {
      "architecture": "",
      "region_of_interest_size": {
        "height_1":{
          "width": 250,
          "height": 250
        },
        "height_2":{
          "width": 100,
          "height": 100
        }
      },
      "filters": {}
    }
    

    In other words, at 60 cm your crop will have 250 pixels in width and height and when placed at 90 cm 100 pixels.

    This is not officially documented but shall be after a future release.