First of all, thank you for a great library!

I tried to use sparkdl in PySpark, but couldn't import sparkdl. Detailed procedure is as follows:

# make sparkdl jar
build/sbt assembly

# run pyspark with sparkdl
pyspark --master local[4] --jars target/scala-2.11/spark-deep-learning-assembly-0.1.0-spark2.1.jar

# import sparkdl
import sparkdl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named sparkdl

After digging a few places, I found that it works if I deflate the jar file as follows.

cd target/scala-2.11
mkdir tmp
cp spark-deep-learning-assembly-0.1.0-spark2.1.jar tmp/
cd tmp
jar xf spark-deep-learning-assembly-0.1.0-spark2.1.jar

pyspark --jars spark-deep-learning-assembly-0.1.0-spark2.1.jar

import sparkdl
Using TensorFlow backend.

Edited-1 : The second method works only in the directory where the jar file is deflated.

Best wishes, HanCheol

What changes are proposed in this pull request?

Creating a Spark MLlib Estimator API for Keras models, with a reference implementation. It provides a taste of how to ingest Image from URI in a DataFrame and use them to train a Keras model.

The changes consist of these components.

1. Extracted a few Params types for Keras Transformers/Estimators.
2. Keras utilities
   • Serialization: model <=> hdf5 <=> bytes (for broadcast)
   • Check avaialble Keras options (optimizers, loss functions, etc.)
3. Keras Estimator.

How is this patch tested?

• [x] Unit tests
• [x] Manual tests

Hi,

I am trying to use this library in jupyter notebook, but I am getting error "no module found".

When I am running the below command pyspark --packages databricks:spark-deep-learning:1.5.0-spark2.4-s_2.11 I am able to import sparkdl in the spark shell.

How can I use it in jupyter notebook?

• bump spark version to 2.3.1
• bump tensorframes version to 0.4.0
• bump keras==2.2.2 and tensorflow==1.10.0 to fix travis issues
• TF_C_API_GRAPH_CONSTRUCTION added as a temp fix
• Drop support for Spark <2.3 and hence Scala 2.10
• add python3 friendly print
• add pooling='avg' in resnet50 testing model beccause keras api changed
• test arrays almost equal with whatever precision 5 in NamedImageTransformerBaseTestCase, test_bare_keras_module, keras_load_and_preproc
• make keras model smaller in test_simple_keras_udf

This is a continued work from https://github.com/databricks/spark-deep-learning/pull/149.

• add test for KerasImageFileEstimator used with CrossValidator
• fix pickling issue
• use new fitMultiple api and ensure thread safety
• fix tests to reflect new api
• bugfix setDefault
• bugfix HasOutputMode
• remove _validateParams
• avoid testing KIFT functionality in KIFEst tests

Use Spark 2.3's ImageSchema as image interface.

• the biggest change is using opposite ordering of color channels - BGR instead of RGB, requires extra reordering in couple of places. -preserved ability to read and resize images in python using PIL to match Keras (resize gives different result but also reading jpegs produced images which were off by 1 on some green pixels)
• needed few tweeks to run with spark 2.3 - notably UDFs are now referenced by SQL identifier and can not have dash as part of the name

[TODO] - In order to run on spark < 2.3, the image schema files have been copied here and need to be removed in the future.

This is the first part of PRs from the TF Transformer API design POC PR.

It introduces parameters needed for the TFTransformer (minus those for TFInputGraph) and corresponding type converters.

• Introducing MLlib model Params for TFTransformer.
• Type conversion utilities for the introduced MLlib Params used in Spark Deep Learning Pipelines.
  • We follow the convention of MLlib to name these utilities "converters", but most of them act as type checkers that return the argument if it is the desired type and raise TypeError otherwise.
  • We follow the practice that a type converter only returns a single type.

In this PR, we

• add python/.pylint/suggested.rc adapted from the default configuration generated by pylint
• allow both camelCase and snake_case using regexes lifted from pylint source code
• increase thresholds for number of arguments, local, variables
• disable checks that are used often in this project: unused-argument, too-many-arguments, no-member, missing-docstring, no-init, protected-access, misplaced-comparison-constant, no-else-return, fixme
• escape some code with # pylint: disable=... because it was hard to refactor without thorough testing

Some style decisions that were discussed are:

• disables are acceptable if there is no other way to do this, in which case a comment must be left explaining that
• other disables should be removed and should be considered similar to todos
• we allow todo marks in code because these are acceptable for this project and should be taken care of in future
• there are 50 todos, fixmes or pylint disables currently, we should aim to bring this down find python/sparkdl | grep ".*\.py$" | xargs egrep -ino --color=auto "(TODO|FIXME|# pylint).*"
• function calls and function defintions that span more than 1 line are left to committer and reviewer's discretion
  • pep8 style:

  long_function_name(
      long_argument_one = "one",
      long_argument_two = "two",
      long_argument_three = "three",
      long_argument_four = "four",
      long_argument_five = "five")
  

  • MLlib style:

  long_function_name(
      long_argument_one = "one", long_argument_two = "two", long_argument_three = "three",
      long_argument_four = "four", long_argument_five = "five")
  

Fix bug in conversions to/from BufferedImage in which we copied raw byte data to BufferedImage rasters using the wrong channel ordering. The fix in this PR is to use BufferedImage.setRGB, BufferedImage.getRGB APIs instead of accessing image raster data directly for three and four-channel images.

Also enhanced an existing unit test to verify that we correctly convert from row image to BufferedImage for one and four-channel images.

This PR:

• Updates conversions from row image to/from Buffered image (spImageToBufferedImage and spImageFromBufferedImage) to support one, three, and four channel images
• Updates resizeImage to use the tgtChannels parameter to determine the number of channels in the output image instead of defaulting to three output channels
• Updates existing tests to verify that resizing, conversions to/from BufferedImage work for one, three, and four-channel images

• Based of Image schema PR, do not merge until Image schema is merged.
• Otherwise mostly straightforward except results will not match keras in general due to different image libraries

I am running the following code to try to fit a model

from sparkdl.xgboost import XgboostClassifier
param = {
    'num_workers': 4, # number of workers on the cluster, adjust as needed
  'missing': 0,
    "objective": "binary:logistic",
    "eval_metric": "logloss",
      'featuresCol':"features", 
      'labelCol':"objective",
      'nthread':32 # equal to the number of cpus on each worker machine
}
  
train, test = data.randomSplit([0.001, 0.001])
xgb_classifier = XgboostClassifier(**param)
xgb_clf_model = xgb_classifier.fit(train)

When I run the model training on my databricks cluster is seems to be getting stuck when it is trying to map partitions. It is using almost zero cpu on each cluster but the memory usage is slowly increasing.

is there anything I can do to get around this issue

Hi,

Per this overflow question one needs to modify /home/user/.local/lib/python3.8/site-packages/sparkdl/transformers/keras_applications.py . This happened in databricks using v 10.3.

Have to change from keras.applications import inception_v3, xception, resnet50

to

from keras.applications import inception_v3, xception from tensorflow.keras.applications import resnet50

https://github.com/hongzimao/decima-sim would you like to help me to integrate this deep learning model into your pipeline> how can I integrate or plug it with your frameworks?

Hi,

I'm developing a neural network using Pytorch in a non-databricks cluster to ensure its functionality prior migrating to a databricks cluster.

Since I'm using Pytorch, I don't need Keras or TensorFlow. I installed successfully Horovod and Sparkdl, however, when I try to run the Spark process I found (for now) three consecutive exceptions related to missing dependencies:

    from sparkdl import HorovodRunner
  File "/opt/conda/default/lib/python3.8/site-packages/sparkdl/__init__.py", line 17, in <module>
    from sparkdl.transformers.keras_image import KerasImageFileTransformer
  File "/opt/conda/default/lib/python3.8/site-packages/sparkdl/transformers/keras_image.py", line 16, in <module>
    import keras.backend as K
  File "/opt/conda/default/lib/python3.8/site-packages/keras/__init__.py", line 21, in <module>
    from tensorflow.python import tf2
ModuleNotFoundError: No module named 'tensorflow'

    from sparkdl import HorovodRunner
  File "/opt/conda/default/lib/python3.8/site-packages/sparkdl/__init__.py", line 17, in <module>
    from sparkdl.transformers.keras_image import KerasImageFileTransformer
  File "/opt/conda/default/lib/python3.8/site-packages/sparkdl/transformers/keras_image.py", line 16, in <module>
    import keras.backend as K
ModuleNotFoundError: No module named 'keras'

This one is DEPRECATED!!:

    from sparkdl import HorovodRunner
  File "/opt/conda/default/lib/python3.8/site-packages/sparkdl/__init__.py", line 17, in <module>
    from sparkdl.transformers.keras_image import KerasImageFileTransformer
  File "/opt/conda/default/lib/python3.8/site-packages/sparkdl/transformers/keras_image.py", line 27, in <module>
    from sparkdl.transformers.tf_image import TFImageTransformer
  File "/opt/conda/default/lib/python3.8/site-packages/sparkdl/transformers/tf_image.py", line 18, in <module>
    import tensorframes as tfs
ModuleNotFoundError: No module named 'tensorframes'

On one hand, I don't understand why should I need these dependencies if I'm not going to use them... Shouldn't it be checked and disabled instead of forcing it to be installed?

On the other hand, if those dependencies are unavoidable, they should be included in the setup.py script to avoid having these errors and losing time, since installing Horovod packages in an ephemeral cluster takes a lot of time just to discover that you cannot run the program...

I'm sure I won't have a problem in a Databricks cluster, but I cannot use it yet and that shouldn't be a problem to test HorovodRunner functionality as stated in the warning message when running a program in a non-databricks cluster...

Kind regards

I find it so many ‘spark-deep-learning’, such as : elephas：https://github.com/maxpumperla/elephas dist-keras：https://github.com/cerndb/dist-keras sparknet：https://github.com/amplab/sparknet dl4j：https://github.com/deeplearning4j/dl4j-spark-ml TensorFlowOnSpark：https://github.com/yahoo/TensorFlowOnSpark spark-deep-learning：https://github.com/databricks/spark-deep-learning H2O：https://github.com/h2oai/sparkling-water/tree/master/ BigDL：https://github.com/intel-analytics/BigDL analytics-zoo：https://github.com/intel-analytics/analytics-zoo

It looks like BigDL is the most active one. I want to start my DeepLearning on spark by using spark-deep-learning, but I afraid others will popular than databricks.spark-deep-learning. So I still hesitate which one to choice.