SBT commands

http://www.scala-sbt.org/release/docs/Running.html

Here are some of the most common sbt commands. For a more complete list, see Command Line Reference.

clean	Deletes all generated files (in the target directory).
compile	Compiles the main sources (in src/main/scala andsrc/main/java directories).
test	Compiles and runs all tests.
console	Starts the Scala interpreter with a classpath including the compiled sources and all dependencies. To return to sbt, type :quit, Ctrl+D (Unix), or Ctrl+Z (Windows).
run *	Runs the main class for the project in the same virtual machine as sbt.
package	Creates a jar file containing the files in src/main/resources and the classes compiled from src/main/scala andsrc/main/java.
help	Displays detailed help for the specified command. If no command is provided, displays brief descriptions of all commands.
reload	Reloads the build definition (build.sbt, project/*.scala,project/*.sbt files). Needed if you change the build definition.

http://www.scala-sbt.org/release/docs/Basic-Def.html

In sbt shell, you can type the name of any task to execute that task. This is why typing compile runs the compile task. compile is a task key.

$ sbt
> tasks

This is a list of tasks defined for the current project.

It does not list the scopes the tasks are defined in; use the 'inspect' command for that.

Tasks produce values.  Use the 'show' command to run the task and print the resulting value.

  clean            Deletes files produced by the build, such as generated sources, compiled classes, and task caches.

  compile          Compiles sources.

  console          Starts the Scala interpreter with the project classes on the classpath.

  consoleProject   Starts the Scala interpreter with the sbt and the build definition on the classpath and useful imports.

  consoleQuick     Starts the Scala interpreter with the project dependencies on the classpath.

  copyResources    Copies resources to the output directory.

  doc              Generates API documentation.

  package          Produces the main artifact, such as a binary jar.  This is typically an alias for the task that actually does the packaging.

  packageBin       Produces a main artifact, such as a binary jar.

  packageDoc       Produces a documentation artifact, such as a jar containing API documentation.

  packageSrc       Produces a source artifact, such as a jar containing sources and resources.

  publish          Publishes artifacts to a repository.

  publishLocal     Publishes artifacts to the local Ivy repository.

  publishM2        Publishes artifacts to the local Maven repository.

  run              Runs a main class, passing along arguments provided on the command line.

  runMain          Runs the main class selected by the first argument, passing the remaining arguments to the main method.

  test             Executes all tests.

  testOnly         Executes the tests provided as arguments or all tests if no arguments are provided.

  testQuick        Executes the tests that either failed before, were not run or whose transitive dependencies changed, among those provided as arguments.

  update           Resolves and optionally retrieves dependencies, producing a report.

More tasks may be viewed by increasing verbosity.  See 'help tasks'.

SBT templates

http://www.scala-sbt.org/0.13/docs/sbt-new-and-Templates.html

Create a new sbt build directory tree using a template.

sbt new sbt/scala-seed.g8

SPARK SimpleApp

https://blog.knoldus.com/2014/06/04/a-simple-application-in-spark-and-scala/

sbt package

$SPARK_HOME/bin/spark-submit   --class "SimpleApp"   --master local[4]   target/scala-2.11/simple-project_2.11-1.0.jar

SBT tips and tricks from Stackoverflow

https://stackoverflow.com/questions/27710887/kafkautils-class-not-found-in-spark-streaming

SPARK: running the examples from https://spark.apache.org

https://spark.apache.org/docs/0.9.0/

./bin/run-example org.apache.spark.examples.SparkPi

./bin/run-example org.apache.spark.examples.streaming.KafkaWordCount

Usage: KafkaWordCount

SSH Shortcuts

http://www.phcomp.co.uk/Tutorials/Unix-And-Linux/ssh-check-server-fingerprint.html

https://help.github.com/articles/github-s-ssh-key-fingerprints/

To compare your local key's finger print against the github fingerprint:

ssh-keygen -l -E md5 -f ~/.ssh/joe_rsa

4096 MD5:7a:95:35:2e:e3:47:43:2a:4c:4d:69:51:9d:bb:83:12 joe.smith@any.com (RSA)

z = keras.layers.add([x, y]) NameError: name 'keras' is not defined

https://keras.io/getting-started/functional-api-guide/#more-examples

from keras.layers import Conv2D, MaxPooling2D, Input
#import keras

input_img = Input(shape=(3, 256, 256))

tower_1 = Conv2D(64, (1, 1), padding='same', activation='relu')(input_img)
tower_1 = Conv2D(64, (3, 3), padding='same', activation='relu')(tower_1)

tower_2 = Conv2D(64, (1, 1), padding='same', activation='relu')(input_img)
tower_2 = Conv2D(64, (5, 5), padding='same', activation='relu')(tower_2)

tower_3 = MaxPooling2D((3, 3), strides=(1, 1), padding='same')(input_img)
tower_3 = Conv2D(64, (1, 1), padding='same', activation='relu')(tower_3)

output = keras.layers.concatenate([tower_1, tower_2, tower_3], axis=1)

$ python concat.py Using TensorFlow backend.

Traceback (most recent call last):

  File "concat.py", line 15, in

    output = keras.layers.concatenate([tower_1, tower_2, tower_3], axis=1)

NameError: name 'keras' is not defined

#######################################

>>> keras

Traceback (most recent call last):

  File "", line 1, in

NameError: name 'keras' is not defined

>>> import keras

>>> keras

>>> 

#######################################

# Fix

from keras.layers import Conv2D, MaxPooling2D, Input
import keras # <<<<< Fix

input_img = Input(shape=(3, 256, 256))

tower_1 = Conv2D(64, (1, 1), padding='same', activation='relu')(input_img)
tower_1 = Conv2D(64, (3, 3), padding='same', activation='relu')(tower_1)

tower_2 = Conv2D(64, (1, 1), padding='same', activation='relu')(input_img)
tower_2 = Conv2D(64, (5, 5), padding='same', activation='relu')(tower_2)

tower_3 = MaxPooling2D((3, 3), strides=(1, 1), padding='same')(input_img)
tower_3 = Conv2D(64, (1, 1), padding='same', activation='relu')(tower_3)

output = keras.layers.concatenate([tower_1, tower_2, tower_3], axis=1)

Saving a model that does not import the backend as "K" breaks model loading. #5088

 https://github.com/fchollet/keras/issues/5088

airalcorn2 commented on Jan 19 • edited

The following code produces this error: NameError: name 'backend' is not defined. from keras import backend from keras.layers import Dense, Input, Lambda from keras.models import load_model, Model x = Input((100, ), dtype = "float32") x_i = Lambda(lambda x: x + backend.epsilon())(x) o = Dense(10, activation = "softmax")(x_i) model = Model(input = [x], output = o) model.compile("sgd", loss = "categorical_crossentropy") model.save("toy_model.h5") model = load_model("toy_model.h5")

 airalcorn2 changed the title from Saving a model that does not import the Keras backend as "K" breaks model loading. to Saving a model that does not import the backend as "K" breaks model loading. on Jan 19

Contributor

joelthchao commented on Jan 19

Quick solution: change backend import name, keras doesn't know your variable in lambda layer from keras import backend as K from keras.layers import Dense, Input, Lambda from keras.models import load_model, Model x = Input((100, ), dtype = "float32") x_i = Lambda(lambda x: x + K.epsilon())(x) o = Dense(10, activation = "softmax")(x_i) model = Model(input = [x], output = o) model.compile("sgd", loss = "categorical_crossentropy") model.save("toy_model.h5") model = load_model("toy_model.h5")

airalcorn2 commented on Jan 19 • edited

@joelthchao - thanks for trying to help, but I had already realized that changing the import name for backend to K would make the error go away (hence, my title). The problem is that Keras allows you to save a model using the actual name of the backend module but is unable to load such a model.

Contributor

bstriner commented on Jan 19

No quick fix but worth looking into. There are similar issues if you try to use some custom imports in your lambda. Another workaround is to do the import within the Lambda. def mylambda(x): from keras import backend import mymodule ... #use backend and mymodule y = Lambda(mylambda)(h)

"Merge" versus "merge", what is the difference? #3921

https://github.com/fchollet/keras/issues/3921

farizrahman4u commented on Sep 29, 2016 • edited

Merge is a layer. Merge takes layers as input Merge is usually used with Sequential models merge is a function. merge takes tensors as input. merge is a wrapper around Merge. merge is used in Functional API Using Merge: left = Sequential() left.add(...) left.add(...) right = Sequential() right.ad(...) right.add(...) model = Sequential() model.add(Merge([left, right])) model.add(...) using merge: a = Input((10,)) b = Dense(10)(a) c = Dense(10)(a) d = merge([b, c]) model = Model(a, d) Can you please confirm that the Keras 1.2.2 code from keras.engine import merge m = merge([init, x], mode='sum') is equivalent to this Keras 2.0.2 code: from keras.layers import add m = add([init, x])

CNN link's

https://cs231n.github.io/understanding-cnn/

https://colah.github.io/posts/2014-07-Understanding-Convolutions/

http://www.robots.ox.ac.uk/~vgg/practicals/cnn/#vgg-convolutional-neural-networks-practical

https://cs.stanford.edu/people/karpathy/ilsvrc/

http://neuralnetworksanddeeplearning.com

https://adeshpande3.github.io/adeshpande3.github.io/A-Beginner's-Guide-To-Understanding-Convolutional-Neural-Networks/

https://adeshpande3.github.io/adeshpande3.github.io/A-Beginner's-Guide-To-Understanding-Convolutional-Neural-Networks-Part-2/

https://adeshpande3.github.io/adeshpande3.github.io/The-9-Deep-Learning-Papers-You-Need-To-Know-About.html

https://gist.github.com/fchollet/0830affa1f7f19fd47b06d4cf89ed44d

SPARK RDD operations

How to print the contents of an RDD?

https://stackoverflow.com/questions/23173488/how-to-print-the-contents-of-rdd

down voteaccepted

If you want to view the content of a RDD, one way is to use collect(): myRDD.collect().foreach(println) That's not a good idea, though, when the RDD has billions of lines. Use take() to take just a few to print out: myRDD.take(n).foreach(println)

How to save the contents of an RDD to a single file?

https://stackoverflow.com/questions/23173488/how-to-print-the-contents-of-rdd

If you want to save in a single file, you can coalesce you RDD into one partition before calling saveAsTextFile, but again this may cause issues.  I think the best option is to write in multiple files in HDFS, then use hdfs dfs --getmerge in order to merge the files – Oussama Jul 21 '15 at 16:10

SPARK Shell

https://stackoverflow.com/questions/27717379/spark-how-to-run-spark-file-from-spark-shell

To load an external file from spark-shell simply do

:load PATH_TO_FILE

https://stackoverflow.com/questions/32808053/spark-shell-command-lines

scala> :help
All commands can be abbreviated, e.g. :he instead of :help.
Those marked with a * have more detailed help, e.g. :help imports.

:cp                  add a jar or directory to the classpath
:help [command]            print this summary or command-specific help
:history [num]             show the history (optional num is commands to show)
:h?                search the history
:imports [name name ...]   show import history, identifying sources of names
:implicits [-v]            show the implicits in scope
:javap <path|class>        disassemble a file or class name
:load                load and interpret a Scala file
:paste                     enter paste mode: all input up to ctrl-D compiled together
:quit                      exit the repl
:replay                    reset execution and replay all previous commands
:reset                     reset the repl to its initial state, forgetting all session entries
:sh <command line>         run a shell command (result is implicitly => List[String])
:silent                    disable/enable automatic printing of results
:fallback                  
disable/enable advanced repl changes, these fix some issues but may introduce others. 
This mode will be removed once these fixes stablize
:type [-v]           display the type of an expression without evaluating it
:warnings                  show the suppressed warnings from the most recent line which had any

As you can see above you can invoke shell commands using :sh. For example:

scala> :sh mkdir foobar
res0: scala.tools.nsc.interpreter.ProcessResult = `mkdir foobar` (0 lines, exit 0)

scala> :sh touch foobar/foo
res1: scala.tools.nsc.interpreter.ProcessResult = `touch foobar/foo` (0 lines, exit 0)

scala> :sh touch foobar/bar
res2: scala.tools.nsc.interpreter.ProcessResult = `touch foobar/bar` (0 lines, exit 0)

scala> :sh ls foobar
res3: scala.tools.nsc.interpreter.ProcessResult = `ls foobar` (2 lines, exit 0)

scala> res3.lines foreach println
bar
foo

Python Numpy Slices

>>> np.arange(12)
array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11])
>>> np.arange(12).reshape(3,4)
array([[ 0,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11]])
>>> a = np.arange(12).reshape(3,4)
>>> a[slice(None, 3, None)]
array([[ 0,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11]])
>>> a[:3]
array([[ 0,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11]])
>>> b = np.arange(24).reshape(3,4,2)
>>> b
array([[[ 0,  1],
        [ 2,  3],
        [ 4,  5],
        [ 6,  7]],

       [[ 8,  9],
        [10, 11],
        [12, 13],
        [14, 15]],

       [[16, 17],
        [18, 19],
        [20, 21],
        [22, 23]]])
>>> b[2:]
array([[[16, 17],
        [18, 19],
        [20, 21],
        [22, 23]]])
>>> b[slice(2, None, None)]
array([[[16, 17],
        [18, 19],
        [20, 21],
        [22, 23]]])
>>> b[2:,3]
array([[22, 23]])
>>> b[2:,1:]
array([[[18, 19],
        [20, 21],
        [22, 23]]])
>>> b[2:,:1]
array([[[16, 17]]])
>>> b[slice(2, None, None), slice(None, 1, None)]
array([[[16, 17]]])