https://janav.wordpress.com/2013/10/27/tf-idf-and-cosine-similarity/
From:
http://stackoverflow.com/questions/24011418/apache-spark-naive-bayes-based-text-classification
From:
http://stackoverflow.com/questions/24011418/apache-spark-naive-bayes-based-text-classification
For example i have two types of Class (C and J)
Train data is :
C, Chinese Beijing Chinese
C, Chinese Chinese Shanghai
C, Chinese Macao
J, Tokyo Japan Chinese
And test data is : Chinese Chinese Chinese Tokyo Japan // What is ist J or C ?
...
But you can do it manually by first creating a dictionary of terms. Then compute IDFs for each term and then convert each documents into vectors using the TF-IDF scores.
...
There any many classification methods (logistic regression, SVMs, neural networks,LDA, QDA...), you can either implement yours or use MLlib classification methods (actually, there are logistic regression and SVM implemented in MLlib)
What you need to do is transform your features to a vector, and labels to doubles.
For examples, your dataset will look like:
1, (2,1,0,0,0,0)
1, (2,0,1,0,0,0)
1, (1,0,0,1,0,0)
0, (1,0,0,0,1,1)
And tour test vector:
(3,0,0,0,1,1)
From:
From:
Another method for creating recs with Spark is the search engine method. This is basically a cooccurrence recommender served by Solr or Elasticsearch. Comparing factorized to cooccurrence is beyond this question so I'll just describe the latter.
You feed interactions (user-id,item-id) into Mahout's
spark-itemsimilarity. This produces a list of similar items for every item seen in the interaction data. It will come out by default as a csv and so can be stored anywhere. But it needs to be indexed by a search engine.
...
One other benefit of either factorization or the search method is that entirely new users and new history can be used to create recs where the older Mahout recommenders could only recommend to users and interactions known when the job was run.
Descriptions here:
http://stackoverflow.com/questions/30089675/clustering-cosine-similarity-matrix?rq=1
http://stackoverflow.com/questions/30089675/clustering-cosine-similarity-matrix?rq=1
https://en.wikipedia.org/wiki/Tf%E2%80%93idf
http://stackoverflow.com/questions/30168166/using-k-means-for-document-clustering-should-clustering-be-on-cosine-similarity?rq=1
http://stackoverflow.com/questions/12120780/euclidean-distance-or-cosine-similarity?rq=1
http://stackoverflow.com/questions/30089675/clustering-cosine-similarity-matrix?rq=1
https://en.wikipedia.org/wiki/Tf%E2%80%93idf
http://stackoverflow.com/questions/30168166/using-k-means-for-document-clustering-should-clustering-be-on-cosine-similarity?rq=1
http://stackoverflow.com/questions/12120780/euclidean-distance-or-cosine-similarity?rq=1
