WHY RECOMMENDER SYSTEMS ?

Due to enormous amount of data present in a web application or shopping carts, recommendations are used to engage users and helps in saving good amount of time to look for similar products which were interested by them. Moreover, Recommender systems helps in maximizing the profits of an organization by suggesting relevant items which might be purchased by the user.

DATA SET USED

Data set has a major role to play in building recommender systems. Data set shouldn’t contain null or empty tuples. For this project we are using beer review data from social website (Beer Advocate). This data set has ratings for each beer in different aspects such as aroma, appearance, taste, palate & overall rating with 1,586,614 records.

• Dataset-Source

ALGORITHM AND IMPLEMENTATION USING PYTHON & SPARK

USER-USER RECOMMENDATION (FIRST COLLABORATIVE ALGORITHM):

• For this recommender system we are using Matrix factorization method with stochastic gradient. The model is used to estimate matrix which approximates to rating matrix (input to the model).
  

• In collaborative filtering , data about the ratings by the user and by other users is leveraged to recommend products. For instance , if a user 'A' rates a product high and some other user'B' also rates the same product high , then it is possible that 'A' may probably like the other products that are highly rated by 'B' and vice-versa.
  
  FORMULA:


R is the rating matrix (This matrix is formed when the user inputs the preferences).
P is a latent feature matrix of the user and features (aroma, taste, appearance, palate, overall rating).
Q is latent feature matrix of Beers and features.
R` is built with matrix factorization.



DATA PRE-PROCESSING -
• BeerId: This column had the unique identifiers given to each beer but the problem with these identifiers was , they were not serialized. The total number of unique beers was 21,467 were as the identifiers were as large as 65,789. using this identifiers could drastically increase the computation of the model.so new identifiers were assigned according to beerID ranging from 0 – 21,466.

• userId: same was the case with this column , so new identifiers were assigned according to userId ranging from 0 -47345. review_appearance: this column had appearance rating given by users, review_aroma:this column had aroma rating given by users, review_palate:this column had palate rating given by users, review_taste:this column had taste rating given by users, review_overall:this column had overall rating given by users

STEP 2 -
• Users are asked to give their preference about taste, palate, appearance and weights are calculated based on the it.

• Rating(weighted) = Rating (overall)/2 +{Rating (Aroma) * weight of aroma + Rating (taste) * weight of taste + Rating (appearance) * weight of appearance} /(sum of all the weights * 2)

STEP 3 -
• Implementation of prediction method: using the biases , prediction for each row is calculated by multiplying a row in P matrix with column in Q matrix.

• Implementation of stochastic gradient method : Prediction is used and error is calculated which is then used to update the biases , P and Q matrices.

• Implementation of complete matrix method: which returns complete matrix by multiplying P and Q.

STEP 4 -
• Implementation square error method : calculates and returns the error between predicted matrix from the complete matrix function and known rating matrix for each iteration.

• Implementation of Train function : this function calls all the above mentioned method for a specified number of iterations.

• Implementation of get_user_recommendation method: this methods takes the output of the above model (in form of a list) and retrieve the names of the beers and prints them to the console.

RESULTS -
• Here are some of the test cases results displayed by the system -

• We parse the arguments in such a way that :
command : python user-user.py johnmichaelsen 2 2 1 2

• The above command display's the recommendations for the user whose profilename is "johnmichaelsen"








ITEM-ITEM RECOMMENDATION (SECOND COLLABORATIVE ALGORITHM):

Item -Item collaborative filtering is one such recommendation techniques used to recommend items similar to those which were rated by the user or interested by the user in the past. This method is similar to the user-user collaborative filtering technique where the similarity between items is measured and predictions were made to those items for a particular user and analyze whether he is interested .



DATA PRE-PROCESSING -
• The data which we chose is beer-data set to build the Recommender system. Though there are many features in the data, we use only the informative ones such as review_overall, review_taste etc. We eliminated other subjective information like brewery name, usernames,beer-style etc.

• Generate unique id's for beers and users using the groupBy method in python and created a new data file.

• Once the data is ready to facilitate the purpose of distributed computing in spark we hosted the data in the cluster.

STEP 2 -
• Make user beer pairs which include ratings. (user, beer name, rating given).

• This rating is aggregate of all the ratings (aroma, appearance, palate, taste) . Rating is calculated in such a way that user need to define their preferences. We assumed that overall rating is given preference by every user and preferences for other features are to be passed by user.





• The above shows some probable scenarios for user input preferences. Then the overall weighted sum of rating is calculated using the below method: Overall weighted rating (for case1)= (2*(review_overall)+2*(review_taste)+1*(review_appearance)+1*(review_aroma)+2*(review_palate))/2+2+1+1+2

STEP 3 (FORMING SIMILARITY PAIRS) -
• Once we have all the user beer pairs we need to form similarity groups for all the beers which were presented in the data set.

• For each user combinations of the items he rated are made and store their ratings as (beer1,beer2) , (rating1,rating2)

• Once for each user the similarity pairs are formed we calculate the similarity using cosine similarity.

• The computational formula for cosine similarity is.




here {i= 1,2,3,....n} which is user's who rated both A and B

STEP 4 (GENERATING RECOMMENDERS) -
• Once the similarity score is formed, we need to build the predictions of beers which were not rated by every user.

• For this, we choose the similarity pair of beer which is rated by the user and check whether the beers in similarity group are rated by user.

• If not, then we compute the predicted rating and store them. This process is used for all the respective similarity groups of beers rated by user.

• Once we are done with all the beers rated by the user, we know to sort the results based on predicted ratings.

• These beers which have highest predicted ratings are most recommended to a user.

• The computational formula of predicted ratings for an item of the user is :




Here S is the similarity score of Item u with item N
R is the rating given by the user u for item N
N can be subset of items which have some similarity score with respect to item i


RESULTS -
• Here are some of the test cases results displayed by the system -

• We parse the arguments in such a way that :
command : spark-submit recommender.py /user/amekala/beer_input/beer_data.csv beer_items.txt 23203 2 1 2 1

• The above command display's the recommendations for the user with id '23203' whose profilename is "northyorksammy"






• You can observe when the user preferred aroma and taste rather than other features then /she are recommended some set of beers whose integrated rating is more. The below results show the beers when the user prefers appearance and taste rather than pallate and aroma. You can observe the results are different as the beers have a different integrated rating with respect to user preferences.

• command : spark-submit recommender.py /user/amekala/beer_input/beer_data.csv beer_items.txt 23203 1 2 1 2








COMPARISON OF MODELS

• Test cases are chosen based on the number of reviews. All the users in these test cases have more than 100 reviews in the dataset.

1. Item-Item recommendations: For user 'northyorksammy' with the weights 1 1 1 1 :
User-User Recommendations:




2. Item-Item recommendations: For user 'mikesgroove' with the weights 1 1 1 1 :


User-User Recommendations:



As seen in the above comparison results, The recommendations from these two models have no or very few matches. This mainly has to do with the way these models are designed and the factors that are taken into consideration to give recommendations.