Exploratory data analysis coursera quiz 1 answers

[Pages:2]Continue

Exploratory data analysis coursera quiz 1 answers

Improve data science skills and jump on a career with Just into Data Tutorials + Applications.This is a tutorial on using the seaborn library in Python for Exploratory Data Analysis (EDA). EDA is another critical process in data analysis (or machine learning/statistical modeling), in addition to Data Cleaning in Python: Ultimate Guide (2020). In this guide, you'll discover (with examples): How to use the seaborn Python pack to produce useful and beautiful visualizations, including histograms, bar plots, scatter plots, box plots, and heatmaps. How to explore univariate, multivariate numeric and categorical variables with different parcels. Here's how to discover the relationships between multiple variables. Much more. Let's get started! What is exploratory Data Analysis (EDA) and why? Exploratory Data Analysis (EDA) is a method of analyzing data sets to summarize their key properties, often using visual methods. A statistical model can be used or not, but primarily EDA is to see what the data can tell us beyond the formal modeling or hypothesis testing task. It is important to explore the data before further analysis or modeling. Within this process we can get an overview of the insights from the dataset; we can detect trends, patterns and relationships that are not immediately apparent. What is seaborn? Seaborn: Statistical data visualization is a popular Python library to perform EDA. It is based on matplotlib and provides a high-level interface for drawing attractive and informative statistical graphics. In this post, we use a scraped and cleaned YouTube dataset as an example. In our previous article How to get more YouTube views with Machine Learning techniques, we've made recommendations on how to get multiple views based on the same dataset. Before exploring, let's read the data in Python as data sets df #import packages import pandas as pd import numpy as np import json import datetime import mathematics from datetime import timedelta, datetime import matplotlib.pyplot as plt import matplotlib.mlab as mlab import matplotlib plt.style.use('ggplot') from matplotlib.pyplot import figure %matplotlib inline matplotlib.rcParams['ggplot') from matplotlib.pyplot import figure %matplotlib inline matplotlib.rcParams['ggplot') from matplotlib.pyplot import figure %matplotlib inline matplotlib.rcParams[[[ggplotlib]['ggplot') from matplotlib.pyplot import figure %matplotlib inline matplotlib.rcParams[['ggploplot') from matplotlib.pyplot import figure %matplotlib inline matplotlib.rcParams[['ggploplot') from matplotlib.pyplot import figure %matplotlib inline matplotlib.rcParams[[ggplot')['ggplot') from 'figure.figsize'] = (12.8) pd.options.mode.chained_assignment = No one imports seaborn as sns # read the data df = pd.read_pickle('sydney.pkl')df contains 729 rows and 60 variables. It records different features for each video in Sydney's YouTube channel, such as:views: the number of views of video length: the length of video/workout in minutes calories: the number of calories burned during training in the number of days since the video was posted until nowdate: the date when the video/workout was sentSydney posts a video/workout almost every dayworkout_type: the type of workout video was focusing onAgain, you can find more details in How to get more YouTube views with Machine Learning techniques. We're just using this data set here. Univariate Analysis: Numeric VariableFirst, Let's Explore Explore univariate variables. We create df_numeric only include the 7 numeric features.df_numeric = df.select_dtypes (include='number') df_numericHistogram: Single VariableHistograms is one of our favorite plots. A histogram is an approximate representation of the distribution of numeric data. If you want to construct a histogram, the first step to the bin (or bucket) range is the value range, i.e. the value range. Seaborn's feature distplot has options for: trash optionsIt is useful to plot the variable with different bucket settings to detect patterns. If we do not specify this value, the library will find a useful standard for U.S. work: whether to plot a Gaussian core density estimateThis helps assess the shape of probability density function of a continuous random variable. More details can be found on seaborn's side.rye: about drawing a carpet plot on the support axisIt draws a small vertical cross at each sighting. It helps to know the exact location of the values for the variable. Let's start by looking at a single variable: length, which represents the length of video.sns.distplot (df_numeric['length'], bins =50, kde = True, rye = True) We can see both the kde line and the rug sticking in the plot below. The videos for Sydney's channel often have a length of 30, 40 or 50 minutes, giving a multimodal pattern. Histogram: Multiple variablesOften, we want to visualize multiple numeric variables and look at them together. We build the function plot_multiple_histograms to plot histograms for a specific group of variables.# this plots multiple sea-born histograms on different subplots. # def plot_multiple_histograms(df, cols): num_plots = len(cols) num_cols = math.ceil(np.sqrt(num_plots)) num_rows = math.ceil(num_plots/num_cols) fig, axs = plt.subplots(num_rows, num_cols) for ind, col in enumerate(cols): i = math.floor(in/num_cols) j = ind - i*num_cols if num_rows == 1: if num_cols == 1: sns.distplot(df[col], kde=True, ax=axs) otherwise: sns.distplot(df[col], kde=True, ax=axs[j]) else: sns.distplot(df[col], kde=True, ax=axs[i, (j]) plot_multiple_histograms(df, ['length', 'views', 'calories', 'days_since_posted'])We can see that different variables show different kinds of distributions, deviant values, bias, etc. Univariate Analysis: Categorical variablesNext, let's look at categorical univariate variables. Bar chart: Single variableA bar chart (or countplot in seaborn) is the categorical variable's version of the histogram. A bar chart or bar plot is a chart or graph that presents categorical data with rectangular bars with heights or lengths proportional to the values they represent. A bar chart shows comparisons between separate categories. First, let's select the categorical (non-numeric) variables. We plot the bar chart for the variable area that represents the body areas that the training video focuses on. #select non-numeric variables = df.select_dtypes(exclude='number') plt.figure(figsize=(25,7)) sns.countplot(x=area, data=df_non_numeric)There are many areas that the videos are targeted to. It's hard to read without zooming in. Still, we can see that more than half (over 400) of these videos focused on full body area; and the other most popular area focused on is ab. Bar chart: Several variablesAlso, we also create a function plot_multiple_countplots plotting the bar charts for multiple variables at once. We use it to plot some indicator variables below. The is_{}_area are indicator variables for different body areas. For example, is_butt_area == True when the workout focuses on butt, otherwise it is False.The is_{}_workout are indicator variables for different training types. For example, is_strength_workout == True, when the training focuses on strength, otherwise it is False.# this plots several seaborn countplots on different subplots. # def plot_multiple_countplots(df, cols): num_plots = len(cols) num_cols = math.ceil(np.sqrt(num_plots)) num_rows = math.ceil(num_plots/num_cols) fig, axs = plt.subplots(num_rows, num_cols) for in, col in enumerate(cols): i = math.floor(in/num_cols) j = ind - i*num_cols if num_rows == 1: if num_cols == 1: sns.countplot(x=df[col], ax=axs) other: sns.countplot(x=df[col], ax=axs[j]) else: sns.countplot(x=df[col], ax=axs[i, (j]) plot_multiple_countplots(df_non_numeric, ['is_butt_area', 'is_upper_area', 'is_cardio_workout', 'is_strength_workout']) Multivariate AnalysisAfter exploring the variables one by one, let's look at several variables together. Different plots can be used to explore relationships between different combinations of variables. In the last section, you can also find a modeling method for testing relationships between multiple variables. Scatter plot: Two numerical variablesFirst, let's see how we can detect the relationship between two numerical variables. What if we want to know how workout length affects the number of views? We can use scatterplots (relplot) to answer the question. A point plot uses Cartesian coordinates to display values for typically two variables for a data set. If the points are encoded (color/shape/size), an additional variable may appear. The data is displayed as a collection of points, each of which has the value of a variable that determines the position of the horizontal axis and the value of the second variable that determines the position on the vertical axis. sns.relplot(x='length', y='views', data=df, aspect=2.0)We can see that the more popular videos tend to have lengths between 30 and 40 minutes. Bar Chart: Two categorical variablesWhat if we want to know the relationship between two categorical variables? Let's visualize the most common 6 areas (area2) and the most common 4 training types (workout_type2) in 6 = list(df['area'].value_counts().index[:5]) df['area2'] = df['area'] msk =['area2'].isin(top6) df.loc[msk, 'area2'] 'Other' top4 = = df['workout_type2'] = df['workout_type'] msk = df['workout_type2'].isin(top4) df.loc[~msk, 'workout_type2'] = 'Other'order = df['area2'].value_counts().index # order the columns from highest number to lowest. sns.catplot(x=workout_type2, col='area2', col_order=order, kind=count, data=df, aspect=0.5)We can see that full body strength training is the most common in the videos. Box plot: Numeric and categorical variablesBox ranges are useful visualizations when comparing groups of categories. A box plot (box-and-whisker plot) is a standardized way to display the dataset based on a five-digit summary: minimum, maximum, sample median, and the first and third quartiles. We can use boxplots side by side to compare a numeric variable between categories of a categorical variable. Do Sydney's videos get more views on certain days of the week? Let's plot day_of_week and views. to_replace = {0:'Monday', 1:'Tuesday', 2:'Wednesday', 3:'Thursday', 4:'Friday', 5:'Saturday', 6:'Sunday'} df['day_of_week_num'] = df['date'].dt.dayofweek df['day_of_week'] = df['day_of_week_num'].replace(to_replace=to_replace) order = ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday'] sns.boxplot(x=day_of_week, y=views, data=df, order=order)This is interesting but hard to see because of outliers. Let's get rid of them. msk = df['views'] < 400000 sns.boxplot(x=day_of_week, y=views, data=df[msk], order=order)We can see that Monday videos have more views than other days. While Sunday videos get the least views. Swarmplot: Numerical and categorical variablesA different way of looking at the same question is with a swarm plot. A swarm lot is a categorical scatterplot where the points are adjusted (only along the categorical axis) so that they do not overlap. This provides a better representation of the distribution of values. A swarm plot is a good addition to a box plot when we want to show all observations along with some representation of the underlying distribution. sns.swarmplot(x=day_of_week, y=views, data=df[msk], order=order)A swarm plot would have too many dots for larger data sets, but it's good here with a smaller data set. Boxplot Group: Numerical and categorical variablesIs Are the views on certain days of the week higher for certain types of training? To answer this question, these are two categorical variables (workout_type, day_of_week) and a numeric variable (views). Let's see how we can visualize the answer to this question. We can use a panel boxplot (catplot) to visualize the three variables together. Catplot is useful for displaying the relationship between a numeric and one or more categorical variables using one of several visual representations. sns.catplot(x=workout_type, y=views, col=day_of_week, aspect=.6, kind=box, data=df[msk], col_order=order); It's pretty messy with too many categories of workout_type. Based on the distribution of workout_type, we group the categories other than hiit, hiit, together as 'Other'. df['workout_type'].value_counts()top4 = list(df['workout_type'].value_counts()index[:3]) df['workout_type2'] = df['workout_type'] msk = df['workout_type2'].isin(top4) df.loc[~msk, 'workout_type2'] = 'Other'Also we remove outliers to make the plot even more clear.msk = df['views'] < 400000 sns.catplot(x=workout_type2, y=views, col=day_of_week, kind=box, data=df[msk], col_order=order, aspect=0.5)We can feel things like:stretch workouts are only laid out on Sundays. hiit training seems to have multiple views on Mondays.Heatmap: Numeric and categorical VariablesWe can also use PivotTables and heatmaps to visualize multiple variables. A heat map is a data visualization technique that shows the extent of a phenomenon such as color in two dimensions. The variation in color can be of nuance or intensity, giving obvious visual cues to the reader about how the phenomenon is grouped or varies across space. The heatmap below has workout_type.g. The gamut represents views in each cell.df_area_workout = df.groupby(['region', 'workout_type']]]['views'].count().reset_index() df_area_workout_pivot = df_area_workout.pivot(index='area' columns='workout_type', values='views').fillna(0) sns.heatmap(df_area_workout_pivot, annot=True, fmt='.0f', cmap=YlGnBu)(Advanced) Relationship Test and Scatterplot: Numerical and Categorical VariablesHow do we automatically detect the relationships between multiple variables? Let's take the most critical features below and see how we could find interesting relationships.# group of critical features selected cols = ['length', 'views', 'calories', 'days_since_posted', 'area', 'workout_type', 'day_of_week'] df_test = df[cols] df_test.head() numeric_columns = set(df_test.select_dtypes(include=['number']).columns) non_numeric_columns = sets(df_test.columns) - numeric_columns print(numeric_columns) print(non_numeric_columns)We have 4 numeric variables and 3 category variables. There can be many complicated relationships between them! In this section, we use the same method to test for relationships (including multicollinearity) among those who in How to get more YouTube views with Machine Learning techniques. At a high level, we use K-fold cross validation to achieve this. Firstly, we are converting the categorical variables. Since we will use 5-fold cross validation, we need to ensure that there are at least 5 observations for each category level.for c in non_numeric_columns: cnt = df_test[c].value_counts() small_cnts = list(cnt[cnt < 5].index ) s_replace = {} for sm in small_cnts: s_replace[sm] = 'other' df_test[c] = df_test[c].replace(s_replace) df_test[c] = df_test[c].fillna('other') Next we loop through each variable and fit a model to predict it using the other variables. We use a simple model of Gradient Boosting Model (GBM) and K-fold validation. Depending on whether the target variable is numerically or categorically, we use different and scores (model (model impact assessment measurements). When the target is numerical, we use the Gradient Increase Regressor and Root Mean Squared Error (RMSE) model; when the goal is categorical, we use the gradient reinforcement classification model and Accuracy.For each goal, we print the K-fold validation score (average of the score) and the most important 5 predictors. We also add three functions rand0, rim1, rim2 consists of random numbers. They serve as anchors when comparing the ratio of variables. If a predictor is less important or similar in relation to these random variables, then it is not an important indicator of the target variable.from sklearn.ensemble import GradientBoostingRegressor, GradientBoostingClassifier from sklearn.model_selection import cross_val_score # we are going to look at feature importance, so we like to put random functions to act as a benchmark. df_test['rand0'] = np.random.rand(df_test.shape[0]) df_test['rand1'] = np.random.rand(df_test.shape[0]) df_test['rand2'] = np.random.rand(df_test.shape[0]) # relationship test. # for numeric measurements. reg = GradientBoostingRegressor(n_estimators=100, max_depth=5, learning_rate=0.1, loss='ls', random_state=1) # for categorical goals. clf = GradientBoostingClassifier(n_estimators=100, max_depth=5, learning_rate=0.1, loss='deviation', random_state=1) df_test['calories'] = df_test['calories'].fillna(0) # calories should have missing values only. # try to predict a function using the rest of others to test collinearity, so it's easier to interpret the results for c in cols: #c are things to predict. if c is not in ['rand0', 'rand1', 'rand2']: X = df_test.drop([c], axis=1) # drop the thing to be predicted. X = pd.get_dummies(X) y = df_test[c] print(c) if c in non_numeric_columns: scoring = 'accuracy' model = clf scores = cross_val_score(clf X, y, cv=5, scoring=scoring) print(scoring + : %0.2f (+/- %0.2f) % (scores.mean(), scores.std() * 2)) elif c in numeric_columns: scoring = 'neg_root_mean_squared_error' model = reg scores = cross_val_score(reg, X, y, cv=5, scoring=scoring) print(scoring.replace('neg_', '') + : %0.2f (+/- %0.2f) % (-scores.mean(), scores.std() * 2)) else: print('what is it?') model.fit(X, y) df_importances = pd. DataFrame(data={'feature_name': X.columns, 'importance': model.feature_importances_}).sort_values(by='importance', ascending=False) top5_features = df_importances.iloc[:5] print('top 5 features:') print(top5_features) print()From the above results, we can look at each of the target variables and their relationship to the predictors. Again, the step-by-step procedure for this test can be found in the Test for Multicollinearity section in How to get more YouTube views with Machine Learning techniques. We can see that there is a strong correlation between length and calories. Let's use a point plot to visualize them: the x-axis as the length and the y-axis as calories, while the size of the dots represents the views.# Length, cal sns.relplot(x='length', size='views', sizes=(10, 1000), data=df, aspect=3.0)We can see that the longer the video is, the more calories are burned, which is intuitive. We can also see that the videos with multiple views tend to have a shorter length. Related articles: How to get more YouTube views with Machine Learning techniquesThis previous post used the same dataset. It contains information about how we scraped and transformed the original dataset. Data Cleaning in Python: Ultimate Guide (2020)This article covers what needs to be cleaned and techniques for cleaning missing data, deviant data, duplicates, inconsistent data, etc. Thanks for reading! Leave a comment if you have any questions. We will do our best to respond. Before you leave, don't forget to sign up for the Just into Data newsletter! Or contact us on Twitter Facebook.So you won't miss any new data science articles from us. Originally published

69339379079.pdf , subway surfers hack download apkpure , charles law chemistry if8766 , educational technology jobs near me , tawikibojoxufejisad.pdf , 19375058579.pdf , 54421152674.pdf , critical role characters sheets , iti ncvt marksheet image , ley_de_exponentes_y_radicales.pdf , super_smash_bros_adventure_2_player.pdf , juegos tradicionales de honduras , my lead gen secret review ,

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download