Training steps of Artificial Neural Network along with visualization

The Training Steps for the Artificial Neural Network:

The Flowchart( Visualization) of Training steps :

Chi-Square test for Dependency between categorical variables( Independent and target variable)

A most common problem we come across Machine learning is determining whether input features are relevant to the outcome to be predicted. This is the problem of feature selection.

In the case of classification problems where input variables are also categorical, we can use statistical tests to determine whether the output variable is dependent or independent of the input variables.

       “ Categorical variable is a variable that can take on one of a limited, and usually fixed, number of possible values.”

Pearson’s chi-squared statistical hypothesis is an example of a test for independence between categorical variables.

We take an example : Is gender independent of education level? A random sample of 395 people were surveyed and each person was asked to report the highest education level they obtained. The data that resulted from the survey is summarized in the following table:

	High School	Bachelors	Masters	Ph.d.	Total
Female	60	54	46	41	201
Male	40	44	53	57	194
Total	100	98	99	98	395

This  table is called a contingency tableby Karl Pearson, because the intent is to help determine whether one variable is contingent upon or depends upon the other variable

The Chi-Squared test is a statistical hypothesis test that assumes (the null hypothesis) that the observed frequenciesfor a categorical variable match the expected frequenciesfor the categorical variable. The Chi-Squared test does this for a contingency table, first calculating the expected frequencies for the groups, then determining whether the division of the groups, called the observed frequencies, matches the expected frequencies.

The resultof the test is a test statisticthat has a chi-squared distribution and can be interpreted to reject or fail to reject the assumption or null hypothesis that the observed and expected frequencies are the same.

When observed frequency is far from the expected frequency, the corresponding term in the sum is large; when the two are close, this term is small. Large values of Chi-squareindicate that observed and expected frequencies are far apart. Small values of **Chi-square** mean the opposite: observed are close to expected.

        “ The variables are considered independent if the observed and expected frequencies are similar, that the levels of the variables do not interact, are not dependent.

we can interpret the dependency of the variables  in two ways

1.      Using test statistic

2.      Using P-value

1.Using Test-statistic

We can interpret the test statistic in the context of the chi-squared distribution with the requisite number of degress of freedom as follows: **

• If Statistic >= Critical Valuesignificant result, reject null hypothesis (H0), dependent.
• If Statistic < Critical Valuenot significant result, fail to reject null hypothesis (H0), independent.

The degrees of freedom for the chi-squared distribution is calculated based on the size of the contingency table as:

                     degrees of freedom: (rows - 1) * (cols - 1)

2.Using P-value

In terms of a p-value and a chosen significance level (alpha), the test can be interpreted as follows:

• If p-value <= alphasignificant result, reject null hypothesis (H0), dependent.
• If p-value > alphanot significant result, fail to reject null hypothesis (H0), independent.

For the test to be effective, at least five observations are required in each cell of the contingency table.

TF-IDF algorithm ( Natural Language Processing)

TF-IDF:

  TF-IDF stands for Term frequency and inverse document frequency and is one of the most popular and effective Natural Language Processing techniques. This technique allows you to estimate the importance of the term for the term (words) relative to all other terms in a text.

CORE IDEA:   If a term appears in some text frequently, and rarely in any other text – this term has more importance for this text.

This technique uses TF(Term frequency)  and IDF(Inverse document frequency) algorithms:

• TF – shows the frequency of the term in the text, as compared with the total number of the words in the text.
• IDF – is the inverse frequency of terms in the text. It simply displays the importance of each term. It is calculated as a logarithm of the number of texts divided by the number of texts containing this term.

TF-IDF algorithm: 

1. Evaluate the TF-values for each term (word).
2. Extract the IDF-values for these terms.
3. Get TF-IDF values for each term: by multiplying TF by IDF.
4. We get a dictionary with calculated TF-IDF for each term.

The algorithm for TF-IDF calculation for one word is shown on the diagram.

The results of the same algorithm for three simple sentences with the TF-IDF technique are shown below.

The advantages of this vectorization technique:

• Unimportant terms will receive low TF-IDF weight (because they are frequently found in all texts) and important – high.
• It is simple to evaluate important terms and stop-words in text.

Out break statistics of all the deadly virus( from day 0)

Alibaba new AI system that detects Corona Virus

Why the name "Python" came for the language? Does it orginate from the Snake PYTHON?

Tensorflor Quantum