Linear Algebra - Basics for Deep Learning

Scalars :

using MXNet, we can work with scalars by creating NDArray with just one element.We will see some addition,multiplication,division and exponentiation in this session.

if you have not installed MXNet package, first install in your PC using the following command at cmd prompt, goto command prompt :
C:\Users\ABC> pip install mxnet

let us take two scalars, x and y.

C:\Users\ABC> python

>>> from mxnet import nd

>>> x = nd.array([3.0])
>>> y = nd.array([2.0])

>>> print('x + y = ', x+y)
x + y = [5.]

>>> print('x * y = ', x*y)
x * y = [6.]

>>> print('x / y = ', x/y)
x / y = [1.5]

>>> print('x ** y = ', nd.power(x,y))
x ** y = [9.]

Vectors :

a vector is a (array) list of numbers, for example [1.0, 3.0, 5.0, 2.0]. These numbers are called as scalars, each of the numbers in the vector consists of a single scalar value. We call these values the entries or components of the vector. 

in MXNet, we work with vectors via 1D NDArrays.

>>> x = nd.arange(4)

>>> print('x = ', x)

x = [0. 1. 2. 3.]

for Example if we want the 3rd element in a vector,use

>>> x[3]

[3.]

Length, Dimensionality and Shape :

The length of a Vector is called as its dimension. in NDArray we can access a vector's length using .shape attribute. like this
>>> x.shape
(4,)
The shape is a tuple that lists the dimensionality of the NDArray along each of its axes. Because a vector can only be indexed along one axis, its shape has just one element.

Note that a scalar would have 0 dimensions and a vector would have 1 dimension.
so you can think of 2D array as 2 axes and 3D array as 3 axes, and so on.

let's see some examples,

>>> a = 2
>>> x = nd.array([1,2,3])
>>> y = nd.array([10,20,30])
>>> print(a * x)
[2. 4. 6.]

>>> print(a * x + y)
[12. 24. 36.]

Matrices :

Matrices are 2D arrays, it can be denoted with capital letter like, A, B, C etc.

>>> A = nd.arange(20).reshape((5,4))
>>> print(A)
[[0. 1. 2. 3.]
 [4. 5. 6. 7.]
 [8. 9. 10. 11.]
 [12. 13. 14. 15.]
 [16. 17. 18. 19.]]

we can transpose the matrix through T.

>>> print(A.T)
[[0. 4. 8. 12. 16.]
[1. 5. 9. 13. 17.]
[2. 6. 10. 14. 18.]
[3. 7. 11. 15. 19.]]

Tensors :

Tensors give us a generic way of discussing arrays with an arbitrary number of axes.

for example, Vectors are first-order tensors, and matrices are second-order tensors.

Using tensors, images ( 3D data structures) its axes corresponding to height, width and three (RGB) color channels we can work with it.

>>> X = nd.arange(24).reshape((2, 3, 4))

>>> print('X.shape =', X.shape)

X.shape = (2, 3, 4)

>>> print('X =', X)

X =

[[[ 0.  1.  2.  3.]

  [ 4.  5.  6.  7.]

  [ 8.  9. 10. 11.]]

 [[12. 13. 14. 15.]

  [16. 17. 18. 19.]

  [20. 21. 22. 23.]]]

Basic properties of tensor arithmetic :

for all tensors, multiplication by a scalar produces a tensor of the same shape.

>>> a = 2
>>> x = nd.ones(3)
>>> y = nd.zeros(3)

>>> print(x.shape)
(3,)
>>> print(y.shape)
(3,)
>>> print((a * x).shape)
(3,)
>>> print((a * x + y).shape)
(3,)

Sums and means :

>>> print(x)

[1. 1. 1.]

>>> print(nd.sum(x))

[3.]

>>> print(A)

[[ 0. 1. 2. 3.]

[ 4. 5. 6. 7.]

[ 8. 9. 10. 11.]

[12. 13. 14. 15.]

[16. 17. 18. 19.]]

>>> print(nd.sum(A))

[190.]

Mean : it is an average. 

Mean = sum / total number of elements.

>>> print(nd.mean(A))

[9.5]

>>> print(nd.sum(A) / A.size)

[9.5]

Dot product :

>>> x = nd.arange(4)
>>> y = nd.ones(4)
>>> print(x, y, nd.dot(x, y))
[0. 1. 2. 3.]
[1. 1. 1. 1.]
[6.]

where, nd.dot(x, y) is equivalently to nd.sum(x * y) this gives same result.

Dot products are useful in a wide range of contexts. For Example, given a set of weights, the weighted sum of some values could be expressed as the dot product. 
when the weights are non-negative and sum to one, the dot product expresses a weighted average.
when two vectors each have length one. dot products can also capture the cosine of the angle between them.

Matrix-vector product :

>>> nd.dot(A, x)
[14. 38. 62. 86. 110.]
 Note that the column dimension of A must be the same as the dimension of x.

Matrix-matrix multiplication :

>>> B = nd.ones(shape=(4,3))
>>> nd.dot(A, B)
[[ 6. 6. 6.]
[22. 22. 22.]
[38. 38. 38.]
[54. 54. 54.]
[70. 70. 70.]]

Norms :

Norms are operators in linear algebra, they tell us how big a vector or matrix is. 

we represent norms with a notation ||.||, where the . is just a placeholder.

for example , a vector X is ||X|| and matrix A is ||A||.

l1 norm is simply the sum of the absolute values.
the Euclidean distance sqrt (x1**2+ x2**2+....) is l2-norm.

>>> nd.norm(x)
[3.7416573]

to calculate L1-norm we can simply perform the absolute value and then sum over the elements.

>>> nd.sum(nd.abs(x))
[6.]

Norms and objectives :

In machine learning we are often trying to solve optimization problems: like (a). Maximize the probability assigned to observed data. (b). Minimize the distance between predictions and the ground-truth observations. Assign vector representations to items ( like words, products, or news articles) such that the distance between similar items is minimized, and the distance between dissimilar items is maximized. oftentimes, these objectives, perhaps the most important component of a machine learning algorithm are expressed as norms.