PyTorch Beginner Tutorial - Tensors

Introduction to Pytorch

PyTorch is a high-level framework for efficiently creating and training deep learning architectures such as Feed-Forward Neural Networks (FFNN), RNN, and CNN. It is an incredibly useful tool because it allows you to perform nifty natural language processing (NLP) and computer vision (CV) tasks. You can use PyTorch to create models that perform NLP tasks such as sentiment analysis, translation, summarization, and even text generation (smart speech bots). Some CV tasks that you can perform using PyTorch are object classification/detection, semantic segmentation, and real-time image processing. Of course, PyTorch can be used for other applications including audio files, medical files, and time-series forecasting.

Contents:

• Tensor Creation and Attributes
• Tensor Operations
• Using right Hardware

Tensor Creation and Attributes

In this tutorial, we explain the building block of PyTorch operations: Tensors. Tensors are essentially PyTorch's implementation of arrays. Since machine learning is moslty matrix manipulation, you will need to be familiar with tensor operations to be a great PyTorch user. Tensors are similar to Numpy arrays. So, if you have previous experience using Numpy, you will have an easy time working with tensors right away.

Let's start by importing PyTorch and Numpy.

In [1]:
import torch
import numpy as np


Next, let's create a 2x3 random tensor to experiment with.

In [2]:
tens = torch.rand(2,3) #2 is the number of rows, 3 is the number of columns
tens

Out[2]:
tensor([[0.4819, 0.8504, 0.5589],
[0.6966, 0.0651, 0.3366]])

Now that we have a tensor, let's check out some of its important attributes. The two most important tensor attributes that you will often check are its shape and the data type.

In [3]:
print(f"This is the shape of our tensor: {tens.shape}")

This is the shape of our tensor: torch.Size([2, 3])

In [4]:
print(f"This is the data type of our tensor: {tens.dtype}")

This is the data type of our tensor: torch.float32


You will often check the shape of tensors after performing operations to make sure the end result is as expected. There are many data types for numbers in a tensor. You can find the full list here: https://pytorch.org/docs/stable/tensor_attributes.html#torch.torch.dtype

However, you need data types simply because most utilities in PyTorch require a certain data type. For instance, when working with CV utilities, you should your data in float.

You can easily change the data type of a tensor using the .to() method as follows:

In [5]:
int_tens = tens.to(torch.uint8)
int_tens.dtype

Out[5]:
torch.uint8
In [6]:
int_tens

Out[6]:
tensor([[0, 0, 0],
[0, 0, 0]], dtype=torch.uint8)

You can see that tensor has integer data now, and the values are rounded down to zeros.

Notice that I created the tensor using using torch.rand, but there are other ways to create tensors:

Create an an empty tensor with zeros.

In [7]:
torch.zeros(2,3)

Out[7]:
tensor([[0., 0., 0.],
[0., 0., 0.]])
In [8]:
#An ones tensor
torch.ones(2,3)

Out[8]:
tensor([[1., 1., 1.],
[1., 1., 1.]])

Create a Tensor from Python list

In [9]:
torch.tensor([[1, 2, 3], [4, 5, 6]])

Out[9]:
tensor([[1, 2, 3],
[4, 5, 6]])

If your data is in Numpy, you can also convert it in to a tensor:

In [10]:
arr = np.array([[1, 2, 3], [4, 5, 6]])
tens = torch.from_numpy(arr)
tens

Out[10]:
tensor([[1, 2, 3],
[4, 5, 6]])

You can also convert tensors back to Numpy arrays:

In [11]:
tens.numpy()

Out[11]:
array([[1, 2, 3],
[4, 5, 6]])

Note that you can set the dtype of a tensor while creating it:

In [12]:
torch.zeros(2,3, dtype=torch.double)

Out[12]:
tensor([[0., 0., 0.],
[0., 0., 0.]], dtype=torch.float64)

So far, so good! Now let's explore what kind of tensor manipulations we need to be familiar with.

Tensor Operations

There are many tensor operations in PyTorch, but I like to group them into 2 categories: slice and math.
• Slice operations allow you to extract or write to any section of a tensor, such as a row, column, or submatrix. These are very useful.
• Math operations allow you to change the values of the tensor mathematically.

Access Operations

Let's create a tensor in order to experiment.

In [13]:
tens = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
tens

Out[13]:
tensor([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])
In [14]:
#To access a single value in the tensor (Keep in mind that Python indexing starts at 0):
print(f"Value in cell 1, 0: {tens[1,0]}")
print(f"Value in cell 2, 2: {tens[2,2]}")

Value in cell 1, 0: 4
Value in cell 2, 2: 9

In [15]:
#To access a row in the tensor:
print(f"Row 0: {tens[0]}")
print(f"Row 2: {tens[2]}")

Row 0: tensor([1, 2, 3])
Row 2: tensor([7, 8, 9])

In [16]:
#To access a column in the tensor:
print(f"Column 0: {tens[:, 0]}")
print(f"Column 1: {tens[:, 1]}")

Column 0: tensor([1, 4, 7])
Column 1: tensor([2, 5, 8])

In [17]:
#To access a subtensor in the tensor:
tens[1:, 1:2]

Out[17]:
tensor([[5],
[8]])
In [18]:
tens[:2, 1:3]

Out[18]:
tensor([[2, 3],
[5, 6]])

Tensor Mathematical Operations

We will explore the commonly used operations. For the full list of math operations: https://pytorch.org/docs/stable/torch.html#math-operations

Let's create 2 tensors from the original using .clone():

In [19]:
tens1 = tens.clone()
tens2 = tens.clone()


For basic arithmetic operations, you can use math symbols or torch functions:

Tensor Addition

In [20]:
tens1 + tens2

Out[20]:
tensor([[ 2,  4,  6],
[ 8, 10, 12],
[14, 16, 18]])
In [21]:
#Addition
torch.add(tens1, tens2)

Out[21]:
tensor([[ 2,  4,  6],
[ 8, 10, 12],
[14, 16, 18]])

Tensor Subtraction

In [22]:
tens1 - tens2

Out[22]:
tensor([[0, 0, 0],
[0, 0, 0],
[0, 0, 0]])
In [23]:
#Subtraction
torch.sub(tens1, tens2)

Out[23]:
tensor([[0, 0, 0],
[0, 0, 0],
[0, 0, 0]])

Tensor Multiplication

In [24]:
tens1 * tens2

Out[24]:
tensor([[ 1,  4,  9],
[16, 25, 36],
[49, 64, 81]])
In [25]:
#Multiplication
torch.mul(tens1, tens2)

Out[25]:
tensor([[ 1,  4,  9],
[16, 25, 36],
[49, 64, 81]])

Tensor Division

In [26]:
tens1 / tens2

Out[26]:
tensor([[1., 1., 1.],
[1., 1., 1.],
[1., 1., 1.]])
In [27]:
#Division
torch.div(tens1, tens2)

Out[27]:
tensor([[1., 1., 1.],
[1., 1., 1.],
[1., 1., 1.]])

For true matrix multiplication, use torch.matmul()

In [28]:
#Matrix Multiplication
torch.matmul(tens1, tens2)

Out[28]:
tensor([[ 30,  36,  42],
[ 66,  81,  96],
[102, 126, 150]])

When concatenating 2 tensors, you specify the dimension along which the concatenation should happen. Again, dim = 0 means along rows, dim = 1 means along columns, etc.

Matrix Concatenation

In [29]:
torch.cat([tens1, tens2], dim=1)

Out[29]:
tensor([[1, 2, 3, 1, 2, 3],
[4, 5, 6, 4, 5, 6],
[7, 8, 9, 7, 8, 9]])

Taking the transpose is a common operation when dealing with data. It can be done in 2 ways:

In [30]:
tens1.T

Out[30]:
tensor([[1, 4, 7],
[2, 5, 8],
[3, 6, 9]])
In [31]:
tens1.t()

Out[31]:
tensor([[1, 4, 7],
[2, 5, 8],
[3, 6, 9]])
Other common math operations done on a single tensor are:
• Mean
• Min
• Max
• Argmin
• Argmax
• Sigmoid
• Tanh

Mean accepts only float dtypes, so we must first convert to float.

In [32]:
flt_tens = tens.to(torch.float32)
torch.mean(flt_tens)

Out[32]:
tensor(5.)

As shown above the mean output is a single-element tensor. We can get this value by using .item():

In [33]:
torch.mean(flt_tens).item()

Out[33]:
5.0

Tensor Min value

In [34]:
torch.min(tens).item()

Out[34]:
1

Tensor Max value

In [35]:
torch.max(tens).item()

Out[35]:
9
Argmin and argmax operations give you the index of the element that is max or min respectively.
In [36]:
#Argmin
torch.argmin(tens).item()

Out[36]:
0
In [37]:
#Argmax
torch.argmax(tens).item()

Out[37]:
8
Sigmoid and tanh are common activation functions in neural networks. There are more advanced ways to use these 2 activation functions in PyTorch, but following is the simplest way ...
In [38]:
#Sigmoid
torch.sigmoid(tens)

Out[38]:
tensor([[0.7311, 0.8808, 0.9526],
[0.9820, 0.9933, 0.9975],
[0.9991, 0.9997, 0.9999]])
In [39]:
#Tanh
torch.tanh(tens)

Out[39]:
tensor([[0.7616, 0.9640, 0.9951],
[0.9993, 0.9999, 1.0000],
[1.0000, 1.0000, 1.0000]])
Note that most transformation operations in PyTorch can also be done in-place. Typically, the in-place version of the function has the same name but ends with an underscore (). For example, sigmoid, tanh_, etc.:
In [40]:
#In-place sigmoid
torch.sigmoid_(tens.to(torch.float32))

Out[40]:
tensor([[0.7311, 0.8808, 0.9526],
[0.9820, 0.9933, 0.9975],
[0.9991, 0.9997, 0.9999]])

Here, since we are applying the transformation in-place, we must change the dtype of the input to match that of the output.

The final function we explore is .view(), which allows us to reshape a tensor. This will be used a lot when working with data.

.view() takes in the new dimensions of the tensor. Note that the new dimensions should be compatible with the original. For example, our tensor (tens) is a 3x3 tensor. That means that the only reshapes possible are 9x1 and 1x9:
In [41]:
tens.view(9, 1)

Out[41]:
tensor([[1],
[2],
[3],
[4],
[5],
[6],
[7],
[8],
[9]])
In [42]:
tens.view(1, 9)

Out[42]:
tensor([[1, 2, 3, 4, 5, 6, 7, 8, 9]])

Another way to reshape a tensor into a 1xN vector is to use (1, -1) shape. The -1 means that this dimension should be inferred from the others. If the other is 1, that means that the columns must be 9. This is a dynamic way of reshaping tensors.

In [43]:
tens.view(1, -1)

Out[43]:
tensor([[1, 2, 3, 4, 5, 6, 7, 8, 9]])

Using Right Hardware for PyTorch

When training large models in PyTorch, you will need to use GPUs. A GPU speeds up the training process by 49 or more times (according to this repo https://github.com/jcjohnson/cnn-benchmarks). So, it is important to make sure the GPU is being used when training.

To do so, we must first set the device:

In [44]:
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
device

Out[44]:
device(type='cpu')

This line dynamically sets the device depending on whether or not a GPU is available. Next, we must send the model we are working with to device.

I will create a simple neural network to demonstrate GPU usage.

In [45]:
import torch.nn as nn
import torch.nn.functional as F

class NeuralNet(nn.Module):
def __init__(self):
super(NeuralNet, self).__init__()
self.fc1 = nn.Linear(30, 120)
self.fc2 = nn.Linear(120, 64)
self.fc3 = nn.Linear(64, 5)

def forward(self, x):
x = F.relu(self.fc1(x))
x = F.relu(self.fc2(x))
x = self.fc3(x)
return x


Now that we have written the model, we can initizalize it as such:

In [46]:
model = NeuralNet()
print(model)

NeuralNet(
(fc1): Linear(in_features=30, out_features=120, bias=True)
(fc2): Linear(in_features=120, out_features=64, bias=True)
(fc3): Linear(in_features=64, out_features=5, bias=True)
)


After initialization, we send the model to the device, where it is CPU or GPU:

In [47]:
model = model.to(device)

Please note that when working with a GPU, it is not enough to send the model to the GPU. The data must also be sent to the GPU. Since the GPU has limited space, we typically create batches of data (for example a batch of 16 images) in order to train.

You can send the data to the device using the same .to() operation:

In [49]:
tens = tens.to(device)