Skip to main content

Tensors

A PyTorch tensor is a NumPy-like array with a dtype, a shape and a device, stored as one row of memory read through strides. Meet the float32 default, memory shared with NumPy, views against copies, broadcasting, and the forward pass of DL01.2 rewritten in torch.
DL01.7.1L2 Intermediate25 min
In review

The maths and code on this page are checked by automated tests on every change, but an independent reader has not signed it off yet. Spot a mistake or a confusing step? Tell us.

Before this lesson: DL01.6 Build an autodiff engine from scratch DL01.2 Multilayer perceptrons and universal approximation P02.3 Indexing, slicing and views

What you will be able to do
  • Create tensors and read their dtype, shape and device, and explain why torch.tensor(0.1) is not the float64 number 0.1.
  • Predict when a tensor and a NumPy array share memory (torch.from_numpy, .numpy()) and when they are copies (torch.tensor, .clone()).
  • Compute the strides and the storage offset of a transpose, a slice, unsqueeze and expand, and find any element’s place in memory from them.
  • Decide whether view() can work on a tensor or whether reshape() or contiguous() must copy.
  • Rewrite a NumPy forward pass in torch and check it with torch.testing.assert_close.

The question

You have built every part of a network in NumPy. PyTorch’s tensor looks like the same array with a new name, yet torch.tensor(0.1) is not quite 0.1, a tensor made from a NumPy array changes when the array changes, and x.T.view(12) stops with an error while x.view(12) works. What is a tensor, underneath, that explains all three?

Intuition

A tensor is two things. One is a storage: a single row of numbers in memory, all of one type. The other is a small record that says how to read that row as a grid: a shape, a stride per axis, and a starting place called the storage offset.

At the top, a row of twelve blue memory cells holding 0 to 11, labelled storage: one row of memory. Below on the left, the 3 by 4 grid x with rows 0 1 2 3, 4 5 6 7 and 8 9 10 11, labelled shape (3, 4) and stride (4, 1). Below on the right, the 4 by 3 grid x.T with rows 0 4 8, 1 5 9, 2 6 10 and 3 7 11, labelled shape (4, 3) and stride (1, 4). The cell holding 6 is yellow in all three places: x[1, 2] is at 1 times 4 plus 2 times 1, which is 6, and x.T[2, 1] is at 2 times 1 plus 1 times 4, also 6. A note says both grids read the same twelve numbers and nothing was moved.
DL01.7.1-F1

One storage, two ways of reading it: only the strides differ. The yellow cell is x[1, 2] and x.T[2, 1] at once.

The stride of an axis is how many cells you jump in memory to move one step along that axis. In x a step to the right is one cell and a step down is four. The transpose x.T swaps the two strides and touches no number. Many operations work this way: they write a new record and keep the storage. The result is a view. Others must write new numbers into new memory: a copy.

Two more facts belong to the record. The dtype says what one cell holds, and PyTorch’s default for decimals is float32, four bytes, where NumPy’s is float64, eight bytes. The device says where the storage lives. On this site that is always the CPU, but the code names it once, in one variable, so that the same lesson would run anywhere.

Build it up

Step 1. Making tensors. torch.tensor([1.0, 2.0]) copies Python numbers into a new tensor. torch.zeros(3, 4), torch.arange(12) and torch.randn(2, 3) make tensors from a shape, as NumPy’s functions do. Python decimals become float32 and Python whole numbers become int64.

Step 2. Fewer digits. A float32 keeps about 7 significant decimal digits, a float64 about 16. The gap between 1 and the next number up is \(2^{-23} \approx 1.19 \times 10^{-7}\) in float32 and \(2^{-52} \approx 2.22 \times 10^{-16}\) in float64.

Two rows of digit boxes. Top row, float64, 8 bytes, NumPy's default: 0.1 is stored as 0.1000000000000000055 and the first seventeen digits after the point, green, agree with 0.1. Bottom row, float32, 4 bytes, PyTorch's default: 0.1 is stored as 0.1000000014901161194 and only the first eight digits agree. Beside each row, the gap after 1: 2.2e-16 and 1.2e-7. At the bottom, print(torch.tensor(0.1)) gives tensor(0.1000), because printing rounds to 4 decimals, while torch.tensor(0.1).item() gives 0.10000000149011612.
DL01.7.1-F2

The same decimal, 0.1, as each type stores it. Green digits agree with the true 0.1. Printing hides the difference, because a tensor prints with 4 decimals by default; .item() shows the stored value.

So torch.tensor(1 / 3) prints tensor(0.3333) and np.array(1 / 3) prints 0.3333333333333333. The 4 decimals are only the print setting (torch.set_printoptions); the stored float32 is \(0.3333333432\ldots\). Networks train in float32 because it halves the memory and the work, and 7 digits are plenty for weights that gradient descent nudges anyway.

Step 3. Sharing memory with NumPy. torch.from_numpy(a) wraps the storage of the array a without copying it, and keeps its dtype, so a float64 array gives a float64 tensor. Change a and the tensor changes. t.numpy() goes the other way and also shares. torch.tensor(a) copies, and so does t.clone().

Step 4. Views through strides. x = torch.arange(12).view(3, 4) reads the twelve numbers as three rows of four, with strides \((4, 1)\). Then:

  • x.T has shape \((4, 3)\) and strides \((1, 4)\);
  • x[:, 1::2], every second column from column 1, has offset 1 and strides \((4, 2)\);
  • x.unsqueeze(0) adds an axis of length 1 at the front: shape \((1, 3, 4)\);
  • x[:, :1].expand(3, 4) repeats column 0 four times with stride 0 along the columns.
Five rows, each with the same strip of twelve memory cells numbered 0 to 11. In each row the cells the view reads are filled and the others faded, and small vermillion numbers under the cells give the position at which the view reads each one. x reads all twelve in order, stride (4, 1). x.T reads cell 0 first, cell 4 second, cell 8 third and cell 1 fourth, stride (1, 4). x[:, 1::2] reads cells 1, 3, 5, 7, 9 and 11, offset 1, stride (4, 2). x.unsqueeze(0) reads all twelve in order, shape (1, 3, 4), stride (12, 4, 1). x[:, :1].expand(3, 4) reads only cells 0, 4 and 8, each four times, stride (4, 0). A note says all five are views of the same twelve numbers.
DL01.7.1-F3

Five views of one storage. Each reads the cells it needs by fixed jumps, the strides. None of them copies a number; expand even reads the same cell several times, with stride 0.

Step 5. Broadcasting. Tensors broadcast as NumPy arrays do: line the shapes up from the right; each pair of lengths must be equal, or one must be 1 or missing. Adding a row of shape \((4,)\) to x reads the row with stride 0 down the three rows, as expand does, and writes the sum into new memory.

The formal statement

NoteDefinition: a strided tensor

A tensor with storage \(s\) (a row of numbers \(s[0], s[1], \dots\)), storage offset \(o\), shape \((n_1, \dots, n_k)\) and strides \((\sigma_1, \dots, \sigma_k)\) has, at index \((i_1, \dots, i_k)\) with \(0 \le i_j < n_j\), the element

\[ s\Big[\, o + \sum_{j=1}^{k} i_j\, \sigma_j \Big]. \]

PyTorch counts strides in elements. NumPy, in P02.3.4, counts them in bytes; divide by the item size to compare.

Contiguous strides. A tensor is contiguous when it reads its storage from the offset onwards in plain row-major order, the last index moving fastest. For shape \((n_1, \dots, n_k)\) the last axis then has stride 1, and stepping axis \(j\) by one must skip a whole block of the later axes:

\[ \sigma_k = 1, \qquad \sigma_j = \sigma_{j+1}\, n_{j+1}. \]

For shape \((3, 4)\) that gives \(\sigma_2 = 1\) and \(\sigma_1 = 1 \cdot 4 = 4\), the strides \((4, 1)\). (Axes of length 1 are never stepped along, so their stride does not matter for this test.)

Transpose swaps strides. Let \(\mat{A}\) have strides \((\sigma_1, \sigma_2)\). By definition \(\mat{A}\T[j, i] = \mat{A}[i, j]\), and the right side sits at \(o + i\sigma_1 + j\sigma_2\). Read as an index \((j, i)\) of the transpose, that is \(o + j\sigma_2 + i\sigma_1\): strides \((\sigma_2, \sigma_1)\), the same offset, the same storage.

Slices shift and stretch. Take the slice start::step on axis \(j\). Its index \(i\) is the old index \(\text{start} + i \cdot \text{step}\), so the term \(i_j \sigma_j\) becomes

\[ \begin{aligned} &(\text{start} + i\,\text{step})\,\sigma_j \\ &\quad = \text{start}\,\sigma_j + i\,(\text{step}\,\sigma_j). \end{aligned} \]

The offset grows by \(\text{start}\,\sigma_j\) and the stride becomes \(\text{step}\,\sigma_j\). Again no number moves.

Expand uses stride 0. An axis of length 1 can be stretched to length \(m\) with stride 0: every index \(i\) adds \(i \cdot 0 = 0\), so all \(m\) positions read the same cell. That is how broadcasting reads a small tensor without copying it.

When view works. view gives a new shape over the same storage and the same offset, so it needs strides that walk the elements in the old row-major order with fixed jumps. On a contiguous tensor this always works. On a tensor such as x.T the walk in row-major order is \(0, 4, 8, 1, 5, \ldots\), and no single stride makes it, so view(12) raises an error. reshape returns a view when one exists and otherwise a copy; contiguous() returns the tensor itself if it is already contiguous, and otherwise a contiguous copy.

Go deeper: the exact condition for flattening a tensor with view

Merge two neighbouring axes \(j\) and \(j+1\) into one of length \(n_j n_{j+1}\). Walking the merged axis in row-major order means stepping \(i_{j+1}\) from 0 to \(n_{j+1} - 1\), each step \(\sigma_{j+1}\), and then stepping \(i_j\) by one, a jump of \(\sigma_j\) from the start of the row. For one fixed stride \(\sigma_{j+1}\) to cover both, the jump to the next row must equal \(n_{j+1}\) ordinary steps: \(\sigma_j = n_{j+1}\,\sigma_{j+1}\). So view(-1) works exactly when this holds for every pair of neighbouring axes, after leaving out axes of length 1. For x.T the test asks whether \(\sigma_1 = 1\) equals \(n_2 \sigma_2 = 3 \cdot 4 = 12\); it does not.

Worked example

Take x = torch.arange(12).view(3, 4), so the storage holds \(s[m] = m\) and every element tells you where it lives.

Find x.T[2, 1]. The strides of x.T are \((1, 4)\) and the offset is 0, so the element is at \(0 + 2 \cdot 1 + 1 \cdot 4 = 6\). Check from the other side: x.T[2, 1] is x[1, 2], at \(1 \cdot 4 + 2 \cdot 1 = 6\).

Read y = x[:, 1::2]. On axis 2 the slice has start 1 and step 2, so the offset is \(0 + 1 \cdot 1 = 1\) and the strides are \((4, 2 \cdot 1) = (4, 2)\), with shape \((3, 2)\). Its six elements sit at the places

\[ 1 + 4i + 2j, \]

for \(i = 0, 1, 2\) and \(j = 0, 1\): that is \(1, 3\) in the first row, \(5, 7\) in the second and \(9, 11\) in the third. Element y[2, 1] is at \(1 + 8 + 2 = 11\), which is x[2, 3]. Write y[0, 0] = 100 and x[0, 1] becomes 100 too.

Two parts. Top, y = x[:, 1::2], a view with offset 1 and stride (4, 2): a 3 by 2 grid holding 1, 3, 5, 7, 9 and 11, the formula place = 1 + 4i + 2j, and for each row i the arithmetic, such as 1 + 4 + 0 = 5 and 1 + 4 + 2 = 7. Below it the memory strip with cells 1, 3, 5, 7, 9 and 11 filled and brackets marking a jump of +2 for the next j and +4 for the next i. Bottom, x.T.view(12): the twelve places x.T visits in row-major order, 0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11, with the jumps between them, +4, +4, then a bold -7, repeating. A note says no fixed stride walks these jumps, so view raises an error, and reshape(12) copies the numbers into new memory instead.
DL01.7.1-F4

Left: the slice is a view, and the formula \(1 + 4i + 2j\) finds every element. Right: flattening x.T would need the jumps \(+4, +4, -7, \ldots\), which no single stride gives, so view refuses and reshape copies.

Can x.T be flattened by view? Reading x.T in row-major order visits \(0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11\). The jumps are \(+4, +4, -7, +4, +4, -7, \ldots\). A 1-D view has one stride, so this walk is impossible, and x.T.view(12) raises an error. x.T.reshape(12) copies the twelve numbers into the new order and returns a tensor with its own memory.

Code

Start with the record: dtype, shape and device. The device is named once, so that moving the whole lesson elsewhere would mean changing one line.

import numpy as np
import torch

torch.manual_seed(0)
device = torch.device("cpu")              # the one place the device is named

a = torch.tensor([1.0, 2.0, 3.0], device=device)
n = np.array([1.0, 2.0, 3.0])
print(a.dtype, tuple(a.shape), a.device, "|", n.dtype, n.shape)
print(torch.tensor([1, 2]).dtype)
print(torch.tensor(1 / 3), np.array(1 / 3))
print(f"0.1 stored as float32: {torch.tensor(0.1).item():.17f}")
print(f"0.1 stored as float64: {np.float64(0.1):.17f}")
print("spacing near 1:", torch.finfo(torch.float32).eps, torch.finfo(torch.float64).eps)
torch.float32 (3,) cpu | float64 (3,)
torch.int64
tensor(0.3333) 0.3333333333333333
0.1 stored as float32: 0.10000000149011612
0.1 stored as float64: 0.10000000000000001
spacing near 1: 1.1920928955078125e-07 2.220446049250313e-16

Now memory shared with NumPy, and memory that is not.

X_np = np.arange(6.0).reshape(2, 3)
shared = torch.from_numpy(X_np)           # wraps the same memory, keeps float64
copied = torch.tensor(X_np)               # a copy
X_np[0, 0] = 100.0
print(shared[0, 0].item(), copied[0, 0].item(), shared.dtype)

back = shared.numpy()                     # shares again
back[1, 2] = -1.0
print(X_np[1, 2])
100.0 0.0 torch.float64
-1.0

Strides, the storage offset and contiguity for the views of the figures, then the address formula written from scratch and checked against PyTorch’s own indexing.

x = torch.arange(12).view(3, 4)
views = {"x": x, "x.T": x.T, "x[:, 1::2]": x[:, 1::2],
         "x.unsqueeze(0)": x.unsqueeze(0), "x[:, :1].expand(3, 4)": x[:, :1].expand(3, 4)}
for name, v in views.items():
    print(f"{name:22} shape {tuple(v.shape)!s:10} stride {v.stride()!s:11} "
          f"offset {v.storage_offset()}  contiguous {v.is_contiguous()}")

storage = torch.arange(12)                # x's storage, as a plain row

def element(v, index):
    """Read v[index] from the storage by the formula s[o + sum of i_j * stride_j]."""
    place = v.storage_offset() + sum(i * s for i, s in zip(index, v.stride()))
    return storage[place].item()

def same_memory(u, v):
    return u.untyped_storage().data_ptr() == v.untyped_storage().data_ptr()

print(element(x.T, (2, 1)), x.T[2, 1].item(), element(x[:, 1::2], (2, 1)))
try:
    x.T.view(12)
except RuntimeError as error:
    print("view failed:", str(error)[:46])
print(same_memory(x, x.T), same_memory(x, x.T.reshape(12)), same_memory(x, x.view(12)))
x                      shape (3, 4)     stride (4, 1)      offset 0  contiguous True
x.T                    shape (4, 3)     stride (1, 4)      offset 0  contiguous False
x[:, 1::2]             shape (3, 2)     stride (4, 2)      offset 1  contiguous False
x.unsqueeze(0)         shape (1, 3, 4)  stride (12, 4, 1)  offset 0  contiguous True
x[:, :1].expand(3, 4)  shape (3, 4)     stride (4, 0)      offset 0  contiguous False
6 6 11
view failed: view size is not compatible with input tensor'
True False True

Broadcasting follows NumPy’s rule, and an expanded tensor reads its cells with stride 0.

r = torch.tensor([0, 10, 20, 30])
total = x + r
print(tuple(total.shape), r.expand(3, 4).stride(), same_memory(total, x))
print(torch.broadcast_shapes((3, 1), (1, 4)), np.broadcast_shapes((3, 1), (1, 4)))
(3, 4) (0, 1) False
torch.Size([3, 4]) (3, 4)

Finally the forward pass of DL01.2.2, ReLU hidden layers and a sigmoid output, written once for NumPy and once for torch. With the course’s \((\text{out}, \text{in})\) weights the body is the same line, A @ W.T + b; only the activations change their names. The weights are those of DL01.2.2’s init, for a 2-16-16-1 network run on the 200 points of make_moons.

from sklearn.datasets import make_moons

def forward_np(A, layers):
    for l, (W, b) in enumerate(layers, start=1):
        Z = A @ W.T + b
        A = np.maximum(Z, 0.0) if l < len(layers) else 1 / (1 + np.exp(-Z))
    return A

def forward_torch(A, layers):
    for l, (W, b) in enumerate(layers, start=1):
        Z = A @ W.T + b
        A = torch.relu(Z) if l < len(layers) else torch.sigmoid(Z)
    return A

def init(sizes, seed=0):                  # DL01.2.2: weights (out, in), zero biases
    rng = np.random.default_rng(seed)
    return [(rng.normal(0, 1 / np.sqrt(n_in), (n_out, n_in)), np.zeros(n_out))
            for n_in, n_out in zip(sizes[:-1], sizes[1:])]

X_np, _ = make_moons(n_samples=200, noise=0.2, random_state=0)
layers_np = init([2, 16, 16, 1])
out_np = forward_np(X_np, layers_np)

to_t = lambda arr, dtype: torch.as_tensor(arr, dtype=dtype, device=device)
layers_64 = [(to_t(W, torch.float64), to_t(b, torch.float64)) for W, b in layers_np]
out_64 = forward_torch(to_t(X_np, torch.float64), layers_64)
torch.testing.assert_close(out_64, torch.from_numpy(out_np))       # float64 against float64

layers_32 = [(to_t(W, torch.float32), to_t(b, torch.float32)) for W, b in layers_np]
out_32 = forward_torch(to_t(X_np, torch.float32), layers_32)
gap = (out_32.double() - torch.from_numpy(out_np)).abs().max().item()
print(tuple(out_64.shape), out_64.dtype, out_32.dtype, f"largest float32 gap {gap:.1e}")
try:
    torch.testing.assert_close(out_32, torch.from_numpy(out_np))
except AssertionError as error:
    print("assert_close:", str(error).splitlines()[0][:45])
(200, 1) torch.float64 torch.float32 largest float32 gap 5.9e-08
assert_close: The values for attribute 'dtype' do not match

The float64 run agrees with NumPy to rounding. The float32 run differs in about the seventh significant digit (the largest gap here is below \(10^{-6}\) on outputs between 0 and 1), as the spacing \(2^{-23}\) predicts, and assert_close refuses to compare a float32 with a float64 at all until one is converted: its default checks the dtype as well as the values.

Explore

DL01.7-X1

Shapes and strides

Pick a shape for the twelve numbers and an operation, and watch which memory cells the result reads and in which order. Then add view(-1): predict from the strides whether it can work before you look at the read-out.

Five rows, each with the same strip of twelve memory cells. Filled cells show what x, x.T, x[:, 1::2], x.unsqueeze(0) and x[:, :1].expand(3, 4) read, with the read order underneath and the strides beside each row.

Pitfalls

Two panels. Left, marked with a cross and titled shared memory: X_np and the tensor X both point at one blue strip of memory; the code reads X = torch.from_numpy(X_np), then X_np /= 16; the strip now holds the scaled values and a note says the tensor changed too, so the training data was scaled twice when the tensor was scaled again. Right, marked with a tick and titled a copy: X_np points at one strip and X at a second strip of its own; the code reads X = torch.tensor(X_np), then X_np /= 16; only the first strip changed and a note says the tensor kept its values.
DL01.7.1-F5

Wrong beside right. With from_numpy the tensor and the array are one memory, so scaling the array in place scales the tensor too. torch.tensor makes a copy that later changes cannot reach.

“I changed the NumPy array, not the tensor.”

After X = torch.from_numpy(X_np) there is only one block of memory. An in-place change to either name, such as X_np /= 16 to scale pixels, changes both. Use torch.tensor(X_np) or torch.from_numpy(X_np).clone() when the tensor must keep its own values, and remember that from_numpy keeps NumPy’s float64, which the default float32 weights of a network will not accept in a matrix product.

“view and reshape are the same thing.”

view never copies: it fails when the strides cannot walk the new shape, as after a transpose. reshape silently copies in that case, so a later write to the result does not reach the original. When code needs to know, use view, or call contiguous() first on purpose.

“Two tensors that print the same are equal.”

Printing rounds to 4 decimals, and a float32 result differs from a float64 one around the seventh digit. Compare with torch.testing.assert_close, which picks tolerances by dtype (\(1.3 \times 10^{-6}\) relative for float32, \(10^{-7}\) for float64), never with ==.

Check yourself

Recall

1. What three things describe what a tensor holds and where, besides the numbers themselves?

Show answer

Its dtype (the type of one cell), its shape (with the strides and offset that read the storage), and its device (where the storage lives).

2. What is the default dtype of torch.tensor(0.5), and of np.array(0.5)?

Show answer

torch.float32 (four bytes, about 7 digits) and float64 (eight bytes, about 16 digits).

3. Which of torch.from_numpy(a), torch.tensor(a) and t.numpy() share memory?

Show answer

torch.from_numpy(a) and t.numpy() share memory; torch.tensor(a) copies.

4. A contiguous tensor has shape \((2, 3, 4)\). What are its strides?

Show answer

\(\sigma_3 = 1\), \(\sigma_2 = 1 \cdot 4 = 4\), \(\sigma_1 = 4 \cdot 3 = 12\): strides \((12, 4, 1)\).

5. Why does x.T.view(12) fail while x.T.reshape(12) works?

Show answer

Reading x.T in row-major order needs the jumps \(+4, +4, -7, \ldots\), which no single stride gives, and view never copies. reshape falls back to copying the numbers into new memory.

Exercises

1. ★ x = torch.arange(24).view(2, 3, 4). Give the shape, strides and offset of x[1], of x[:, 2] and of x.permute(2, 0, 1), and say which are contiguous.

Show solution

x has strides \((12, 4, 1)\). x[1] fixes the first index at 1: offset \(1 \cdot 12 = 12\), shape \((3, 4)\), strides \((4, 1)\), contiguous. x[:, 2] fixes the second index at 2: offset \(2 \cdot 4 = 8\), shape \((2, 4)\), strides \((12, 1)\), not contiguous (a contiguous \((2, 4)\) would have strides \((4, 1)\)). x.permute(2, 0, 1) puts the axes in the order 3, 1, 2: shape \((4, 2, 3)\), strides \((1, 12, 4)\), offset 0, not contiguous.

2. ★★ For x = torch.arange(12).view(3, 4), find the strides and offset of x[1:, ::3], list the storage places it reads in row-major order, and say what writing x[1:, ::3] += 1 does to x.

Show solution

On axis 1 the slice 1: adds \(1 \cdot 4 = 4\) to the offset and keeps stride 4; on axis 2 the step 3 makes the stride \(3 \cdot 1 = 3\). So the offset is 4, the strides \((4, 3)\) and the shape \((2, 2)\) (columns 0 and 3). The places are \(4 + 4i + 3j\): 4, 7, 8, 11. The slice is a view, so the write adds 1 to x[1, 0], x[1, 3], x[2, 0] and x[2, 3] of x itself.

3. ★★★ A tensor w of shape \((16, 2)\) holds the weights of a layer in the course’s (out, in) convention. A colleague builds it as torch.from_numpy(W_in_out).T, where W_in_out is a contiguous float64 array of shape \((2, 16)\). List everything about w that differs from a tensor made by torch.zeros(16, 2), and give one line that fixes all of it.

Show solution

Four things. Its dtype is float64, not float32. It shares memory with W_in_out, so a change to the array changes the weights. It is not contiguous: its strides are \((1, 16)\) instead of \((2, 1)\), so w.view(32) fails. And its storage is owned by NumPy. The line w = torch.from_numpy(W_in_out).T.contiguous().float() fixes all four: contiguous() copies the numbers in row-major order into new memory with strides \((2, 1)\), which also cuts the link to NumPy, and .float() converts the copy to float32. Beware the shorter torch.tensor(W_in_out.T, dtype=torch.float32): it copies and converts, but keeps the transposed strides \((1, 16)\), as clone() does, so it fixes only three of the four.

Summary card

A tensor is one row of memory read through a shape and strides.

The anchor figure again: one strip of twelve memory cells read as the 3 by 4 grid x with strides (4, 1) and as the 4 by 3 grid x.T with strides (1, 4), with the cell 6 yellow in all three places.
DL01.7.1-F1

One storage, many views.

  • Address. Element \((i_1, \dots, i_k)\) is at \(o + \sum_j i_j \sigma_j\).
  • Contiguous. \(\sigma_k = 1\) and \(\sigma_j = \sigma_{j+1} n_{j+1}\).
  • Views. Transpose swaps strides, a slice shifts the offset and multiplies a stride, expand uses stride 0.
  • Copies. torch.tensor, .clone(), and reshape or contiguous() when no view exists.
  • Defaults. Decimals are float32; from_numpy and .numpy() share memory.

Sources and next steps

Sources. Each was opened and checked against this lesson on 15 November 2026.

  • PyTorch 2.15 documentation, Tensor Views: a view shares the underlying data with its base; basic slicing, transpose, permute, view, unsqueeze and expand return views; contiguous() returns the tensor itself when it is already contiguous and otherwise copies; reshape and flatten may return a view or a copy.
  • PyTorch 2.15 documentation, torch.Tensor.stride: the stride is the jump needed to go from one element to the next in a dimension.
  • PyTorch 2.15 documentation, torch.from_numpy: the tensor and the array share memory, and changes to one show in the other.
  • PyTorch 2.15 documentation, Broadcasting semantics: shapes are compared from the trailing dimension, and sizes must be equal, 1, or missing, as in NumPy.
  • PyTorch 2.15 documentation, torch.set_default_dtype (the default floating-point dtype is initially float32, and Python floats become it), torch.set_printoptions (printing precision 4 by default) and torch.testing (assert_close uses rtol \(1.3 \times 10^{-6}\) and atol \(10^{-5}\) for float32, \(10^{-7}\) and \(10^{-7}\) for float64, and checks the dtype by default).
  • A. Zhang, Z. C. Lipton, M. Li and A. J. Smola, Dive into Deep Learning, section 2.1: broadcasting (2.1.4), saving memory with in-place operations (2.1.5), and a torch tensor and a NumPy array sharing their underlying memory after conversion (2.1.6).

Next. DL01.7.2 Autograd adds the one thing NumPy arrays do not have: a record of how each tensor was computed, so that PyTorch can run the graph backwards as your engine of DL01.6 did.

This lesson unlocks

DL01.7.2 Autograd

Your progress is saved in this browser only. No account, no tracking.
Back to top