![]() RuntimeError Traceback (most recent call last) This is probably because view() requires that the tensor to be contiguously stored so that it can do fast reshape in memory. The answer is it the view() function cannot be applied to a discontiguous tensor. So now the question is: what happens if I use a discontiguous tensor? Interestingly, repeat() and view() does not make it discontiguous. Ok, we can find that transpose(), narrow() and tensor slicing, and expand() will make the generated tensor not contiguous. # means the singleton dimension is repeated d3 timesįff = aaa.unsqueeze(2).repeat(1,1,8).view(2,-1,2) # if a tensor has a shape, it can only be expanded using "expand(d1,d2,d3)", which Now we try apply come functions to the tensor: bbb = aaa.transpose(0,1)Ĭcc = aaa.narrow(1,1,2) # equivalent to matrix slicing aaaĭdd = aaa.repeat(2,1) # The first dimension repeat once, the second dimension repeat twice This indicates that the elements in the tensor are stored contiguously. When moving along the second dimension (column by column), we need to move 1 step in the memory. The stride() return (3,1) means that: when moving along the first dimension by each step (row by row), we need to move 3 steps in the memory. ![]() First, let's create a contiguous tensor: aaa = torch.Tensor( ,] ) The contiguous() function is usually required when we first transpose() a tensor and then reshape (view) it. ntiguous() will create a copy of the tensor, and the element in the copy will be stored in the memory in a contiguous way. You're generally safe to assume everything will work, and wait until you get a RuntimeError: input is not contiguous where PyTorch expects a contiguous tensor to add a call to contiguous(). Normally you don't need to worry about this. ![]() When you call contiguous(), it actually makes a copy of the tensor such that the order of its elements in memory is the same as if it had been created from scratch with the same data. Here bytes are still allocated in one block of memory but the order of the elements is different! Note that the word "contiguous" is a bit misleading because it's not that the content of the tensor is spread out around disconnected blocks of memory. In the example above, x is contiguous but y is not because its memory layout is different to that of a tensor of same shape made from scratch. This is where the concept of contiguous comes in. In this example, the transposed tensor and original tensor share the same memory: x = torch.randn(3,2) Narrow(), view(), expand() and transpose()įor example: when you call transpose(), PyTorch doesn't generate a new tensor with a new layout, it just modifies meta information in the Tensor object so that the offset and stride describe the desired new shape. There are a few operations on Tensors in PyTorch that do not change the contents of a tensor, but change the way the data is organized. So obviously calling contiguous() in this example is necessary. RuntimeError: invalid argument 1: input is not contiguous at. I tried to leave out calling contiguous(), but this leads to an error message when computing the loss. How could targets be non-contiguous and inputs still be contiguous? So in general my question is, what does contiguous() do and why do I need it?įurther I don't understand why the method is called for the target sequence and but not the input sequence as both variables are comprised of the same data. So as a simple example, when using batch size 1 and seq_length 10 inputs and targets looks like this: inputs Variable containing: for i in range(0, ids.size(1) - seq_length, seq_length): But I'm still struggling to understand what calling contiguous() does, which occurs several times in the code.įor example in line 74/75 of the code input and target sequences of the LSTM are created.ĭata (stored in ids) is 2-dimensional where first dimension is the batch size. What it does in general is pretty clear to me. I was going through this example of a LSTM language model on github (link).
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |