How do I use the model with my own image stream and pose data?

[moaazassali]

I think I might be over complicating the process, but here is what I tried so far:

I have tried calling the Neural_SLAM_Module() alone first to test out using my own images, but I can't seem to get that to work let alone the other models for exploration.

test.py file:

from model import RL_Policy, Local_IL_Policy, Neural_SLAM_Module
from PIL import Image
import torch
import numpy as np
from arguments import get_args

device = torch.device("cuda:0")

map_size_cm = 1000
map_resolution_cm = 5
map_size = map_size_cm // map_resolution_cm

full_w, full_h = map_size, map_size
local_w, local_h = full_w // 4, full_h // 4

# Initializing full and local map
full_map = torch.zeros(1, 4, full_w, full_h).float().to(device)
local_map = torch.zeros(1, 4, local_w, local_h).float().to(device)

# Initial full and local pose
full_pose = torch.zeros(1, 3).float().to(device)
local_pose = poses = torch.zeros(1, 3).float().to(device)

img = Image.open("test_img.jpg")

img = np.transpose(img, (2, 0, 1))

obs = torch.from_numpy(np.transpose(np.expand_dims(img, axis=0), (0, 1, 3, 2))).float().to(device)
last_obs = obs

print(obs.size()) # (1,3,128,128)

args = get_args()
args.device = device
args.frame_height = 128 # image height
args.frame_width = 128 # image height
args.map_resolution = map_resolution_cm
args.map_size_cm = map_size


nslam_module = Neural_SLAM_Module(args).to(device)
_, _, local_map[:, 0, :, :], local_map[:, 1, :, :], _, local_pose = \
                nslam_module(last_obs, obs, poses, local_map[:, 0, :, :],
                             local_map[:, 1, :, :], local_pose, build_maps=True)

This gives me the following error:

/home/.../Desktop/Neural-SLAM/test.py:27: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /opt/conda/conda-bld/pytorch_1678402411778/work/torch/csrc/utils/tensor_numpy.cpp:206.)
obs = torch.from_numpy(np.transpose(np.expand_dims(img, axis=0), (0, 1, 3, 2))).float().to(device)
torch.Size([1, 3, 128, 128])
Auto GPU config:
Number of processes: 24
Number of processes on GPU 0: 8
Number of processes per GPU: 16
/home/.../anaconda3/envs/neuralslam/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
warnings.warn(
/home/.../anaconda3/envs/neuralslam/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing weights=ResNet18_Weights.IMAGENET1K_V1. You can also use weights=ResNet18_Weights.DEFAULT to get the most up-to-date weights.
warnings.warn(msg)
/home/.../anaconda3/envs/neuralslam/lib/python3.10/site-packages/torch/nn/functional.py:4298: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
warnings.warn(
Traceback (most recent call last):
File "/home/.../Desktop/Neural-SLAM/test.py", line 44, in
nslam_module(last_obs, obs, poses, local_map[:, 0, :, :],
File "/home/.../anaconda3/envs/neuralslam/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/.../Desktop/Neural-SLAM/model.py", line 209, in forward
rot_mat, trans_mat = get_grid(st_poses,
File "/home/.../Desktop/Neural-SLAM/utils/model.py", line 40, in get_grid
rot_grid = F.affine_grid(theta1, torch.Size(grid_size))
File "/home/.../anaconda3/envs/neuralslam/lib/python3.10/site-packages/torch/nn/functional.py", line 4341, in affine_grid
return torch.affine_grid_generator(theta, size, align_corners)
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [1, 3] but got: [24, 3].