Info / Feature request for correction of equirectangular panoramas

[testor2897]

Hi,

I am not sure wheter equilib is supporting my request.

I would like to programme a panorama correction tool which is able to:

• align the panorama to a new vertical axis
• lower or lift the horizon
• shift the forward direction of the panorama (already solved with PIL [pillow] with picture shifting)

The input is an (all around = 360°) equirectangular panorama picture (2:1).
The output shall be the same panorama with corrected axis, corrected horizon and corrected forward direction in equirectangular format.

Analogue to my current tool (Bildshifter = picture shifter) I would like to do the following:

• convert the equirectangular panorama picture to a small version for a faster preview
• show panorama picture in browser (flask / flaskwebgui) with javascript panorama viewer (PanoradoViewer.js)
• interactively change the vertical angle, horizontal line and forward direction line in browser
• take the values of Theta (angle of vertical [=upright] correction in starting view), Gamma (angle of horizon correction) and Phi (shifting angle for new fwd direction) and calculate a new equirectangular panorama in 3 steps:

1. correct vertical angle
2. correct horizon
3. shift fwd view position (picture shifting already coded)

I would use an interactive web page (with flask / flaskwebgui) with a javascript viewer (panorado) for the ineractive part (already working with picture shifting).
The small preview picture for the web page is created using PIL (pillow).

Is the recalculation possible with equilib and numpy?
Is there example code doing these operations (mainly step 1 and 2) already existing?

I would be grateful for any help.

Greetings

Sebastian

[haruishi43]

@testor2897 Thanks if it is a simple vertical or horizontal rotation, equilib does support this. I might not be fully understanding your needs, but if you are talking about projection or lens distortion correction, equilib does not currently have those functions. Here is a simple script to rotate via equi2equi (https://github.com/haruishi43/equilib/blob/master/scripts/equi2equi_numpy.py)

[testor2897]

Hi,
Thanks a lot, that what I was looking for.

Do I understand you right that the following code is sufficient for the correction 1 and 2:

    # Rotation:
    rot = {"roll": 0, "pitch": 0, "yaw": np.pi / 4}  # vertical, horizontal

    # Initialize equi2equi
    equi2equi = Equi2Equi(height=320, width=640, mode="bilinear")

    # Open Image
    src_img = Image.open(path)
    src_img = preprocess(src_img)

    out_img = equi2equi(src=src_img, rots=rot)

Is the following interpretation correct:

• roll around FWD axis?
• pitch around left axis?
• yaw around bottom axis?

I tried to picturize this:

Sorry for the awfull yaw arrow...

Or do you use a different version of roll, pitch and yaw?

What are the following lines doing?

    out_img = np.transpose(out_img, (1, 2, 0))
    out_img = Image.fromarray(out_img)

[haruishi43]

The coordinate system uses a right hand rule where the x-axis faces forward and z-axis faces up.

• roll: counter-clockwise rotation about the x-axis
• pitch: counter-clockwise rotation about the y-axis
• yaw: counter-clockwise rotation about the z-axis

    out_img = np.transpose(out_img, (1, 2, 0))
    out_img = Image.fromarray(out_img)

The lines here just changes the dimension order because equilib uses the pattern common in deep learning (CHW), but images are usually loaded as (HWC), where C=3 (R, G, B) when using RGB images.

[testor2897]

Fantastic.
I will test it. Thanks a lot for your help.

That sounds easy enough.

[testor2897]

Hi,

I tested it out and it works fantastic for smaller and medium size panoramas.

Unfortunately I would like to use it for panoramas size 20288 x 10144 (24 Bits).
The transposed image (20288 x 10144) has the size of 602934.0 Kbytes.

For this size the app throws the following error:

...
File "H:\Python\Bildshifter\mit_Session_Drehung\testapp.py", line 252, in hello2
im_new = equi2equi(src=im_new, rots=rot)
File "D:\venv\equilib\lib\site-packages\equilib\equi2equi\base.py", line 55, in call
**kwargs,
File "D:\venv\equilib\lib\site-packages\equilib\equi2equi\base.py", line 110, in equi2equi
**kwargs,
File "D:\venv\equilib\lib\site-packages\equilib\equi2equi\numpy.py", line 169, in run
out = numpy_grid_sample(img=src, grid=grid, out=out, mode=mode)
File "D:\venv\equilib\lib\site-packages\equilib\grid_sample\numpy\grid_sample.py", line 36, in grid_sample
out = bilinear(img, grid, out)
File "D:\venv\equilib\lib\site-packages\equilib\grid_sample\numpy\bilinear.py", line 49, in bilinear
out[i, ...] = interp2d(p00, p10, p01, p11, dy, dx)
File "D:\venv\equilib\lib\site-packages\equilib\grid_sample\numpy\bilinear.py", line 15, in interp2d
return interp(f0, f1, dy, 1)
File "D:\venv\equilib\lib\site-packages\equilib\grid_sample\numpy\bilinear.py", line 9, in interp
return v0 * (1 - d) / L + v1 * d / L
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 4.60 GiB for an array with shape (3, 10144, 20288) and data type float64

I tried to use np.unit8 as datatype in the preprocessing but it didn't help.

My code is (only using pillow image source):

def preprocess(img: Image.Image, is_cv2: bool = False) -> np.ndarray:
    """Preprocesses image"""
    # Sometimes images are RGBA
    img = img.convert("RGB")
    img = np.asarray(img, dtype=np.uint8)
    assert len(img.shape) == 3, "input must be dim=3"
    assert img.shape[-1] == 3, "input must be HWC"
    img = np.transpose(img, (2, 0, 1))
    print("img transposed:", img.nbytes / 1024, "Kbytes")
    return img

and the main code is:

...
          im = Image.open(fileName)
          if im.mode in ("RGBA", "P"):
            im = im.convert("RGB")

...

          # Rotation:
          rollValue = rollAngle * np.pi / 180
          yawValue = yawAngle * np.pi / 180
          rot = {"roll": rollValue, "pitch": 0, "yaw": yawValue}  # vertical, horizontal
          
          # Initialize equi2equi
          equi2equi = Equi2Equi(height=actHeight, width=actWidth, mode="bilinear")
          
          im_new = preprocess(im)
          im_new = equi2equi(src=im_new, rots=rot)

          im_new = np.transpose(im_new, (1, 2, 0))
          im_new = Image.fromarray(im_new)

          im_new.save(newOutFile, "JPEG", quality=95)

Is there a way to use bigger size panoramas? Do I do anything wrong?

[haruishi43]

I've never used a panorama image that is 20k x 10k. Perhaps it might require more RAM to handle such extreme size.
And your logs does state that you couldn't allocate 4.6GiB of memory:

Unable to allocate 4.60 GiB for an array with shape (3, 10144, 20288) and data type float64

First, can you check the RAM size and see if it has more than 8GiB?
Then, can you try using float32 instead of uint8?

def preprocess(img: Image.Image, is_cv2: bool = False) -> np.ndarray:
    """Preprocesses image"""
    # Sometimes images are RGBA
    img = img.convert("RGB")
    img = np.asarray(img, dtype=np.float32)
    assert len(img.shape) == 3, "input must be dim=3"
    assert img.shape[-1] == 3, "input must be HWC"
    img = np.transpose(img, (2, 0, 1))
    print("img transposed:", img.nbytes / 1024, "Kbytes")
    return img

Also, using pytorch on cpu mode might be useful since in pytorch I supported float16 which should consume less memory. The script is scripts/equi2equi_torch.py, but you might have to tinker with the code a little bit:

• remove the device = torch.device(cuda) if you want to use cpu
• add img = img.to(torch.float16) after line 40. This can also be torch.float32 or any other data types.

[testor2897]

Hi,

Thanks a lot for the hints.
I assume my memory problem is connected to the fact that I use a venv environment with command prompt (Windows 10 - 64 bit).
In principle I do have lots of memory (64GB installed).

With your tips I found a solution for the problem.
I now use the pytorch approach (without cuda):

def preprocess_torch(img: Union[np.ndarray, Image.Image], is_cv2: bool = False) -> torch.Tensor:
    """Preprocesses image"""
    if isinstance(img, np.ndarray) and is_cv2:
        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    if isinstance(img, Image.Image):
        # Sometimes images are RGBA
        img = img.convert("RGB")

    to_tensor = transforms.Compose([transforms.ToTensor()])
    img = to_tensor(img)
    assert len(img.shape) == 3, "input must be dim=3"
    assert img.shape[0] == 3, "input must be HWC"
    return img


def postprocess(img: torch.Tensor, to_cv2: bool = False) -> Union[np.ndarray, Image.Image]:
    if to_cv2:
        img = np.asarray(img.to("cpu").numpy() * 255, dtype=np.uint8)
        img = np.transpose(img, (1, 2, 0))
        img = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)
        return img
    else:
        to_PIL = transforms.Compose([transforms.ToPILImage()])
        img = img.to("cpu")
        img = to_PIL(img)
        return img

...

          if im.mode in ("RGBA", "P"):
            im = im.convert("RGB")

          # Rotation:
          rollValue = rollAngle * np.pi / 180
          yawValue = yawAngle * np.pi / 180
          rot = {"roll": rollValue, "pitch": 0, "yaw": yawValue}  # vertical, horizontal
          
          # Initialize equi2equi
          equi2equi = Equi2Equi(height=actHeight, width=actWidth, mode="bilinear")
          
          im_new = preprocess_torch(im)
          im_new = equi2equi(src=im_new, rots=rot)

          im_new = postprocess(im_new)
          
          im_new.save(newOutFile, "JPEG", quality=95)

Thank you very much for the help.

The other approach with:
img = np.asarray(img, dtype=np.float32)

results in an error:

Traceback (most recent call last):
  File "D:\venv\equilib\lib\site-packages\PIL\Image.py", line 3089, in fromarray
    mode, rawmode = _fromarray_typemap[typekey]
KeyError: ((1, 1, 3), '<f4')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\venv\equilib\lib\site-packages\flask\app.py", line 2529, in wsgi_app
    response = self.full_dispatch_request()
  File "D:\venv\equilib\lib\site-packages\flask\app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "D:\venv\equilib\lib\site-packages\flask\app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "D:\venv\equilib\lib\site-packages\flask\app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "H:\Python\Bildshifter\mit_Session_Drehung\testapp.py", line 260, in hello2
    im_new = Image.fromarray(im_new)
  File "D:\venv\equilib\lib\site-packages\PIL\Image.py", line 3092, in fromarray
    raise TypeError(msg) from e
TypeError: Cannot handle this data type: (1, 1, 3), <f4

I also tried to use grayscale images (8bit PNG files), but even with a adapted preprocessing the function

equi2equi(src=im_new, rots=rot)

reports the error that rots is not list.

I wondered if it possible to process the RGB channels seperately and to "simulate" each channel would be a separate grayscale image.
Is each channel calculated on its own or is there any dependency between the RGB channels?

Is there a possibility to process grayscale images (8bit PNG files) in principle?
if it is possible, do you have an example code incl. preprocessing?

Again thanks a lot for your support the new editor is now working. Great library.