CLIP: Input image size (490x490) doesn't match model (336x336).

[Lv996331209]

Hi,
There is an error when I run quickstart.py, which as following:
File "/root/.cache/huggingface/modules/transformers_modules/model/modeling_chartmoe.py", line 122, in encode_img
img_embeds, atts_img, img_target = self.img2emb(image)
File "/root/.cache/huggingface/modules/transformers_modules/model/modeling_chartmoe.py", line 126, in img2emb
img_embeds = self.vision_proj(self.vit(image.to(self.device)))
File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/model/build_mlp.py", line 133, in forward
image_forward_outs = self.vision_tower(
File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 1171, in forward
return self.vision_model(
File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 1094, in forward
hidden_states = self.embeddings(pixel_values, interpolate_pos_encoding=interpolate_pos_encoding)
File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 244, in forward
raise ValueError(
ValueError: Input image size (490490) doesn't match model (336336).

It seems like that the img_size is 490(from config.json) but the size of clip is 336(clip-vit-large-patch14-336)。Where did I operate incorrectly? Thank you so much!

[Lv996331209]

I read the issue #4 and notice the same error.

Both model and config are downloaded from HF, but this error still appears.

[Coobiw]

Hi, @Lv996331209 . 490 is indeed correct resolution we used in ChartMoE. Probably, you can check the version of core python packages like transformers according to requirements.txt. If you can give me more details to reproduce your bug, that would be really better~. I've never encountered this problem. But #4 has been solved. You can also discuss it with the author.

Thanks for you attention! If any question exists (like the version of some python package?), I'm very willing to communicate with you～!

[Coobiw]

@Lv996331209 . Hi, can you provide the versions of packages mentioned in requirements.txt. I will try to reproduce this bug~. Thanks!

[Coobiw]

@Lv996331209 . Hi, you can try using the same version of transformers as requirements.txt. I think it will help you~

[Lv996331209]

@Coobiw Hi, thanks for your help! It was the wrong version of transformers package, due to my unintentional upgrade. After I reverted the version as requirements.txt, the error was resolved. That was good!

Thank you so much for your suggestion! BTW, looking forward to your dataset. :)

[Coobiw]

Thanks for your kind words. I will add this problem into FAQs. Whole training pipeline codes and dataset are coming! I am organizing these contents.

[pengyu965]

You only need to add the args, interpolate_pos_encoding == True to the build_mlp.py on line 132.
Basically change it to:
image_forward_outs = self.vision_tower( images.to(device=self.device, dtype=self.dtype), output_hidden_states=True, interpolate_pos_encoding=True)

The new version of transformers add a assert error there and handle the interpolate of position encoding based on the different input image size there. So you don't need to write your own interpolation.