I can't reproduce your results

[DingdongD]

As for the baseline perp, I got the perp higher than 3000. I dont know what happend, can you tell me why

8
9
10
11
Warning: Large NLL detected, perplexity may be unreliable
Perplexity: 7452.839
ptb
Evaluating ...
0
1
2
3
4
5
6
7
8
9
10
11
Warning: Large NLL detected, perplexity may be unreliable
Perplexity: 5195.345
c4
Evaluating ...
0
1
2
3
4
5
6
7
8
9
10
11
Warning: Large NLL detected, perplexity may be unreliable
Perplexity: 6933.685

[ldfrancis]

Hello @DingdongD I think this has to do with the transformers version you are using. Versions above 4.50.3 do not work well with OPT models, I'm still trying to figure this out. Try using version 4.50.3

You can run pip install transformers==4.50.3

[jimmy-adams]

Hello @DingdongD I think this has to do with the transformers version you are using. Versions above 4.50.3 do not work well with OPT models, I'm still trying to figure this out. Try using version 4.50.3

You can run pip install transformers==4.50.3

Hello,
Have you tried Llama2-7B? Does it work?

[jsvir]

I tried llama2-7B and llama3.2-1b and I also get high perplexity numbers. I think there is an issue with position_ids parameter which is missing in the code (llama.py)
For OPT-1.3b model I've got:

wikitext2: 17.46 ppl
ptb; 25.47 ppl
c4: 19.24

I use transformers==4.45.0