这是indexloc提供的服务,不要输入任何密码
Skip to content

使用Randeng-Pegasus-523M-Summary-Chinese生成摘要报错PegasusTokenizer' object has no attribute 'vocab #461

@flystarts

Description

@flystarts

我的代码来自 https://huggingface.co/IDEA-CCNL/Randeng-Pegasus-523M-Summary-Chinese

from transformers import PegasusForConditionalGeneration
from tokenizers_pegasus import PegasusTokenizer

model = PegasusForConditionalGeneration.from_pretrained("IDEA-CCNL/Randeng-Pegasus-523M-Summary-Chinese")
tokenizer = PegasusTokenizer.from_pretrained("IDEA-CCNL/Randeng-Pegasus-523M-Summary-Chinese")

text = "据微信公众号“界面”报道"
inputs = tokenizer(text, max_length=1024, return_tensors="pt")

summary_ids = model.generate(inputs["input_ids"])
print(tokenizer.batch_decode(summary_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0])

执行时报错:
tokenizer = PegasusTokenizer.from_pretrained("IDEA-CCNL/Randeng-Pegasus-523M-Summary-Chinese")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Administrator\Downloads\Fengshenbang-LM-main\Fengshenbang-LM-main.venv\Lib\site-packages\transformers\tokenization_utils_base.py", line 2110, in from_pretrained
return cls._from_pretrained(
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Administrator\Downloads\Fengshenbang-LM-main\Fengshenbang-LM-main.venv\Lib\site-packages\transformers\tokenization_utils_base.py", line 2336, in _from_pretrained
tokenizer = cls(*init_inputs, **init_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Administrator\Downloads\Fengshenbang-LM-main\Fengshenbang-LM-main\fengshen\examples\pegasus\tokenizers_pegasus.py", line 156, in init
super().init(
File "C:\Users\Administrator\Downloads\Fengshenbang-LM-main\Fengshenbang-LM-main.venv\Lib\site-packages\transformers\tokenization_utils.py", line 367, in init
self._add_tokens(
File "C:\Users\Administrator\Downloads\Fengshenbang-LM-main\Fengshenbang-LM-main.venv\Lib\site-packages\transformers\tokenization_utils.py", line 467, in _add_tokens
current_vocab = self.get_vocab().copy()
^^^^^^^^^^^^^^^^
File "C:\Users\Administrator\Downloads\Fengshenbang-LM-main\Fengshenbang-LM-main\fengshen\examples\pegasus\tokenizers_pegasus.py", line 209, in get_vocab
return dict(self.vocab, **self.added_tokens_encoder)
^^^^^^^^^^
AttributeError: 'PegasusTokenizer' object has no attribute 'vocab'

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions