evaluation accuracy remains 0 #14
Description
Thanks for your great work in creating this dataset, I have questions while evaluating llama2-7b-chat on this dataset.
- The accuracy of the
llama2-7b-chatoutput remains 0 when the training goes, here is my code:
def acc(eval_preds:EvalPrediction):
logits, labels = eval_preds
preds = tokenizer.batch_decode(logits, skip_special_tokens=True)
labels = tokenizer.batch_decode(labels, skip_special_tokens=True)
save_results(preds, labels) # save results to json file
preds = [last_boxed_only_string(s) for s in preds]
correct = 0
total = 0
for pred, label in zip(preds, labels):
if is_equiv(pred, label):
correct += 1
total += 1
return {"accuracy": correct / total}
return acc
- whether the preprocess is required or recommended to use?