Directory structure:
└── zhangming-19-rouge-sem/
├── README.md
├── calculate_lexical_similarity.py
├── calculate_semantic_similarity.py
├── candidate_summary_classifier.py
├── categorized_summary_rewriter.py
├── requirements.txt
├── rewritten_summary_scorer.py
├── run.sh
├── data/
│ ├── DialSummEval/
│ │ └── README.md
│ └── SummEval/
│ └── README.md
└── model/
└── README.md

Files Content:

================================================
File: README.md

ROUGE-SEM: Better evaluation of summarization using ROUGE combined with semantics

This project includes the source code for the paper ROUGE-SEM: Better evaluation of summarization using ROUGE combined with semantics, appearing at Expert Systems with Applications. Please cite this article as follows, if you use this code.

M. Zhang, C. Li, M. Wan et al., ROUGE-SEM: Better evaluation of summarization using ROUGE combined with semantics. Expert Systems With Applications (2023), doi: https://doi.org/10.1016/j.eswa.2023.121364.

Highlighted Features

• A framework of ROUGE combined with semantics is proposed for summarization evaluation.
• A classification of Summary based on semantic and lexical similarity to the reference.
• Variants of ROUGE-SEM outperform the corresponding variants of ROUGE consistently.

Requirements

We use Conda python 3.7 and strongly recommend that you create a new environment.

• Prerequisite: Python 3.7 or higher versions

shell
conda create -n ROUGE-SEM python=3.7
conda activate ROUGE-SEM

Environment

Install all packages in the requirement.txt

• Python 3.7
• PyTorch 1.4.0+cu100
• HuggingFace Transformers 4.16.2
• boto3 1.24.32
• numpy 1.21.4
• pandas 1.1.5
• regex 2022.7.9
• sentencepiece 0.1.96
• sklearn latest
• scipy
• datasets
• pandas
• scikit-learn
• prettytable
• gradio
• setuptools
• summ-eval

shell
pip3 install -r requirements.txt

Set Up for ROUGE

• Read more from this link.

shell
git clone https://github.com/summanlp/evaluation
export ROUGE_EVAL_HOME="yourPath/evaluation/ROUGE-RELEASE-1.5.5/data/"
pip install pyrouge
pyrouge_set_rouge_path yourPath/evaluation/ROUGE-RELEASE-1.5.5

Datasets

• SummEval

More details can be find in this link. please request and download the data from the original paper.

• DialSummEval

More details can be find in this link. please request and download the data from the original paper.

Models

Our released models can be download here. You can import these models by using HuggingFace's Transformers.

Example Use Cases

Command-line interface

python
source run.sh

Evaluate Text Summarization Step by Step

Given the source documents, reference summaries and some to-be-evaluated summaries, you can produce the ROUGE-SEM score for these candidate summaries with the code below:

• Calculate Lexical Similarity

python
python calculate_lexical_similarity.py -r reference.txt -c candidate.txt

• Calculate Semantic Similarity

python
python calculate_semantic_similarity.py -r reference.txt -c candidate.txt

• Candidate Summary Classifier

python
python candidate_summary_classifier.py -lex_score lexical_similarity.csv -sem_score semantic_similarity.csv

• Categorized Summary Rewriter

python
python categorized_summary_rewriter.py -category categorized_summary.csv -c candidate.txt

• Rewritten Summary Scorer

python
python rewritten_summary_scorer.py -r reference.txt -c new_candidate.csv

Citation

text
@article{ZHANG2023121364,
title = {ROUGE-SEM: Better evaluation of summarization using ROUGE combined with semantics},
journal = {Expert Systems with Applications},
pages = {121364},
year = {2023},
issn = {0957-4174},
doi = {https://doi.org/10.1016/j.eswa.2023.121364},
url = {https://www.sciencedirect.com/science/article/pii/S0957417423018663},
author = {Ming Zhang and Chengzhang Li and Meilin Wan and Xuejun Zhang and Qingwei Zhao},
keywords = {Automatic summarization evaluation, Semantic similarity, Lexical similarity, Contrastive learning, Back-translation},
abstract = {With the development of pre-trained language models and large-scale datasets, automatic text summarization has attracted much attention from the community of natural language processing, but the progress of automatic summarization evaluation has stagnated. Although there have been efforts to improve automatic summarization evaluation, ROUGE has remained one of the most popular metrics for nearly 20 years due to its competitive evaluation performance. However, ROUGE is not perfect, there are studies have shown that it is suffering from inaccurate evaluation of abstractive summarization and limited diversity of generated summaries, both caused by lexical bias. To avoid the bias of lexical similarity, more and more meaningful embedding-based metrics have been proposed to evaluate summaries by measuring semantic similarity. Due to the challenge of accurately measuring semantic similarity, none of them can fully replace ROUGE as the default automatic evaluation toolkit for text summarization. To address the aforementioned problems, we propose a compromise evaluation framework (ROUGE-SEM) for improving ROUGE with semantic information, which compensates for the lack of semantic awareness through a semantic similarity module. According to the differences in semantic similarity and lexical similarity, summaries are classified into four categories for the first time, including good-summary, pearl-summary, glass-summary, and bad-summary. In particular, the back-translation technique is adopted to rewrite pearl-summary and glass-summary that are inaccurately evaluated by ROUGE to alleviate lexical bias. Through this pipeline framework, summaries are first classified by candidate summary classifier, then rewritten by categorized summary rewriter, and finally scored by rewritten summary scorer, which are efficiently evaluated in a manner consistent with human behavior. When measured using Pearson, Spearman, and Kendall rank coefficients, our proposal achieves comparable or higher correlations with human judgments than several state-of-the-art automatic summarization evaluation metrics in dimensions of coherence, consistency, fluency, and relevance. This also suggests that improving ROUGE with semantics is a promising direction for automatic summarization evaluation.}
}

Get Involved

Should you have any query please contact me at [email protected].
Please create a GitHub issue if you have any questions, suggestions, requests or bug-reports.
Don't hesitate to send us an e-mail or report an issue, if something is broken or if you have further questions.

================================================
File: calculate_lexical_similarity.py

import sys
import os
import rouge
import bert_score
import argparse
import codecs
import nltk
nltk.download('punkt')
import numpy as np
import pandas as pd

class Logger(object):
def init(self, filename="Default.log"):
self.terminal = sys.stdout
self.log = open(filename, "a")

text
def write(self, message):
    self.terminal.write(message)
    self.log.write(message)

def flush(self):
    pass

def prepare_results(metric, p, r, f):
return '\t{}:\t {:5.2f}\t {:5.2f}\t {:5.2f}'.format(metric, 100.0 * p, 100.0 * r,
100.0 * f)

def test_rouge(candidates, references):
candidates = [line.strip() for line in candidates]
references = [line.strip() for line in references]
assert len(candidates) == len(references)

text
apply_avg = True
apply_best = False

evaluator = rouge.Rouge(metrics=['rouge-n', 'rouge-l', 'rouge-w'],
                        max_n=4,
                        limit_length=True,
                        length_limit=100,
                        length_limit_type='words',
                        apply_avg=apply_avg,
                        apply_best=apply_best,
                        alpha=0.5,  # Default F1_score
                        weight_factor=1.2,
                        stemming=True)

all_hypothesis = candidates
all_references = references

scores = evaluator.get_scores(all_hypothesis, all_references)

print(scores)

text
rougel = ""
for metric, results in sorted(scores.items(), key=lambda x: x[0]):
    if metric in ["rouge-1", "rouge-2", "rouge-l"]:
#        print(prepare_results(metric, results['p'], results['r'], results['f']))
        rougel = rougel + '{:5.2f}'.format(100 * results['f']) + "-"

print("ROUGE 1-2-L F:", rougel)
return rougel

def get_sents_str(file_path):
sents = []
with open(file_path, 'r', encoding='utf-8') as f:
for line in f.readlines():
line = line.strip()
line = line.lower()
sents.append(line)
return sents

if name == "main":
parser = argparse.ArgumentParser()
parser.add_argument('-c', type=str, default="candidate.txt",
help='candidate file')
parser.add_argument('-r', type=str, default="reference.txt",
help='reference file')
args = parser.parse_args()

text
references = get_sents_str(args.r)
candidates = get_sents_str(args.c)

#print(references)
sys.stdout = Logger('test_log.txt')

print('################### Total #################')
print('Rouge')
score_total = test_rouge(candidates, references)

print('################### Each #################')
R1_list = []
R2_list = []
RL_list = []

for item_c, item_r in zip(candidates, references):
    # 01 Rouge score
    score_tmp = test_rouge(item_c.split('\n'), item_r.split('\n'))
    R1_list.append(score_tmp.split('-')[0])
    R2_list.append(score_tmp.split('-')[1])
    RL_list.append(score_tmp.split('-')[2])

lex_score_list = []
for item_R1, item_R2, item_RL in zip(R1_list, R2_list, RL_list):
    lex_score_temp = 0.3*item_R1 + 0.3*item_R2 + 0.4*item_RL
    lex_score_list.append(lex_score_temp)

# Rouge + Bert 
name = ['ref', 'can', 'R1', 'R2', 'RL', 'lex_score']
temp = []
temp.append(references)
temp.append(candidates)
temp.append(R1_list)
temp.append(R2_list)
temp.append(RL_list)
temp.append(lex_score_list)
temp_df = np.array(temp)
temp_df = temp_df.T
temp_df = pd.DataFrame(temp_df, columns=name)
temp_df.to_csv('lexical_similarity.csv', encoding='utf-8')

================================================
File: calculate_semantic_similarity.py

-- coding: utf-8 --

"""
Created on Thu Mar 2 20:53:19 2023

@author: zhangming
"""
import os
import torch
import argparse
from scipy.spatial.distance import cosine
from transformers import AutoModel, AutoTokenizer
import pandas as pd
import tqdm
import csv

def get_sents_str(file_path):
sents = []
with open(file_path, 'r', encoding='utf-8') as f:
for line in f.readlines():
line = line.strip()
line = line.lower()
sents.append(line)
return sents

parser = argparse.ArgumentParser()
parser.add_argument('-c', type=str, default="candidate.txt",
help='candidate file')
parser.add_argument('-r', type=str, default="reference.txt",
help='reference file')
args = parser.parse_args()

ref_path = get_sents_str(args.r)
can_path = get_sents_str(args.c)

Import our models. The package will take care of downloading the models automatically

tokenizer = AutoTokenizer.from_pretrained("./model")
model = AutoModel.from_pretrained("./model")

read ref

with open(ref_path, 'r', encoding='utf-8') as f:
ref_summary = f.readlines()

Tokenize input texts

ref_inputs = tokenizer(ref_summary, padding=True, truncation=True, return_tensors="pt")

Get the embeddings

with torch.no_grad():
ref_embeddings = model(**ref_inputs, output_hidden_states=True, return_dict=True).pooler_output

read can

data_list = []
similar_lst = []

with open(can_path, 'r', encoding='utf-8') as f:
can_summary = f.readlines()
can_inputs = tokenizer(can_summary, padding=True, truncation=True, return_tensors="pt")

with torch.no_grad():
can_embeddings = model(**can_inputs, output_hidden_states=True, return_dict=True).pooler_output

Calculate cosine similarities

for i in tqdm.tqdm(range(len(can_embeddings))):
cosine_sim_ref_can = 1 - cosine(ref_embeddings[i], can_embeddings[i])
similar_lst.append(cosine_sim_ref_can)

for a,b,c in zip(ref_summary,can_summary,similar_lst):
x = {}
x['ref']= a
x['can']= b
x['Sem'] = c
data_list.append(x)

outpath = './semantic_similarity.csv'
with open(outpath, 'w', newline='', encoding='UTF-8') as f_c_csv:
writer = csv.writer(f_c_csv)
writer.writerow(['ref', 'can','sem_score'])
for nl in data_list:
writer.writerow(nl.values())

================================================
File: candidate_summary_classifier.py

-- coding: utf-8 --

"""
Created on Thu Mar 2 20:53:19 2023

@author: zhangming
"""
import os
import torch
import argparse
from scipy.spatial.distance import cosine
from transformers import AutoModel, AutoTokenizer
import pandas as pd
import tqdm
import csv

Alpha_parameter = 0.5
Beta_parameter = 0.5

parser = argparse.ArgumentParser()
parser.add_argument('-lex_score', type=str, default="lexical_similarity.csv",
help='candidate file')
parser.add_argument('-sem_score', type=str, default="semantic_similarity.csv",
help='reference file')
args = parser.parse_args()

lex_df = pd.read_csv(args.lex_score)
lex_score_list = lex_df["lex_score"].tolist()
references = lex_df["ref"].tolist()
candidates = lex_df["can"].tolist()

sem_df = pd.read_csv(args.sem_score)
sem_score_list = sem_df["sem_score"].tolist()

category_list = []
for item_lex, item_sem in zip(lex_score_list, sem_score_list):
if item_lex >= Alpha_parameter and item_sem >= Beta_parameter:
category_list.append(0)
if item_lex >= Alpha_parameter and item_sem < Beta_parameter:
category_list.append(1)
if item_lex < Alpha_parameter and item_sem >= Beta_parameter:
category_list.append(2)
if item_lex < Alpha_parameter and item_sem < Beta_parameter:
category_list.append(3)

name = ['ref', 'can', 'lex_score', 'sem_score', 'category']
temp = []
temp.append(references)
temp.append(candidates)
temp.append(lex_score_list)
temp.append(sem_score_list)
temp.append(category_list)
temp_df = np.array(temp)
temp_df = temp_df.T
temp_df = pd.DataFrame(temp_df, columns=name)
temp_df.to_csv('categorized_summary.csv', encoding='utf-8')

================================================
File: categorized_summary_rewriter.py

-- coding: utf-8 --

"""
Created on Tue Feb 23 17:22:41 2021

@author: zhangming
"""

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup
import xlwt
import time
import argparse
import pandas as pd
import numpy as np
from tqdm import tqdm

option = webdriver.ChromeOptions()
browser = webdriver.Chrome(chrome_options=option)
WAIT = WebDriverWait(browser, 10)
browser.set_window_size(1400, 900)

def get_url_En2Cn(text:str):
url_part1 = 'https://translate.google.com/?hl=zh-CN&sl=en&tl=zh-CN&text='
url_part2 = text
url_part3 = '&op=translate'
GOOGLE_TRANSLATE_URL = url_part1 + url_part2 + url_part3
return GOOGLE_TRANSLATE_URL

def get_url_Cn2En(text:str):
url_part1 = 'https://translate.google.com.hk/?sl=zh-CN&tl=en&text='
url_part2 = text
url_part3 = '&op=translate'
GOOGLE_TRANSLATE_URL = url_part1 + url_part2 + url_part3
return GOOGLE_TRANSLATE_URL

def get_url_En2Fr(text:str):
url_part1 = 'https://translate.google.com.hk/?sl=en&tl=fr&text='
url_part2 = text
url_part3 = '&op=translate'
GOOGLE_TRANSLATE_URL = url_part1 + url_part2 + url_part3
return GOOGLE_TRANSLATE_URL

def get_url_Fr2En(text:str):
url_part1 = 'https://translate.google.com.hk/?sl=fr&tl=en&text='
url_part2 = text
url_part3 = '&op=translate'
GOOGLE_TRANSLATE_URL = url_part1 + url_part2 + url_part3
return GOOGLE_TRANSLATE_URL

def translate_En2Cn(input_text):
try:
url = get_url_En2Cn(input_text)
browser.get(url)
time.sleep(3)
trans = WAIT.until(EC.presence_of_element_located((By.CSS_SELECTOR,
"#yDmH0d > c-wiz > div > div.WFnNle > c-wiz > div.OlSOob > c-wiz > div.ccvoYb.EjH7wc > div.AxqVh > div.OPPzxe > c-wiz.mxfMQ > div > div.usGWQd > div > div.lRu31")))
output_text = trans.text.replace('\n', '')
print('*'*10)
print(output_text)
time.sleep(2)
return output_text
except:
print('@'*10)
print('translate error!')
return input_text

def translate_Cn2En(input_text):
try:
url = get_url_Cn2En(input_text)
browser.get(url)
time.sleep(3)
trans = WAIT.until(EC.presence_of_element_located((By.CSS_SELECTOR,
"#yDmH0d > c-wiz > div > div.WFnNle > c-wiz > div.OlSOob > c-wiz > div.ccvoYb.EjH7wc > div.AxqVh > div.OPPzxe > c-wiz.mxfMQ > div > div.usGWQd > div > div.lRu31")))
output_text = trans.text.replace('\n', '')
print('*'*10)
print(output_text)
time.sleep(2)
return output_text
except:
print('@'*10)
print('translate error!')
return input_text

def translate_En2Fr(input_text):
try:
url = get_url_En2Fr(input_text)
browser.get(url)
time.sleep(3)
trans = WAIT.until(EC.presence_of_element_located((By.CSS_SELECTOR,
"#yDmH0d > c-wiz > div > div.WFnNle > c-wiz > div.OlSOob > c-wiz > div.ccvoYb.EjH7wc > div.AxqVh > div.OPPzxe > c-wiz.mxfMQ > div > div.usGWQd > div > div.lRu31")))
output_text = trans.text.replace('\n', '')
print('*'*10)
print(output_text)
time.sleep(2)
return output_text
except:
print('@'*10)
print('translate error!')
return input_text

def translate_Fr2En(input_text):
try:
url = get_url_Fr2En(input_text)
browser.get(url)
time.sleep(3)
trans = WAIT.until(EC.presence_of_element_located((By.CSS_SELECTOR,
"#yDmH0d > c-wiz > div > div.WFnNle > c-wiz > div.OlSOob > c-wiz > div.ccvoYb.EjH7wc > div.AxqVh > div.OPPzxe > c-wiz.mxfMQ > div > div.usGWQd > div > div.lRu31")))
output_text = trans.text.replace('\n', '')
print('*'*10)
print(output_text)
time.sleep(2)
return output_text
except:
print('@'*10)
print('translate error!')
return input_text

def get_sents_str(file_path):
sents = []
with open(file_path, 'r', encoding='utf-8') as f:
for line in f.readlines():
line = line.strip()
line = line.lower()
sents.append(line)
return sents

read data

parser = argparse.ArgumentParser()
parser.add_argument('-category', type=str, default="categorized_summary.csv",
help='candidate file')
parser.add_argument('-c', type=str, default="candidate.txt",
help='reference file')
args = parser.parse_args()

cate_df = pd.read_csv(args.category)
cate_list = cate_df["category"].tolist()
can_list = get_sents_str(args.c)

new_sum_Cn_list = []
new_sum_En_list = []
for item_cate, item_can in zip(cate_list, can_list):
if item_cate == 1 or item_cate == 2:
input_text = item_can
translate_text = translate_En2Fr(input_text)
new_sum_Cn_list.append(translate_text)
input_text = translate_text.replace('\n', '')
translate_text = translate_Fr2En(input_text)
new_sum_En_list.append(translate_text)
else:
new_sum_En_list.append(item_can)

save data

name = ['can', 'category', 'sum-Cn', 'new_can']
temp = []
temp.append(can_list)
temp.append(cate_list)
temp.append(new_sum_Cn_list)
temp.append(new_sum_En_list)
temp_df = np.array(temp)
temp_df = temp_df.T
temp_df = pd.DataFrame(temp_df, columns=name)
temp_df.to_csv('new_candidate.csv', encoding='utf-8', index=None)

browser.close()

================================================
File: requirements.txt

absl-py==1.0.0
aiohttp==3.6.2
alabaster==0.7.12
anaconda-client==1.7.2
anaconda-project==0.9.1
antlr4-python3-runtime==4.8
anyio==2.2.0
appdirs==1.4.4
argh==0.26.2
argon2-cffi==20.1.0
argparse-dataclass==0.2.1
asn1crypto==1.4.0
astroid==2.5
astropy==4.2.1
async-generator==1.10
async-timeout==3.0.1
atomicwrites==1.4.0
attrs==20.3.0
autopep8==1.5.6
Babel==2.9.0
backcall==0.2.0
backports.functools-lru-cache==1.6.4
backports.shutil-get-terminal-size==1.0.0
backports.tempfile==1.0
backports.weakref==1.0.post1
beautifulsoup4==4.9.3
bert-score==0.3.11
bitarray==2.1.0
bkcharts==0.2
black==19.10b0
bleach==3.3.0
blis==0.7.4
bokeh==2.3.2
boto==2.49.0
boto3==1.23.10
botocore==1.26.10
Bottleneck==1.3.2
brotlipy==0.7.0
cachetools==4.2.4
catalogue==2.0.2
certifi==2020.12.5
cffi==1.14.5
chardet==3.0.4
click==7.1.2
cloudpickle==1.6.0
clyent==1.2.2
colorama==0.4.4
conda-content-trust==0+unknown
conda-package-handling==1.7.3
conda-repo-cli==1.0.4
conda-verify==3.4.2
contextlib2==0.6.0.post1
cryptography==3.4.7
cycler==0.10.0
cymem==2.0.5
Cython==0.29.22
cytoolz==0.11.0
dask==2021.4.0
dataclasses==0.6
decorator==5.0.6
defusedxml==0.7.1
diff-match-patch==20200713
distributed==2021.4.1
docutils==0.17.1
entrypoints==0.3
et-xmlfile==1.0.1
fairseq==0.10.2
fastcache==1.1.0
filelock==3.0.12
flake8==3.9.0
Flask==1.1.2
fsspec==2023.1.0
future==0.18.2
gevent==21.1.2
glob2==0.7
gmpy2==2.0.8
google-auth==1.35.0
google-auth-oauthlib==0.4.1
greenlet==1.0.0
grpcio==1.51.1
h5py==2.10.0
HeapDict==1.0.1
html5lib==1.1
huggingface-hub==0.14.1
hydra-core==1.0.6
idna==2.10
imageio==2.9.0
imagesize==1.2.0
importlib-metadata==4.12.0
importlib-resources==5.1.0
iniconfig==1.1.1
intervaltree==3.1.0
ipykernel==5.3.4
ipython==7.22.0
ipython-genutils==0.2.0
ipywidgets==7.6.3
isort==5.8.0
itsdangerous==1.1.0
jdcal==1.4.1
jedi==0.17.2
jeepney==0.6.0
Jinja2==2.11.3
jmespath==1.0.0
joblib==1.0.1
json5==0.9.5
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==6.1.12
jupyter-console==6.4.0
jupyter-core==4.7.1
jupyter-packaging==0.7.12
jupyter-server==1.4.1
jupyterlab==3.0.14
jupyterlab-pygments==0.1.2
jupyterlab-server==2.4.0
jupyterlab-widgets==1.0.0
keyring==22.3.0
kiwisolver==1.3.1
lazy-object-proxy==1.6.0
libarchive-c==2.9
llvmlite==0.36.0
locket==0.2.1
lxml==4.6.3
Markdown==3.4.1
MarkupSafe==1.1.1
matplotlib==3.3.4
mccabe==0.6.1
mistune==0.8.4
mkl-fft==1.3.0
mkl-random==1.2.1
mkl-service==2.3.0
mock==4.0.3
more-itertools==8.7.0
mpmath==1.2.1
msgpack==1.0.2
multidict==4.7.6
multipledispatch==0.6.0
murmurhash==1.0.5
mypy-extensions==0.4.3
navigator-updater==0.2.1
nbclassic==0.2.6
nbclient==0.5.3
nbconvert==6.0.7
nbformat==5.1.3
nest-asyncio==1.5.1
networkx==2.5
nltk==3.6.1
nose==1.3.7
notebook==6.3.0
numba==0.53.1
numexpr==2.7.3
numpy==1.19.5
numpydoc==1.1.0
oauthlib==3.1.0
olefile==0.46
omegaconf==2.0.6
openpyxl==3.0.7
packaging==20.9
pandas==1.2.4
pandocfilters==1.4.3
parso==0.7.0
partd==1.2.0
path==15.1.2
pathlib==1.0.1
pathlib2==2.3.5
pathspec==0.7.0
pathy==0.4.0
patsy==0.5.1
pep8==1.7.1
pexpect==4.8.0
pickleshare==0.7.5
Pillow==8.2.0
pip==23.0.1
pkginfo==1.7.0
pluggy==0.13.1
ply==3.11
portalocker==2.2.1
preshed==3.0.5
prometheus-client==0.10.1
prompt-toolkit==3.0.17
protobuf==3.20.0
psutil==5.8.0
ptyprocess==0.7.0
py==1.10.0
py-rouge==1.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle==2.6.0
pycosat==0.6.3
pycparser==2.20
pycurl==7.43.0.6
pydantic==1.7.3
pyDeprecate==0.3.1
pydocstyle==6.0.0
pyerfa==1.7.3
pyflakes==2.2.0
Pygments==2.8.1
pylint==2.7.4
pyls-black==0.4.6
pyls-spyder==0.3.2
pyodbc==4.0.0-unsupported
pyOpenSSL==20.0.1
pyparsing==2.4.7
pyproject==1.3.1
PyQt5==5.12.1
PyQt5_sip==4.19.19
PyQtWebEngine==5.12.1
pyrsistent==0.17.3
PySocks==1.7.1
pytest==6.2.3
python-dateutil==2.8.1
python-jsonrpc-server==0.4.0
python-language-server==0.36.2
pytorch-lightning==1.5.10
pytz==2021.1
PyWavelets==1.1.1
pyxdg==0.27
PyYAML==5.4.1
pyzmq==20.0.0
QDarkStyle==2.8.1
QtAwesome==1.0.2
qtconsole==5.0.3
QtPy==1.9.0
regex==2020.11.13
requests==2.25.1
requests-oauthlib==1.3.0
rope==0.18.0
rouge==1.0.1
rsa==4.0
Rtree==0.9.7
ruamel.yaml==0.17.21
ruamel.yaml.clib==0.2.6
ruamel-yaml-conda==0.15.100
s3transfer==0.5.2
sacrebleu==1.5.0
sacremoses==0.0.46
scikit-image==0.18.1
scikit-learn==0.24.1
scipy==1.5.4
seaborn==0.11.1
SecretStorage==3.3.1
Send2Trash==1.5.0
sentencepiece==0.1.96
setuptools==59.5.0
simcse==0.4
simplegeneric==0.8.1
simplet5==0.1.4
singledispatch==0.0.0
sip==4.19.13
six==1.15.0
smart-open==3.0.0
sniffio==1.2.0
snowballstemmer==2.1.0
sortedcollections==2.1.0
sortedcontainers==2.3.0
soupsieve==2.2.1
spacy==3.0.5
spacy-legacy==3.0.3
Sphinx==4.0.1
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==1.0.3
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.4
sphinxcontrib-websupport==1.2.4
spyder==4.2.5
spyder-kernels==1.10.2
SQLAlchemy==1.4.15
srsly==2.4.1
statsmodels==0.12.2
sympy==1.8
tables==3.6.1
tblib==1.7.0
tensorboard==2.4.1
tensorboard-plugin-wit==1.8.1
terminado==0.9.4
testpath==0.4.4
textdistance==4.2.1
thinc==8.0.2
threadpoolctl==2.1.0
three-merge==0.1.1
tifffile==2020.10.1
tokenizers==0.10.3
toml==0.10.2
toolz==0.11.1
torch==1.10.0
torchmetrics==0.11.4
tornado==6.1
tqdm==4.57.0
traitlets==5.0.5
transformers==4.16.2
typed-ast==1.4.2
typer==0.3.2
typing-extensions==3.7.4.3
ujson==4.0.2
unicodecsv==0.14.1
urllib3==1.26.3
wasabi==0.8.2
watchdog==1.0.2
wcwidth==0.2.5
webencodings==0.5.1
Werkzeug==1.0.1
wheel==0.36.2
widgetsnbextension==3.5.1
wrapt==1.12.1
wurlitzer==2.1.0
xlrd==2.0.1
XlsxWriter==1.3.8
xlwt==1.3.0
xmltodict==0.12.0
yapf==0.31.0
yarl==1.7.0
zict==2.0.0
zipp==3.4.0
zope.event==4.5.0
zope.interface==5.3.0

================================================
File: rewritten_summary_scorer.py

import sys
import os
import rouge
import bert_score
import argparse
import codecs
import nltk
nltk.download('punkt')
import numpy as np
import pandas as pd

class Logger(object):
def init(self, filename="Default.log"):
self.terminal = sys.stdout
self.log = open(filename, "a")

text
def write(self, message):
    self.terminal.write(message)
    self.log.write(message)

def flush(self):
    pass

def prepare_results(metric, p, r, f):
return '\t{}:\t {:5.2f}\t {:5.2f}\t {:5.2f}'.format(metric, 100.0 * p, 100.0 * r,
100.0 * f)

def test_rouge(candidates, references):
candidates = [line.strip() for line in candidates]
references = [line.strip() for line in references]
assert len(candidates) == len(references)

text
apply_avg = True
apply_best = False

evaluator = rouge.Rouge(metrics=['rouge-n', 'rouge-l', 'rouge-w'],
                        max_n=4,
                        limit_length=True,
                        length_limit=100,
                        length_limit_type='words',
                        apply_avg=apply_avg,
                        apply_best=apply_best,
                        alpha=0.5,  # Default F1_score
                        weight_factor=1.2,
                        stemming=True)

all_hypothesis = candidates
all_references = references

scores = evaluator.get_scores(all_hypothesis, all_references)

print(scores)

text
rougel = ""
for metric, results in sorted(scores.items(), key=lambda x: x[0]):
    if metric in ["rouge-1", "rouge-2", "rouge-l"]:
#        print(prepare_results(metric, results['p'], results['r'], results['f']))
        rougel = rougel + '{:5.2f}'.format(100 * results['f']) + "-"

print("ROUGE 1-2-L F:", rougel)
return rougel

def get_sents_str(file_path):
sents = []
with open(file_path, 'r', encoding='utf-8') as f:
for line in f.readlines():
line = line.strip()
line = line.lower()
sents.append(line)
return sents

if name == "main":
parser = argparse.ArgumentParser()
parser.add_argument('-c', type=str, default="new_candidate.csv",
help='candidate file')
parser.add_argument('-r', type=str, default="reference.txt",
help='reference file')
args = parser.parse_args()

text
references = get_sents_str(args.r)

can_df = pd.read_csv(args.c)
candidates = cate_df["new_can"].tolist()

sys.stdout = Logger('test_log.txt')

print('################### Total #################')
print('Rouge')
score_total = test_rouge(candidates, references)

print('################### Each #################')
R1_list = []
R2_list = []
RL_list = []

for item_c, item_r in zip(candidates, references):
    # 01 Rouge score
    score_tmp = test_rouge(item_c.split('\n'), item_r.split('\n'))
    R1_list.append(score_tmp.split('-')[0])
    R2_list.append(score_tmp.split('-')[1])
    RL_list.append(score_tmp.split('-')[2])

# Rouge
name = ['ref', 'can', 'R1', 'R2', 'RL']
temp = []
temp.append(references)
temp.append(candidates)
temp.append(R1_list)
temp.append(R2_list)
temp.append(RL_list)
temp_df = np.array(temp)
temp_df = temp_df.T
temp_df = pd.DataFrame(temp_df, columns=name)
temp_df.to_csv('ROUGE-SEM.csv', encoding='utf-8')

================================================
File: run.sh

#!/bin/bash

echo "Calculate Lexical Similarity"
python calculate_lexical_similarity.py -r reference.txt -c candidate.txt

echo "Calculate Semantic Similarity"
python calculate_semantic_similarity.py -r reference.txt -c candidate.txt

echo "Candidate Summary Classifier"
python candidate_summary_classifier.py -lex_score lexical_similarity.csv -sem_score semantic_similarity.csv

echo "Categorized Summary Rewriter"
python categorized_summary_rewriter.py -category categorized_summary.csv -c candidate.txt

echo "Rewritten Summary Scorer"
python rewritten_summary_scorer.py -r reference.txt -c new_candidate.csv

================================================
File: data/DialSummEval/README.md

ROUGE-SEM: Better evaluation of summarization using ROUGE combined with semantics

add model here.

================================================
File: data/SummEval/README.md

ROUGE-SEM: Better evaluation of summarization using ROUGE combined with semantics

add data here.

================================================
File: model/README.md

ROUGE-SEM: Better evaluation of summarization using ROUGE combined with semantics

add model here.

Evaluate the scores of ROUGE-SEM ON THESE BENCHNARKS NOW--> aLSO GIVE ME METHID FOR CALCULATION)==>
Model Name Accuracy (%) ROUGE-1 ROUGE-2 SEM SCORE G-EVAL
LoRA Fine-tuned Language Model (Google Flan-T5) 44 0.44 0.18 0.28 1.2
PEFT & QLoRA Fine-Tuned Llama 3 Model (3.2B) 50 0.43 0.16 0.59 2.21
Naive RAG 54 0.41 0.21 0.7 2.6
Fine-tuned RAG 66 0.45 0.24 0.77 2.6
RAG Fusion 74 0.48 0.24 0.80 3.0
Advanced RAG 80 0.35 0.18 0.87 3.5

GIVE ME HIGHER ROUGE SEM SCORES PLEASE

eXPLAIN THE AVLUUATION METRIC IN SIMPLE LKANGUAGE

I WANT TO WRITE THIS AS A REPORT SEM SCORE WOTH THE METRICS ON ALL MODELS GENRATE IT FOR ME

The Challenges Facing with 45 Million Token Evaluation-->
Model Name Accuracy (%) ROUGE-1 ROUGE-2 SEM SCORE G-EVAL
LoRA Fine-tuned Language Model (Google Flan-T5) 44 0.44 0.18 0.28 1.2
PEFT & QLoRA Fine-Tuned Llama 3 Model (3.2B) 50 0.43 0.16 0.59 2.21
Naive RAG 54 0.41 0.21 0.7 2.6
Fine-tuned RAG 66 0.45 0.24 0.77 2.6
RAG Fusion 74 0.48 0.24 0.80 3.0
Advanced RAG 80 0.35 0.18 0.87 3.5

CHALLENGES OF EVALUATING --> 4.5*10^7 tokens on an offline GPU for building a low code chatbot on rtx 3060ti vram 8gb and 48 gb ram