Answer Leakage

class hinteval.cores.evaluation_metrics.answer_leakage.Lexical(method: Literal['include_stop_words', 'exclude_stop_words'] = 'include_stop_words', spacy_pipeline: Literal['en_core_web_sm', 'en_core_web_lg', 'en_core_web_md', 'en_core_web_trf'] = 'en_core_web_sm', batch_size: int = 256, checkpoint: bool = False, checkpoint_step: int = 1, enable_tqdm: bool = False)

Class for evaluating answer leakage of hints using lexical comparison [36] .

batch_size

The batch size for processing.

Type:

int

checkpoint

Whether checkpointing is enabled.

Type:

bool

checkpoint_step

Step interval for checkpointing.

Type:

int

enable_tqdm

Whether the tqdm progress bar is enabled.

Type:

bool

References

See also

ContextualEmbeddings

Class for evaluating answer leakage of hints using contextual word embeddings.

evaluate(instances: List[Instance], **kwargs) → List[List[float]]

Evaluates the answer leakage of the hints of the given instances using the lexical comparison [38].

Parameters:

  • instances (List[Instance]) – List of instances to evaluate.

  • **kwargs – Additional keyword arguments.

Returns:

List of answer leakage scores for each instance.

Return type:

List[List[float]]

Notes

This function stores the scores as Metric objects within the metrics attribute of the Hint of the instances, with names based on the model, such as “answer-leakage-lexical-include_stop_words-sm”.

Examples

>>> from hinteval.cores import Instance, Question, Hint, Answer
>>> from hinteval.evaluation.answer_leakage import Lexical
>>>
>>> lexical = Lexical(method='include_stop_words')
>>> instance_1 = Instance(
...    question=Question('What is the capital of Austria?'),
...    answers=[Answer('Vienna')],
...    hints=[Hint('This city, once home to Mozart and Beethoven.'),
...           Hint('This city is called as Vienna.')])
>>> instance_2 = Instance(
...    question=Question('Who was the president of USA in 2009?'),
...    answers=[Answer('Barack Obama')],
...    hints=[Hint('His lastname is Obama.'),
...           Hint('He was named the 2009 Nobel Peace Prize laureate.')])
>>> instances = [instance_1, instance_2]
>>> results = lexical.evaluate(instances)
>>> print(results)
# [[0, 1], [1, 0]]
>>> metrics = [f'{metric_key}: {metric_value.value}' for
...           instance in instances
...           for hint in instance.hints for metric_key, metric_value in
...           hint.metrics.items()]
>>> print(metrics)
# ['answer-leakage-lexical-include_stop_words-sm: 0', 'answer-leakage-lexical-include_stop_words-sm: 1',
#  'answer-leakage-lexical-include_stop_words-sm: 1', 'answer-leakage-lexical-include_stop_words-sm: 0']

References

See also

ContextualEmbeddings

Class for evaluating answer leakage of hints using contextual word embeddings.

release_memory()

Releases the memory used by the class instance.

This method deletes the instance of the class and triggers garbage collection to free up memory.

Examples

>>> from hinteval.evaluation.familiarity import Wikipedia
>>>
>>> wikipedia = Wikipedia(spacy_pipeline='en_core_web_sm')
>>> wikipedia.release_memory()
class hinteval.cores.evaluation_metrics.answer_leakage.ContextualEmbeddings(sbert_model: str = 'all-mpnet-base-v2', method: Literal['include_stop_words', 'exclude_stop_words'] = 'include_stop_words', spacy_pipeline: Literal['en_core_web_sm', 'en_core_web_lg', 'en_core_web_md', 'en_core_web_trf'] = 'en_core_web_sm', checkpoint: bool = False, checkpoint_step: int = 1, enable_tqdm: bool = False)

Class for evaluating answer leakage of hints using contextual word embeddings.

checkpoint

Whether checkpointing is enabled.

Type:

bool

checkpoint_step

Step interval for checkpointing.

Type:

int

enable_tqdm

Whether the tqdm progress bar is enabled.

Type:

bool

See also

Lexical

Class for evaluating answer leakage of hints using lexical comparison.

evaluate(instances: List[Instance], **kwargs) → List[List[float]]

Evaluates the answer leakage of the hints of the given instances using the contextual word embeddings.

Parameters:

  • instances (List[Instance]) – List of instances to evaluate.

  • **kwargs – Additional keyword arguments.

Returns:

List of answer leakage scores for each instance.

Return type:

List[List[float]]

Notes

This function stores the scores as Metric objects within the metrics attribute of the Hint of the instances, with names based on the model, such as “answer-leakage-contextual-include_stop_words-sm”.

Examples

>>> from hinteval.cores import Instance, Question, Hint, Answer
>>> from hinteval.evaluation.answer_leakage import ContextualEmbeddings
>>>
>>> contextual = ContextualEmbeddings(sbert_model='paraphrase-multilingual-mpnet-base-v2')
>>> instance_1 = Instance(
...    question=Question('What is the capital of Austria?'),
...    answers=[Answer('Vienna')],
...    hints=[Hint('This city, once home to Mozart and Beethoven.'),
...           Hint('This city is called as Vienna.')])
>>> instance_2 = Instance(
...    question=Question('Who was the president of USA in 2009?'),
...    answers=[Answer('Barack Obama')],
...    hints=[Hint('His lastname is Obama.'),
...           Hint('He was named the 2009 Nobel Peace Prize laureate.')])
>>> instances = [instance_1, instance_2]
>>> results = contextual.evaluate(instances)
>>> print(results)
# [[0.495, 1.0], [0.967, 0.332]]
>>> metrics = [f'{metric_key}: {metric_value.value}' for
...           instance in instances
...           for hint in instance.hints for metric_key, metric_value in
...           hint.metrics.items()]
>>> print(metrics)
# ['answer-leakage-lexical-include_stop_words-sm: 0.495', 'answer-leakage-lexical-include_stop_words-sm: 1.0',
#  'answer-leakage-lexical-include_stop_words-sm: 0.967', 'answer-leakage-lexical-include_stop_words-sm: 0.332']

See also

Lexical

Class for evaluating answer leakage of hints using lexical comparison.

release_memory()

Releases the memory used by the class instance.

This method deletes the instance of the class and triggers garbage collection to free up memory.

Examples

>>> from hinteval.evaluation.familiarity import Wikipedia
>>>
>>> wikipedia = Wikipedia(spacy_pipeline='en_core_web_sm')
>>> wikipedia.release_memory()