Model

class hinteval.cores.model.model.AnswerAware(model_name: str, api_key: str = None, base_url: str = 'https://api.together.xyz/v1', num_of_hints: int = 5, parse_llm_response: Callable[[str], List[str]] = None, temperature: float = 0.7, top_p: float = 1.0, max_tokens: int = 512, batch_size: int = 2, checkpoint: bool = False, checkpoint_step: int = 1, enable_tqdm=False)

Class for automatically generating hints for questions that are aware of their answers [39].

checkpoint

Whether checkpointing is enabled.

Type:

bool

checkpoint_step

Step interval for checkpointing.

Type:

int

enable_tqdm

Whether the tqdm progress bar is enabled.

Type:

bool

References

[39]

Jamshid Mozafari, Anubhav Jangra, and Adam Jatowt. 2024. TriviaHG: A Dataset for Automatic Hint Generation from Factoid Questions. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ‘24). Association for Computing Machinery, New York, NY, USA, 2060–2070. https://doi.org/10.1145/3626772.3657855

See also

AnswerAgnostic

Class for automatically generating hints for questions that are unaware of their answers.

generate(instances: List[Instance], **kwargs) → List[List[str]]

Generates hints for a list of instances using the configured large language model [42].

Parameters:

• instances (List[Instance]) – A list of instances, where each instance contains a question and its corresponding answer.

• **kwargs – Additional keyword arguments.

Returns:

A list of lists containing generated hints for each instance.

Return type:

List[List[str]]

Notes

This function stores the generated hints as Hint objects within the instances attribute of the Subset.

Examples

>>> from hinteval.cores import Instance, Question, Answer
>>> from hinteval.model import AnswerAware
>>>
>>> answer_aware = AnswerAware(model_name='meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo',
...                           api_key='your_api_key',
...                           base_url='base_url',
...                           num_of_hints=2)
>>> instance_1 = Instance(question=Question('What is the capital of Austria?'),
...                      answers=[Answer('Vienna')], hints=[])
>>> instance_2 = Instance(question=Question('Who was the president of USA in 2009?'),
...                      answers=[Answer('Barack Obama')], hints=[])
>>> instances = [instance_1, instance_2]
>>> results = answer_aware.generate(instances)
>>> print(results)
# [
#     ['The city is celebrated for its cultural landmarks like palaces and museums.',
#      'It is historically significant, located near the Danube River.'],
#     ['The president Obama, inaugurated in 2009, was the first African American to hold the office.',
#      'He won the Nobel Peace Prize in 2009 for enhancing international diplomacy.']
# ]

Raises:

RuntimeError – If hint generation fails due to model or API issues.

References

[42]

Jamshid Mozafari, Anubhav Jangra, and Adam Jatowt. 2024. TriviaHG: A Dataset for Automatic Hint Generation from Factoid Questions. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ‘24). Association for Computing Machinery, New York, NY, USA, 2060–2070. https://doi.org/10.1145/3626772.3657855

See also

AnswerAgnostic

Class for automatically generating hints for questions that are unaware of their answers.

release_memory()

Releases the memory used by the class instance.

This method deletes the instance of the class and triggers garbage collection to free up memory.

Examples

>>> from hinteval.model import AnswerAware
>>>
>>> answer_aware = AnswerAware(model_name='meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo',
...                           api_key='your_api_key',
...                           base_url='base_url',
...                           num_of_hints=2)
>>> answer_aware.release_memory()

class hinteval.cores.model.model.AnswerAgnostic(model_name: str, api_key: str = None, base_url: str = 'https://api.together.xyz/v1', num_of_hints: int = 5, parse_llm_response: Callable[[str], List[str]] = None, temperature: float = 0.7, top_p: float = 1.0, max_tokens: int = 512, batch_size: int = 2, checkpoint: bool = False, checkpoint_step: int = 1, enable_tqdm=False)

Class for automatically generating hints for questions that are unaware of their answers [43].

checkpoint

Whether checkpointing is enabled.

Type:

bool

checkpoint_step

Step interval for checkpointing.

Type:

int

enable_tqdm

Whether the tqdm progress bar is enabled.

Type:

bool

References

[43]

Jamshid Mozafari, Anubhav Jangra, and Adam Jatowt. 2024. TriviaHG: A Dataset for Automatic Hint Generation from Factoid Questions. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ‘24). Association for Computing Machinery, New York, NY, USA, 2060–2070. https://doi.org/10.1145/3626772.3657855

See also

AnswerAware

Class for automatically generating hints for questions that are aware of their answers.

generate(instances: List[Instance], **kwargs) → List[List[str]]

Generates hints for a list of instances using the configured large language model [46].

Parameters:

• instances (List[Instance]) – A list of instances, where each instance contains a question.

• **kwargs – Additional keyword arguments.

Returns:

A list of lists containing generated hints for each instance.

Return type:

List[List[str]]

Notes

This function stores the generated hints as Hint objects within the instances attribute of the Subset.

Examples

>>> from hinteval.cores import Instance, Question
>>> from hinteval.model import AnswerAgnostic
>>>
>>> answer_agnostic = AnswerAgnostic(model_name='meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo',
...                                 api_key='your_api_key',
...                                 num_of_hints=2)
>>> instance_1 = Instance(question=Question('What is the capital of Austria?'), answers=[], hints=[])
>>> instance_2 = Instance(question=Question('Who was the president of USA in 2009?'), answers=[], hints=[])
>>> instances = [instance_1, instance_2]
>>> results = answer_agnostic.generate(instances)
>>> print(results)
# [
#     ["The city you're looking for is located along the Danube River.",
#      "It's a city famous for its grand palaces, opera houses, and classical music heritage."],
#     ['The person who held the office in 2009 was the first African American to hold the position.',
#      'The president who took office in 2009 was a member of the Democratic Party and served two consecutive terms from 2009 to 2017.']
# ]

Raises:

RuntimeError – If hint generation fails due to model or API issues.

References

[46]

Jamshid Mozafari, Anubhav Jangra, and Adam Jatowt. 2024. TriviaHG: A Dataset for Automatic Hint Generation from Factoid Questions. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ‘24). Association for Computing Machinery, New York, NY, USA, 2060–2070. https://doi.org/10.1145/3626772.3657855

See also

AnswerAware

Class for automatically generating hints for questions that are aware of their answers.

release_memory()

Releases the memory used by the class instance.

This method deletes the instance of the class and triggers garbage collection to free up memory.

Examples

>>> from hinteval.model import AnswerAware
>>>
>>> answer_aware = AnswerAware(model_name='meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo',
...                           api_key='your_api_key',
...                           base_url='base_url',
...                           num_of_hints=2)
>>> answer_aware.release_memory()