evals.retrievals

Helper functions for evaluating the retrieval step of retrieval-augmented generation.

compute_precisions_at_k(relevance_classifications)

Given a list of relevance classifications, computes precision@k for k = 1, 2, …, n, where n is the length of the input list.

Parameters:

relevance_classifications (List[Optional[bool]]) – A list of relevance classifications for a set of retrieved documents, sorted by order of retrieval (i.e., the first element is the classification for the first retrieved document, the second element is the classification for the second retrieved document, etc.). The list may contain None values, which indicate that the relevance classification for the corresponding document is unknown.

Returns:

A list of precision@k values for k = 1, 2, …, n, where n is the

length of the input list. The first element is the precision@1 value, the second element is the precision@2 value, etc. If the input list contains any None values, those values are omitted when computing the precision@k values.

Return type:

List[Optional[float]]