evals.generate#

llm_generate(dataframe, template, model, system_instruction=None, verbose=False, output_parser=None, include_prompt=False, include_response=False, run_sync=False, concurrency=None)#

Generates a text using a template using an LLM. This function is useful if you want to generate synthetic data, such as irrelevant responses :param dataframe: A pandas dataframe in which each row

represents a record to be used as in input to the template. All template variable names must appear as column names in the dataframe (extra columns unrelated to the template are permitted).

Parameters:

  • template (Union[PromptTemplate, str]) – The prompt template as either an instance of PromptTemplate or a string. If the latter, the variable names should be surrounded by curly braces so that a call to .format can be made to substitute variable values.

  • model (BaseEvalModel) – An LLM model class.

  • system_instruction (Optional[str], optional) – An optional system message.

  • verbose (bool, optional) – If True, prints detailed information to stdout such as model invocation parameters and retry info. Default False.

  • output_parser (Callable[[str, int], Dict[str, Any]], optional) – An optional function that takes each generated response and response index and parses it to a dictionary. The keys of the dictionary should correspond to the column names of the output dataframe. If None, the output dataframe will have a single column named “output”. Default None.

  • include_prompt (bool, default=False) – If True, includes a column named prompt in the output dataframe containing the prompt used for each generation.

  • include_response (bool, default=False) – If True, includes a column named response in the output dataframe containing the raw response from the LLM prior to applying the output parser.

  • run_sync (bool, default=False) – If True, forces synchronous request submission. Otherwise evaluations will be run asynchronously if possible.

  • concurrency (Optional[int], default=None) – The number of concurrent evals if async submission is possible. If not provided, a recommended default concurrency is set on a per-model basis.

Returns:

A dataframe where each row

represents the generated output.

Return type:

generations_dataframe (pandas.DataFrame)

On this page

This Page