Code String Match

Definitions

Code String Match measures how close the generated code string is to the ground truth code string.

It outputs both the binary exact match score and the fuzzy match score in the range of (0.0 - 1.0).

Example Usage

Required data items: answer, ground_truth_answers

from continuous_eval.metrics.code import CodeStringMatch

datum = {
    "answer": "def function(x, y):\n  return x + y",
    "ground_truth_answers": [
        "def foo(x, y):\n  return x * y",
        "def foo(x, y):\n  return x + y",
    ],
},

metric = CodeStringMatch()
print(metric(**datum))

Example Output

{
    "Exact_Match_Score": 0,
    "Fuzzy_Match_Score": 0.89
}