API reference¶
Import every public name from ebaml, for example from ebaml import describe_ebm. The sections below name the module that defines each name.
Describing graphs and models¶
ebaml.llm.descriptors
¶
Describe EBM graphs and models using LLMs.
The prompts and the chat conversations live in baml_src/talk_to_ebm.baml; this module extracts the graphs from the
EBM, converts them to text, and calls the generated baml_sdk functions. Each function takes one request
from ebaml.llm.requests, and the model field of the request chooses the LLM of the call.
EBMDescription
¶
The description of an EBM, and the parts of the model that it leaves out.
A reader of text learns from the text itself what the description leaves out. A program reads the same
information from the two lists, with no need to parse a log or the text.
Attributes:
| Name | Type | Description |
|---|---|---|
text |
str
|
The description. Pass it as |
skipped_terms |
list[str]
|
The names of the interaction terms that the description leaves out. The list is empty for a model without interactions. |
skipped_features |
list[str]
|
The names of the features that have no main-effect term, because |
LocalExplanation
¶
The explanation of one prediction: three kinds of explanation in words, and the facts they come from.
describe_graph(request)
¶
Ask the LLM to describe a graph. Uses chain-of-thought reasoning.
A reply cut off by max_tokens is retried with a simplified graph, up to 4 times. The logged usage leaves out the
cut-off attempts, because the provider error carries no token count. It is a lower bound, and the usage log
event has the number of those attempts in truncated_attempts. The description covers the main effect of the
feature. The prompt names the interaction terms of the feature, so that the description says that it leaves them
out, and an interactions_skipped warning names them too.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
request
|
DescribeGraphRequest
|
The EBM, the index of the feature, the number of sentences, the options of the graph, the prompts and the model. |
required |
Returns:
| Type | Description |
|---|---|
str
|
The description of the graph. |
sample_graph_tokens(graph, model=None, *, prompt=_DEFAULT_PROMPT_OPTIONS)
¶
Send one request with the text of a graph and return its token samples.
The request is the first turn of describe_graph. Each sample has the body of the request and the input tokens
that the provider counted. Set max_output_tokens in the model options to a small number, such as 16, to keep the
cost low: a reply that the limit cuts off is not an error here. scripts/llm/token_weights.py uses the function
to fit the token weights of a model.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
graph
|
str
|
The text of the graph, as |
required |
model
|
ModelOptions | str | None
|
The LLM of the call, as |
None
|
prompt
|
PromptOptions
|
The descriptions that the prompt uses. |
_DEFAULT_PROMPT_OPTIONS
|
Returns:
| Type | Description |
|---|---|
list[TokenSample]
|
One sample for each request that the provider counted. The list is empty when the provider reports no count. |
chat_with_mcp(command, arguments, model=None, *, max_steps=12)
¶
Chat in the terminal with an agent that reads an EBM through an MCP server.
BAML starts the server once and keeps it for the whole chat. Each message that you type gets an answer from the LLM, which calls the tools of the server as it needs them. The chat reads standard input and ends on an empty line.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
command
|
str
|
The program that starts the MCP server, for example the path of |
required |
arguments
|
Sequence[str]
|
The arguments of the program, for example |
required |
model
|
ModelOptions | str | None
|
The LLM of the chat, as |
None
|
max_steps
|
int
|
The most rounds of tool calls for one answer. |
12
|
Returns:
| Type | Description |
|---|---|
int
|
The number of questions that the chat answered. |
describe_ebm(request)
¶
Ask the LLM to describe an EBM.
Each feature graph is described like in describe_graph (concurrently, at most max_concurrency at a time,
each retried with a simplified graph if the reply is cut off by max_tokens), and then the LLM summarizes the
descriptions. The descriptions skip the interaction terms, and a feature that exclude or remove_terms left
out, because it has no graph. The summary prompt names both, so that the text says what it leaves out. An
interactions_skipped and a features_skipped warning name them too.
The logged usage leaves out the cut-off attempts, because the provider error carries no token count. It is a lower
bound, and the usage log event has the number of those attempts in truncated_attempts.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
request
|
DescribeEBMRequest
|
The EBM, the number of sentences, the number of features described at the same time, whether to show a progress bar on the error stream (one step for each feature and one for the summary; without a terminal, the bar writes one line for each step), the options of the graph, the prompts and the model. |
required |
Returns:
| Type | Description |
|---|---|
EBMDescription
|
The description of the EBM, and the names of the interaction terms and the features that it leaves out. |
explain_instance(request)
¶
Explain one prediction of an EBM with three explanations, computed exactly and put in words by the LLM.
The facts come from ebaml.local.compute_local_facts: the minimum change that flips the prediction
(counterfactual), a minimum set of inputs that fixes it (sufficient reason), and the share of completions of a
partially known input that get the same prediction (count completions). The global explanation of the model, e.g.
the output of describe_ebm, is prepended to each prompt.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
request
|
ExplainInstanceRequest
|
The EBM, the instance, the global explanation, the audience, the number of sentences, the language, the local options, the prompts and the model. |
required |
Returns:
| Type | Description |
|---|---|
LocalExplanation
|
The three explanations and the facts they come from. |
Options¶
ebaml.llm.options
¶
Options of the functions in ebaml.llm.descriptors.
Each class holds the settings of one part of the work: GraphOptions for the graph and its text, PromptOptions
for the prompts, ModelOptions for the LLM. ebaml.local.LocalOptions holds the settings of the local facts. A
default of an option is the default of the function that reads it, so a call with no options does what the function
does alone.
GraphOptions
¶
Options of EBMGraph.from_ebm and EBMGraph.to_text, and the token learner of the budget check.
Attributes:
| Name | Type | Description |
|---|---|---|
normalization |
Normalization | str
|
See |
use_feature_bounds |
bool
|
See |
classes |
Sequence[str] | None
|
See |
include_description |
bool
|
See |
feature_format |
FeatureType | str | None
|
See |
x_axis_precision |
int | None
|
See |
y_axis_precision |
AxisPrecision
|
See |
confidence_bounds |
bool
|
See |
confidence_level |
float
|
See |
max_tokens |
int
|
See |
overbudget_percent |
int
|
See |
token_learner |
TokenLearner | None
|
The token estimators of the budget check, and the weights that they learn from the calls. |
PromptOptions
¶
The descriptions that the prompts use. A value of None keeps the BAML default.
Attributes:
| Name | Type | Description |
|---|---|---|
expert_description |
str | None
|
Who the LLM plays. |
graph_description |
str | None
|
What a graph is and how to read it. |
dataset_description |
str | None
|
What the data is about. A good description helps the LLM. |
task_description |
str | None
|
What the model predicts. |
ModelOptions
¶
The LLM that a call uses. A field of None keeps the default of the call.
The key of the model never sits in this class. api_key_env names the environment variable that holds it, and
EBaMl reads the variable only when it sends a request. BAML lists the valid providers. describe_ebm and the
other functions raise InvalidOptionError before the first paid call if BAML does not know the provider.
Attributes:
| Name | Type | Description |
|---|---|---|
provider |
str
|
The BAML provider prefix, such as |
model |
str
|
The name of the model at the provider. |
api_key_env |
str | None
|
The name of the environment variable that holds the key. |
base_url |
str | None
|
The address of the server, for a gateway, a local server or Azure. |
temperature |
float | None
|
The sampling temperature. |
max_output_tokens |
int | None
|
The most tokens of one reply. A reasoning model counts its hidden tokens in this limit. |
headers |
Mapping[str, str]
|
Headers to add to each request. |
parse(value)
classmethod
¶
Read the "provider/model" shorthand, or pass a ModelOptions on.
The string splits at the first / only, so "openrouter/openrouter/free" is the provider openrouter
with the model openrouter/free.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
value
|
ModelOptions | str
|
The options, or the shorthand. |
required |
Returns:
| Type | Description |
|---|---|
ModelOptions
|
The options. |
Raises:
| Type | Description |
|---|---|
InvalidOptionError
|
If the string has no |
Examples:
>>> options = ModelOptions.parse("openrouter/openai/gpt-4o-mini")
>>> options.provider, options.model
('openrouter', 'openai/gpt-4o-mini')
Retry after a cut-off reply¶
ebaml.llm.retry
¶
Retry an LLM call with a smaller graph when the reply is cut off.
This module holds the policy only. It imports no LLM client, so the caller passes the call, the truncation test, the
token estimate and the graph rebuild. ebaml.llm.descriptors passes the real baml_sdk functions.
ShrinkingGraphCall
¶
The call and the services that call_with_shrinking_graph needs.
Attributes:
| Name | Type | Description |
|---|---|---|
call |
Callable[[str], R]
|
Sends the graph text to the LLM and returns the reply. |
graph |
str
|
The text of the graph for the first call. |
error_type |
type[E]
|
The class of the error that |
is_truncated |
Callable[[E], bool]
|
Tells whether an error means that the reply hit the |
estimate_tokens |
Callable[[str], int]
|
Estimates the number of tokens of a text. |
rebuild |
Callable[[int], str]
|
Returns the graph text again for a token budget. It raises |
on_truncated |
Callable[[E], None]
|
Called with the error of each attempt that was cut off, the last one included, before the retry or the raise. The caller counts the attempts with it, because the error carries no token usage. |
label |
str
|
The name of the feature, for the log. |
call_with_shrinking_graph(request)
¶
Call the LLM with a graph text. If the reply is cut off, ask again with a smaller graph, up to 4 times.
Each retry asks request.rebuild for a graph of 75% of the previous estimated size. The function raises the
error of the LLM call if the error is not a truncation, if 4 retries failed, or if the graph cannot shrink.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
request
|
ShrinkingGraphCall[R, E]
|
The call, the first graph text, and the services that the retry uses. |
required |
Returns:
| Type | Description |
|---|---|
R
|
The reply of the first call that succeeds. |
Local explanations¶
ebaml.local.facts
¶
Deterministic local explanations of one EBM prediction.
The three explanations are the explainability queries of Barceló, Monet, Pérez and Subercaseaux, "Model Interpretability through the Lens of Computational Complexity" (arXiv:2010.12265):
- Minimum change required (counterfactual): the fewest inputs to change so that the prediction flips.
- Minimum sufficient reason: the fewest inputs that fix the prediction whatever the other inputs are.
- Count completions: the share of completions of a partially known input that get the same prediction.
An EBM without interactions is a sum of one score per feature. The decision of a prediction is a margin: a sum of one
term per feature, positive while the decision holds (ebaml.local.local_kinds). The first two explanations are solved
exactly by sorting the features by how much they move the margin, and the third by sampling the unknown features from
their training distribution with a fixed seed. A decision with several margins, and the counterfactual of a chosen
class, are covering problems that sorting does not solve. A CoveringSolver of ebaml.local.covering_solvers
solves them. Nothing here calls an LLM, and nothing here renders text: ebaml.local.fact_text puts the facts in words
for the prompts in baml_src/local_explanations.baml.
The proof that the sorting algorithms are exact is in proofs/papers/local-explanations/paper.typ. It covers a binary
classifier, a multiclass classifier and a regressor, which use the same algorithms on the margin of their decision. For
a covering problem, the paper states what a result of a solver proves and what it does not.
The confidence of the model enters through confidence_level, as a number z of standard deviations. Each claim
names the scores that move by z deviations, always in the direction that makes the claim harder to prove:
- A counterfactual moves the score of the new bin of each changed feature against the flip. The features that do not change keep their fitted scores.
- A sufficient reason moves every score against the decision: a fixed feature at its bin, and a free feature at its worst plausible bin.
- The interval of the output moves the total score. Its deviation is that of a sum of independent errors.
- A margin of a multiclass classifier is the difference of two class scores. Its deviation is the sum of the two deviations, because an EBM stores no covariance between two classes.
A claim holds under its own model only. confidence_level=0 gives z = 0, so no score moves.
compute_local_facts(ebm, instance, options=None)
¶
Compute the three local explanations of one prediction of a classifier or a regressor.
The classifier is binary or multiclass. The function raises an UnsupportedModelError for an EBM that is not a
classifier or a regressor, and for an EBM with interactions. It raises an InvalidInstanceError for an instance
that lacks a value or has an unknown category. A regressor explains the answer to LocalOptions.decision.
A counterfactual and a sufficient reason are the smallest sets only if their minimality says so. Sorting proves
it for a decision with one margin. A decision with several margins and the counterfactual of a target class
depend on LocalOptions.search.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ebm
|
ExplainableBoostingModel
|
The fitted EBM. |
required |
instance
|
Mapping[str, FeatureValue]
|
The value of every feature, by feature name. |
required |
options
|
LocalOptions | None
|
Confidence level, sampling and support options. |
None
|
Returns:
| Type | Description |
|---|---|
LocalFacts
|
The prediction, the contributions and the three explanations. |
Raises:
| Type | Description |
|---|---|
OptionRangeError
|
If the confidence level is not in [0, 1). |
Audiences of the local explanations¶
ebaml.local.audience
¶
The audience of a local explanation: who reads it decides which facts the LLM gets and how it is told to write.
Audience is the base class. A subclass fills in the rules of one kind of reader:
guidanceis the instruction that tells the LLM how to write for this reader.contribution,prediction,change,flip,class_output,margin,guarantee,support,lower_bound,shareandsamplereturn the keys that the audience adds to the facts of the prompt. Each is abstract, so the author of an audience decides on every model number.flip,guarantee,supportandsamplestate the conditions of the facts: whether the scores moved by their uncertainty, whether rare values were left out, and how large the sample is. The prompts read these keys, so the text claims a condition only when the options of the facts set it.
Layperson and DataScientist are the two audiences that ship with the package. To change one rule, subclass one
of them and override one method. Pass an instance of any subclass as audience= to explain_instance or to
facts_to_texts.
Audience
¶
Bases: ABC
The base class of the audiences: one subclass for each kind of reader.
facts_to_texts builds the keys that all audiences share. The methods here return the keys that one audience
adds. All methods are abstract, so a new audience cannot leak a model number that its author forgot. To change one
rule of a built-in audience, subclass Layperson or DataScientist.
Examples:
To change only the language of the explanation, pass language= to explain_instance. An audience is for a
reader with its own rules. This one is a chief executive: no model numbers, and the bottom line first:
from ebaml.local.audience import Layperson
class Executive(Layperson):
def guidance(self) -> str:
return (
"The reader is a chief executive who has one minute, so open with the bottom line in one sentence. "
"Then name the one or two inputs that matter most, because the reader decides on the result."
)
guidance()
abstractmethod
¶
Return the instruction that tells the LLM how to write for this reader, and why.
contribution(contribution, prediction)
abstractmethod
¶
Return the keys that describe the size of one contribution.
prediction(prediction, confidence_level)
abstractmethod
¶
Return the keys that describe how sure the model is of the prediction.
change(change)
abstractmethod
¶
Return the keys that describe the size of one change of the counterfactual.
flip(confidence_level)
abstractmethod
¶
Return the keys that tell how sure the flip of the counterfactual is.
class_output(output, output_kind)
abstractmethod
¶
Return the key that tells the output after the changes: the probability of a class, or a predicted value.
margin(reason, output_kind)
abstractmethod
¶
Return the keys that describe the margin of the sufficient reason.
guarantee(confidence_level)
abstractmethod
¶
Return the keys that tell whether the sufficient reason holds under the uncertainty of the scores.
support(min_support)
abstractmethod
¶
Return the keys that tell whether the counterfactual and the sufficient reason left out rare values.
lower_bound(lower_bound, items)
abstractmethod
¶
Return the keys that give the fewest items ("changes" or "features") that any set needs.
share(share)
abstractmethod
¶
Return the share of completions with the same prediction.
sample(n_samples)
abstractmethod
¶
Return the keys that tell how large the sample behind the shares of the completions is.
Layperson
¶
DataScientist
¶
likelihood_word(probability)
¶
Return the words for a probability, from almost certain to almost impossible.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
probability
|
float
|
The probability, in |
required |
Returns:
| Type | Description |
|---|---|
str
|
The words. Each word starts at its threshold: |
strength_word(score)
¶
Return the word for the size of a score, from strongly to slightly.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
score
|
float
|
The score. Only its size counts, so |
required |
Returns:
| Type | Description |
|---|---|
str
|
|
sample_size_word(n_samples)
¶
Return the word for the size of a sample, from large to small.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
n_samples
|
int
|
The number of samples. |
required |
Returns:
| Type | Description |
|---|---|
str
|
|
round_value(value)
¶
Round a number to three decimals, the precision that the prompts show.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
value
|
float
|
The number. |
required |
Returns:
| Type | Description |
|---|---|
float
|
The rounded number. |
points_towards(contribution, prediction)
¶
Tell whether a contribution pushes towards the prediction.
A score points towards the prediction if it raises the score while the decision says that the score is high, and if it lowers the score otherwise. The decision says that the score is high when the score is above the threshold of the decision: the log-odds zero for a classifier. A score on the threshold belongs to the low side.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
contribution
|
Contribution
|
The contribution of one feature. |
required |
prediction
|
Prediction
|
The prediction. |
required |
Returns:
| Type | Description |
|---|---|
bool
|
True if the contribution supports the prediction. |
resolve_audience(audience)
¶
Return the audience for an Audience or for the name of a built-in one.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
audience
|
Audience | str
|
An audience, or |
required |
Returns:
| Type | Description |
|---|---|
Audience
|
The audience. |
Raises:
| Type | Description |
|---|---|
InvalidOptionError
|
If the name is not the name of a built-in audience. |
Text of the local explanations¶
ebaml.local.fact_text
¶
Put the local facts of one EBM prediction in words, as JSON texts for the LLM prompts.
ebaml.local computes the facts. This module only renders them, for one Audience (ebaml.local.audience): a
layperson gets words in place of the model numbers, so the LLM has no number to repeat, and a data scientist gets the
numbers. The prompts that read the texts are in baml_src/local_explanations.baml.
FactTexts
¶
The JSON text of each explanation, as the prompts read them.
facts_to_texts(facts, audience=_DEFAULT_AUDIENCE)
¶
Render the facts as one JSON text per explanation, for the prompts.
The audience decides which keys each fact gets. For a layperson the model numbers (scores, probabilities, intervals) are replaced by words, so the LLM has none to repeat. A data scientist gets the numbers.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
facts
|
LocalFacts
|
The facts of the prediction. |
required |
audience
|
Audience
|
Who will read the explanation. Pass an instance of any subclass of |
_DEFAULT_AUDIENCE
|
Returns:
| Type | Description |
|---|---|
FactTexts
|
The texts of the prediction, the counterfactual, the sufficient reason and the completions. |
MCP server¶
ebaml.mcp.server
¶
Create the MCP server, register the tools and start it from the command line.
Each tool is a thin adapter. It calls a function of a lower layer and turns the result into JSON text. A package error
(EBaMlError) becomes a tool error with its message.
create_server(store, llm=None)
¶
Create the server and register the tools.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
store
|
ModelStore
|
The session. |
required |
llm
|
ModelOptions | None
|
The LLM. Without it, the server has no LLM tools. |
None
|
Returns:
| Type | Description |
|---|---|
MCPServer
|
The server. |
serve(*, model, data=None, llm=None, transport=Transport.STDIO, host='127.0.0.1', port=8000, output_dir=None, edits=None, labels=None)
¶
Load the EBM and run the MCP server.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
Path
|
EBM in a skops file, which runs no code, or in a pickled or joblib file, which does. |
required |
data
|
Path | None
|
Parquet or CSV file with the rows for the row argument. |
None
|
llm
|
str | None
|
PROVIDER/MODEL of the LLM tools. Without it the server has none. |
None
|
transport
|
Transport
|
How the server talks to the client. |
STDIO
|
host
|
str
|
The address that the |
'127.0.0.1'
|
port
|
int
|
The port that the |
8000
|
output_dir
|
Path | None
|
The directory where the tool |
None
|
edits
|
Path | None
|
The history file of an earlier session ( |
None
|
labels
|
str | None
|
The column of the data that holds the true outcome of each row. The effect tools compute the metrics from it. |
None
|
main(argv=None)
¶
Run the ebaml-mcp command.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
argv
|
Sequence[str] | None
|
The arguments. The default is the command line. |
None
|
ebaml.mcp.model_store
¶
The state of one server session: the EBM, the optional data and the optional LLM options.
The user gives the paths at start. No tool argument names a path, so an agent cannot make the server load a file.
RevisionRead
¶
A copy of the model at one revision, and the state of the session when the tool read it.
Attributes:
| Name | Type | Description |
|---|---|---|
model |
ExplainableBoostingModel
|
A copy of the model at |
info |
RevisionInfo
|
The fixed name of the revision, the head, and the number of edits that the head holds. |
ModelStore
¶
The EBM of the session and what the tools need besides it.
The store never changes the loaded model. The edit session works on a copy, so each read of a revision gives the model with the edits of that revision.
Attributes:
| Name | Type | Description |
|---|---|---|
original |
ExplainableBoostingModel
|
The fitted EBM that the user loaded. The constructor takes it as |
data |
Frame | None
|
The rows that the |
session |
EditSession | None
|
The edits of the session. The default is a new session on |
output_directory |
Path | None
|
The directory where |
label_column |
str | None
|
The column of |
read(revision=CURRENT)
¶
Read the model at a revision. The read never changes the model of the session.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
revision
|
str
|
|
CURRENT
|
Returns:
| Type | Description |
|---|---|
RevisionRead
|
A copy of the model at the revision, and the state of the session. |
Raises:
| Type | Description |
|---|---|
EditError
|
If the model has no edit session and the revision is not |
labels()
¶
Read the true outcome of each row of the data.
Returns:
| Type | Description |
|---|---|
NDArray[generic] | None
|
The values of the label column, or |
Raises:
| Type | Description |
|---|---|
DataFileError
|
If the server has a label column but no data, or the data has no such column. |
editing()
¶
Give the edit session.
Returns:
| Type | Description |
|---|---|
EditSession
|
The session that the edit tools use. |
Raises:
| Type | Description |
|---|---|
UnsupportedModelError
|
If the model is one that the edits do not support. The message of |
described(revision)
¶
Give the kept description of the model if it describes this revision.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
revision
|
str
|
The fixed name of a revision, as |
required |
Returns:
| Type | Description |
|---|---|
str | None
|
The output of |
keep_description(revision, text)
¶
Keep the description of a revision, but only if the head is still this revision.
An edit during the slow LLM call moves the head. The description then describes a model that is no longer the current one, so the store drops it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
revision
|
str
|
The fixed name of the revision that the text describes. |
required |
text
|
str
|
The output of |
required |
feature_index(feature)
¶
Find the index of a feature by its name.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
feature
|
str
|
The name of the feature. |
required |
Returns:
| Type | Description |
|---|---|
int
|
The index of the feature in |
Raises:
| Type | Description |
|---|---|
InvalidInstanceError
|
If the model has no feature with this name. |
row_instance(row)
¶
Read one row of the data as an instance of the model.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
row
|
int
|
The position of the row in the data of |
required |
Returns:
| Type | Description |
|---|---|
dict[str, FeatureValue]
|
The value of each feature of the model. |
Raises:
| Type | Description |
|---|---|
InvalidInstanceError
|
If the server has no data, the row does not exist, or the data lacks a feature. |
load_model(path)
¶
Load an EBM from a skops file, or from a pickled or joblib file, of the path that the user gave.
A skops file is a zip archive. The loader trusts only the EBM classes, so a skops file runs no code. A pickled or joblib file runs code from the file. Call this function only with a path from the command line of the user, never with a path from a tool argument.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
PathLike
|
The file of the fitted EBM. |
required |
Returns:
| Type | Description |
|---|---|
ExplainableBoostingModel
|
The EBM. |
Raises:
| Type | Description |
|---|---|
UnsupportedModelError
|
If the file does not hold an EBM, or a skops file holds an untrusted type. |
load_data(path)
¶
Read the rows from a Parquet file, or from a CSV file.
The function reads the first and the last bytes of the file, not its extension. A file that starts and ends with
the Parquet mark PAR1 is a Parquet file, and Parquet keeps the type of each column. A file of UTF-8 text is a
CSV file, and polars infers the type of each column from the text.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
PathLike
|
The Parquet file, or the CSV file. |
required |
Returns:
| Type | Description |
|---|---|
Frame
|
The rows. |
Raises:
| Type | Description |
|---|---|
DataFileError
|
If the file is empty, is cut off, is neither Parquet nor UTF-8 CSV, or the reader fails. The message names the file and says how to save it. Polars reads a binary file as a CSV file of no rows, so the check of the bytes comes first. |
ebaml.mcp.read_tools
¶
The tools that read an EBM: its summary, its features and the text of a graph.
Each function calls code of a lower layer and returns an attrs class. The server turns the class into JSON.
ModelSummary
¶
The facts about the model that an agent needs first.
Attributes:
| Name | Type | Description |
|---|---|---|
kind |
str
|
|
outputs |
list[str]
|
The names of the outputs of the model. |
classes |
list[ClassLabel]
|
The class labels, in the order of the scores. The list is empty for a regressor. |
score_meaning |
str
|
What a score of a graph means. |
intercept |
list[float]
|
The intercept: one value, or one value per class for a multiclass classifier. |
feature_count |
int
|
The number of features that have a graph. |
skipped_terms |
list[str]
|
The interaction terms. The tools skip them. |
session |
RevisionInfo
|
The revision that the summary reads, the head, and the number of edits that the head holds. |
FeatureInfo
¶
One feature of the model.
FeatureList
¶
The features with a graph, the most important first, and the revision that the list reads.
model_summary(store, revision=CURRENT)
¶
Summarize the model.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
store
|
ModelStore
|
The session. |
required |
revision
|
str
|
|
CURRENT
|
Returns:
| Type | Description |
|---|---|
ModelSummary
|
The kind, the outputs, the intercept, the number of features and the interaction terms that the tools skip. |
list_features(store, revision=CURRENT)
¶
List the features with a graph, the most important first.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
store
|
ModelStore
|
The session. |
required |
revision
|
str
|
The revision to list. See |
CURRENT
|
Returns:
| Type | Description |
|---|---|
FeatureList
|
The name, the type and the importance of each feature, sorted by importance, and the revision that the list reads. |
graph_text(store, feature, *, revision=CURRENT, normalization=Normalization.NONE, x_axis_precision=None, y_axis_precision=None, max_tokens=3000)
¶
Write the graph of a feature as text.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
store
|
ModelStore
|
The session. |
required |
feature
|
str
|
The name of the feature. |
required |
revision
|
str
|
|
CURRENT
|
normalization
|
Normalization | str
|
See |
NONE
|
x_axis_precision
|
int | None
|
See |
None
|
y_axis_precision
|
int | None
|
See |
None
|
max_tokens
|
int
|
See |
3000
|
Returns:
| Type | Description |
|---|---|
str
|
A line that names the revision and the edits that the head holds, then the text of |
ebaml.mcp.explain_tools
¶
The tool that explains one prediction with the exact local facts. It calls no LLM.
PredictionExplanation
¶
The facts of one prediction and their texts.
Attributes:
| Name | Type | Description |
|---|---|---|
facts |
LocalFacts
|
The prediction, the contributions and the three local explanations. |
texts |
FactTexts
|
The facts as the JSON texts that the prompts read, for the audience of the call. |
session |
RevisionInfo
|
The revision that the facts explain, the head, and the number of edits that the head holds. |
PredictionRequest
¶
What to explain, and how.
Attributes:
| Name | Type | Description |
|---|---|---|
instance |
(Mapping[str, FeatureValue], optional)
|
The value of every feature, by feature name. |
row |
(int, optional)
|
The position of a row of the data of |
confidence_level |
float, default=0.95
|
See |
audience |
str, default="layperson"
|
The name of a built-in audience. It decides the keys of the texts. |
revision |
str, default="current"
|
|
explain_prediction(store, request)
¶
Compute the local facts of one prediction.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
store
|
ModelStore
|
The session. |
required |
request
|
PredictionRequest
|
The instance or the row, the revision, the confidence level and the audience. |
required |
Returns:
| Type | Description |
|---|---|
PredictionExplanation
|
The facts and their texts. |
Raises:
| Type | Description |
|---|---|
InvalidInstanceError
|
If the request gives both or none of |
ebaml.mcp.llm_tools
¶
The tools that ask the LLM to describe the model. The server registers them only when the user gives --llm.
describe_feature(store, llm, feature)
¶
Ask the LLM to describe the graph of one feature.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
store
|
ModelStore
|
The session. |
required |
llm
|
ModelOptions
|
The LLM. |
required |
feature
|
str
|
The name of the feature. |
required |
Returns:
| Type | Description |
|---|---|
str
|
The output of |
describe_model(store, llm)
¶
Ask the LLM to describe the whole model at the head. The store keeps the text for the revision that it describes.
The LLM call runs on a revision with no lock. If an edit moves the head during the call, the store drops the text, so the next call describes the new head.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
store
|
ModelStore
|
The session. |
required |
llm
|
ModelOptions
|
The LLM. |
required |
Returns:
| Type | Description |
|---|---|
str
|
The output of |
Graphs¶
ebaml.graphs.ebm_graph
¶
Graphs for EBM models.
TokenBudget
¶
BudgetCheck
¶
Bases: Protocol
A function that estimates the tokens of a text and compares them with a budget.
ebaml.graphs.token_budget.check_token_budget is such a function.
__call__(*, text, max_tokens, overbudget_percent)
¶
Compare the estimated tokens of text with max_tokens, and with the tolerance above it.
EBMGraph
¶
A data structure for the graphs of Explainable Boosting Machines.
A graph of a binary classifier or a regressor has one score per bin, and classes is empty. A graph of a
multiclass classifier has one row per bin with one score per class, and classes names the columns.
from_ebm(ebm, feature_index, normalization=Normalization.NONE, *, use_feature_bounds=True, classes=None)
classmethod
¶
Extract the graph of a feature from an Explainable Boosting Machine.
This is a low-level function. It does not return the final format in which the graph is presented to the LLM.
The purpose of this function is to extract the graph from the internals of the EBM and return it in a format that is easy to work with.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ebm
|
ExplainableBoostingModel
|
The EBM model. |
required |
feature_index
|
int
|
The index of the feature in the EBM. A feature that |
required |
normalization
|
Normalization | str
|
How to normalize the graph (shift on the y-axis). |
NONE
|
use_feature_bounds
|
bool
|
If True, the first and last x-axis bins are the min and max values of the feature stored in the EBM. If false, the first and last value are -inf and inf. |
True
|
classes
|
Sequence[str] | None
|
The classes to keep in the graph of a multiclass classifier, in the order to show them. If not provided,
the graph has every class. The text of a model with many classes is long, so a subset keeps it short.
A model with one output raises |
None
|
Returns:
| Type | Description |
|---|---|
EBMGraph
|
The graph. |
Raises:
| Type | Description |
|---|---|
InvalidOptionError
|
If the feature type is not continuous or nominal. |
from_enum
|
If the normalization option is not valid. |
EBMGraphShapeError
|
If the number of bins does not match the number of scores. |
Notes
A multiclass classifier has one score per bin and per class. Softmax does not change when the same constant is
added to the scores of all the classes of a bin, so the graph centres the scores of each bin over all the
classes of the model, before it selects classes. The standard deviations stay those of the EBM.
The normalization then shifts all the scores by one constant, so the differences between the classes of a bin
do not change.
proofs/papers/multiclass-scores/paper.typ proves that these graphs keep the predictions of the EBM.
simplify(min_variation_per_cent=0.0)
¶
Simplify the graph: remove the redundant (flat) bins.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
min_variation_per_cent
|
float
|
The degree of simplification. If min_variation_per_cent>0, the function simplifies the graph by removing bins that correspond to a less than min_variation_per_cent change in the score, considering the overall min/max difference of score for the feature as 100%. A bin of a multiclass graph merges only if the score of every class changes that little. |
0.0
|
Returns:
| Name | Type | Description |
|---|---|---|
EBMGraph |
Self
|
The simplified graph. |
Raises:
| Type | Description |
|---|---|
InvalidOptionError
|
If the feature type is not continuous. |
OptionRangeError
|
If min_variation_per_cent is not between 0 and 1. |
to_body_text(*, feature_format, x_axis_precision, y_axis_precision, confidence_bounds, confidence_level)
¶
Render the lines of text (name, type, means and optional bounds) that follow the preamble.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
feature_format
|
FeatureType
|
The format of the feature. |
required |
x_axis_precision
|
int | None
|
The precision of the values on the x-axis. If None, the precision is not adjusted. |
required |
y_axis_precision
|
int
|
The precision of the values on the y-axis. |
required |
confidence_bounds
|
bool
|
Whether to include confidence bounds. |
required |
confidence_level
|
float
|
The desired confidence level of the bounds. |
required |
Returns:
| Type | Description |
|---|---|
str
|
The textual representation of the graph, without the preamble. A multiclass graph has a |
to_text(*, include_description=True, feature_format=None, x_axis_precision=None, y_axis_precision=None, confidence_bounds=True, confidence_level=0.95, max_tokens=3000, overbudget_percent=50, check_budget=None)
¶
Convert the graph to text. This is the text that we then pass to the LLM.
The function takes care of a variety of different formatting issues that can arise in the process of converting a graph to text.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
include_description
|
bool
|
Whether to include a short descriptive preamble that describes the graph to the LLM. The preamble of a multiclass graph also says how to read the series of the classes. |
True
|
feature_format
|
FeatureType | str | None
|
The format of the feature (continuous, categorical, boolean). If not provided, auto-detection is attempted based on the feature type and the x-axis values. A boolean format needs exactly two x-axis values, and raises an EBMGraphShapeError otherwise. |
None
|
x_axis_precision
|
int | None
|
The precision of the values on the x-axis. If not provided, the precision is not adjusted. |
None
|
y_axis_precision
|
AxisPrecision
|
The precision of the values on the y-axis. If not provided or "auto", the precision is auto-detected based on the range of the y-axis |
None
|
confidence_bounds
|
bool
|
Whether to include confidence bounds. |
True
|
confidence_level
|
float
|
The desired confidence level of the bounds. |
0.95
|
max_tokens
|
int
|
The token budget of the text of the graph. The token count is a rough estimate (there is no tokenizer before the LLM call), so exceeding the budget only emits a warning. |
3000
|
overbudget_percent
|
int
|
How far above max_tokens (in percent) the estimate may go before the graph is considered obviously too large. Beyond that bound the function simplifies the graph, or raises if it cannot. The LLM provider's own error remains the real limit. |
50
|
check_budget
|
BudgetCheck | None
|
Estimates the tokens of a text and compares them with |
None
|
Returns:
| Type | Description |
|---|---|
str
|
The textual representation of the graph. |
Raises:
| Type | Description |
|---|---|
EBMGraphSimplificationError
|
If the graph cannot be simplified enough to fit into the max_tokens limit. |
EBMGraphTokenLimitError
|
If the graph exceeds the max_tokens limit even after simplification. |
from_text(text)
classmethod
¶
Parse the textual representation of a graph back into a graph. The inverse of to_text.
See ebaml.graphs.graph_text.parse_graph_text for the details of the format.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
text
|
str
|
The text of the graph, with the confidence bounds included. |
required |
Returns:
| Type | Description |
|---|---|
EBMGraph
|
The graph. |
Terms of the features¶
ebaml.graphs.terms
¶
The term of an EBM that holds the main effect of each feature.
interpret keeps the scores, the standard deviations, the bin weights and the importances by term, and the names, the
types and the bins by feature. Term i is not always feature i: a feature that exclude leaves out at fit
time, or that remove_terms removes later, has no term, and the terms after it move. So other code finds the term
of a feature here, and does not index the term attributes with the index of a feature.
main_term(ebm, feature_index)
¶
Find the term that holds the main effect of a feature.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ebm
|
ExplainableBoostingModel
|
The fitted EBM. |
required |
feature_index
|
int
|
The index of the feature in |
required |
Returns:
| Type | Description |
|---|---|
int
|
The index of the term in |
Raises:
| Type | Description |
|---|---|
UnsupportedModelError
|
If the feature has no main-effect term, because |
main_effect_features(ebm)
¶
List the features that have a main-effect term.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ebm
|
ExplainableBoostingModel
|
The fitted EBM. |
required |
Returns:
| Type | Description |
|---|---|
list[int]
|
The index of each such feature in |
Token budget¶
ebaml.graphs.token_budget
¶
The token budget of a text: an estimate of its tokens, compared with a limit.
No tokenizer runs before the LLM call, so the estimate comes from the counts of the text. TokenEstimator turns the
words, bytes and digits of a text into a number of tokens, and adds a margin, so that an estimate that is too low is
rarer than one that is too high. A text that the estimate calls too large only gets a simpler graph, but a text that it
calls small enough can make the provider refuse the request. The provider error stays the real limit. The module needs
no baml_sdk, so code that calls no LLM can check a budget too.
The estimator can learn. TokenEstimator.observe moves the weights toward the token counts of real requests, by
recursive least squares with a prior at the starting weights. Learning is off by default, because learned weights change
the estimate, and so the graph text, from one call to the next.
check_token_budget = TokenEstimator()
module-attribute
¶
The budget check with the default weights.
The estimator never learns, so it gives the same status for the same text in every call. Call it with text,
max_tokens and overbudget_percent. It is a BudgetCheck, so EBMGraph.to_text accepts it as
check_budget. Pass a TokenEstimator instead to choose the weights or to learn them.
Examples:
>>> str(check_token_budget(text="1.5 2.5 " * 12, max_tokens=100, overbudget_percent=50).status)
'warn'
>>> str(check_token_budget(text="1.5 2.5 " * 12, max_tokens=1000, overbudget_percent=50).status)
'ok'
>>> str(check_token_budget(text="1.5 2.5 " * 12, max_tokens=50, overbudget_percent=50).status)
'over'
BudgetStatus
¶
The estimated tokens of a text, and how they compare with the budget.
Attributes:
| Name | Type | Description |
|---|---|---|
estimated_tokens |
int
|
The estimated number of tokens of the text, with the margin. |
status |
BudgetLevel
|
|
EstimatorState
¶
What a learning estimator knows after its samples. TokenEstimator.restore takes it back.
Attributes:
| Name | Type | Description |
|---|---|---|
weights |
TokenWeights
|
The learned weights. |
covariance |
tuple[float, ...]
|
The nine entries of the 3 by 3 matrix of the recursive least squares, row by row. It holds how sure the estimator is about each weight. |
samples_seen |
int
|
The number of samples that the estimator learned from. |
TokenEstimator
¶
Estimate the tokens of a text, and compare the estimate with a budget.
An instance is a BudgetCheck, so EBMGraph.to_text(check_budget=estimator) works. The estimate is an upper
bound: the fit plus margin times relative_spread of the fit.
Attributes:
| Name | Type | Description |
|---|---|---|
weights |
TokenWeights
|
The starting weights, and the center of the prior of the learning. |
relative_spread |
float
|
The root mean square of the relative error of |
margin |
float
|
How many spreads the estimate lies above the fit. |
learning |
bool
|
Whether |
prior_samples |
float
|
How many samples of a typical graph the prior of |
Examples:
>>> estimator = TokenEstimator()
>>> str(estimator(text="1.5 2.5 " * 12, max_tokens=100, overbudget_percent=50).status)
'warn'
current_weights
property
¶
The weights that the estimate uses now. They equal weights until a sample changes them.
samples_seen
property
¶
The number of samples that changed the weights.
__attrs_post_init__()
¶
Start the learning at the prior.
estimate(text)
¶
Estimate the tokens of a text, with the margin, rounded up.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
text
|
str
|
The text. |
required |
Returns:
| Type | Description |
|---|---|
int
|
The estimated number of tokens. It is 0 for a text of no words. |
Examples:
>>> TokenEstimator(weights=TokenWeights(words=2.0, bytes=0.0, digits=0.0), relative_spread=0.0).estimate(
... "a b c"
... )
6
__call__(*, text, max_tokens, overbudget_percent)
¶
Compare the estimated tokens of a text with a budget and with the tolerance above it.
The method makes an estimator a BudgetCheck, so EBMGraph.to_text(check_budget=estimator) works.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
text
|
str
|
The text. |
required |
max_tokens
|
int
|
The budget. |
required |
overbudget_percent
|
int
|
How far above |
required |
Returns:
| Type | Description |
|---|---|
BudgetStatus
|
The estimate and the status. |
observe(sample)
¶
Move the weights toward the token count of one real request, if learning is on.
The step is one of recursive least squares with the prior at weights. A projection then keeps every
weight at zero or above. The method is safe to call from several threads.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sample
|
TokenSample
|
The counts of the text of a request and the input tokens that the provider reported for it. |
required |
Returns:
| Type | Description |
|---|---|
bool
|
Whether the sample changed the weights. It is |
Examples:
>>> TokenEstimator().observe(TokenSample(TextFeatures(words=10, bytes=50, digits=5), 60))
False
>>> learner = TokenEstimator(learning=True)
>>> learner.observe(TokenSample(TextFeatures(words=10, bytes=50, digits=5), 60))
True
state()
¶
Take a copy of what the estimator learned.
Returns:
| Type | Description |
|---|---|
EstimatorState
|
The learned weights, the matrix of the recursive least squares and the number of samples. |
restore(state)
¶
Take back what state made, for example after a restart.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
state
|
EstimatorState
|
A value that |
required |
Returns:
| Type | Description |
|---|---|
TokenEstimator
|
The estimator itself. |
estimate_tokens(text)
¶
Estimate the tokens of a text with the default weights, with the margin, rounded up.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
text
|
str
|
The text. |
required |
Returns:
| Type | Description |
|---|---|
int
|
The estimated number of tokens. |
Examples:
>>> estimate_tokens("")
0
>>> estimate_tokens("0.1 0.2 0.3 0.4") > estimate_tokens("0.1 0.2")
True
ebaml.graphs.token_weights
¶
The word, byte and digit counts of a text, and the weights that turn them into a token estimate.
The estimate of the tokens of a text is words * w_words + bytes * w_bytes + digits * w_digits. Graph text is mostly
digits and punctuation, so a digit count separates it from prose better than a character count. The three counts grow
together, so a plain least-squares fit can give a weight below zero. fit_weights keeps every weight at zero or
above. The module needs no tokenizer, no file and no baml_sdk.
TextFeatures
¶
The counts of a text that the token estimate reads.
Attributes:
| Name | Type | Description |
|---|---|---|
words |
int
|
The number of parts of the text that whitespace separates. |
bytes |
int
|
The number of bytes of the UTF-8 encoding of the text. |
digits |
int
|
The number of characters of the text that are digits. |
of(text)
classmethod
¶
Count the words, bytes and digits of a text.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
text
|
str
|
The text. |
required |
Returns:
| Type | Description |
|---|---|
TextFeatures
|
The counts. |
Examples:
>>> TextFeatures.of("x: 12 é")
TextFeatures(words=3, bytes=8, digits=2)
as_vector()
¶
Return the counts in the order of TokenWeights.as_vector.
Returns:
| Type | Description |
|---|---|
tuple[float, ...]
|
The words, the bytes and the digits. |
TokenWeights
¶
The tokens that one word, one byte and one digit add to the estimate of a text.
Attributes:
| Name | Type | Description |
|---|---|---|
words |
float
|
The tokens for each word. |
bytes |
float
|
The tokens for each byte. |
digits |
float
|
The tokens for each digit. |
as_vector()
¶
Return the weights in the order of TextFeatures.as_vector.
Returns:
| Type | Description |
|---|---|
tuple[float, ...]
|
The weights of the words, the bytes and the digits. |
estimate(features)
¶
Estimate the tokens of a text from its counts.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
features
|
TextFeatures
|
The counts of the text. |
required |
Returns:
| Type | Description |
|---|---|
float
|
The estimated number of tokens, before any margin. |
Examples:
>>> TokenWeights(words=2.0, bytes=0.5, digits=1.0).estimate(TextFeatures(words=3, bytes=10, digits=2))
13.0
TokenSample
¶
One real request: the counts of its text and the input tokens that the provider reported for it.
Attributes:
| Name | Type | Description |
|---|---|---|
features |
TextFeatures
|
The counts of the text of the messages of the request. |
tokens |
int
|
The input tokens that the provider counted, with the cached tokens included. It is at least 1. |
ModelKey
¶
The provider and the name of the model that answered a request.
Attributes:
| Name | Type | Description |
|---|---|---|
provider |
str
|
The provider that the BAML runtime resolved, for example |
model |
str
|
The model at the provider, for example |
WeightsEntry
¶
The weights of one model, and what they are worth.
Attributes:
| Name | Type | Description |
|---|---|---|
weights |
TokenWeights
|
The weights that fit the model. |
relative_spread |
float
|
The root mean square of the relative error of |
samples |
int
|
The number of samples that fitted |
update_allowed |
bool
|
Whether a learning estimator may change |
prose_bytes_per_token |
float
|
The bytes of English prose that the model counts as one token, such as the prompt template. A request holds the graph and also prose, and the learner takes the prose away to learn the weights of the graph alone. |
WeightsTable
¶
The default weights and the weights of the models that were fitted.
Attributes:
| Name | Type | Description |
|---|---|---|
default |
WeightsEntry
|
The entry for a model that the table does not list. |
models |
Mapping[ModelKey, WeightsEntry]
|
The entry of each model that was fitted. |
entry_for(key)
¶
Look up the entry of a model.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key
|
ModelKey | None
|
The model, or |
required |
Returns:
| Type | Description |
|---|---|
WeightsEntry
|
The entry of the model, or |
with_entry(key, entry)
¶
Add the entry of a model, or replace the one that the table has.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key
|
ModelKey
|
The model. |
required |
entry
|
WeightsEntry
|
Its entry. |
required |
Returns:
| Type | Description |
|---|---|
WeightsTable
|
A new table. The table itself does not change. |
from_bytes(data)
classmethod
¶
Read a table from its JSON text.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
bytes
|
The JSON text that |
required |
Returns:
| Type | Description |
|---|---|
WeightsTable
|
The table. |
Raises:
| Type | Description |
|---|---|
TokenFitError
|
If the text is not JSON, or lacks a field of the table. |
to_bytes()
¶
Write the table as JSON text, with the models in order, so that the text is the same for the same table.
Returns:
| Type | Description |
|---|---|
bytes
|
The JSON text. |
fit_weights(samples)
¶
Fit the weights by least squares, with every weight at zero or above.
The fit tries each set of features that can have a weight above zero. It solves the least-squares problem for the set, and keeps the set with the smallest squared error among those whose weights are all at zero or above. No proof backs that this is the best fit with weights of zero or above.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
samples
|
Sequence[TokenSample]
|
The samples to fit. |
required |
Returns:
| Type | Description |
|---|---|
TokenWeights
|
The weights that fit the samples, with every weight at zero or above. |
Raises:
| Type | Description |
|---|---|
TokenFitError
|
If |
Examples:
>>> samples = [
... TokenSample(TextFeatures(words=words, bytes=0, digits=digits), 2 * words + digits)
... for words, digits in [(1, 5), (4, 1), (3, 3)]
... ]
>>> [round(weight, 6) for weight in fit_weights(samples).as_vector()]
[2.0, 0.0, 1.0]
relative_spread(weights, samples)
¶
Measure how far the estimates are from the counts, as the root mean square of the relative error.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
weights
|
TokenWeights
|
The weights to judge. |
required |
samples
|
Sequence[TokenSample]
|
The samples to judge them on. |
required |
Returns:
| Type | Description |
|---|---|
float
|
The root mean square of |
Examples:
>>> relative_spread(TokenWeights(words=1.0, bytes=0.0, digits=0.0), [TokenSample(TextFeatures(1, 0, 0), 2)])
0.5
ebaml.graphs.request_text
¶
The text of the messages in the body of an LLM request.
The tokenizer of a provider counts the prompt, not the JSON around it. The body of a request holds the model name, the
sampling parameters and the structure of the messages besides the text. So the token learner reads the text of the
messages only. Providers shape the body differently (OpenAI, Anthropic and Gemini differ), so the function collects
the text under the keys that carry it. It needs no baml_sdk.
message_text(body)
¶
Collect the text of the messages from the body of a request, one part on each line.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
body
|
str
|
The JSON text of the request body. |
required |
Returns:
| Type | Description |
|---|---|
str
|
The text of the messages. It is empty when the body is not JSON or holds no text. |
Examples:
>>> message_text('{"model": "m", "messages": [{"role": "user", "content": "hello there"}]}')
'hello there'
>>> body = '{"system": "be brief", "messages": [{"role": "user", "content": [{"type": "text", "text": "hi"}]}]}'
>>> print(message_text(body))
be brief
hi
ebaml.llm.token_learning
¶
Token estimators that learn from the token counts of real LLM calls, one for each model.
The BAML runtime reports the input tokens of each request and the model that answered. TokenLearner keeps one
TokenEstimator for each model. It starts each from the shipped table of weights, and moves it toward the tokens of
the graph in the requests when learning is on. The tokens of the prose in a request (the template and earlier replies)
are taken away first. Learning is off by default, because learned weights change the estimate, and so the
text of a graph, from one call to the next.
When learning is on, the weights persist between sessions in a JSON file. By default the file is
$XDG_CACHE_HOME/ebaml/token_weights.json, or ~/.cache/ebaml/token_weights.json. The learner reads and writes the
file only when learning is on. A model that the table marks as final (update_allowed false) never learns.
RequestSample
¶
Bases: Protocol
One counted request, as the BAML runtime reports it. baml_sdk.TokenSample is such a sample.
TokenLearner
¶
Keep a token estimator for each model, and learn its weights from the counts of real requests.
Pass a learner as GraphOptions(token_learner=...). The budget check of describe_graph and describe_ebm
then uses the estimator of the model that you chose, and the calls feed their token counts back to it. The BAML
runtime names the model that answered, which can differ from the one that you chose (openrouter/free picks a
model for each request). So a learner learns the weights of each model that answered. Its estimate before a call
uses the model that you chose, and the default weights when the table does not list it.
Attributes:
| Name | Type | Description |
|---|---|---|
learning |
bool
|
Whether the estimators change their weights. It is off by default. |
persist |
bool
|
Whether learned weights persist in |
path |
PathLike | None
|
The file of the learned weights. |
table |
WeightsTable
|
The weights that each estimator starts from. The default is the table that ships with the package. |
prior_samples |
float
|
See |
margin |
float
|
See |
estimator_for(key)
¶
Get the estimator of a model, and create it from the table on the first request.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key
|
ModelKey | None
|
The model, or |
required |
Returns:
| Type | Description |
|---|---|
TokenEstimator
|
The estimator of the model. The learner keeps it, so later calls get the same object. |
observe(samples, graph)
¶
Teach the estimators the token counts of the graph in requests, and save what they learned.
A request holds the graph and also prose: the prompt template and, in later turns, earlier replies. The
provider counts both. The method takes the prose away, at the bytes for each token that the table gives for the
model, and teaches the estimator what is left for the graph. A request that does not hold graph as it is,
or that has no tokens left, gives no sample. A model whose entry is final learns nothing. The method does
nothing when learning is off.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
samples
|
Iterable[RequestSample]
|
The counted requests, for example |
required |
graph
|
str
|
The text of the graph that the requests carried. |
required |
Returns:
| Type | Description |
|---|---|
int
|
The number of samples that changed an estimator. |
save()
¶
Write the learned weights to the file. Nothing happens when learning or persistence is off.
A file that cannot be written logs a warning and does not stop the call.
default_table()
cached
¶
Read the table of weights that ships with the package.
Returns:
| Type | Description |
|---|---|
WeightsTable
|
The default weights and the weights of the models that were fitted. |
default_weights_path()
¶
Name the file where learned weights persist when the learner gets no path.
Returns:
| Type | Description |
|---|---|
Path
|
|
Text of the graphs¶
ebaml.graphs.graph_text
¶
Helpers for the textual representation of EBM graphs: the JSON lines, and how to read a graph text back.
ParsedGraph
¶
The fields of a graph that parse_graph_text reads from a text. They are the fields of EBMGraph.
xy_to_json(x_vals, y_vals)
¶
Convert a sequence of x_vals and y_vals to a json string.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
x_vals
|
Sequence[str | tuple[float | str, float | str]]
|
The x values. The graph text passes interval ends as rounded strings. |
required |
y_vals
|
Iterable[float | str]
|
The y values. The graph text passes them as rounded strings. |
required |
Returns:
| Type | Description |
|---|---|
str
|
A json string representation of the x and y values. The keys are escaped as JSON strings. The y values are written as they are, to keep their exact digits. |
Examples:
>>> xy_to_json([("0.0", "1.5"), "red"], ["0.25", "-1.0"])
'{"(0.0, 1.5)": 0.25, "red": -1.0}'
>>> print(xy_to_json(['Master "B"'], ["0.5"]))
{"Master \"B\"": 0.5}
parse_str_tuple_to_float_tuple(str_tuple)
¶
Parse a string tuple to a float tuple.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
str_tuple
|
str
|
The string tuple to parse. |
required |
Returns:
| Type | Description |
|---|---|
Interval
|
The parsed float tuple. |
Raises:
| Type | Description |
|---|---|
GraphTextError
|
If the text is not two numbers in brackets. |
Examples:
>>> parse_str_tuple_to_float_tuple("(0.5, 2)")
(0.5, 2.0)
>>> parse_str_tuple_to_float_tuple("(a, b)")
Traceback (most recent call last):
...
ebaml.core.exceptions.GraphTextError: '(a, b)' is not an interval of two numbers.
parse_graph_text(text)
¶
Read the fields of a graph from its textual representation. The inverse of EBMGraph.to_text.
The text may contain other lines (such as the preamble or an LLM prompt around the graph). Only the lines
that EBMGraph.to_text writes are read: the name, the type, the means and both confidence bounds. The standard
deviations are estimated from the bounds with the normal approximation, at the confidence level that the
bounds are labeled with, so they are only as precise as the rounding of the text.
A graph of a multiclass classifier has one series per class: a Class: line, then the means and the bounds of
that class. The scores and the standard deviations then have one row per bin and one column per class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
text
|
str
|
The text of the graph, with the confidence bounds included. |
required |
Returns:
| Type | Description |
|---|---|
ParsedGraph
|
The feature name, the feature type, the x-axis intervals, the scores, the standard deviations and the classes. |
Raises:
| Type | Description |
|---|---|
GraphTextError
|
If a line of the graph is missing, is not valid JSON, the means and the bounds do not share the same
intervals, or a multiclass text has a series before its first |
InvalidOptionError
|
If the feature type is not continuous. |
Types¶
ebaml.core.custom_types
¶
Types for EBM models, and the coercion of options to them.
Normalization
¶
Bases: StrEnum
Normalization options for EBM graphs.
FeatureType
¶
Bases: StrEnum
Feature types for EBM graphs.
MonotoneDirection
¶
Bases: StrEnum
The direction of a monotone graph.
Direction
¶
Bases: StrEnum
The direction that an edit gives a graph. AUTO lets the edit pick the better MonotoneDirection.
BudgetLevel
¶
Bases: StrEnum
How an estimated number of tokens compares with the budget.
Transport
¶
Bases: StrEnum
How an MCP server talks to its client.
normalize_feature_type(feature_type)
¶
Normalize a feature type to the FeatureType enum.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
feature_type
|
str
|
The feature type to normalize. |
required |
Returns:
| Type | Description |
|---|---|
FeatureType
|
The normalized feature type. |
Raises:
| Type | Description |
|---|---|
from_enum
|
If an invalid feature type is provided. |
coerce_enum(enum_class, value)
¶
Convert a value (a member or its string value) to a member of an enum.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
enum_class
|
type[E]
|
The enum class. |
required |
value
|
E | str
|
The member or the value to convert. |
required |
Returns:
| Type | Description |
|---|---|
E
|
The enum member. |
Raises:
| Type | Description |
|---|---|
from_enum
|
If the value is not a member of the enum. |
normalize_feature_types(feature_types, reject_set=None)
¶
Normalize a list of feature types to the FeatureType enum.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
feature_types
|
Iterable[str]
|
The feature types. |
required |
reject_set
|
Set[FeatureType] | None
|
Set of feature types to reject. If a feature type is in this set, the function raises an InvalidOptionError. |
None
|
Returns:
| Type | Description |
|---|---|
list[FeatureType]: List of normalized feature types.
|
|
Raises:
| Type | Description |
|---|---|
InvalidOptionError
|
If an invalid feature type is provided or if a feature type is in the reject set. |
Exceptions¶
ebaml.core.exceptions
¶
Exceptions for EBM models.
Every error that ebaml raises on purpose derives from EBaMlError, so except EBaMlError catches all of them.
An error for a bad value also derives from ValueError, so code that catches ValueError keeps working.
EBaMlError
¶
Bases: Exception
The base class of the errors that ebaml raises.
InvalidOptionError(option, valid_options)
¶
Bases: EBaMlError, ValueError
Exception raised for invalid options.
Build the error message from the invalid option and the valid ones.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
option
|
str
|
The invalid option provided. |
required |
valid_options
|
Collection[str]
|
The options that are valid. |
required |
from_enum(option, validation_enum)
classmethod
¶
Create an InvalidOptionError from an enum.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
option
|
str
|
The invalid option provided. |
required |
validation_enum
|
type[Enum]
|
The enum class that defines the valid options. |
required |
Returns:
| Type | Description |
|---|---|
InvalidOptionError
|
An instance of InvalidOptionError with a message indicating the invalid option and the valid options. |
OptionRangeError
¶
Bases: EBaMlError, ValueError
Exception raised when a number option is outside its range.
EBMGraphError
¶
Bases: EBaMlError
Exception raised for errors in the EBM graph processing.
EBMGraphSimplificationError
¶
Bases: EBMGraphError
Exception raised when an EBM graph cannot be simplified enough to fit into the max_tokens limit.
EBMGraphTokenLimitError
¶
Bases: EBMGraphError
Exception raised when an EBM graph exceeds the max_tokens limit even after simplification.
EBMGraphShapeError
¶
Bases: EBMGraphError, ValueError
Exception raised when the bins of an EBM graph do not fit the scores or the requested format.
GraphTextError
¶
Bases: EBaMlError, ValueError
Exception raised when the textual representation of a graph cannot be read.
UnsupportedModelError
¶
Bases: EBaMlError, ValueError
Exception raised when a function does not support the kind of EBM, such as one with interactions.
DataFileError
¶
Bases: EBaMlError, ValueError
Exception raised when a data file is empty, is neither Parquet nor UTF-8 CSV, or fails to read.
InvalidInstanceError
¶
Bases: EBaMlError, ValueError
Exception raised when the instance to explain lacks a value or has a value that the EBM did not see.
SDKUnavailableError
¶
Bases: EBaMlError, ImportError
Exception raised when the generated baml_sdk package is missing and ebaml cannot generate it.
The message lists the steps that fix the cause. The error also derives from ImportError, so code that catches
the import error of a missing package catches it too.
LLMRequestError
¶
Bases: EBaMlError
Exception raised when the provider of the LLM fails a request.
The message names the cause and lists the steps that fix it. The error that the BAML runtime raised is the
__cause__.
MissingAPIKeyError
¶
Bases: LLMRequestError
Exception raised when a provider needs an API key and the environment has none.
APIKeyRejectedError
¶
Bases: LLMRequestError
Exception raised when a provider answers a request with "unauthorized" or "forbidden".
RateLimitedError
¶
Bases: LLMRequestError
Exception raised when a provider limits the rate of the requests.
ProviderUnreachableError
¶
Bases: LLMRequestError
Exception raised when a request does not reach the provider.
TokenFitError
¶
Bases: EBaMlError, ValueError
Exception raised when the token weights cannot be fitted or read, for example from an empty list of samples.
SearchLimitError
¶
Bases: EBaMlError
Exception raised when a covering search ends at its limit with no set and no proof that no set exists.
EditError
¶
Bases: EBaMlError, ValueError
Exception raised when an edit of an EBM cannot run, for example an edit that needs an order on a nominal feature.
The message names the cause and lists the steps that fix it.
SelectionError
¶
Bases: EditError
Exception raised when the bins of an edit do not exist, or the feature type does not allow the selection.
StaleRevisionError
¶
Bases: EBaMlError
Exception raised when the head of an edit session is not the revision that the caller expected.
The message names the current head. The caller reads the session again and decides what to do. The session never retries by itself.
UnreviewedEditError
¶
Bases: EBaMlError
Exception raised when a save needs the review of an edit that nobody reviewed.
Benchmark¶
ebaml.benchmark
¶
Benchmark of how well an LLM reads the graphs of Explainable Boosting Machines.
Port of the benchmark/ of TalkToEBM. The questions are stored in benchmark/data as JSON lists of
[prompt, solution] pairs. Each question is answered in two LLM turns (see answer_benchmark_question in
baml_src/talk_to_ebm.baml): the model reasons step by step, then states its final answer in a format that this
module parses and compares with the solution. TalkToEBM's notebooks printed the answers next to the solutions and
compared them by eye; here the comparison is automatic. The module imports no LLM client: the caller passes the
function that asks the question, so the parsing and the scoring run without the generated baml_sdk.
Usage
¶
Reply
¶
AskQuestion
¶
Bases: Protocol
A function that asks the LLM a question of the benchmark.
baml_sdk.answer_benchmark_question_sync is such a function.
__call__(*, question, answer_format)
¶
Ask the question and ask for the final answer in the format of answer_format.
Task
¶
Bases: StrEnum
The tasks of the benchmark. The value is the name of the data file.
answer_format
property
¶
How the model is asked to state its final answer, to complete "answering with ...".
Returns:
| Type | Description |
|---|---|
str
|
The description of the answer format of the task. |
Question
¶
A question of the benchmark.
Outcome
¶
The answer of the LLM to a question.
TaskResult
¶
The outcomes of a task, with the tokens that the provider reported for them.
correct
property
¶
The number of questions that were answered correctly.
Returns:
| Type | Description |
|---|---|
int
|
The number of correct outcomes. |
accuracy
property
¶
The fraction of the questions that were answered correctly.
Returns:
| Type | Description |
|---|---|
float
|
The accuracy, between 0 and 1. A task with no outcomes has the accuracy 0. |
load_questions(task, data_dir)
¶
parse_answer(task, text)
¶
Parse the answer to a question of a task, or its solution.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task
|
Task
|
The task, which sets the format of the answer. |
required |
text
|
str
|
The text of the answer. |
required |
Returns:
| Type | Description |
|---|---|
Answer | None
|
The parsed answer, or None if the text does not hold an answer in the format of the task. |
is_correct(task, response, solution)
¶
Whether the response of the LLM states the solution.
The comparison is made on the parsed answers, so the formatting of the response (a or (a), 0.50 or
.5, markdown, a trailing period) does not matter, but the values must be exactly the same.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task
|
Task
|
The task, which sets the format of the answer. |
required |
response
|
str
|
The final answer of the LLM. |
required |
solution
|
str
|
The solution of the question. |
required |
Returns:
| Type | Description |
|---|---|
bool
|
True if the response holds an answer, and it is the solution. |
Examples:
>>> is_correct(Task.MONOTONICITY, "The graph is **increasing**.", "increasing")
True
>>> is_correct(Task.READ_VALUE, "The answer is 0.50.", "0.5")
True
>>> is_correct(Task.READ_VALUE, "The answer is 0.25.", "0.5")
False
run_task(task, questions, ask, *, progress=False)
¶
Ask the LLM the questions of a task, one at a time, and score its answers.
The questions are asked sequentially, because the free models are rate limited.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task
|
Task
|
The task, which sets the answer format and the scoring. |
required |
questions
|
Sequence[Question]
|
The questions to ask, for example from |
required |
ask
|
AskQuestion
|
Asks the LLM one question, for example |
required |
progress
|
bool
|
Whether to show a progress bar of the questions on the error stream. If that stream is not a terminal, the bar writes one line for each question. |
False
|
Returns:
| Type | Description |
|---|---|
TaskResult
|
The outcomes and the token usage. If there are no questions, there are no outcomes and the accuracy is 0. |
format_results(results)
¶
Format the results of the tasks as a table.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
results
|
Sequence[TaskResult]
|
The results, one per task. |
required |
Returns:
| Type | Description |
|---|---|
str
|
A table with a row per task: the correct answers, the accuracy and the tokens used. |