Matrix Runner¶
pikit.matrix ¶
Batch matrix evaluation — run attacks × defenses × channels × agents.
The :class:MatrixRunner automates the combinatorial experiment: for every
combination of attack, defense, channel, and agent, it crafts the injection,
runs the agent, and applies a judge. Results are collected into a list of
:class:ExperimentResult objects and can be saved to JSON or CSV.
This is the engine that makes pikit useful for research: instead of running
one combination at a time via demos/run.py, you define a matrix in a
TOML file (see :class:~pikit.config.ExperimentConfig) and let it run.
Example¶
from pikit import matrix from pikit.config import ExperimentConfig cfg = ExperimentConfig( ... attacks=["naive"], defenses=["none"], agents=["chat"], ... target_spec="mock", judge_type="rule", ... ) results = matrix.run(cfg) len(results) 1 isinstance(results[0], matrix.ExperimentResult) True
ExperimentResult
dataclass
¶
ExperimentResult(attack: str, defense: str, channel: str, agent: str, target: str, task: str, success: bool, confidence: str, partial_success: bool = False, success_level: str = 'none', reason: str = '', signals: List[str] = list(), final_text: str = '', sink_fired: bool = False, trace: str = '', timestamp: str = '', repeat_index: int = 0, success_count: int = 0, total_runs: int = 1, run_id: str = '', case_id: str = '', seed: Optional[int] = None, generation_config: Dict[str, Any] = dict(), method_specs: Dict[str, Dict[str, Any]] = dict(), evidence: List[Dict[str, Any]] = list(), trace_data: Dict[str, Any] = dict(), metadata: Dict[str, Any] = dict(), outcome: Outcome = Outcome.NOT_REACHED, model_complied: Optional[bool] = None, runtime_blocked: bool = False)
The outcome of one matrix cell (one combination).
Attributes¶
attack, defense, channel, agent:
The combination keys ("" for no channel/direct).
target:
The target spec used.
task:
The attacker's injected instruction.
success:
Judge verdict — whether the injection succeeded.
confidence:
Judge confidence level.
reason:
Judge explanation.
signals:
Judge signal names.
final_text:
The agent's final output text.
sink_fired:
Whether any sink tool was called.
trace:
Full trace as a string (for debugging / inspection).
timestamp:
When this result was produced.
MatrixRunner ¶
Run a batch experiment defined by an :class:ExperimentConfig.
Parameters¶
config: The experiment specification. verbose: Print progress to stderr as each combination runs.
Examples¶
from pikit.matrix import MatrixRunner from pikit.config import ExperimentConfig cfg = ExperimentConfig(target_spec="mock", judge_type="rule") runner = MatrixRunner(cfg) results = runner.run() len(results) >= 1 True
run_one ¶
run_one(attack_key: str, defense_key: str, channel_key: str, agent_key: str, *, attack_kwargs: Optional[Dict[str, Any]] = None, defense_kwargs: Optional[Dict[str, Any]] = None, channel_kwargs: Optional[Dict[str, Any]] = None) -> ExperimentResult
Run a single combination and return its result.
run ¶
Run all combinations in the matrix.
When config.repeats > 1, each combination is run that many
times. Each individual run produces its own
:class:ExperimentResult (with repeat_index set), and a
summary row is appended with aggregate success_count and
total_runs.
Returns¶
List[ExperimentResult] One result per combination (and per repeat), in the order they were run. When repeats > 1, a summary row is appended per combination.
save_json ¶
Save results to a JSON file.
save_csv ¶
Save results to a CSV file (flat, no trace column).
save_jsonl ¶
Save one full structured experiment result per JSONL line.
run ¶
Convenience: create a runner and execute it.
pikit.matrix.MatrixRunner ¶
Run a batch experiment defined by an :class:ExperimentConfig.
Parameters¶
config: The experiment specification. verbose: Print progress to stderr as each combination runs.
Examples¶
from pikit.matrix import MatrixRunner from pikit.config import ExperimentConfig cfg = ExperimentConfig(target_spec="mock", judge_type="rule") runner = MatrixRunner(cfg) results = runner.run() len(results) >= 1 True
run_one ¶
run_one(attack_key: str, defense_key: str, channel_key: str, agent_key: str, *, attack_kwargs: Optional[Dict[str, Any]] = None, defense_kwargs: Optional[Dict[str, Any]] = None, channel_kwargs: Optional[Dict[str, Any]] = None) -> ExperimentResult
Run a single combination and return its result.
run ¶
Run all combinations in the matrix.
When config.repeats > 1, each combination is run that many
times. Each individual run produces its own
:class:ExperimentResult (with repeat_index set), and a
summary row is appended with aggregate success_count and
total_runs.
Returns¶
List[ExperimentResult] One result per combination (and per repeat), in the order they were run. When repeats > 1, a summary row is appended per combination.
pikit.matrix.ExperimentResult
dataclass
¶
ExperimentResult(attack: str, defense: str, channel: str, agent: str, target: str, task: str, success: bool, confidence: str, partial_success: bool = False, success_level: str = 'none', reason: str = '', signals: List[str] = list(), final_text: str = '', sink_fired: bool = False, trace: str = '', timestamp: str = '', repeat_index: int = 0, success_count: int = 0, total_runs: int = 1, run_id: str = '', case_id: str = '', seed: Optional[int] = None, generation_config: Dict[str, Any] = dict(), method_specs: Dict[str, Dict[str, Any]] = dict(), evidence: List[Dict[str, Any]] = list(), trace_data: Dict[str, Any] = dict(), metadata: Dict[str, Any] = dict(), outcome: Outcome = Outcome.NOT_REACHED, model_complied: Optional[bool] = None, runtime_blocked: bool = False)
The outcome of one matrix cell (one combination).
Attributes¶
attack, defense, channel, agent:
The combination keys ("" for no channel/direct).
target:
The target spec used.
task:
The attacker's injected instruction.
success:
Judge verdict — whether the injection succeeded.
confidence:
Judge confidence level.
reason:
Judge explanation.
signals:
Judge signal names.
final_text:
The agent's final output text.
sink_fired:
Whether any sink tool was called.
trace:
Full trace as a string (for debugging / inspection).
timestamp:
When this result was produced.
pikit.matrix.save_json ¶
Save results to a JSON file.
pikit.matrix.save_csv ¶
Save results to a CSV file (flat, no trace column).
pikit.matrix.save_jsonl ¶
Save one full structured experiment result per JSONL line.
pikit.matrix.run ¶
Convenience: create a runner and execute it.