Testing
Plugin
criterium
Benchmarking library for clojure
Plugin Signal60/100Solid
maintained73
capable14
documented100
adopted66
Install on Claude Code
> /plugin marketplace add hugoduncan/criterium> /plugin install criterium@criterium
What's inside
README
1Criterium
Alpha version
Master is now on 0.5.x, which should be considered ALPHA, subject to breaking changes on new features.
See ALPHA Readme
If you try it, please give feedback on any breaking changes with 0.4.6.
Stable version 0.4.6
Criterium measures the computation time of an expression. It is designed to address some of the pitfalls of benchmarking, and benchmarking on the JVM in particular.
This includes:
- statistical processing of multiple evaluations
- inclusion of a warm-up period, designed to allow the JIT compiler to optimise its code
- purging of gc before testing, to isolate timings from GC state prior to testing
- a final forced GC after testing to estimate impact of cleanup on the timing results
Installation
Leiningen
Add the following to your :dependencies:
[criterium "0.4.6"]
Maven
<dependency>
<groupId>criterium</groupId>
<artifactId>criterium</artifactId>
<version>0.4.6</version>
</dependency>
Usage
The top level interface is in criterium.core.
(use 'criterium.core)
Use bench to run a benchmark in a simple manner.
(bench (Thread/sleep 1000))
=>
Execution time mean : 1.000803 sec
Execution time std-deviation : 328.501853 us
Execution time lower quantile : 1.000068 sec ( 2.5%)
Execution time upper quantile : 1.001186 sec (97.5%)
By default bench is quiet about its progress. Run with-progress-reporting to
get progress information on *out*.
(with-progress-reporting (bench (Thread/sleep 1000) :verbose))
(with-progress-reporting (quick-bench (Thread/sleep 1000) :verbose))
Lower level functions are available, that separate benchmark statistic generation and reporting.
(report-result (benchmark (Thread/sleep 1000) {:verbose true}))
(report-result (quick-benchmark (Thread/sleep 1000)))
Note that results are returned to the user to prevent JIT from recognising that the results are not used.
Measurement Overhead Estimation
Criterium will automatically estimate a time for its measurement
overhead. The estimate is normally made once per session, and is
available in the criterium.core/estimated-overhead-cache var.
If the estimation is made while there is a lot of other processing
going on, then benchmarking quick functions may report small negative
times. You can force a recalculation of the overhead by calling
criterium.core/estimated-overhead!.
If you want consistency across JVM processes, it might be prudent to
explicitly set criterium.core/estimated-overhead! to a constant
value.
References
API Documentation Annotated Source
See Elliptic Group for a Java benchmarking library. The accompanying article describes many of the JVM benchmarking pitfalls.
See Criterion for a Haskell benchmarking library that applies many of the same statistical techniques.
Todo
Serial correlation detection. Multimodal distribution detection. Use kernel density estimators?
Non-Goals
Expressions are evaluated inside a function call. We deliberately do not seek ultimate accuracy for very quick expressions where the function overhead may be significant.
Releasing
To release, run the release.sh script. This requires that you have
git-flow enabled your git repository with git flow init, and that
you have configured your
credentials for clojars.
YourKit
YourKit is kindly supporting open source projects with its full-featured Java Profiler.
YourKit, LLC is the creator of innovative and intelligent tools for profiling Java and .NET applications. Take a look at YourKit's leading software products:
- YourKit Java Profiler and
- YourKit .NET Profiler.
License
Licensed under EPL
File tree
473hugoduncan/criteriumView on GitHub
.claude/
.claude-plugin/
marketplace.json
plugin.json
CLAUDE.md
settings.json
.clj-kondo/
config.edn
hooks/
impl.clj
.cljfmt.edn
.dir-locals.el
.githooks/
pre-commit
pre-push
.github/
workflows/
agent-cpp.yml
release.yml
tests.yml
.gitignore
.line-breaker.edn
.lsp/
config.edn
.nrepl.edn
.travis.yml
agent-cpp/
.clang-tidy
agent.cpp
CMakeLists.txt
compile_commands.json
include/
agent_state.h
agent_types.h
alloc_rec.h
call_tree.h
jni_operations.h
jvmti_operations.h
message_queue.h
state_transitions.h
utils.h
test/
agent_state_test.cpp
all_tags_test.cpp
call_tree_test.cpp
CMakeLists.txt
integration_test.cpp
is_system_class_test.cpp
message_queue_test.cpp
method_tracing_state_transitions_test.cpp
mocks.h
state_transitions_test.cpp
test_main.cpp
bases/
agent/
deps.edn
resources/
agent/
.keep
src/
clj/
criterium/
agent/
agent.clj
core.clj
instrumentation.clj
loader.clj
platform.clj
runtime.clj
wrapper.clj
java/
criterium/
agent/
Agent.java
Allocation.java
MethodCall.java
test/
criterium/
agent/
agent_test.clj
core_test.clj
degradation_test.clj
integration_test.clj
loader_test.clj
method_call_test.clj
platform_test.clj
arg-gen/
deps.edn
resources/
clj-kondo.exports/
org.hugoduncan/
arg-gen/
clj_kondo/
hooks.clj
config.edn
src/
criterium/
arg_gen.clj
test/
criterium/
arg_gen_test.clj
blackhole/
deps.edn
resources/
blackhole/
.keep
src/
clj/
criterium/
blackhole.clj
java/
criterium/
blackhole/
Blackhole.java
test/
criterium/
blackhole_test.clj
criterium/
.clj-kondo/
config.edn
deps.edn
resources/
clj-kondo.exports/
org.hugoduncan/
criterium/
clj_kondo/
criterium/
hooks.clj
config.edn
src/
criterium/
allocation/
allocation.clj
analysis.clj
analyse/
analyse.clj
call_graph.clj
digest_samples.clj
methods.clj
metrics_samples.clj
bench/
bench_plans.clj
bench.clj
config.clj
impl.clj
benchmark.clj
call_graph/
call_graph.clj
plans.clj
collect_plan/
collect_plan.clj
config.clj
impl.clj
collect.clj
collector/
collector_configs.clj
collector.clj
fns.clj
impl.clj
metrics.clj
core.clj
domain/
domain_plans.clj
domain.clj
analysis.clj
builder.clj
types.clj
instrument_fn.clj
instrument.clj
jvm/
jvm.clj
impl.clj
measured/
measured.clj
impl.clj
metric.clj
platform.clj
sampled_fn.clj
sampler.clj
trigger/
trigger.clj
impl.clj
util/
blackhole.clj
bootstrap.clj
debug.clj
format.clj
forms.clj
helpers.clj
invariant.clj
output.clj
sampled_stats.clj
t_digest/
t_digest.clj
merging_digest.clj
scale.clj
units.clj
view.clj
viewer/
call_graph.clj
common/
common_charts/
autocorrelation.clj
comparison.clj
distribution_ascii.clj
distribution.clj
profile.clj
quantile.clj
regression.clj
samples.clj
tail_ascii.clj
tail.clj
util.clj
allocation.clj
ascii_chart.clj
autocorrelation.clj
bootstrap.clj
core.clj
distribution.clj
domain/
comparison.clj
detection.clj
extract.clj
modal.clj
regression.clj
shape.clj
tail.clj
kindly/
kindly.clj
allocation.clj
autocorrelation.clj
core.clj
distribution.clj
domain.clj
modal.clj
shape.clj
tail.clj
portal/
portal.clj
allocation.clj
autocorrelation.clj
core.clj
distribution.clj
domain.clj
modal.clj
shape.clj
tail.clj
pprint.clj
print/
print.clj
allocation.clj
autocorrelation.clj
core.clj
distribution.clj
domain.clj
modal.clj
shape.clj
table.clj
tail.clj
data_readers.clj
test/
criterium/
allocation/
allocation_test.clj
analysis_test.clj
analyse/
analyse_test.clj
call_graph_test.clj
digest_samples_test.clj
metrics_samples_test.clj
array_allocation_test.clj
bench/
bench_plans_test.clj
bench_test.clj
config_test.clj
benchmark_test.clj
call_graph/
call_graph_test.clj
plans_test.clj
collect_plan/
collect_plan_test.clj
config_test.clj
impl_test.clj
collect_test.clj
collector/
collector_test.clj
fns_test.clj
metrics_test.clj
core_test.clj
data/
air_quality.clj
r_validation/
adjbox.clj
medcouple.clj
suicide.clj
wage2.clj
domain/
domain_plans_test.clj
domain_test.clj
analysis_test.clj
builder_test.clj
test_util.clj
types_test.clj
instrument_fn_test.clj
instrument_test.clj
jvm/
jvm_test.clj
impl_test.clj
measured_test.clj
metric_test.clj
platform_test.clj
probability_test
sampled_fn_test.clj
test_data.clj
test_utils.clj
trigger_test.clj
util/
bootstrap_test.clj
format_test.clj
forms_test.clj
helpers_test.clj
histogram_test.clj
invariant_test.clj
probability_test.clj
sampled_stats_test.clj
t_digest/
merging_digest_test.clj
MergingDigest.java
scale_test.clj
viewer/
call_graph_test.clj
common/
common_charts/
autocorrelation_test.clj
bar_test.clj
box_test.clj
distribution_test.clj
line_test.clj
profile_test.clj
quantile_test.clj
regression_test.clj
samples_test.clj
tail_ascii_test.clj
tail_test.clj
util_test.clj
allocation_test.clj
ascii_chart_test.clj
core_tail_test.clj
domain/
comparison_test.clj
detection_test.clj
extract_test.clj
regression_test.clj
completeness_test.clj
kindly/
kindly_accumulator_test.clj
kindly_allocation_views_test.clj
kindly_autocorrelation_views_test.clj
kindly_basic_views_test.clj
kindly_clay_test.clj
kindly_distribution_views_test.clj
kindly_domain_views_test.clj
kindly_tail_views_test.clj
kindly_test_helpers.clj
allocation_test.clj
autocorrelation_test.clj
core_test.clj
distribution_test.clj
domain_test.clj
modal_test.clj
shape_test.clj
tail_test.clj
portal/
portal_autocorrelation_views_test.clj
portal_test.clj
allocation_test.clj
autocorrelation_test.clj
core_test.clj
distribution_test.clj
domain_test.clj
modal_test.clj
shape_test.clj
tail_test.clj
pprint_test.clj
print/
allocation_test.clj
autocorrelation_test.clj
core_test.clj
distribution_test.clj
domain_test.clj
modal_test.clj
shape_test.clj
tail_test.clj
schema_validation_test.clj
schema_validation.clj
validate-vega-lite.mjs
tests.edn
validation/
criterium/
validation/
... 273 moreFAQ
criterium is a Claude Code plugin with 1 hand-picked skill for testing work, indexed on Flowy. Install it with the command on its page. It includes criterium. Its skills do not fire on their own yet. Request auto-invocation to have Flowy route them as you prompt. Free and open source.
- What is criterium?
- Benchmarking library for clojure
- How do I install criterium?
- Run these in Claude Code: /plugin marketplace add hugoduncan/criterium then /plugin install criterium@criterium. Then prompt normally.
- Does criterium auto-invoke its skills?
- Not yet. It is indexed on Flowy as a plain plugin. Request auto-invocation on its page and Flowy will route its skills for you as you prompt.
- Is criterium free?
- Yes. Flowy is free and open source, with nothing gated. You can read every skill in full before you install.