Contributing¶
What this repository holds in common with the others of the organization — the toolchain, the lint gate, the tool tables behind it, the workflow set and the branch rules — is stated once in the btclib-org repository standard, each rule with the alternative it was decided against. It binds this repository, so a change departing from it is a divergence, and one filed as an issue in that repository rather than here: a difference between two repositories belongs to neither of them.
This file is the same in every repository of the organization up to its last section. What is true of one tree only — the commands that build its environment, the gates it runs, which of its workflows decide a merge — is under that heading, and the comparison stops there.
How the organization decides, and who holds which role, is
GOVERNANCE.md; what it intends to do, and what it
deliberately does not, is ROADMAP.md. Both are the
organization’s, one copy each beside the standard.
The issue tracker¶
Where an issue is filed, and what an alignment finding has to name, is the standard’s What this repository is: an issue spanning repositories, or whose subject is the standard, goes to btclib-org/.github, and one about this tree alone stays here.
A finding noticed while doing something else goes where REVIEWING.md’s
What is filed, and what is not says, for an author as much as for a
reviewer: a pull request answering two questions cannot be accepted for
either.
An issue labelled good first issue is one set aside for a first
contribution: small and self-contained.
One search lists the open ones across the organization, so
a newcomer need not know in advance which repository to look in.
Documentation and comments¶
Section 9 of the standard is the prose style, and it governs the prose this tree ships — comments, docstrings and markdown. It is not restated here: a second wording is the one that goes stale, which is that section’s own One fact in one place.
A commit message is prose this tree ships too, though section 9 does not
say so: the only merge method the rule accepts puts it on main
as the landing commit’s body, so what is written in one is read there
long after the branch is gone.
Pull requests¶
What main accepts, and what it refuses to everyone, is section 11 of
the standard. Run the gates locally before opening anything —
the last section of this file says which they are — because CI runs
exactly them, so a red run there is a local run that was not done.
Every commit of a pull request carries a Signed-off-by: trailer
naming its author, which certifies the Developer Certificate of
Origin. git commit -s adds it, and git rebase --signoff <base>
adds it to commits already made. The standard’s Signatures
says why a signature does not replace it, and which commits the
Sign-off job skips.
What a pull request’s title and description have to say about the issues it closes, and why a manual link in the Development panel is a trap neither of them shows, is the standard’s What a pull request says it is. Read it before opening one; it is the rule most often found broken after the fact.
Before it is opened, the branch’s own commit subjects and bodies are read against that same rule. The description does not exist yet to disagree with them, and the standard has the command that scans the branch’s own commit text for a verb in front of a reference.
The two spellings are named here as well as there, against section 9’s
One fact in one place, the paragraph above naming the section
and not the forms, which are the half a citation is got wrong in:
(closes #N) cites an issue the change closes, wherever the citation
sits — the title, the commit subject where Merge method makes
that the thing that lands, and a CHANGELOG.md entry — and (issue #N)
cites, in those same places, an issue the change advances and does not
close. One token holds one meaning whichever file it sits in, so the
pair is chosen by what is true of the change rather than by which file
is being written, and a tree’s own landed subjects are not what to copy
it from: nothing already landed is rewritten, so what a repository wrote
before the rule stays where it is.
REVIEWING.md is the standard a review is written against, and is this
file’s other half. Read before opening a pull request, it is what the
pull request will be answered against.
CHANGELOG.md gets an entry for anything a reader would notice, and the
release notes move only for something a user has to act on, in the
repositories that publish.
Where that entry goes is section 9’s — the end of the open
section — and no gate reads it: check-changelog is handed the file and
no base, so it cannot tell which entry the branch wrote. The open
section’s headings, in the order the file holds them, a branch’s own
last:
awk '/^## /{n++} n==1 && /^### /' CHANGELOG.md
n==1 takes the open section, from the first ## heading to the next,
and the scan is /^## / rather than /^## v/: a section headed
## Unreleased is no match for /^## v/, which counts from the first
release heading instead and prints a released section’s entries — or
nothing, where the tree has released nothing — while reading as a
check that passed.
One subject, opened as soon as it is written¶
A pull request answers one question. Issues that share a subject are one pull request, closing each of them; issues that do not are one pull request each, however small either of them is.
It is opened the moment it is written and verified — not held for the previous one to be reviewed or to land, and not batched with the next. A batch arrives as one reviewing job with several subjects, which is the shape that costs the most to read; a finished pull request held back is review that could have started and did not.
Working this way stacks branches, which is fine and costs one rule: a child whose base was amended is moved with the old base named,
git rebase --onto <new-base> <old-base-sha> <child>
because a plain rebase replays the base’s old commit inside the child,
and the forge then shows the base’s old text as additions with nothing
red anywhere. Read the child’s diff afterwards rather than trusting the
rebase, and retarget each child onto main as its parent lands.
The landing queue¶
Where more than one pull request is open against this repository, only
one is carried to main at a time: rebased onto the tip, reviewed on
that head, and landed, while every other one waits, untouched, for its
turn. This governs which of several already open pull requests reaches
main next; One subject, opened as soon as it is written above governs
the moment before that, when a finished one is opened — the two do not
conflict, since a pull request is still opened without delay and still
waits its turn once several are open.
The reason is CI throughput, not the ack a waiting pull request keeps —
REVIEWING.md’s The verdict states what an ack belongs to, and
Landing it below states which rebase voids one. Every rebase queues
this repository’s whole check matrix against the organization’s ceiling
on concurrent jobs, so rebasing every waiting pull request after each
landing spends that capacity on runs the next landing invalidates
anyway, and delays the one pull request that is actually next: work
spent on a pull request that is not next is work that delays the one
that is. The ceiling’s figure is REPOSITORY.md’s, under Plan-gated
settings, beside the command that re-derives it.
Order is cheapest and least contended first, most invasive last, so that a large change does not sit at the head blocking everything behind it.
The maintainer may declare a bounded exception — several pull requests in flight against one repository, for a named piece of work — trading the cost above for throughput; it is recorded as a comment in btclib-org/.github, by The issue tracker above, and holds only for the work it names.
The review¶
A review is given promptly and on local evidence. It does not wait for CI, does not report a check as a finding, and does not discuss a run at all: whether CI is green is the author’s business, once, at landing time.
The exchange is anchored to a sha rather than to a branch, a branch being free to move under a review:
the author hands off by naming the sha pushed and the evidence run against it, then leaves that head alone;
the reviewer answers with findings — where, what is wrong, how they know it, and whether each is blocking;
the author accepts what is reasonable, declines the rest with a reason in the thread, and pushes the answer without waiting for CI;
the reviewer resolves the threads they opened, that being what says a finding is closed, and re-reviews the delta rather than the branch.
What ends the loop is the ack of record, and the author does not
supply their own. A reading that says what it found and delivers no
verdict is a review too and ends nothing; the standard’s Review
has which is which, and REVIEWING.md has how each is written. A
disagreement that survives a second exchange goes to the maintainer
instead of into a third round.
While claude-review.yml is off, there is no ack of record
(decision 70a of ISS 452). The maintainer lands their own pull
requests through the bypass below. What stands in for the ack is a local
review of a named sha, by a reviewer other than the author, before the
pull request opens. The OpenSSF Best Practices criterion
two_person_review is unmet (decision 1 of ISS 1362).
Landing it¶
CI is read once, and this is where. Rebase onto main’s tip, push that
head so the checks run on the tree that will land, and only then wait for
them: checks read before a rebase describe a tree nobody is landing. A
rebase that moved nothing but the base leaves the ack standing; one that
resolved a conflict does not, that resolution being a change no reviewer
has seen.
Then squash, the only method the rule accepts.
The maintainer’s bypass is not automatic — it has to be invoked, and
gh pr merge cannot invoke it, refusing client-side before it asks
GitHub anything:
Pull request is not mergeable: the base branch policy prohibits the merge
The merge endpoint applies it server-side, and it is the same endpoint the merge button asks:
gh api -X PUT repos/{owner}/{repo}/pulls/<n>/merge \
-f merge_method=squash -f sha=<the head the checks ran on>
The sha is not optional. Reading the ack and merging are two
calls, and the head is free to move between them — the push that would
move it comes out of the same round the verdict does. Unpinned, the
command takes whatever sits at the head when it runs; pinned, the
endpoint answers 409 where the head has moved, and a round
lost that way is cheaper than a tree nobody has read reaching main.
The review above anchors the exchange to a sha and section 11
has an ack name one: the pin is that rule reaching the call that
performs the landing.
Verify what landed rather than trusting the answer, the signature the standard asks for being a valid one rather than a particular signer’s:
gh api repos/{owner}/{repo}/commits/main \
--jq '.commit.verification | {verified, reason}'
What it closed is read again here too, from the landed sha rather than from the pull request: the standard’s What a pull request says it is has the second read, and why the first alone does not reach a squash subject composed after it runs.
The forge deletes the head branch itself, per the setting section 11
names. What is still yours is bringing every checkout sitting on main
up to date,
that being where the next session starts from and a stale one being where
a branch gets built on a base that has moved. REPOSITORY.md carries the
settings and why they are what they are.
This repository in particular¶
Everything above is the same file in every repository of the organization; everything below is this one’s, and the comparison stops at this heading.
The environment and the gates¶
btclib is managed with uv, the only tool that must be installed on the development machine: see the uv installation instructions.
A virtual environment with btclib (in editable way) and all the development tools, including those needed to build the documentation, is then created with:
uv sync
Every dependency is required but the bindings, which are the secp256k1
extra, and every one of them is a floor with no upper bound, and the absence
of a ceiling is a decision. Which releases those floors name is
pyproject.toml, next to the reason each one is where it is: a floor
moves whenever this tree starts calling something newer, so a copy of the
number here would be a second place to remember and the first to go
stale. typing-extensions is the backport of what the 3.11 floor does
not have, and its floor is the release adding the latest name this tree
imports from it. The others are btclib-org projects developed by the
same people, and the bindings’ whole purpose is to be the bindings this
library calls, so a breaking change there is coordinated with the release
here —
which is what a version ceiling substitutes for when it cannot be. A
<0.9 ceiling would cost a btclib
release for every bindings minor, the ones that break nothing included, and
would make a published artifact refuse a version it in fact works with; a
<1 ceiling constrains nothing, pre-1.0 semver putting the breaking
changes in the minor.
bitcoin-core-rpc is required and the bindings are an extra, and that
was weighed rather than inherited. Making bitcoin-core-rpc an extra
was considered and refused: an optional dependency whose absence changes
a speed is a different object from one whose absence removes a
capability, and only the first is what an extra is for. The bindings
are the first — btclib answers without them, on a pure-Python arm that
is supported, covered by CI and documented. bitcoin-core-rpc is the
second: btclib.p2p.magic takes Core’s chain names and a signet’s
message start from it, and nothing in btclib stands behind them.
Every sibling floor names a release PyPI serves, so pyproject.toml
carries no [tool.uv.sources] table and uv resolves the siblings from
the index like anything else: uv.lock pins a version and its wheels,
uv sync downloads rather than compiling libsecp256k1 in each
environment, and every job that installs from the lock passes --locked.
The floor is then the whole of the coordination with that sibling, and
it carries weight the specifier alone does not show: btclib_ecc imports the bindings’
surface in a single try whose except ImportError sets
INSTALLED = False, so an install that resolves one release short of
what that package imports does not lose the entry point it is missing —
it loses every delegation, quietly, on to the Python arithmetic that
SECURITY.md publishes as slower and not constant-time. btclib’s own
imports of the bindings are guarded one by one instead, so a shortfall
there fails at the call rather than degrading. Calling something new is
therefore the same commit that raises the floor in both of its places,
with the reason written beside it.
The jobs that do not pass --locked are deliberate exceptions.
uv build, uv run --no-project and, on a rehearsal, the dev-version
re-lock run in the job that builds the published files
(reusable-build.yml, which release.yml calls). None installs the
project’s dependencies:
test.yml’sdistjob installs the wheel under constraints exported from the lock, which pins the same versions, and on a release asks once, withuv run --isolated --with, whether the newest bindings work, after the build job and on the files that job uploaded; it publishes nothing.deps-latest.ymlrunsuv lock --upgradeto test the newest versions,deps-oldest.ymlre-locks to each declared floor, andpypi-install.ymlrunsuv pip installto test the published release.The
dev-versionaction re-locks after writing a rehearsal’s version suffix;uv lockkeeps the versions already locked.uv buildanduv run --no-projectinstall nothing from the lock.public-api, in btclib-org/.github’sreusable-public-api.yml, runsuvx griffe==2.2.0: a pinned version, with its own dependencies unlocked. It is not the build job.fuzz.yml’s ClusterFuzzLite build installs.clusterfuzzlite/requirements.txt, the lock exported with its hashes by theuv-exporthook, withpip3 install --require-hashes.
Calling an entry point merged upstream but not yet released is what a
[tool.uv.sources] entry pointing a sibling at a commit of its main
branch is for, and it is a state to leave rather than to keep.
[tool.uv] is not distribution metadata, so dependencies still
publishes a plain floor and PyPI still accepts a release — where a
direct reference written into dependencies instead would have to be
written back over before publication, which is what RELEASING.md’s step
1 refuses. What the entry costs is the wheels: a git source has none, so
every environment builds that sibling from source, for the bindings
their submodule and C library included, and uv.lock pins the resolved
commit, so the branch
is followed only when the lock is regenerated. One release upstream
clears it — the source table goes and the floor moves to the version
that carries what is being called.
What the policy does not cover: dependency metadata is baked into every wheel already published, so coordinating the two projects protects the current pair and not an artifact that went out before. Only the latest release is supported (SECURITY.md) and nothing is backported, so the promise is about that pair — a bindings release keeps the runtime API the supported btclib needs, and an older btclib may one day stop installing or running against the newest bindings.
The bound is the oldest final release this version supports, and a prerelease of it sorts below it rather than satisfying it. What a resolver does with prereleases is its own policy — pip and uv alike prefer a stable candidate and reach for a prerelease only when no stable one satisfies the constraints — so the bound states support and nothing else. See the version-specifiers page and uv’s prerelease handling.
Every command is run inside that environment prefixing it with uv run
(e.g., uv run pytest); alternatively, activate the environment once with
source .venv/bin/activate (.venv\Scripts\activate on Windows).
The dependency groups defined in pyproject.toml can also be installed
individually, e.g. uv sync --no-default-groups --group test.
The interpreter .python-version pins is the one uv sync picks and the
one uv run uses. To reproduce a failure that only one cell of the
matrix shows, name the interpreter instead of editing that file:
uv sync --python 3.11
uv run --python 3.11 pytest
--python rebuilds .venv, and that has a consequence worth knowing
before it bites. An interpreter other than the one already there makes uv
report Removed virtual environment at: .venv and create it again — so
whatever groups the command asks for are the only ones installed
afterwards. With the group-restricted commands under “Reproducing what CI
runs” below, that leaves an environment holding one group where uv sync
installs them all, and pre-commit is not in it. uv sync puts the gate
back, and so does any uv run that is not group-restricted, which syncs
the environment before it runs anything; the group-restricted ones leave
it out again. A shell that entered .venv with
source .venv/bin/activate finds no pre-commit at all until uv sync.
To leave .venv alone in the first place, put the other interpreter’s
environment somewhere else:
UV_PROJECT_ENVIRONMENT=.venv-3.11 \
uv run --locked --no-default-groups --group test --python 3.11 pytest \
--no-cov
.gitignore does not mention that directory and does not need to: uv
writes a .gitignore of its own, holding *, inside every environment it
creates, so the tree stays clean without a rule per interpreter.
As an annotated Python3 project, btclib is very strict on code formatting and linting (ruff, which replaces autoflake, bandit, black, docformatter, flake8, isort, pydocstyle, pylint, pyupgrade, and yesqa) and proper type definition (mypy): warnings are not tolerated and should be taken care of. This might be annoying at first, but enforcing formatting rules can be done easily once you’re finished with coding or, even better, automatically taken care of while coding if you configure your development environment. Type definition improves code readability and helps in spotting bugs.
Moreover,
the pytest unit tests
must pass at any time, with
coverage
of both the library and the test suite at the fail_under ratchet in
pyproject.toml — 100%, and coverage takes that literally: it
special-cases the value, so 99.999% is a red build where the 99.99 this
replaces passed everything above 99.985%, and the 99.9 before it ten
times as much again. No slack, so a statement no test reaches is either
covered by patching what stands in the way, as the ripemd160 fallback
and electrum’s round-trip check are, or marked pragma: no cover --
with the reason on that same line, and a reason too long for the line
above it as well, the inline half naming the case. --cov is in
pyproject.toml’s addopts, so uv run pytest prints the total and
enforces the ratchet on every whole run, on the interpreter
.python-version pins, which the gate is checked on — a run that
selects a subset reports without gating, since fail_under would
otherwise fail it on the tree’s coverage rather than on its own. Which
flags select is section 8 of the organization standard’s, and
tests/README.md is where this tree states it.
See Tests, code coverage, and profiling.
The ratchet needs the release tags in the clone.
tests/changelog_immutability_test.py and tests/declared_version_test.py
read the v* tags. A clone holding none — --no-tags, or a --depth that
reaches no tagged commit — skips what they compare, and a skipped test’s
body counts as uncovered lines of tests/, so uv run pytest fails the
ratchet with no test failing. A --depth that reaches a tagged commit holds
only the tags it reaches, and tests/changelog_immutability_test.py then
fails for every release it reads whose tag the clone lacks. git fetch --tags
cures both without deepening a shallow history, and those modules need
nothing more.
These requirements are easily checked (and partially fixed) with:
uv run ruff check --fix
uv run ruff format
uv run mypy src/btclib tests .github/scripts
uv run pytest
or, in one go, with pre-commit:
uv run pre-commit run --all-files
The suite writes nothing: it runs against a read-only checkout, and from an
installed sdist. The one exception is deliberate and has to be asked for.
Several modules compare a dataclass’s to_dict() against a json file
committed beside them, under tests/**/_generated_files/, so that a change
to a serialized form is a failing test rather than something to notice in a
git diff. When such a change is the intended one:
BTCLIB_REGENERATE_GOLDEN=1 uv run pytest
rewrites those files, and the diff it leaves is the review the change wants. The failure message names the command, so there is nothing to remember.
That second command is not a convenience: it is the lint gate itself.
The lint workflow runs this very configuration, so what CI enforces is
what that command enforces, mark-down included
(markdownlint-cli2 is
one of the hooks, as are ruff, mypy, yamllint, actionlint, and the checks
on packaging metadata and on uv.lock).
The lint gate is not installed as a git hook. pre-commit install
writes into the common git directory, which every worktree of this
repository shares: git -C <worktree> rev-parse --git-path hooks answers
with the primary checkout’s .git/hooks from every one of them, so one
session installing it installs it for every other. The hook names an
interpreter by absolute path, the one that ran the install, so which
environment a commit runs through is decided by where the hook was
installed rather than by which tree the commit is made in. Run the gate
by hand before committing, the uv run pre-commit run --all-files above.
One of those hooks needs maintenance, and only one. The test vectors under
tests/ecc/_data/ and tests/script/_data/ are private keys by the
hundred, so they are recorded in .secrets.baseline as
already reviewed —
rather than excluded from the scan, which would leave those files unwatched
for a credential that has no business being there.
tests/block/_generated_files/block_481824.json is in the baseline for a
narrower reason: ACCA is one of the AWS key prefixes and is also four hex
digits, so the block’s own signatures match the detector wherever a script
is rendered as upper-case hex. Adding a vector to one of those files, or
changing what a golden file holds, means regenerating the baseline:
uvx --from detect-secrets detect-secrets scan --baseline .secrets.baseline
That preserves the plugin selection the file already carries, which is deliberate: the two entropy plugins are off, because in vectors made of 64-character hex strings a new high-entropy string is what a legitimate addition looks like.
Read the resulting diff before committing it. That is the whole point of a baseline rather than an exclusion: what appears in it is what nobody has looked at yet, so regenerating without reading turns the review into a rubber stamp and the hook into decoration.
The editor¶
.vscode/settings.json and .vscode/extensions.json are tracked, and they
hold no preference: the recommended extensions are the tools
.pre-commit-config.yaml already runs, and the settings put the fixing ones
on save. Installing them is optional and changes nothing about what a local
run enforces — what they buy is learning of a finding while typing rather
than at the local run that trips over it.
Anything machine-local — an interpreter path, a telemetry answer, a theme — belongs in the editor’s own user settings instead, those two files being read by every checkout of this repository.
What runs when¶
workflow |
when |
what it varies |
|---|---|---|
|
pull request, push |
— |
|
pull request, push |
— |
|
pull request, push, weekly |
a node |
|
pull request, and |
— |
|
pull request, push to main, and weekly |
2 languages |
|
weekly, and by hand before a release |
— |
|
weekly, push to main |
— |
|
weekly, a release |
ubuntu images and interpreters |
|
weekly, a release |
macOS images and interpreters |
|
weekly, a release |
Windows images and interpreters |
|
weekly |
platforms sampled, deps upgraded |
|
weekly |
the floor interpreter, deps at their floors |
|
weekly |
— |
|
weekly |
the pin ledgers |
|
weekly, a release |
what PyPI serves |
|
weekly |
the latest release’s sdist, rebuilt |
|
weekly, push to main |
the arm-authority table |
|
a tag |
the workflows |
That last cell is a pointer rather than a list on purpose: which workflows a release calls is read out of the file, where a list here goes stale the next time one is added or dropped.
grep -n 'uses: \./\.github/workflows/' .github/workflows/release.yml
The test, lint/docs and integration-bitcoind rows are what a
merge waits for, and between them they report the required checks, which
REPOSITORY.md names. They run one image on one interpreter: ubuntu-latest,
and the version .python-version names.
Which day each of the rest runs is section 10 of the organization
standard, in btclib-org/.github, and not this file’s to restate — one
calendar in one place is one thing to keep true, where a copy of it in
every repository would be one more.
Every workflow the tree holds has a row above, and
tests/what_runs_when_test.py is what holds it to that: it compares the
first column with .github/workflows/ both ways, so a workflow added
without a row fails the suite, and so does a row left behind by one
renamed or removed. What the cells say is nobody’s to check but the
reader’s.
Why so little gates is one number: the ceiling GitHub Free puts on an
organization’s concurrent jobs, shared across every repository in it.
REPOSITORY.md measures what a wider gate cost against that ceiling,
and the consequence is this table’s: at that ceiling a pull request’s
wall clock is the wait for a slot rather than the suite. os-macos.yml and
os-windows.yml each carry the measurement for their own cells, the
queueing and the runner seconds. codeql spends against that same
ceiling on every pull request too now, for the OpenSSF Scorecard’s
SAST check rather than for this table’s arithmetic — REPOSITORY.md has
that trade — but it still does not gate the merge: no matrix cell of its
analyze job is nameable in the branch rule on its own, and the
aggregate that is nameable, codeql: every job passed, is a name the
rule does not ask for. zizmor in lint reads
these same workflow files for an injected expression on every pull
request, which is a different question from what codeql asks.
The trade, stated here rather than discovered later: the gate does not
refuse a regression on 3.11, on arm, on PyPy or on a platform. It sits
on main until the sentinel for it runs, at most six days.
What a sentinel varies, it varies whole. os-ubuntu runs the images and
the interpreters the gate leaves alone and the cell it spends. A matrix
with the gate’s cell cut out of it is one nobody can read the shape of,
and whoever asked what ran would have to re-derive the hole from
test.yml.
os-ubuntu, os-macos and os-windows hold the dependencies at the lock and move
the platform; deps-latest moves both. Red in one of the three with deps-latest
green is that platform; red in both is the upgrade. Every workflow in the
table also takes workflow_dispatch, the gates included: a branch whose
pull request is not open yet has no other way to ask, and for the three
platform workflows it is the only way to ask about a branch at all.
claude-review and scorecard are the exceptions, and take none:
claude-review’s job calls a workflow whose two jobs each read the pull
request or the comment that triggered them, so a manual dispatch would
start a run with nothing to read; scorecard’s triggers are
ossf/scorecard-action’s own, whose README names push and schedule as
supported and calls workflow_dispatch experimental.
Reproducing what CI runs¶
Every job of every workflow is a uv command, and uv fetches what it
needs: no interpreter, no linter and no packaging tool has to be
installed by hand.
lint / Lint and type-check is lint.yml, calling btclib-org/.github’s
reusable-lint.yml, whose command is this, in full — the same pre-commit
the lock pins, which is what uv run above gives you too:
uv run --locked --only-group lint \
pre-commit run --all-files --show-diff-on-failure
lint / Dependency review is another job of the same call, and has no local
command: it asks the forge’s dependency graph what a pull request adds.
docs / Build the documentation is docs.yml, calling
btclib-org/.github’s reusable-docs.yml, and its command is the one
below under “The documentation”.
One cell of the platform sweep the python-versions list os-ubuntu.yml,
os-macos.yml and os-windows.yml each pass to btclib-org/.github’s
reusable-os-suite.yml names. The interpreter is chosen with --python,
which accepts any of the ones those workflows list, 3.14t and
pypy3.11 included, and downloads it if the machine has none:
uv run --locked --no-default-groups --group test --python 3.11 pytest --no-cov
That one rebuilds .venv with the test group alone, which is what leaves
the environment without pre-commit until the next uv sync: see the note
under “The environment and the gates” above, and UV_PROJECT_ENVIRONMENT for
running it without touching .venv. CI has no .venv to lose.
--no-cov is the matrix asking about the platform and not about the
number: it undoes the --cov addopts carries, so what a cell reports is
whether that (os, architecture, interpreter) triple passes. The job below
is where coverage is measured, which is the reason reusable-os-suite.yml’s
header gives, beside section 8’s floor. deps-latest.yml does not pass the
flag: it runs no PyPy cell and has no coverage job of its own, so the
ratchet meeting an upgraded coverage.py is one of the things that
workflow exists to find out.
The ubuntu-latest cell of those matrices on the interpreter
.python-version pins is the gate, and the job below is it: same image,
same interpreter, same suite, and the only difference is the
instrumentation. Reproducing that cell is therefore the
coverage command rather than this one.
The coverage job, gated by fail_under in pyproject.toml:
COVERAGE_FILE=coverage-data-bindings \
uv run --locked --no-default-groups --group test pytest
The job types pytest and nothing after it, which is section 8 of the
organization standard: addopts carries --cov, and what it measures and
how it reports are tool.coverage.run’s source and
tool.coverage.report in pyproject.toml. So this command and the bare
uv run pytest above are the same measurement in a clone carrying the
tags the job’s checkout fetches, the job being unable to gate on a scope a
contributor’s run does not have. COVERAGE_FILE is
--data-file’s environment variable, coverage’s own, and names here what
the step’s own env: names there, not a local-reproduction addition: the
two artifacts this job and no-bindings below produce would overwrite
each other under the default .coverage name once both land in the
coverage-union job that reads them. The job then uploads the data file
this command wrote as an artifact, for coverage-union to read; that
step has no command of its own to reproduce, being a plain
actions/upload-artifact.
The no-bindings job runs the suite against a btclib that has no
btclib_secp256k1 to delegate to. CI starts it from an empty
environment; locally uv run syncs additively, so a .venv any other
command in this section left behind — uv sync above included — still
holds the bindings, and the assert below fires on it. --exact is what
prunes them back to the one group this job runs with:
uv sync --exact --no-default-groups --group harness
uv run --locked --no-default-groups --group harness \
python -c "import importlib.util; \
assert importlib.util.find_spec('btclib_secp256k1') is None, \
'btclib_secp256k1 is installed'; \
from btclib_ecc.curves import is_libsecp256k1_serving; \
assert not is_libsecp256k1_serving()"
COVERAGE_FILE=coverage-data-no-bindings \
uv run --locked --no-default-groups --group harness \
pytest --cov-fail-under=0
uv sync
The closing uv sync restores the environment --exact just pruned to
harness alone, the same restoration the note above makes for any
group-restricted command in this section.
harness and not test: test is harness plus bindings, and uv’s
--no-group suppresses a group that was selected rather than one another
group includes, so the split in pyproject.toml is what this job is.
--group test in place of harness installs the bindings back and runs
the whole suite with them present, reporting it as a passing no-bindings
run with nothing to say it was not one.
--cov-fail-under=0, not --no-cov (issue #1002): the delegated arms
are still unreachable in this configuration by construction, so a report
of this run alone would still fail the 100% gate for the one reason
the run exists to create — --cov-fail-under=0 collects the data
without gating this run on it. It is the one argument the job types, and
it is the job’s own rather than a second copy: nothing in pyproject.toml
names it, where --cov is in addopts. The tests that hold both
implementations and compare them carry the bindings marker and skip
themselves; everything else runs. This job’s data is uploaded as an
artifact too, under the name COVERAGE_FILE gave it above, so it does
not collide with the coverage job’s when both are downloaded into the
same job.
The coverage-union job, which combines the two data files above and
gates their union at 100% as well — beside the coverage job’s own
gate on the delegated run alone, not instead of it, so that a line
covered only by the no-bindings run cannot pass in silence just because
the union reached it:
uv run --locked --no-default-groups --group harness \
coverage combine --rcfile=pyproject.toml \
coverage-data-bindings coverage-data-no-bindings
uv run --locked --no-default-groups --group harness \
coverage report --rcfile=pyproject.toml
Reproducing this one locally needs the two data files the commands above
already produced, under the two names they already wrote them with — no
renaming step, since COVERAGE_FILE on each command is what named them
that way in the first place. The two commands above run in two passes
either way, one .venv not being able to hold and lack the bindings at
once.
The report step types coverage report and nothing after it but
--rcfile, which is section 8 of the organization standard as the
coverage job’s pytest step is: fail_under in pyproject.toml is
what gates the union, and --rcfile names no number of its own, so a
change to the ratchet leaves both commands above untouched. Run from
the repository root it names the file coverage already finds unasked,
so it changes nothing there; run from anywhere else, coverage report
with no --rcfile reads no configuration at all and still reports, at
no floor, where with the flag it fails outright — Couldn't read 'pyproject.toml' as a config file — rather than passing over a
measurement the tree’s own ratchet never saw. combine carries the
same flag for the same reason, one command short of where the wrong
directory would otherwise first show. Naming the file also means a
configuration file added at the root later, ahead of
pyproject.toml in coverage’s own search order, cannot start
shadowing it for either command
(btclib-org/.github#443, issue #2072).
tests/docs_commands_test.py reads the block above against this job’s
combine and report steps, and the COVERAGE_FILE in front of each
command further up against the env: test.yml’s coverage and
no-bindings jobs set. What it compares there is the value: a workflow
writes the assignment as a yaml mapping where a shell writes it as a
prefix, so that is the half of the spelling the two sites cannot share.
The dist job, which checks the distribution files and then installs one. A
pull request builds its own files there. release.yml’s test job calls this
workflow with use-signed-dist, and the job then checks the files
reusable-build.yml built, signed and uploaded instead, which the publish jobs
download — so what the checks below judge is what an index ends up serving,
byte for byte. The first two commands after the build do not make two checkouts
of one commit agree: uv_build ignores SOURCE_DATE_EPOCH and writes the same
fixed metadata into the wheel and the sdist either way, so those two archives
are already byte for byte the same file across checkouts before either command
runs. What the two buy instead is that the published bytes are this
repository’s own choice rather than the backend’s — the first pins mtime to
the commit date, and the second, normalize_sdist.py, rewrites the sdist’s
member metadata to it — so a rebuild of a released tag reproduces what was
published rather than only reproducing itself. sha256sum after them is the
digest a rebuild from the tag is compared against. The two after that read the
members of the two archives, which the three checks further down do not: an
allowlist of what may be in a wheel and an sdist, and the bill of materials a
release attaches. That allowlist is stated in prose in the package-content
policy, which the suite compares
against the script’s own constants, so changing a rule means changing both.
Both are written before anything is installed: the twine, check-wheel-contents
and pyroma steps below install tools that run third-party code, so the worst
a compromised one can do there is fail the job. The two
smoke tests ask for the wheel and nothing else, so what pulls btclib-secp256k1
in is the Requires-Dist the wheel carries for the secp256k1 extra — which
they name on both sides, a wheel installed without it declaring nothing to
resolve. The first pins the lock as constraints, which bind a version without
requesting a package, so a release of the bindings cannot turn a required check
red while the wheel’s own metadata still does the work; the second is
unconstrained and release-only, asking instead whether the newest published
bindings still satisfy the wheel, which is what a user installing it resolves —
no pull request waits on it, only release.yml’s call sets the input that
turns it on. That the bindings then serve is asserted rather than assumed in
both: with the extra resolving to nothing, or to a release this tree cannot
import, btclib falls back to the Python arithmetic and answers the version and
the signature correctly — the supported configuration no-bindings runs the
whole suite in — so the assertion is the only thing between that resolution and
a green check. They run from an empty directory, or the import finds the source
tree instead of the wheel:
export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
uv build
uv run --no-project --python 3.15 .github/scripts/normalize_sdist.py dist/
sha256sum dist/*
uv run --no-project --python 3.15 \
.github/scripts/verify_dist_contents.py dist/
# generate_sbom.py is btclib-org/.github's, served from main
served=$(mktemp -d)
git clone --depth 1 --filter=blob:none --sparse \
https://github.com/btclib-org/.github "$served"
git -C "$served" sparse-checkout set .github/scripts
uv run --no-project --python 3.15 \
"$served"/.github/scripts/generate_sbom.py dist/ sbom/
uv run --locked --only-group check twine check --strict dist/*
uv run --locked --only-group check check-wheel-contents dist/*.whl
uv run --locked --only-group check pyroma --min 10 dist/*.tar.gz
tmp=$(mktemp -d)
uv export --locked --no-dev --extra secp256k1 --no-emit-project \
--no-hashes -o "$tmp"/constraints.txt
cd "$tmp" && uv venv &&
set -- "$OLDPWD"/dist/*.whl &&
uv pip install --constraints constraints.txt "$1[secp256k1]" &&
.venv/bin/python -c "import btclib; \
from btclib_ecc.curves import is_libsecp256k1_serving; \
from btclib_ecc.ecc import dsa; \
from btclib.key import PrvKeyData; \
print(btclib.__version__); \
assert btclib.__version__ != 'unknown'; \
assert is_libsecp256k1_serving(), \
'the secp256k1 extra resolved, the bindings do not serve'; \
assert dsa.verify(b'btclib', PrvKeyData(1).pub.sec, \
dsa.sign(b'btclib', 1))"
cd "$OLDPWD" &&
uv run --isolated --no-project --with "$(echo dist/*.whl)[secp256k1]" \
python -c "import btclib; \
from btclib_ecc.curves import is_libsecp256k1_serving; \
from btclib_ecc.ecc import dsa; \
from btclib.key import PrvKeyData; \
print(btclib.__version__); \
assert btclib.__version__ != 'unknown'; \
assert is_libsecp256k1_serving(), \
'the secp256k1 extra resolved, the bindings do not serve'; \
assert dsa.verify(b'btclib', PrvKeyData(1).pub.sec, \
dsa.sign(b'btclib', 1))"
A rehearsal’s build, in reusable-build.yml, first runs
.github/actions/dev-version, which rewrites pyproject.toml’s version
with the .dev<run*100+attempt> suffix version-check computed and
re-locks, so that uv build ships a version TestPyPI has not already
seen. None
of the three checks minds — what they judge is metadata syntax, README
rendering, wheel layout and metadata quality, none of which a .dev<N>
suffix changes — and both smoke tests do mind, installing the wheel that
suffix names.
Last, the dist job unpacks the sdist outside the checkout and runs the suite
there, gated at 100% like the coverage job’s run: a shipped test that
reads a file the archive leaves out, or matches a pattern against the
normalized pyproject.toml the archive carries in place of the
committed one, fails there and nowhere else (issue #2252). From the
checkout’s root:
tmp=$(mktemp -d)
tar -xzf dist/*.tar.gz -C "$tmp" --strip-components=1
cd "$tmp"
uv run --locked --no-default-groups --group test pytest
cd "$OLDPWD"
The checks release.yml’s version-check job runs before anything is
built, the first of them being where the tag came from rather than what
it says:
git merge-base --is-ancestor HEAD origin/main
uv lock --check
uv version --short
RELEASING.md’s “Rebuild a release from its tag” has the reproducibility
commands above in the order a verifier runs them, with the
gh attestation verify that gives the rebuild a verdict and the two
bounds on what that verdict means.
The deps-latest workflow, which upgrades every dependency uv resolves before
running the suite, the lint gate and the packaging checks. The upgrade
rewrites uv.lock, and here too git checkout uv.lock restores it; the
commands after it are the ones already listed above, so only the first is
worth repeating:
uv lock --upgrade
That workflow has a second job, suite-bindings-latest, upgrading only
the bindings and re-running the suite — narrower than the upgrade above,
which moves every dependency, so a red run here names the one responsible
instead of burying it behind a dozen others. That the newest release
then serves is asserted before the suite rather than assumed: a release
this tree cannot import leaves btclib on the Python arithmetic, the
suite skipping every bindings-marked test, and the run failing on
coverage — which reads as a dependency this tree has not caught up with
rather than as the answer the job was asked for:
uv lock --upgrade-package btclib-secp256k1
uv run --locked --no-default-groups --group test python -c \
"from btclib_ecc.curves import is_libsecp256k1_serving; \
assert is_libsecp256k1_serving(), \
'the newest btclib_secp256k1 does not import'"
uv run --locked --no-default-groups --group test pytest
The pypi-install workflow, weekly, on demand and as part of a release,
has two jobs, entitled to different answers about what the index
serves.
install-published installs btclib itself from PyPI, nothing checked
out, and asks whether it works rather than whether it installs:
import btclib runs __init__.py alone, and the files under
src/btclib/_data/ are read by path when btclib.network is first
imported, so a wheel missing one would pass import btclib and fail only
here. This install names no extra, so
the bindings are not asked for and the job is not entitled to assert they
serve — the supported no-bindings configuration of issues #990, #991
and #992. The check below is version-independent, a signature whose value
is fixed forever:
python -m pip install btclib
python -c "import btclib; \
from btclib_ecc.ecc import dsa; \
from btclib.key import PrvKeyData; \
assert dsa.verify(b'btclib', PrvKeyData(1).pub.sec, \
dsa.sign(b'btclib', 1))"
install-published-secp256k1 installs btclib[secp256k1] instead,
wheels only for both btclib and btclib-secp256k1, so a wheel the
index does not have for a platform fails the job rather than falling
back to a source build that answers a different question. Naming the
extra is what entitles it to the assertion the other job cannot make: a
resolution that installs and imports but does not serve — issue #1116,
live on main once already — is invisible to a signature check alone,
so this one asks is_libsecp256k1_serving() first, and the signature
after it is meaningful because of that:
python -m pip install --only-binary btclib --only-binary btclib-secp256k1 \
"btclib[secp256k1]"
python -c "import btclib; \
from btclib_ecc.curves import is_libsecp256k1_serving; \
from btclib_ecc.ecc import dsa; \
from btclib.key import PrvKeyData; \
assert is_libsecp256k1_serving(), \
'the secp256k1 extra resolved, the bindings do not serve'; \
assert dsa.verify(b'btclib', PrvKeyData(1).pub.sec, \
dsa.sign(b'btclib', 1))"
The links workflow, weekly and on demand, which is the one that needs a
tool uv does not provide: lychee is a rust binary, and the workflow uses
the action. It gates nothing — a link rots without anybody touching the
repository, so a red merge would be somebody else’s weather — and a
failing run is read in the Actions tab. .lycheeignore holds the URLs a
checker cannot judge, each with the reason.
The mutation workflow, weekly and on demand, which gates nothing either
and for a different reason: it asks whether the suite would notice a
wrong line, where coverage only says the line ran, and a surviving mutant
is a test nobody has written rather than a regression somebody just
caused. One parallel job per profile, each under its own budget, and no
list of them here: .github/mutation/ is the list, and every file in it
states what it mutates and what judges it — the consensus code and the
wire format it reads, the key, codec and script layers, the signature
schemes, and the boundaries where somebody else’s bytes arrive. Where a
session has already weighed survivors, the same header names the scheduled
run and the *-survivors.txt artifact it uploaded, never a restated
count. Those configurations are also what a local run reads, so there is
one statement of what is mutated and what judges it:
uv run --locked --no-default-groups --group test --group mutation \
cosmic-ray baseline .github/mutation/parsers.toml
uv run --locked --no-default-groups --group test --group mutation \
cosmic-ray init .github/mutation/parsers.toml parsers.sqlite
uv run --locked --no-default-groups --group test --group mutation \
cr-filter-operators parsers.sqlite .github/mutation/parsers.toml
uv run --locked --no-default-groups --group test --group mutation \
cosmic-ray exec .github/mutation/parsers.toml parsers.sqlite
uv run --locked --no-default-groups --group test --group mutation \
cr-report --surviving-only --show-diff parsers.sqlite
baseline first, always: it runs the configured test command against the
unmutated tree, and without it a stale path or a renamed test file fails
every mutant identically and the session reports a perfect kill rate,
which is the one failure mode of a mutation run that looks like good news.
cr-filter-operators marks as skipped the mutants a configuration
excludes by operator, and is a no-op for one that excludes none, so the
same five commands run any of them: parsers.toml is the one that
excludes a family, and it states which and why. sig_hash.toml or
engine.toml in place of it is the consensus profile, minutes of cpu
against hours; the parser profile is the one that finishes fastest. Each
configuration carries its own arithmetic — cosmic-ray init enumerates
the mutant count and cosmic-ray baseline reports the cpu cost for
whichever one is run, rather than a figure fixed here that the source can
grow past. The report is --surviving-only, which is the whole of what
anybody acts on: a killed mutant is the suite doing its job, and printing
every one of them buries the handful that are not.
Three things to know before starting one. The session mutates the source
file in place and restores it afterwards, so nothing else may read the
tree while it runs — no second session, no pytest in another shell, and
a git status in the middle is a working tree with a mutant in it.
exec is resumable, running whatever the session still has pending, so
interrupting one costs only the mutant it was on. And the .sqlite
sessions are the artifact the workflow uploads: cr-report, cr-html and
cr-rate all read one, and a downloaded one can be finished locally.
For the counts, read the session with the workflow’s own script —
btclib-org/.github’s mutation_counts.py, checked out beside this tree
as standard/ the way reusable-mutation.yml does — rather than
cr-rate:
uv run --locked --no-default-groups \
python standard/.github/scripts/mutation_counts.py parsers.sqlite
cr-rate’s is_killed is test_outcome != SURVIVED, so a mutant the
operator filter skipped counts as a kill and the rate divides by every
result: on a filtered profile that reads 0.97% where the executed mutants
are 1.32%, and on a session nothing has run yet it reads a perfect 0.00%.
cr-report’s summary line is wrong the same way. The script prints killed,
survived, skipped and never-run beside the rate over what ran, names any
outcome that is no verdict — INCOMPETENT, or a worker that raised — and
exits non-zero for one, which is the only thing the workflow is red about.
Its docstring says why it reads the session file rather than cosmic-ray dump, which cannot read one of these sessions at all.
The integration-bitcoind workflow, which gates and is the exception here: it runs
on every pull request, on every push to main and before a release, and
regtest / Regtest against Bitcoin Core is required on main. It runs
weekly too, and that workflow’s own header says what the weekly run asks
that the others do not: it pins one Core release, so the tree is not the
question a schedule puts to it – whether bitcoincore.org still serves that
release is. It asks the one question the rest of CI cannot, whether Bitcoin Core
accepts what btclib built. It downloads a pinned Core release, verifies
its published sha256, and runs the tests tests/README.md documents with
the binary named rather than found on PATH:
BTCLIB_INTEGRATION=1 BTCLIB_BITCOIND=/path/to/bitcoind \
uv run --locked --no-default-groups --group test \
pytest tests/integration --junitxml=integration.xml
A step after it reads that report and fails the job if a regtest test skipped: pytest exits 0 for a module that skipped itself, so a job whose fixture stopped finding the node would stay green while asking Core nothing.
The documentation, which the docs job of docs.yml runs with this same
command by calling reusable-docs.yml, as read the docs does. -W is what
makes an automodule whose module does not import a failure rather than an
empty page — and what catches invalid reStructuredText in a docstring, which no
hook can: markdownlint does not read .rst, and ruff’s pydocstyle rules check
the form of a docstring rather than whether its body parses. A name ending in
an underscore is the trap to know about, rst reading it as a link reference, so
write it in double backticks. -n is what turns an unresolved cross-reference
— a renamed class in a :class: role, a moved function — into a warning for
-W to fail the build on, rather than a link that resolves to nothing on a
green build; conf.py’s intersphinx_mapping is what it resolves the standard
library against, and its nitpick_ignore carries the entries that reason
cannot reach, each with the reason beside it:
uv run --locked --no-default-groups --group docs \
sphinx-build -n -W -b html docs/source docs/build/html
That job’s second step, reusable-docs.yml’s own now that docs.yml calls it,
reads the pages the first one wrote and must find nothing — the job fails if
this grep exits 0:
grep -rn 'href="#\.\.\?/' docs/build/html --include='*.html'
A link between the root markdown files — ./SECURITY.md in README.md, the
spelling GitHub and PyPI need and the one lychee checks — is
one sphinx cannot resolve on its own, and what MyST emits for a target it
cannot resolve is a warning, before falling back to an anchor on the page it
is already on. docs/source/conf.py resolves those links and suppresses no
myst.xref_missing, so -W fails on the next one that has no target; the
grep asks the same question of the HTML.
The grep is a second defense rather than the only one: conf.py’s
suppress_warnings staying empty is a configuration it does not depend on,
and a myst-parser upgrade could stop warning unconditionally the way the
pinned one does.
Unlike three other repositories of the organization,
.pre-commit-config.yaml’s local-link-prefix hook here accepts a
../-prefixed destination as well as a ./-prefixed one, so a link may
still begin either way. The grep above matches #../ as well as #./,
the way this tree’s own step did before it moved to reusable-docs.yml:
a broken ../ link already fails -W on its own, through
docs/source/conf.py’s RootFileLinks (issue #1562), and the grep is
the artifact-level backstop for that case too, reusable-docs.yml
itself having widened to catch it (issue btclib-org/.github#35).
A link into a heading of another root file carries a fragment spelled the
way GitHub derives it from the heading text, as in
the gates are the evidence,
and docs/source/conf.py’s myst_heading_anchors is what makes that
spelling resolve here too.
Both steps ask whether a page renders and whether its links resolve, and
nothing more. Whether the worked examples on it are still true is a
different question, asked by tests/docs_examples_test.py: any page under
docs/source/ carrying a >>> prompt is run as a doctest by the suite, so
an example is edited by running uv run pytest tests/docs_examples_test.py
and pasting back what the library answered — never by writing what it ought
to answer. Keep them deterministic, which for ssa.sign means passing
aux; and keep every key on
those pages a published test vector, so that a reader who copies one copies
something already known to the world.
The only check with no local equivalent is codeql: the analysis needs the
CodeQL bundle and a database, which is a download rather than a uv
command, so .github/workflows/codeql.yml is where it is configured and
GitHub’s runners are where it happens. It runs on every pull request now,
alongside push to main and its weekly schedule, so its findings reach
a branch before a merge rather than only after one; REPOSITORY.md has
why it is still not a required check.
Breaking a caller is not an argument¶
A refactoring that is reasonable gets made, however much it breaks. Do not weigh “this renames something callers use” against it, and do not propose keeping an inconsistency and gating it going forward instead: consistency across a family — one naming shape, one contract per prefix, one way to read the same kind of answer — is itself the reason, and a release is where a break is reported rather than a reason not to make it.
So a census that finds thirty names of one shape and six of another has
found six to change, not a trade-off to price. What the measurement is
for is knowing the blast radius and writing RELEASE_NOTES.md’s
breaking-changes entry, which is how a user is told: read
RELEASE_NOTES.md for what that entry looks like,
and note that v2026.8.7’s list is long on purpose.
The one thing this does not license is a break nobody can act on. An entry says the old spelling, the new one, and what a caller does about it; a rename with no note is the defect, not the rename.
There is no deprecation window. Versions are calendar-based
(YYYY.M.D), a checkout of main is a release cycle in progress with no
promise attached to any name in it, and a source-breaking change is
announced by its RELEASE_NOTES.md breaking-changes
entry and by nothing earlier — a warning-then-removal mechanism, a
deprecated alias kept alongside its replacement, a DeprecationWarning,
is deliberately not provided. What the promise covers where it applies at
all is The public surface below: __all__, at every depth.
A security fix is exempt from all of the above. It may break anything this section otherwise asks for, on the same terms as any other reasonable refactoring; SECURITY.md is where a reader goes for what is covered and how a fix reaches a release.
The public surface¶
Every module and every package declares __all__, at every depth of
the tree. A name is public here because a list says so, not because it
happens to lack a leading underscore: import * and the sphinx pages then
stop depending on an import section, and a helper cannot grow into a name
callers depend on without somebody editing that list. An empty list is a
legitimate answer, for a module with nothing public of its own; declaring
nothing is not.
For a package the list is what the __init__ re-exports, submodules
included: a module named there is a module a caller — and the command line
of btclib-org/.github#1235, which reads the command tree off __all__ —
can descend into. For a module it is what the module itself defines: a
name it imported belongs to the module that defines it, and
btclib.alias.Octets is the spelling of Octets.
btclib.__all__ is the root of that tree: every top-level module and
package, imported on demand by the module __getattr__ in
src/btclib/__init__.py. A module added to src/btclib/ is added there
too, and the suite says so — the list is written out rather than discovered,
because a discovered one would publish a new module without anybody
deciding to.
The two halves are independent. A module declares its own surface whether
or not its parent publishes an edge to it, which is how
btclib.p2p.limits says what it offers without becoming
anybody’s API: the module states the offer, the
parent decides whether it is reachable.
A public name kept out of the list is a decision, and the docstring
says why. The datadir of btclib.network and btclib’s own
name are two such names, and UNEXPORTED in tests/all_test.py records
each; a name kept out stays
importable from the module that defines it, which is where the test suite
takes it from. tests/all_test.py checks all of this and finds the
modules rather than listing them, so a new public name fails the suite
until it is exported or recorded in that file’s UNEXPORTED table.
Every public function validates its inputs. Whatever it is handed — a
string, octets, or an object somebody built earlier — a name a caller can
reach checks it before acting on it, and a malformed argument leaves as a
BTClibTypeError or a BTClibValueError, which is what the callers of
this library are written to catch. A function of btclib’s own handing an
argument on to btclib_ecc leaves with that package’s
BTClibEccTypeError or BTClibEccValueError instead: each derives from
the same built-in, and neither from BTClibException (issue #2282). The
work itself may be deferred to a private twin that does not validate; that
twin is then what the library composes internally, where the inputs have
already been checked and checking them a second time buys nothing.
A function that answers a bool is total over the types it declares,
and only those. is_p2sh answers False for bytes that are not a p2sh
script, and dsa.verify answers False for a signature that does not
verify: that is what the bool is for, and a caller filtering a list of
them wants an answer and not an exception. A value of a type the
signature does not declare is not such an answer — it is the caller’s
own mistake, it is a call mypy already refuses, and it leaves as a
BTClibTypeError like any other. So dsa.verify(msg, 12, sig) raises,
12 being a private key in this library and never a public one, where a
well-formed public key that simply did not sign is False.
bms.verify carves one case out of that, as btclib_ecc.ecc.musig2
and btclib_ecc.ecc.frost do: a value of a declared type whose size or
encoding makes it impossible to read as a signature, a key, an address, a
digest or an opening raises rather than answering False. The function is
not saying the signature is forged, it is saying it has no way to find out
(issue #2170). So btclib_ecc.ecc.dsa.verify(msg, "not a key", sig) raises
BTClibEccValueError, and bms.verify(msg, "not an address", sig)
raises BTClibValueError, while a signature that is well formed and
simply does not verify is False.
What decides is whether the parameter declares a size, which is what
makes the carve-out one rule rather than a judgement per call site. A
message of any length is a message, so nothing about it can be the wrong
length and the rule never fires: ssa.verify_ answers False for a short
one, an ordinary message that was not signed. A digest is of its hash
function’s size and a BIP374 message is of 32 octets, so dsa.verify_
and dleq.verify_proof refuse any other length — 31 octets are no
SHA-256 digest, and the equation has nothing to be about. BIP374’s own
reference reads the same way: dleq_verify_proof asserts
len(proof) == 64, and the dleq_challenge it reaches at its last step
asserts len(m) == 32, while s >= GE.ORDER and a challenge that does
not match are its False.
btclib_ecc.ecc.dsa’s malformed DER encoding is the one thing on the other
side of the line, and it is there because a measurement put it there rather than
because the rule reaches it: its own wycheproof vectors ask that question
over profiles built for it, and answer False.
The line is the annotation, and deliberately not which built-in a helper
happens to derive from: those two coincide only by accident, which is
what issue #745 found and this replaces. The assert_* twin beside each
of these is the spelling that says why a value was refused, and the two
are how a caller chooses between an answer and a reason.
Both halves are gated, and both hold. tests/input_validation_test.py
drives the two rules over every public function whose required parameters
are all library input types. tests/bool_contract_test.py drives them
from hand-written fixtures over the verifications that walk cannot reach —
a signature verification needs a valid message, key and signature, and a
Sig | Octets no vocabulary of wrong values can build.
tests/built_object_contract_test.py does the same for a function whose
parameter is an object the caller already built — a CmpctBlock handed to
p2p.compact_blocks.reconstruct — which is the family
check_validity=False makes reachable, an invalid object being something
a caller may legitimately hold. tests/curve_parameter_test.py drives the
same rules over a parameter that carries a default: reaching one means
every argument in front of it has to be valid, which is the table of valid
values the automatic walk exists to do without, so ec is driven from a
table there as hf and network are driven where their own checks live.
None of these test files has an exemption list, which is the state to
keep: a finding is a red test, to be fixed or to be given a reason of its
own beside the two families that have one.
A bool parameter is a kind or a truth, and only the first is
type-checked. A flag that decides what is computed refuses a non-bool,
musig2._flag stating the reason: a kind written down and read back —
json, a configuration file, a coordinator’s message — arrives as whatever
it was written as, and "false" is true, so
btclib_wallet.wallet.KeyGroup(verify=) would compute every address of
the other script rather than raise. A flag that decides only whether a
check runs is read for its truth: check_validity either runs a check or
skips one, and never changes an answer. tests/check_validity_test.py
owns that convention.
A truth’s True has to be its conservative value, which is the second
half of the same rule and the one issue #884 asked for. Every wrong value
is true, so the misreading is never “the flag was off”: it is always the
one the flag’s True stands for. That is what makes a truth safe —
verify_checksum="no" checks the checksum, strict="no" is strict,
forbid_zero_size="no" forbids — and it is a fact about the name, not
about the check. So a flag whose True is the permissive value is a kind
however little it computes, because the misreading waives the very refusal
it was written to make: verified suppressing the script engine’s NULLFAIL
is the sharp one, with allow_partial accepting an input nobody signed and
hybrid parsing the prefixes it was written to keep out. verify_script’s
final beside verified is the contrast that shows the rule is about
direction and not about the engine: it demands a true stack, so a non-bool
there fails a script rather than passing one.
Which of the two a flag is, is written down for every one of them.
tests/bool_parameter_test.py is the census: two tables, a kind driven
until it refuses "no", 0 and 1, a truth driven until it accepts all
three, and a walk that fails on a bool parameter in neither table. So a
flag added anywhere is a decision somebody makes rather than one the
default makes for them — and 0 and 1 are in that vocabulary because
bool is a subclass of int, which is what leaves isinstance(value, int) no check at all here. utils.assert_type is the refusal, and
check_validity is the one name subtracted by the walk, its own file
holding it.
Four shapes were what the second gate found when it was written, and they
are worth knowing because each is a way of being handed an argument
without looking at it: a sequence parameter walked before it is
checked, so a None is “not iterable” from underneath the library; a
signature reaching Sig.b64decode, which strips before it decodes; an
argument handed straight to the bindings, whose own message names a C
parameter; and a reduction outside the try, which refuses a message
verify_ would answer False about.
Which of those a function is, its name says, and the vocabulary is
closed: a public function that answers a bool carries one of the four
prefixes below, or is one of the English predicates
tests/name_contract_test.py names — BasicBlockFilter.match among them,
where the name is the standard’s and is_ would cost the reading. A bool
that is neither fails that gate, so a new one is a prefix or a decision
somebody wrote down.
A member that takes nothing but self is a @property, whatever it
answers, and that is gated: it is read off the object rather than called,
so tx.is_segwit and not tx.is_segwit(), tx.vsize and not
tx.vsize(). functools.cached_property is one of these — a read
that parses once, as Script.asm does.
What is not a read is exempt by shape rather than by a list of names:
assert_* refuses, and a property that refuses is a trap, since reading
obj.assert_valid evaluates the method and throws it away; get_* talks
to a node or an explorer, so the prefix is the warning a property would
hide; to_* hands back a new object; and close is an action. The one
exemption named individually is alias.HashObject’s digest,
hexdigest and copy, because hashlib calls them that way — and hashlib
draws the same line itself, block_size and digest_size being
attributes there as they are properties here.
The bool half of the same rule: Thirty were properties and six were methods,
which made the six the exception; they are properties now. It also makes
the forgotten-parentheses bug unsayable — if tx.is_segwit: on a method
is a bound method, and every bound method is true. mypy’s
truthy-function names that, and mypy is a gate here, but a shape that
cannot go wrong beats a checker that catches it going wrong.
What the four prefixes buy is a promise read off the name:
assert_*refuses and returnsNone, and is the reason half of the pair above.is_*answers aboolabout a value, and is total over the declared types:is_p2shis False for bytes that are not a p2sh script.verify*answers aboolabout a signature or a proof, on the same terms.assert_as_validbeside it says why.check_*answers abooland refuses what cannot be an answer – the one prefix that warns a caller it still needs anexcept:script.engine.script.check_pub_key, where a wrong length is False but a hybrid prefix under STRICTENC is the offence itself, which is how Core’sCheckPubKeyEncodingsplits it — andscript.taproot.check_output_pubkey, where a malformed control block is no proof rather than a disproof.
Nine functions carried check_ while refusing and returning None, and
two more while returning bytes and a pair of bools, which left the prefix
saying four things and therefore nothing (issue #814). A converter is
named for what it returns – bytes_from_octets,
b32.bytes_from_witness_program – and a query for what it answers.
check_validity is a parameter and not one of those four, which is
worth saying now that the prefix means something: there check is a verb
governing an object — “check the validity” — where a prefix on a function
name classifies what the call answers, and a caller passes this one
rather than reading a result off it. What it gates is assert_valid, a
refusal, so the vocabulary above would name it after that if it named it
at all. It keeps the name it has: validate says no more, and the
alternative is renaming a keyword across every public signature that
carries it, for a reading nobody has to make twice.
check_validity=False is not an exemption from this. It says “do not
check now”, not “this object is exempt from here on”: these are mutable
dataclasses whose fields are public and get reassigned in place, so
validity at construction is not validity at use. Passing it is supported —
tests/check_validity_test.py exercises the flag across the library — so
a public function that takes an already-built object and asks it nothing
is one a caller can reach with an object the library itself would refuse.
btclib_wallet.bip32 is the shape to copy. derive validates and calls
_derive, which does not, and _key_data_from_bip32_key is the one place
a BIP32Key of any spelling becomes a validated BIP32KeyData, every
public wrapper going through it. btclib_wallet.descriptors then composes
the twins directly — _xpub_from_xprv(_derive(_key_data_from_bip32_key(xprv), prefix, None)) — paying one validation instead of three, and no base58
round trip between them. The leading underscore those three carry means
what it means anywhere in Python: private. __all__ is what decides
publicity, so it is a reader’s hint rather than the rule, but an
unvalidating twin belongs on the private side of that list, and calling one
asserts that its caller validated.
A private function takes no default argument either, which is the same
absence of an outside caller read the other way: a default is written for
one the author cannot see, and there is none here. So the value the call is
made with is at the call site, where it is read — _derive’s None above
is the version it forces on the derived key, and _deserialize_scalar’s
strict decides whether BIP66’s minimal encoding is enforced at all. A
flag added to a private function is then a question asked again at each of
its call sites, instead of answered for the ones nobody revisited.
tests/private_defaults_test.py is the gate, and its docstring carries
what the rule does not reach.
A trailing underscore is the other convention, and it is public. It
marks the spelling whose input the caller has already prepared: dsa.sign_
takes the hash sign would compute, musig2.nonce_gen_ the randomness
nonce_gen draws, musig2.partial_sig_verify_ the session context
partial_sig_verify aggregates. Both names of the pair are in __all__,
so it is an offer to skip work the caller has already paid for, not a way
past the validation above: sign_ checks its msg_hash as sign checks
its msg. The two conventions are independent, and
btclib_ecc.ecc.dsa._assert_as_valid_ carries both.
Publishing both names is what makes their agreement a promise rather than
a tidiness. A keyword added to verify is added to verify_ or to
neither, with the same default in both: a flag that changes the answer on
one and is missing or defaulted differently on the other makes the pair
two functions instead of two spellings, and a caller who reduced the
message first quietly gets a different rule. The prepared argument is the
one difference the pair exists for.
Prepared is not unchecked, and a bare int the trailing underscore takes
instead of an Integer still refuses a bool: utils.is_integer is the
policy int_from_integer carries for every coerced parameter, and the
underscore spelling asks it of what it is handed already prepared rather
than skipping it – ssa.challenge_’s x_Q and x_K, and
dsa.recover_pub_key_’s key_id, both pinned in
tests/integer_policy_test.py’s census (issue #1248).
The exception is a boundary at which nothing can be invalid, and it is
named rather than passed over: where a class’s invariants are exactly the
widths of its fields, the decoding enforces them by construction and the
check is unreachable by design rather than missing. src/btclib/utils.py’s
docstring is where that rule is written down, OutPoint and TxIn being
the classes it holds for.
A _var suffix means the operand decides the work¶
A function whose duration follows the value it is given ends in _var,
and the plain name beside it is the one a secret may be handed. It is
libsecp256k1’s convention and it is used there for the same two halves:
secp256k1_scalar_inverse_var sits beside secp256k1_scalar_inverse,
secp256k1_gej_add_var beside secp256k1_gej_add_ge. The point of it is
that the question is answered at the call site rather than in the
docstring of the thing being called. A leading underscore changes none of
this: the suffix states the implementation’s timing, for a private
function as for a public one.
What “the work” is depends on the function, and each of these was
measured rather than assumed — on secp256k1’s order or its p, over
random operands of 256 bits down to 64, the ratio between the two ends:
the count of point operations, for a scalar multiplication:
curve_group._mult_jac_varandcurve_group_2._double_mult_w_NAF_varagainst_mult_regular_window,_mult_fixed_baseand_double_mult_regular_window, which make the same number for every scalar of the curve.btclib_ecc.curves.double_mult_varandbtclib_ecc.curves.multi_mult_varcarry it for that reason andbtclib_ecc.curves.multdoes notthe iteration count of an extended Euclid:
mod_inv_varat 4.21x,xgcd_varat 4.84x,mod_inv_batch_varat 2.01xthe length of a loop over the operand:
legendre_symbol_varat 4.26x, andtonelli_var, whose 1.54x is spread within one operand size rather than across sizes — Tonelli-Shanks iterates on the value, not on its length.mod_sqrt_varreaches that loop only for apof 1 mod 4: on a 3 mod 4 one it is a single exponentiation with a fixed exponent and measures 1.01x, and the catalogue holds both kinds —secp224k1,secp224r1andnistp224are 1 mod 4 — so the caller picks which it gets and the suffix states the worse caseone modular inverse, in the layer above:
aff_from_jac_varat 2.96x,x_aff_from_jac_varat 1.93x,aff_from_jac_batch_varat 1.67x andy_aff_from_jac_varat 1.42x follow theZthey are handed;double_aff_varat 1.32x,add_aff_varat 1.23x and theadd_varover them at 1.22x follow the coordinates.y_varand its three tiebreakers follow theirxon the 1 mod 4 curves above, 1.84x onsecp224k1, where on secp256k1 they are flat
The suffix is measured, never inherited, and that is what stops it.
Every function that reaches one of these through some chain of calls
would otherwise take it: most of the library’s own call graph, public
functions included, right down to hex_string, ripemd160 and
p2pkh — a suffix on the library, saying nothing. Two things break
the chain, and both are facts about the caller rather than about the
callee. Blinding is one: mult calls
aff_from_jac_var and is 0.99x in its scalar, because _blinded_jac
randomized the Z that inverse is timed on. A public operand is the
other: point_from_octets measures 1.05x, its y_even_var being one
fixed exponent on secp256k1 and its _is_x_coordinate_var reaching the
bindings. So the question to ask of a new name is not what it calls, but
what its own clock does with what it was handed — and the answer is a
measurement.
The census is at a measured zero, and these are the exceptions it
leaves. Every function that spends one of these primitives has been
timed over random inputs of one class, against add_jac as the floor —
1.05x, the group law having nothing to branch on. What measures at that
floor keeps its plain name, and each is here so that the next reader does
not have to re-derive it: dsa._assert_as_valid_ 1.07x and
dsa._recover_pub_key_ 1.05x, ssa._assert_as_valid_ 1.07x and
ssa._recover_pub_key_ 1.06x, ellswift._try_sqrt 1.05x,
sec_point.point_from_octets 1.01x, ssa.point_from_bip340pub_key
1.03x, Sig.assert_valid 1.03x and ellswift.xdh 1.01x. The inverse
or the root inside each of them is real, and it is diluted: an inverse
varying by 1.32x is 10 us of a 40 us verification, and xdh is
dominated by a mult that is blinded. ellswift._constants is
memoized on the curve and receives no value at all.
The one number worth keeping in view is ellswift._xswiftec_inv_var, at
29.62x — 28 calls of 50 returning under 10 us and 22 over. That is an
early return and not a slower sum, which is what a data-dependent branch
looks like on a clock, and it is why the SwiftEC map carries the suffix
where the ECDH built on it does not.
Three tiers of duration follow, of which only the first is constant-time:
constant-time is libsecp256k1’s alone. Every claim of it in this library is a claim about the C behind the bindings, which is why
SECURITY.mdstates argument by argument when a call crosses that boundary. Nothing written in Python is constant-time, and no name here should be read as promising it.regular, or blinded: the operand does not decide the duration, and something else does.
_mult_regular_window,_mult_fixed_baseand_double_mult_regular_windowmake the same operations for every scalar of the curve;_blinded_jacleaves the intermediate values a function of a random factor rather than of the scalar alone; andmod_invleaves an extended Euclid’s iteration count following a random factor rather than the caller’s operand, at 1.02x wheremod_inv_varis 2.06x over the same range. This is decorrelation and not uniformity — the duration still varies, and what changes is what it varies with.variable: the operand decides. Everything suffixed above, and every reduction whose cost is the size of what it reduces.
So a secret takes the plain name and a public value takes the _var
one, which is the opposite of an optimization default and is meant to
be: the fail-safe direction is that forgetting to choose gives the slower
and safer call. dsa’s signing inverts the nonce with mod_inv while
its r, its s and a verification’s Z take mod_inv_var, and those
four sit within twenty lines of each other; a reviewer should be able to
tell which is which without tracing where each value came from.
Add the suffix when adding such a function, having measured that it earns one, and do not add it to the recodings, the tables or the group law: those take no operand whose size varies, and a suffix on everything says nothing about anything.
What none of this does is make the Python path safe, and SECURITY.md’s
limitations section is what says so at length: a table is still indexed
by a secret digit, and that is out of reach from bytecode. A name in the
second tier closes one measured channel, and the honest form of the claim
is that one — which channel, measured how, and what is left.
Prose in this tree¶
Section 9 of the standard is the whole of the prose style, and Documentation and comments above says why it is not restated. What follows is what this tree adds to it, and it holds for docstrings, comments, the sphinx pages and a pull request reply alike.
Most readers of a docstring here are new to btclib: write for them.
A table of measured timings belongs in the CHANGELOG, not in a docstring. Measure, don’t assert wants the command beside the number, and a timing is the one figure that has none here: the benchmarks are their own repository, and no gate re-measures — a timing gate on a shared runner is a flake. What is left to do instead is put it where it is read as what it is. A docstring is read as a statement about the code as it stands, so a figure in one is a claim about now; a CHANGELOG entry is read as the history of a release, and calendar versioning puts the release day in the heading over it once the release is cut. So the docstring keeps the number that carries the reason, the one a reader needs to follow the decision — “half of what a signature costs”, “twice as fast, flat in n, no crossover” — and the matrix per size or per caller goes in the entry that took it. Two figures were found stale in one week by the branch standing on them, and neither would have been noticed otherwise (issue #940).
Every change starts with an issue¶
Open one before the pull request, however small the change is. The standard’s What a pull request says it is has what the title and the description then do with it.
A security vulnerability is the exception and does not go in the tracker at all. SECURITY.md is what to follow instead: it asks for a private advisory, or for an email if you would rather not open one.
What a squash writes here¶
What the landing commit says is a repository setting and not a decision
made once per pull request, so there is one place to read it from, and
it is REPOSITORY.md’s Merge methods: the two squash_merge_commit_*
settings are recorded there, beside the call that answers them.
A branch of one commit therefore lands under that commit’s own subject, a branch of several under the pull request’s title with its number, and the body is the branch’s commit messages either way — never the pull request’s body, which stays on the pull request. A squash made by hand follows the same convention, the setting being the statement of what the message should say rather than only of what a press produces. REPOSITORY.md has what the other two merge methods would have cost.