Settings¶
Hypothesis tries to have good defaults for its behaviour, but sometimes that’s not enough and you need to tweak it.
The mechanism for doing this is the settings
object.
You can set up a @given
based test to use this using a settings
decorator:
@given
invocation is as follows:
from hypothesis import given, settings
@given(integers())
@settings(max_examples=500)
def test_this_thoroughly(x):
pass
This uses a settings
object which causes the test to receive a much larger
set of examples than normal.
This may be applied either before or after the given and the results are the same. The following is exactly equivalent:
from hypothesis import given, settings
@settings(max_examples=500)
@given(integers())
def test_this_thoroughly(x):
pass
Available settings¶
-
class
hypothesis.
settings
(parent=None, **kwargs)¶ A settings object controls a variety of parameters that are used in falsification. These may control both the falsification strategy and the details of the data that is generated.
Default values are picked up from the settings.default object and changes made there will be picked up in newly created settings.
-
database
¶ An instance of hypothesis.database.ExampleDatabase that will be used to save examples to and load previous examples from. May be
None
in which case no storage will be used,":memory:"
for an in-memory database, or any path for a directory-based example database.default value:
(dynamically calculated)
-
deadline
¶ If set, a duration (as timedelta, or integer or float number of milliseconds) that each individual example (i.e. each time your test function is called, not the whole decorated test) within a test is not allowed to exceed. Tests which take longer than that may be converted into errors (but will not necessarily be if close to the deadline, to allow some variability in test run time).
Set this to None to disable this behaviour entirely.
default value:
timedelta(milliseconds=200)
-
derandomize
¶ If this is True then hypothesis will run in deterministic mode where each falsification uses a random number generator that is seeded based on the hypothesis to falsify, which will be consistent across multiple runs. This has the advantage that it will eliminate any randomness from your tests, which may be preferable for some situations. It does have the disadvantage of making your tests less likely to find novel breakages.
default value:
False
-
max_examples
¶ Once this many satisfying examples have been considered without finding any counter-example, falsification will terminate.
The default value is chosen to suit a workflow where the test will be part of a suite that is regularly executed locally or on a CI server, balancing total running time against the chance of missing a bug.
If you are writing one-off tests, running tens of thousands of examples is quite reasonable as Hypothesis may miss uncommon bugs with default settings. For very complex code, we have observed Hypothesis finding novel bugs after several million examples while testing SymPy.
default value:
100
-
phases
¶ Control which phases should be run. See the full documentation for more details
default value:
(Phase.explicit, Phase.reuse, Phase.generate, Phase.target, Phase.shrink)
-
print_blob
¶ If set to True, Hypothesis will print code for failing examples that can be used with
@reproduce_failure
to reproduce the failing example.default value:
False
-
report_multiple_bugs
¶ Because Hypothesis runs the test many times, it can sometimes find multiple bugs in a single run. Reporting all of them at once is usually very useful, but replacing the exceptions can occasionally clash with debuggers. If disabled, only the exception with the smallest minimal example is raised.
default value:
True
-
stateful_step_count
¶ Number of steps to run a stateful program for before giving up on it breaking.
default value:
50
-
suppress_health_check
¶ A list of
HealthCheck
items to disable.default value:
()
-
timeout
¶ The timeout setting has been deprecated and no longer does anything.
default value:
not_set
The timeout setting can safely be removed with no effect.
-
verbosity
¶ Control the verbosity level of Hypothesis messages
default value:
Verbosity.normal
-
Controlling What Runs¶
Hypothesis divides tests into four logically distinct phases:
- Running explicit examples provided with the @example decorator.
- Rerunning a selection of previously failing examples to reproduce a previously seen error
- Generating new examples.
- Mutating examples for targeted property-based testing.
- Attempting to shrink an example found in previous phases (other than phase 1 - explicit examples cannot be shrunk). This turns potentially large and complicated examples which may be hard to read into smaller and simpler ones.
The phases setting provides you with fine grained control over which of these run,
with each phase corresponding to a value on the Phase
enum:
-
class
hypothesis.
Phase
¶
Phase.explicit
controls whether explicit examples are run.Phase.reuse
controls whether previous examples will be reused.Phase.generate
controls whether new examples will be generated.Phase.target
controls whether examples will be mutated for targeting.Phase.shrink
controls whether examples will be shrunk.
The phases argument accepts a collection with any subset of these. e.g.
settings(phases=[Phase.generate, Phase.shrink])
will generate new examples
and shrink them, but will not run explicit examples or reuse previous failures,
while settings(phases=[Phase.explicit])
will only run the explicit
examples.
Seeing intermediate result¶
To see what’s going on while Hypothesis runs your tests, you can turn up the verbosity setting.
>>> from hypothesis import find, settings, Verbosity
>>> from hypothesis.strategies import lists, integers
>>> @given(lists(integers())
... @settings(verbosity=Verbosity.verbose))
... def f(x): assert not any(x)
... f()
Trying example: []
Falsifying example: [-1198601713, -67, 116, -29578]
Shrunk example to [-1198601713]
Shrunk example to [-1198601600]
Shrunk example to [-1191228800]
Shrunk example to [-8421504]
Shrunk example to [-32896]
Shrunk example to [-128]
Shrunk example to [64]
Shrunk example to [32]
Shrunk example to [16]
Shrunk example to [8]
Shrunk example to [4]
Shrunk example to [3]
Shrunk example to [2]
Shrunk example to [1]
[1]
The four levels are quiet, normal, verbose and debug. normal is the default, while in quiet mode Hypothesis will not print anything out, not even the final falsifying example. debug is basically verbose but a bit more so. You probably don’t want it.
If you are using pytest, you may also need to disable output capturing for passing tests.
Building settings objects¶
Settings can be created by calling settings
with any of the available settings
values. Any absent ones will be set to defaults:
>>> from hypothesis import settings
>>> settings().max_examples
100
>>> settings(max_examples=10).max_examples
10
You can also pass a ‘parent’ settings object as the first argument, and any settings you do not specify as keyword arguments will be copied from the parent settings:
>>> parent = settings(max_examples=10)
>>> child = settings(parent, deadline=None)
>>> parent.max_examples == child.max_examples == 10
True
>>> parent.deadline
200
>>> child.deadline is None
True
Default settings¶
At any given point in your program there is a current default settings,
available as settings.default
. As well as being a settings object in its own
right, all newly created settings objects which are not explicitly based off
another settings are based off the default, so will inherit any values that are
not explicitly set from it.
You can change the defaults by using profiles.
settings Profiles¶
Depending on your environment you may want different default settings. For example: during development you may want to lower the number of examples to speed up the tests. However, in a CI environment you may want more examples so you are more likely to find bugs.
Hypothesis allows you to define different settings profiles. These profiles can be loaded at any time.
-
class
hypothesis.
settings
(parent=None, **kwargs) A settings object controls a variety of parameters that are used in falsification. These may control both the falsification strategy and the details of the data that is generated.
Default values are picked up from the settings.default object and changes made there will be picked up in newly created settings.
-
static
load_profile
(name)[source]¶ Loads in the settings defined in the profile provided.
If the profile does not exist, InvalidArgument will be raised. Any setting not defined in the profile will be the library defined default for that setting.
-
static
register_profile
(name, parent=None, **kwargs)[source]¶ Registers a collection of values to be used as a settings profile.
Settings profiles can be loaded by name - for example, you might create a ‘fast’ profile which runs fewer examples, keep the ‘default’ profile, and create a ‘ci’ profile that increases the number of examples and uses a different database to store failures.
The arguments to this method are exactly as for
settings
: optionalparent
settings, and keyword arguments for each setting that will be set differently to parent (or settings.default, if parent is None).
-
static
Loading a profile changes the default settings but will not change the behavior of tests that explicitly change the settings.
>>> from hypothesis import settings
>>> settings.register_profile("ci", max_examples=1000)
>>> settings().max_examples
100
>>> settings.load_profile("ci")
>>> settings().max_examples
1000
Instead of loading the profile and overriding the defaults you can retrieve profiles for specific tests.
>>> settings.get_profile("ci").max_examples
1000
Optionally, you may define the environment variable to load a profile for you. This is the suggested pattern for running your tests on CI. The code below should run in a conftest.py or any setup/initialization section of your test suite. If this variable is not defined the Hypothesis defined defaults will be loaded.
>>> import os
>>> from hypothesis import settings, Verbosity
>>> settings.register_profile("ci", max_examples=1000)
>>> settings.register_profile("dev", max_examples=10)
>>> settings.register_profile("debug", max_examples=10, verbosity=Verbosity.verbose)
>>> settings.load_profile(os.getenv(u'HYPOTHESIS_PROFILE', 'default'))
If you are using the hypothesis pytest plugin and your profiles are registered
by your conftest you can load one with the command line option --hypothesis-profile
.
$ pytest tests --hypothesis-profile <profile-name>