pytest2/py/doc/_build/html/test-features.html

371 lines
25 KiB
HTML

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>1.2. Features &mdash; py lib v1.0.0b1 documentation</title>
<link rel="stylesheet" href="_static/default.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '',
VERSION: '1.0.0b1',
COLLAPSE_MODINDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<link rel="top" title="py lib v1.0.0b1 documentation" href="index.html" />
<link rel="up" title="1. py.test" href="test.html" />
<link rel="next" title="1.3. Included plugins" href="test-plugins.html" />
<link rel="prev" title="1.1. Quickstart" href="test-quickstart.html" />
</head>
<body>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="test-plugins.html" title="1.3. Included plugins"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="test-quickstart.html" title="1.1. Quickstart"
accesskey="P">previous</a> |</li>
<li><a href="index.html">py lib v1.0.0b1 documentation</a> &raquo;</li>
<li><a href="test.html" accesskey="U">1. py.test</a> &raquo;</li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body">
<div class="section" id="features">
<h1>1.2. Features<a class="headerlink" href="#features" title="Permalink to this headline"></a></h1>
<p>py.test is a standalone-tool that collects and runs tests for
your Python application and modules. py.test works across
linux, windows and osx and on Python 2.3 - Python 2.6.</p>
<p>It aims to support <em>unit-tests</em> and <em>functional tests</em> written
in Python and is used in projects that run more than 10000
tests regularly.</p>
<p>py.test presents a clean and powerful command line interface
and strives to generally make testing a fun effort.</p>
<div class="section" id="automatically-collects-and-executes-tests">
<h2>1.2.1. automatically collects and executes tests<a class="headerlink" href="#automatically-collects-and-executes-tests" title="Permalink to this headline"></a></h2>
<p>py.test discovers tests automatically by inspect specified
directories or files. By default, it collects all python
modules a leading <tt class="docutils literal"><span class="pre">test_</span></tt> or trailing <tt class="docutils literal"><span class="pre">_test</span></tt> filename.
From each test module every function with a leading <tt class="docutils literal"><span class="pre">test_</span></tt>
or class with a leading <tt class="docutils literal"><span class="pre">Test</span></tt> name is collected.</p>
</div>
<div class="section" id="load-balance-tests-to-multiple-cpus">
<h2>1.2.2. load-balance tests to multiple CPUs<a class="headerlink" href="#load-balance-tests-to-multiple-cpus" title="Permalink to this headline"></a></h2>
<p>For large test suites you can distribute your
tests to multiple CPUs by issuing for example:</p>
<div class="highlight-python"><pre>py.test -n 3</pre>
</div>
<p>Read more on <a class="reference external" href="test-dist.html">distributed testing</a>.</p>
</div>
<div class="section" id="distribute-tests-across-machines">
<h2>1.2.3. Distribute tests across machines<a class="headerlink" href="#distribute-tests-across-machines" title="Permalink to this headline"></a></h2>
<p>py.test supports the sending of tests to
remote ssh-accounts or socket servers.
It can <cite>ad-hoc run your test on multiple
platforms one a single test run</cite>. Ad-hoc
means that there are <strong>no installation
requirements whatsoever</strong> on the remote side.</p>
</div>
<div class="section" id="extensive-debugging-support">
<h2>1.2.4. extensive debugging support<a class="headerlink" href="#extensive-debugging-support" title="Permalink to this headline"></a></h2>
<div class="section" id="testing-starts-immediately">
<h3>1.2.4.1. testing starts immediately<a class="headerlink" href="#testing-starts-immediately" title="Permalink to this headline"></a></h3>
<p>Testing starts as soon as the first <tt class="docutils literal"><span class="pre">test</span> <span class="pre">item</span></tt>
is collected. The collection process is iterative
and does not need to complete before your first
test items are executed.</p>
</div>
<div class="section" id="support-for-modules-containing-tests">
<h3>1.2.4.2. support for modules containing tests<a class="headerlink" href="#support-for-modules-containing-tests" title="Permalink to this headline"></a></h3>
<p>As <tt class="docutils literal"><span class="pre">py.test</span></tt> operates as a separate cmdline
tool you can easily have a command line utility and
some tests in the same file.</p>
</div>
<div class="section" id="debug-with-the-print-statement">
<h3>1.2.4.3. debug with the <tt class="docutils literal"><span class="pre">print</span></tt> statement<a class="headerlink" href="#debug-with-the-print-statement" title="Permalink to this headline"></a></h3>
<p>By default, <tt class="docutils literal"><span class="pre">py.test</span></tt> catches text written to stdout/stderr during
the execution of each individual test. This output will only be
displayed however if the test fails; you will not see it
otherwise. This allows you to put debugging print statements in your
code without being overwhelmed by all the output that might be
generated by tests that do not fail.</p>
<p>Each failing test that produced output during the running of the test
will have its output displayed in the <tt class="docutils literal"><span class="pre">recorded</span> <span class="pre">stdout</span></tt> section.</p>
<p>The catching of stdout/stderr output can be disabled using the
<tt class="docutils literal"><span class="pre">--nocapture</span></tt> option to the <tt class="docutils literal"><span class="pre">py.test</span></tt> tool. Any output will
in this case be displayed as soon as it is generated.</p>
</div>
<div class="section" id="test-execution-order">
<h3>1.2.4.4. test execution order<a class="headerlink" href="#test-execution-order" title="Permalink to this headline"></a></h3>
<p>Tests usually run in the order in which they appear in the files.
However, tests should not rely on running one after another, as
this prevents more advanced usages: running tests
distributedly or selectively, or in &#8220;looponfailing&#8221; mode,
will cause them to run in random order.</p>
</div>
<div class="section" id="assert-with-the-assert-statement">
<h3>1.2.4.5. assert with the <tt class="docutils literal"><span class="pre">assert</span></tt> statement<a class="headerlink" href="#assert-with-the-assert-statement" title="Permalink to this headline"></a></h3>
<p><tt class="docutils literal"><span class="pre">py.test</span></tt> allows to use the standard python
<tt class="docutils literal"><span class="pre">assert</span> <span class="pre">statement</span></tt> for verifying expectations
and values in Python tests. For example, you can
write the following in your tests:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="k">assert</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="s">&#39;attribute&#39;</span><span class="p">)</span>
</pre></div>
</div>
<p>to state that your object has a certain <tt class="docutils literal"><span class="pre">attribute</span></tt>. In case this
assertion fails you will see the value of <tt class="docutils literal"><span class="pre">x</span></tt>. Intermediate
values are computed by executing the assert expression a second time.
If you execute code with side effects, e.g. read from a file like this:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="k">assert</span> <span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">()</span> <span class="o">!=</span> <span class="s">&#39;...&#39;</span>
</pre></div>
</div>
<p>then you may get a warning from pytest if that assertions
first failed and then succeeded.</p>
</div>
<div class="section" id="asserting-expected-exceptions">
<h3>1.2.4.6. asserting expected exceptions<a class="headerlink" href="#asserting-expected-exceptions" title="Permalink to this headline"></a></h3>
<p>In order to write assertions about exceptions, you use
one of two forms:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">py</span><span class="o">.</span><span class="n">test</span><span class="o">.</span><span class="n">raises</span><span class="p">(</span><span class="ne">Exception</span><span class="p">,</span> <span class="n">func</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="n">py</span><span class="o">.</span><span class="n">test</span><span class="o">.</span><span class="n">raises</span><span class="p">(</span><span class="ne">Exception</span><span class="p">,</span> <span class="s">&quot;func(*args, **kwargs)&quot;</span><span class="p">)</span>
</pre></div>
</div>
<p>both of which execute the given function with args and kwargs and
asserts that the given <tt class="docutils literal"><span class="pre">Exception</span></tt> is raised. The reporter will
provide you with helpful output in case of failures such as <em>no
exception</em> or <em>wrong exception</em>.</p>
</div>
<div class="section" id="useful-tracebacks-recursion-detection">
<h3>1.2.4.7. useful tracebacks, recursion detection<a class="headerlink" href="#useful-tracebacks-recursion-detection" title="Permalink to this headline"></a></h3>
<p>A lot of care is taken to present nice tracebacks in case of test
failure. Try:</p>
<div class="highlight-python"><pre>py.test py/doc/example/pytest/failure_demo.py</pre>
</div>
<p>to see a variety of tracebacks, each representing a different
failure situation.</p>
<p><tt class="docutils literal"><span class="pre">py.test</span></tt> uses the same order for presenting tracebacks as Python
itself: the oldest function call is shown first, and the most recent call is
shown last. A <tt class="docutils literal"><span class="pre">py.test</span></tt> reported traceback starts with your
failing test function. If the maximum recursion depth has been
exceeded during the running of a test, for instance because of
infinite recursion, <tt class="docutils literal"><span class="pre">py.test</span></tt> will indicate where in the
code the recursion was taking place. You can inhibit
traceback &#8220;cutting&#8221; magic by supplying <tt class="docutils literal"><span class="pre">--fulltrace</span></tt>.</p>
<p>There is also the possibility of using <tt class="docutils literal"><span class="pre">--tb=short</span></tt> to get regular CPython
tracebacks. Or you can use <tt class="docutils literal"><span class="pre">--tb=no</span></tt> to not show any tracebacks at all.</p>
</div>
<div class="section" id="no-inheritance-requirement">
<h3>1.2.4.8. no inheritance requirement<a class="headerlink" href="#no-inheritance-requirement" title="Permalink to this headline"></a></h3>
<p>Test classes are recognized by their leading <tt class="docutils literal"><span class="pre">Test</span></tt> name. Unlike
<tt class="docutils literal"><span class="pre">unitest.py</span></tt>, you don&#8217;t need to inherit from some base class to make
them be found by the test runner. Besides being easier, it also allows
you to write test classes that subclass from application level
classes.</p>
</div>
<div class="section" id="testing-for-deprecated-apis">
<h3>1.2.4.9. testing for deprecated APIs<a class="headerlink" href="#testing-for-deprecated-apis" title="Permalink to this headline"></a></h3>
<p>In your tests you can use <tt class="docutils literal"><span class="pre">py.test.deprecated_call(func,</span> <span class="pre">*args,</span> <span class="pre">**kwargs)</span></tt>
to test that a particular function call triggers a DeprecationWarning.
This is useful for testing phasing out of old APIs in your projects.</p>
</div>
</div>
<div class="section" id="advanced-test-selection-skipping">
<h2>1.2.5. advanced test selection / skipping<a class="headerlink" href="#advanced-test-selection-skipping" title="Permalink to this headline"></a></h2>
<div class="section" id="dynamically-skipping-tests">
<h3>1.2.5.1. dynamically skipping tests<a class="headerlink" href="#dynamically-skipping-tests" title="Permalink to this headline"></a></h3>
<p>If you want to skip tests you can use <tt class="docutils literal"><span class="pre">py.test.skip</span></tt> within
test or setup functions. Example:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">py</span><span class="o">.</span><span class="n">test</span><span class="o">.</span><span class="n">skip</span><span class="p">(</span><span class="s">&quot;message&quot;</span><span class="p">)</span>
</pre></div>
</div>
<p>You can also use a helper to skip on a failing import:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">docutils</span> <span class="o">=</span> <span class="n">py</span><span class="o">.</span><span class="n">test</span><span class="o">.</span><span class="n">importorskip</span><span class="p">(</span><span class="s">&quot;docutils&quot;</span><span class="p">)</span>
</pre></div>
</div>
<p>or to skip if a library does not have the right version:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">docutils</span> <span class="o">=</span> <span class="n">py</span><span class="o">.</span><span class="n">test</span><span class="o">.</span><span class="n">importorskip</span><span class="p">(</span><span class="s">&quot;docutils&quot;</span><span class="p">,</span> <span class="n">minversion</span><span class="o">=</span><span class="s">&quot;0.3&quot;</span><span class="p">)</span>
</pre></div>
</div>
<p>The version will be read from the module&#8217;s <tt class="docutils literal"><span class="pre">__version__</span></tt> attribute.</p>
</div>
<div class="section" id="selecting-unselecting-tests-by-keyword">
<span id="selection-by-keyword"></span><h3>1.2.5.2. selecting/unselecting tests by keyword<a class="headerlink" href="#selecting-unselecting-tests-by-keyword" title="Permalink to this headline"></a></h3>
<p>Pytest&#8217;s keyword mechanism provides a powerful way to
group and selectively run tests in your test code base.
You can selectively run tests by specifiying a keyword
on the command line. Examples:</p>
<div class="highlight-python"><pre>py.test -k test_simple
py.test -k "-test_simple"</pre>
</div>
<p>will run all tests matching (or not matching) the
&#8220;test_simple&#8221; keyword. Note that you need to quote
the keyword if &#8220;-&#8221; is recognized as an indicator
for a commandline option. Lastly, you may use:</p>
<div class="highlight-python"><pre>py.test. -k "test_simple:"</pre>
</div>
<p>which will run all tests after the expression has <em>matched once</em>, i.e.
all tests that are seen after a test that matches the &#8220;test_simple&#8221;
keyword.</p>
<p>By default, all filename parts and
class/function names of a test function are put into the set
of keywords for a given test. You may specify additional
kewords like this:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="nd">@py</span><span class="o">.</span><span class="n">test</span><span class="o">.</span><span class="n">mark</span><span class="p">(</span><span class="n">webtest</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_send_http</span><span class="p">():</span>
<span class="o">...</span>
</pre></div>
</div>
</div>
<div class="section" id="disabling-a-test-class">
<h3>1.2.5.3. disabling a test class<a class="headerlink" href="#disabling-a-test-class" title="Permalink to this headline"></a></h3>
<p>If you want to disable a complete test class you
can set the class-level attribute <tt class="docutils literal"><span class="pre">disabled</span></tt>.
For example, in order to avoid running some tests on Win32:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="k">class</span> <span class="nc">TestPosixOnly</span><span class="p">:</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span> <span class="o">==</span> <span class="s">&#39;win32&#39;</span>
<span class="k">def</span> <span class="nf">test_xxx</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="o">...</span>
</pre></div>
</div>
</div>
</div>
<div class="section" id="generative-tests-yielding-parametrized-tests">
<h2>1.2.6. generative tests: yielding parametrized tests<a class="headerlink" href="#generative-tests-yielding-parametrized-tests" title="Permalink to this headline"></a></h2>
<p><em>Generative tests</em> are test methods that are <em>generator functions</em> which
<tt class="docutils literal"><span class="pre">yield</span></tt> callables and their arguments. This is most useful for running a
test function multiple times against different parameters. Example:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="k">def</span> <span class="nf">test_generative</span><span class="p">():</span>
<span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="p">(</span><span class="mf">42</span><span class="p">,</span><span class="mf">17</span><span class="p">,</span><span class="mf">49</span><span class="p">):</span>
<span class="k">yield</span> <span class="n">check</span><span class="p">,</span> <span class="n">x</span>
<span class="k">def</span> <span class="nf">check</span><span class="p">(</span><span class="n">arg</span><span class="p">):</span>
<span class="k">assert</span> <span class="n">arg</span> <span class="o">%</span> <span class="mf">7</span> <span class="o">==</span> <span class="mf">0</span> <span class="c"># second generated tests fails!</span>
</pre></div>
</div>
<p>Note that <tt class="docutils literal"><span class="pre">test_generative()</span></tt> will cause three tests
to get run, notably <tt class="docutils literal"><span class="pre">check(42)</span></tt>, <tt class="docutils literal"><span class="pre">check(17)</span></tt> and <tt class="docutils literal"><span class="pre">check(49)</span></tt>
of which the middle one will obviously fail.</p>
<p>To make it easier to distinguish the generated tests it is possible to specify an explicit name for them, like for example:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="k">def</span> <span class="nf">test_generative</span><span class="p">():</span>
<span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="p">(</span><span class="mf">42</span><span class="p">,</span><span class="mf">17</span><span class="p">,</span><span class="mf">49</span><span class="p">):</span>
<span class="k">yield</span> <span class="s">&quot;case </span><span class="si">%d</span><span class="s">&quot;</span> <span class="o">%</span> <span class="n">x</span><span class="p">,</span> <span class="n">check</span><span class="p">,</span> <span class="n">x</span>
</pre></div>
</div>
</div>
<div class="section" id="extensible-plugin-system">
<h2>1.2.7. extensible plugin system<a class="headerlink" href="#extensible-plugin-system" title="Permalink to this headline"></a></h2>
<p>py.test itself consists of many plugins
and you can easily write new <a class="reference external" href="test-plugins.html">py.test plugins</a>
for these purposes:</p>
<ul class="simple">
<li>reporting extensions</li>
<li>customizing collection and run of tests</li>
<li>running non-python tests</li>
<li>managing test state setup</li>
</ul>
</div>
</div>
</div>
</div>
</div>
<div class="sphinxsidebar">
<div class="sphinxsidebarwrapper">
<h3><a href="index.html">Table Of Contents</a></h3>
<ul>
<li><a class="reference external" href="">1.2. Features</a><ul>
<li><a class="reference external" href="#automatically-collects-and-executes-tests">1.2.1. automatically collects and executes tests</a></li>
<li><a class="reference external" href="#load-balance-tests-to-multiple-cpus">1.2.2. load-balance tests to multiple CPUs</a></li>
<li><a class="reference external" href="#distribute-tests-across-machines">1.2.3. Distribute tests across machines</a></li>
<li><a class="reference external" href="#extensive-debugging-support">1.2.4. extensive debugging support</a><ul>
<li><a class="reference external" href="#testing-starts-immediately">1.2.4.1. testing starts immediately</a></li>
<li><a class="reference external" href="#support-for-modules-containing-tests">1.2.4.2. support for modules containing tests</a></li>
<li><a class="reference external" href="#debug-with-the-print-statement">1.2.4.3. debug with the <tt class="docutils literal"><span class="pre">print</span></tt> statement</a></li>
<li><a class="reference external" href="#test-execution-order">1.2.4.4. test execution order</a></li>
<li><a class="reference external" href="#assert-with-the-assert-statement">1.2.4.5. assert with the <tt class="docutils literal"><span class="pre">assert</span></tt> statement</a></li>
<li><a class="reference external" href="#asserting-expected-exceptions">1.2.4.6. asserting expected exceptions</a></li>
<li><a class="reference external" href="#useful-tracebacks-recursion-detection">1.2.4.7. useful tracebacks, recursion detection</a></li>
<li><a class="reference external" href="#no-inheritance-requirement">1.2.4.8. no inheritance requirement</a></li>
<li><a class="reference external" href="#testing-for-deprecated-apis">1.2.4.9. testing for deprecated APIs</a></li>
</ul>
</li>
<li><a class="reference external" href="#advanced-test-selection-skipping">1.2.5. advanced test selection / skipping</a><ul>
<li><a class="reference external" href="#dynamically-skipping-tests">1.2.5.1. dynamically skipping tests</a></li>
<li><a class="reference external" href="#selecting-unselecting-tests-by-keyword">1.2.5.2. selecting/unselecting tests by keyword</a></li>
<li><a class="reference external" href="#disabling-a-test-class">1.2.5.3. disabling a test class</a></li>
</ul>
</li>
<li><a class="reference external" href="#generative-tests-yielding-parametrized-tests">1.2.6. generative tests: yielding parametrized tests</a></li>
<li><a class="reference external" href="#extensible-plugin-system">1.2.7. extensible plugin system</a></li>
</ul>
</li>
</ul>
<h4>Previous topic</h4>
<p class="topless"><a href="test-quickstart.html"
title="previous chapter">1.1. Quickstart</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="test-plugins.html"
title="next chapter">1.3. Included plugins</a></p>
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="_sources/test-features.txt"
rel="nofollow">Show Source</a></li>
</ul>
<div id="searchbox" style="display: none">
<h3>Quick search</h3>
<form class="search" action="search.html" method="get">
<input type="text" name="q" size="18" />
<input type="submit" value="Go" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
<p class="searchtip" style="font-size: 90%">
Enter search terms or a module, class or function name.
</p>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
>index</a></li>
<li class="right" >
<a href="test-plugins.html" title="1.3. Included plugins"
>next</a> |</li>
<li class="right" >
<a href="test-quickstart.html" title="1.1. Quickstart"
>previous</a> |</li>
<li><a href="index.html">py lib v1.0.0b1 documentation</a> &raquo;</li>
<li><a href="test.html" >1. py.test</a> &raquo;</li>
</ul>
</div>
<div class="footer">
&copy; Copyright 2009, Holger Krekel.
Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 0.7.
</div>
</body>
</html>