{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Testing" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[Original notebook](https://github.com/profjsb/python-bootcamp/blob/master/Lectures/12_Testing/12_testing.ipynb) by Jarrod Millman, part of the Python-bootcamp.\n", "\n", "Modifications Hans Fangohr, Sept 2013:\n", "\n", "- Add py.test example\n", "- minor edits\n", "\n", "Move to Python 3, Sept 2016.\n", "Reviewed, March 2021." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Motivation\n", "\n", "### Computing is error prone\n", "\n", "> In ordinary computational practice by hand or by desk machines, it is the\n", "> custom to check every step of the computation and, when an error is found,\n", "> to localize it by a backward process starting from the first point where the\n", "> error is noted.\n", ">\n", "> - Norbert Wiener (1948)\n", "\n", "### More computing, more problems\n", "\n", "> The major cause of the **software crisis** is that the machines have become\n", "> several orders of magnitude more powerful! To put it quite bluntly: as long\n", "> as there were no machines, programming was no problem at all; when we had a\n", "> few weak computers, programming became a mild problem, and now we have\n", "> gigantic computers, programming has become an equally gigantic problem.\n", ">\n", "> - Edsger W. Dijkstra (1972)\n", "\n", "## What testing is and is not...\n", "\n", "### Testing and debugging\n", "\n", "- debugging is what you do when you know a program is broken\n", "- testing is a determined, systematic attempt to break a program\n", "- writing tests is more interesting than debugging\n", "\n", "### Program correctness\n", "\n", "> Program testing can be used to show the presence of bugs, but never to show\n", "> their absence!\n", ">\n", "> - Edsger W. Dijkstra (1969)\n", "\n", "### In the imperfect world ...\n", "\n", "- avoid writing code if possible\n", "- write code as simple as possible\n", "- avoid cleverness\n", "- use code to generate code\n", "\n", "### Program languages play an important role\n", "\n", "> Programmers are always surrounded by complexity; we cannot avoid it. Our\n", "> applications are complex because we are ambitious to use our computers in\n", "> ever more sophisticated ways. Programming is complex because of the large\n", "> number of conflicting objectives for each of our programming projects. **If\n", "> our basic tool, the language in which we design and code our programs, is\n", "> also complicated, the language itself becomes part of the problem rather than\n", "> part of its solution.**\n", ">\n", "> --- C.A.R. Hoare - The Emperor's Old Clothes - Turing Award Lecture (1980)\n", "\n", "### Testing and reproducibility\n", "\n", "> In the good old days physicists repeated each other's experiments, just to\n", "> be sure. Today they stick to FORTRAN, so that they can share each other's\n", "> programs, bugs included.\n", ">\n", "> - Edsger W. Dijkstra (1975)\n", "\n", "### Pre- and post-condition tests\n", "\n", "- what must be true *before* a method is invoked\n", "- what must be true *after* a method is invoked\n", "- use assertions\n", "\n", "### Program defensively\n", "\n", "- out-of-range index\n", "- division by zero\n", "- error returns\n", "\n", "### Be systematic\n", "\n", "- incremental\n", "- simple things first\n", "- know what to expect\n", "- compare independent implementations\n", "\n", "### Automate it\n", "\n", "- **regression tests** ensure that changes don't break existing functionality\n", "- verify conservation\n", "- **unit tests** (white box testing)\n", "- measure test coverage\n", "\n", "### Interface and implementation\n", "\n", "- an **interface** is how something is used\n", "- an **implementation** is how it is written\n", "\n", "## Testing in Python\n", "\n", "### Landscape\n", "\n", "- errors, exceptions, and debugging\n", "- `assert`, `doctest`, and unit tests\n", "- `logging`, `unittest`, and `nose`\n", "\n", "### Errors & Exceptions\n", "\n", "#### Syntax Errors\n", "\n", "- Caught by Python parser, prior to execution\n", "- arrow marks the last parsed command / syntax, which gave an error" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "ename": "NameError", "evalue": "name 'true' is not defined", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mwhile\u001b[0m \u001b[0mtrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Hello world'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mNameError\u001b[0m: name 'true' is not defined" ] } ], "source": [ "while true:\n", " print('Hello world')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Exceptions\n", "\n", "- Caught during runtime" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "ename": "ZeroDivisionError", "evalue": "division by zero", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mZeroDivisionError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;36m1\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;31mZeroDivisionError\u001b[0m: division by zero" ] } ], "source": [ "1/0" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "ename": "NameError", "evalue": "name 'factorial' is not defined", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mfactorial\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;31mNameError\u001b[0m: name 'factorial' is not defined" ] } ], "source": [ "factorial" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "ename": "TypeError", "evalue": "can only concatenate str (not \"int\") to str", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;34m'1'\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;31mTypeError\u001b[0m: can only concatenate str (not \"int\") to str" ] } ], "source": [ "'1' + 1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Exception handling" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "No such file\n" ] } ], "source": [ "try:\n", " file = open('filenamethatdoesnotexist.txt')\n", "except FileNotFoundError:\n", " print('No such file')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Raising exceptions" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "ename": "NotImplementedError", "evalue": "Still need to write this code", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mNotImplementedError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;32mraise\u001b[0m \u001b[0mNotImplementedError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Still need to write this code\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mnewfunction\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m\u001b[0m in \u001b[0;36mnewfunction\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mnewfunction\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mNotImplementedError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Still need to write this code\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0mnewfunction\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mNotImplementedError\u001b[0m: Still need to write this code" ] } ], "source": [ "def newfunction():\n", " raise NotImplementedError(\"Still need to write this code\")\n", "\n", "newfunction()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Debugging" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "ename": "ZeroDivisionError", "evalue": "division by zero", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mZeroDivisionError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 7\u001b[0;31m \u001b[0mbar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m\u001b[0m in \u001b[0;36mbar\u001b[0;34m(y)\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mbar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 7\u001b[0m \u001b[0mbar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m\u001b[0m in \u001b[0;36mfoo\u001b[0;34m(x)\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0;36m1\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mbar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mZeroDivisionError\u001b[0m: division by zero" ] } ], "source": [ "def foo(x):\n", " return 1 / x\n", "\n", "def bar(y):\n", " return foo(1 - y)\n", "\n", "bar(1)" ] }, { "cell_type": "code", "execution_count": 61, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "> \u001b[0;32m\u001b[0m(2)\u001b[0;36mfoo\u001b[0;34m()\u001b[0m\n", "\u001b[0;32m 1 \u001b[0;31m\u001b[0;32mdef\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m----> 2 \u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m 3 \u001b[0;31m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m 4 \u001b[0;31m\u001b[0;32mdef\u001b[0m \u001b[0mbar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m 5 \u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\n", "ipdb> p x # can x really be zero?\n", "0\n", "ipdb> up\n", "> \u001b[0;32m\u001b[0m(5)\u001b[0;36mbar\u001b[0;34m()\u001b[0m\n", "\u001b[0;32m 3 \u001b[0;31m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m 4 \u001b[0;31m\u001b[0;32mdef\u001b[0m \u001b[0mbar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m----> 5 \u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m 6 \u001b[0;31m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m 7 \u001b[0;31m\u001b[0mbar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\n", "ipdb> p y # what is y (one function call UP)\n", "1\n", "ipdb> exit \n" ] } ], "source": [ "%debug" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Fixing bugs " ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "inf" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def foo(x):\n", " if x == 0:\n", " return float('Inf')\n", " else:\n", " return 1 / x\n", "\n", "bar(1)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "inf" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def foo(x):\n", " try:\n", " return 1 / x\n", " except ZeroDivisionError:\n", " return float('Inf')\n", "\n", "bar(1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Test as you code\n", "\n", "### Type checking " ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Please enter an integer: 5\n", "Casting 5 to integer.\n" ] } ], "source": [ "s = input(\"Please enter an integer: \") # s is a string\n", "if not isinstance(s, int):\n", " print(\"Casting \", s, \" to integer.\")\n", " i = int(s)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Assert invariants" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "3\n" ] } ], "source": [ "if i % 3 == 0:\n", " print(1)\n", "elif i % 3 == 1:\n", " print(2)\n", "else:\n", " assert i % 3 == 2\n", " print(3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Example\n", "\n", "Let's make a factorial function." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Writing myfactorial.py\n" ] } ], "source": [ "%%file myfactorial.py\n", "\n", "def factorial2(n):\n", " \"\"\" Details to come ...\n", " \"\"\"\n", "\n", " raise NotImplementedError\n", "\n", "def test():\n", " from math import factorial\n", " for x in range(10):\n", " print(\".\", end=\"\")\n", " assert factorial2(x) == factorial(x), \\\n", " \"My factorial function is incorrect for n = %i\" % x" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Let's test it ..." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "." ] }, { "ename": "NotImplementedError", "evalue": "", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mNotImplementedError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mmyfactorial\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mmyfactorial\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtest\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m~/git/mpsd/teaching-python/notebook/myfactorial.py\u001b[0m in \u001b[0;36mtest\u001b[0;34m()\u001b[0m\n\u001b[1;32m 10\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 11\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\".\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 12\u001b[0;31m \u001b[0;32massert\u001b[0m \u001b[0mfactorial2\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mfactorial\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m\\\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 13\u001b[0m \u001b[0;34m\"My factorial function is incorrect for n = %i\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m~/git/mpsd/teaching-python/notebook/myfactorial.py\u001b[0m in \u001b[0;36mfactorial2\u001b[0;34m(n)\u001b[0m\n\u001b[1;32m 4\u001b[0m \"\"\"\n\u001b[1;32m 5\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mNotImplementedError\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 7\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 8\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mtest\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mNotImplementedError\u001b[0m: " ] } ], "source": [ "import myfactorial\n", "myfactorial.test()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Looks like we will have to implement our function, if we want to make any progress..." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Overwriting myfactorial.py\n" ] } ], "source": [ "%%file myfactorial.py\n", "\n", "def factorial2(n):\n", " \"\"\" Details to come ...\n", " \"\"\"\n", "\n", " if n == 0:\n", " return 1\n", " else:\n", " return n * factorial2(n-1)\n", "\n", "def test():\n", " from math import factorial\n", " for x in range(10):\n", " assert factorial2(x) == factorial(x), \\\n", " \"My factorial function is incorrect for n = %i\" % x" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Let's test it ..." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "import importlib\n", "importlib.reload(myfactorial)\n", "myfactorial.test()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Seems to be okay so far. However, calling ``factorial2`` with a negative number, say, will result in infinite loop. Thus:\n", "\n", "### What about preconditions\n", "\n", "What happens if we call `factorial2` with a negative integer? Or something that's not an integer?" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Overwriting myfactorial.py\n" ] } ], "source": [ "%%file myfactorial.py\n", "def factorial2(n):\n", " \"\"\" Find n!. Raise an AssertionError if n is negative or non-integral.\n", " \"\"\"\n", "\n", " assert n >= 0 and type(n) is int, \"Unrecognized input\"\n", "\n", " if n == 0:\n", " return 1\n", " else:\n", " return n * factorial2(n - 1)\n", "\n", "def test():\n", " from math import factorial\n", " for x in range(10):\n", " assert factorial2(x) == factorial(x), \\\n", " \"My factorial function is incorrect for n = %i\" % x" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `doctests` -- executable examples" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[1, 1, 2, 6, 24]" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "importlib.reload(myfactorial)\n", "from myfactorial import factorial2\n", "[factorial2(n) for n in range(5)]" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Overwriting myfactorial.py\n" ] } ], "source": [ "%%file myfactorial.py\n", "def factorial2(n):\n", " \"\"\" Find n!. Raise an AssertionError if n is negative or non-integral.\n", "\n", " >>> from myfactorial import factorial2\n", " >>> [factorial2(n) for n in range(5)]\n", " [1, 1, 2, 6, 24]\n", " \"\"\"\n", "\n", " assert n >= 0. and type(n) is int, \"Unrecognized input\"\n", "\n", " if n == 0:\n", " return 1\n", " else:\n", " return n * factorial2(n - 1)\n", "\n", "def test():\n", " from math import factorial\n", " for x in range(10):\n", " assert factorial2(x) == factorial(x), \\\n", " \"My factorial function is incorrect for n = %i\" % x" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Running doctests" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Trying:\r\n", " from myfactorial import factorial2\r\n", "Expecting nothing\r\n", "ok\r\n", "Trying:\r\n", " [factorial2(n) for n in range(5)]\r\n", "Expecting:\r\n", " [1, 1, 2, 6, 24]\r\n", "ok\r\n", "2 items had no tests:\r\n", " myfactorial\r\n", " myfactorial.test\r\n", "1 items passed all tests:\r\n", " 2 tests in myfactorial.factorial2\r\n", "2 tests in 3 items.\r\n", "2 passed and 0 failed.\r\n", "Test passed.\r\n" ] } ], "source": [ "!python -m doctest -v myfactorial.py" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Real world testing and continuous integration\n", "\n", "### Test fixtures (Unittest)\n", "\n", "- create self-contained tests\n", "- setup: open file, connect to a DB, create datastructures\n", "- teardown: tidy up afterward\n", "- Based on JUnit/XUnit\n", "\n", "### Test runner (nose, pytest)\n", "\n", "- `nosetests`, `py.test`\n", "- test discovery: any callable beginning with `test` in a module\n", " beginning with `test`\n", "\n", "Advice: Unless you have special reason to use `Unittests` or `nose`, use `py.test`.\n", "\n", "### Testing scientific computing libraries\n", "\n", "Such libraries have often testing routines, for example:" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "====================================== test session starts ======================================\n", "platform darwin -- Python 3.8.5, pytest-6.1.1, py-1.9.0, pluggy-0.13.1\n", "rootdir: /Users/fangohr/git/mpsd/teaching-python/notebook\n", "collected 135 items\n", "\n", "tests/test__quad_vec.py .................... [ 14%]\n", "tests/test_banded_ode_solvers.py . [ 15%]\n", "tests/test_bvp.py ............... [ 26%]\n", "tests/test_integrate.py ............................................... [ 61%]\n", "tests/test_odeint_jac.py .. [ 62%]\n", "tests/test_quadpack.py ................................ [ 86%]\n", "tests/test_quadrature.py .................. [100%]\n", "\n", "====================================== 135 passed in 4.12s ======================================\n" ] }, { "data": { "text/plain": [ "True" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import scipy.integrate\n", "scipy.integrate.test()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Assertions revisited - numerical mathematics" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Mathematically\n", "\n", "$ x = (\\sqrt(x))^2$.\n", "\n", "So what is happening here:" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "ename": "AssertionError", "evalue": "", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mAssertionError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mmath\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0;32massert\u001b[0m \u001b[0;36m2\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mmath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msqrt\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;31mAssertionError\u001b[0m: " ] } ], "source": [ "import math\n", "assert 2 == math.sqrt(2)**2" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "2.0000000000000004" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "math.sqrt(2)**2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### NumPy Testing" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "What if we consider x and y almost equal? Can we modify our assertion?" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "np.testing.assert_almost_equal(2, math.sqrt(2) ** 2)" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [], "source": [ "x = 1.000001\n", "y = 1.000002\n", "np.testing.assert_almost_equal(x, y, decimal=5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Testing with py.test" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Going beyond ``doctest`` and Unittest, there are two important frameworks for regression testing: \n", "\n", "* nose (http://nose.readthedocs.org/en/latest/)\n", "\n", "* pytest (http://pytest.org)\n", "\n", "Here, we focus on pytest. \n", "\n", "The example we use is the ``myfactorial.py`` file created earlier:" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [], "source": [ "# %load myfactorial.py\n", "def factorial2(n):\n", " \"\"\" Find n!. Raise an AssertionError if n is negative or non-integral.\n", "\n", " >>> from myfactorial import factorial2\n", " >>> [factorial2(n) for n in range(5)]\n", " [1, 1, 2, 6, 24]\n", " \"\"\"\n", "\n", " assert n >= 0. and type(n) is int, \"Unrecognized input\"\n", "\n", " if n == 0:\n", " return 1\n", " else:\n", " return n * factorial2(n - 1)\n", "\n", "def test():\n", " from math import factorial\n", " for x in range(10):\n", " assert factorial2(x) == factorial(x), \\\n", " \"My factorial function is incorrect for n = %i\" % x" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [], "source": [ "def factorial2(n):\n", " \"\"\" Find n!. Raise an AssertionError if n is negative or non-integral.\n", "\n", " >>> from myfactorial import factorial2\n", " >>> [factorial2(n) for n in range(5)]\n", " [1, 1, 2, 6, 24]\n", " \"\"\"\n", "\n", " assert n >= 0. and type(n) is int, \"Unrecognized input\"\n", "\n", " if n == 0:\n", " return 1\n", " else:\n", " return n * factorial2(n - 1)\n", "\n", "def test():\n", " from math import factorial\n", " for x in range(10):\n", " assert factorial2(x) == factorial(x), \\\n", " \"My factorial function is incorrect for n = %i\" % x" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Providing test functions\n", "\n", "(Addition to original notebook, Hans Fangohr, 21 Sep 2013)\n", "\n", "py.test is an executable that will search through a given file and find all functions that start with ``test``, and execute those. Any failed assertions are reported as errors.\n", "\n", "For example, ``py.test`` can run the ``test()`` function that has been defined already in ``myfactorial``:" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[1m============================= test session starts ==============================\u001b[0m\r\n", "platform darwin -- Python 3.8.5, pytest-6.1.1, py-1.9.0, pluggy-0.13.1\r\n", "rootdir: /Users/fangohr/git/mpsd/teaching-python/notebook\r\n", "\u001b[1mcollecting ... \u001b[0m\u001b[1m\r", "collected 1 item \u001b[0m\r\n", "\r\n", "myfactorial.py \u001b[32m.\u001b[0m\u001b[32m [100%]\u001b[0m\r\n", "\r\n", "\u001b[32m============================== \u001b[32m\u001b[1m1 passed\u001b[0m\u001b[32m in 0.01s\u001b[0m\u001b[32m ===============================\u001b[0m\r\n" ] } ], "source": [ "!py.test myfactorial.py" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This output (the '.' after ``myfactorial.py``) indicates success. We can get a more detailed output using the ``-v`` switch for extra verbosity:" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[1m============================= test session starts ==============================\u001b[0m\n", "platform darwin -- Python 3.8.5, pytest-6.1.1, py-1.9.0, pluggy-0.13.1 -- /Users/fangohr/.pyenv/versions/anaconda3-2020.11/bin/python\n", "cachedir: .pytest_cache\n", "rootdir: /Users/fangohr/git/mpsd/teaching-python/notebook\n", "collected 1 item \u001b[0m\n", "\n", "myfactorial.py::test \u001b[32mPASSED\u001b[0m\u001b[32m [100%]\u001b[0m\n", "\n", "\u001b[32m============================== \u001b[32m\u001b[1m1 passed\u001b[0m\u001b[32m in 0.00s\u001b[0m\u001b[32m ===============================\u001b[0m\n" ] } ], "source": [ "!py.test -v myfactorial.py" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Sometimes, we like having the tests for ``myfactorial.py`` gathered in a separate file, for example in ``test_myfactorial.py``. We create such a file, and within the file we create a number of test functions, each with a name starting with ``test``:" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Writing test_myfactorial.py\n" ] } ], "source": [ "%%file test_myfactorial.py\n", "\n", "from myfactorial import factorial2 \n", "\n", "def test_basics():\n", " assert factorial2(0) == 1\n", " assert factorial2(1) == 1\n", " assert factorial2(3) == 6\n", " \n", "def test_against_standard_lib():\n", " import math\n", " for i in range(20):\n", " assert math.factorial(i) == factorial2(i)\n", " \n", "def test_negative_number_raises_error():\n", " import pytest\n", "\n", " with pytest.raises(AssertionError): # this will pass if \n", " factorial2(-1) # factorial2(-1) raises \n", " # an AssertionError\n", " \n", " with pytest.raises(AssertionError):\n", " factorial2(-10)\n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can now run the tests in this file using" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[1m============================= test session starts ==============================\u001b[0m\n", "platform darwin -- Python 3.8.5, pytest-6.1.1, py-1.9.0, pluggy-0.13.1 -- /Users/fangohr/.pyenv/versions/anaconda3-2020.11/bin/python\n", "cachedir: .pytest_cache\n", "rootdir: /Users/fangohr/git/mpsd/teaching-python/notebook\n", "collected 3 items \u001b[0m\n", "\n", "test_myfactorial.py::test_basics \u001b[32mPASSED\u001b[0m\u001b[32m [ 33%]\u001b[0m\n", "test_myfactorial.py::test_against_standard_lib \u001b[32mPASSED\u001b[0m\u001b[32m [ 66%]\u001b[0m\n", "test_myfactorial.py::test_negative_number_raises_error \u001b[32mPASSED\u001b[0m\u001b[32m [100%]\u001b[0m\n", "\n", "\u001b[32m============================== \u001b[32m\u001b[1m3 passed\u001b[0m\u001b[32m in 0.01s\u001b[0m\u001b[32m ===============================\u001b[0m\n" ] } ], "source": [ "!py.test -v test_myfactorial.py " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The ``py.test`` command can also be given a directory, and it will search all files and files in subdirectories for files starting with ``test``, and will attempt to run all the tests in those. \n", "\n", "Or we can provide a list of test files to work through:\n" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[1m============================= test session starts ==============================\u001b[0m\n", "platform darwin -- Python 3.8.5, pytest-6.1.1, py-1.9.0, pluggy-0.13.1 -- /Users/fangohr/.pyenv/versions/anaconda3-2020.11/bin/python\n", "cachedir: .pytest_cache\n", "rootdir: /Users/fangohr/git/mpsd/teaching-python/notebook\n", "collected 4 items \u001b[0m\n", "\n", "test_myfactorial.py::test_basics \u001b[32mPASSED\u001b[0m\u001b[32m [ 25%]\u001b[0m\n", "test_myfactorial.py::test_against_standard_lib \u001b[32mPASSED\u001b[0m\u001b[32m [ 50%]\u001b[0m\n", "test_myfactorial.py::test_negative_number_raises_error \u001b[32mPASSED\u001b[0m\u001b[32m [ 75%]\u001b[0m\n", "myfactorial.py::test \u001b[32mPASSED\u001b[0m\u001b[32m [100%]\u001b[0m\n", "\n", "\u001b[32m============================== \u001b[32m\u001b[1m4 passed\u001b[0m\u001b[32m in 0.01s\u001b[0m\u001b[32m ===============================\u001b[0m\n" ] } ], "source": [ "!py.test -v test_myfactorial.py myfactorial.py" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "#### Learn more\n", "\n", "* [http://software-carpentry.org](http://software-carpentry.org)\n", "* [http://docs.python.org/library/exceptions.html](http://docs.python.org/library/exceptions.html)\n", "* [http://docs.python.org/library/doctest.html](http://docs.python.org/library/doctest.html)\n", "* [http://docs.python.org/library/unittest.html](http://docs.python.org/library/unittest.html)\n", "* [http://docs.scipy.org/doc/numpy/reference/routines.testing.html](http://docs.scipy.org/doc/numpy/reference/routines.testing.html)\n", "* [http://nedbatchelder.com/code/coverage](http://nedbatchelder.com/code/coverage)\n", "* [http://somethingaboutorange.com/mrl/projects/nose](http://somethingaboutorange.com/mrl/projects/nose)" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[33mcommit ea7db190b55f27ef4356f24c13218f8eea2a75f9\u001b[m\u001b[33m (\u001b[m\u001b[1;36mHEAD -> \u001b[m\u001b[1;32mmaster\u001b[m\u001b[33m, \u001b[m\u001b[1;31morigin/master\u001b[m\u001b[33m)\u001b[m\r\n", "Author: Hans Fangohr \r\n", "Date: Thu Mar 18 18:26:35 2021 +0100\r\n", "\r\n", " tidy up\r\n" ] } ], "source": [ "!git log -1" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" } }, "nbformat": 4, "nbformat_minor": 1 }