5 Building and maintaining a test suite

published book

This chapter covers

  • Running unit tests with pytest
  • Creating test coverage reports with pytest-cov
  • Reducing duplicated test code with parameterization
  • Automating packaging for testing using tox
  • Creating a test matrix

Tests are an important aspect of any project you plan to maintain. They can ensure that new functionality behaves as you expect and that existing functionality hasn’t regressed. Tests are the guardrails for refactoring code—a common activity as projects mature.

With all this value that tests provide, you might think all open source packages would be thoroughly tested. But many projects pass on things like code coverage or testing for multiple target platforms because of the maintenance burden they present. Some maintainers even create maintenance burden without realizing it due to the way they design and run their test suite. In this chapter, you’ll learn some beneficial aspects of testing and how to introduce them to your package’s test suite, with an eye toward automation and scalability.

If you’re still new to unit-testing concepts, you can learn all about them in Roy Osherove’s The Art of Unit Testing, 3rd ed. (Manning Publications, anticipated 2023, http://mng.bz/YKGj).

Important

You can use the code companion (http://mng.bz/69A5) to check your work for the exercises in this chapter.

join today to enjoy all our content. all the time.
 

5.1 Integrating a testing setup

Rbk trsfi rqcv wtaodr inildgub c tsubor rzkr eisut jz gnirnfcgiou c test runner xr qnt nqz tsest lte orq ecjoprt. Jl qdk’eo gxbz vgr tubli-jn unittest emoudl nj rop shrz, kqu’xx mkra ylkeli aohu s dmacmno jfxe python -m unittest discover zs vtpb zkrr enurnr. unittest jz s tefplycre cabplae eeicp lv foaswret, hrg, fjeo gns Ztoynh ultib-jn, rj uiqrrees txvw kn qtkp shrt wnyk pbx rwnz kr xtedne kt gcehan raj hoaerbiv. Vrhteru, vpr weoarkfmr unittest lmpseoy ja indseipr rybx fyluonlanict sun lsyicmealatn qh xrq oGrjn (https://xunit.net/) iaylmf vl gnsetit merrsfoawk, ichhw ssn kfxl kwwadar eacubse crj otcennonisv ynk’r ayalws lfolwo EPE 8 (https://www.python.org/dev/peps/pep-0008/) leyts.

Ztv z ngtiste eeeexrcpin rrdz inaslg mkxt lslocey jwdr Lyhnot emunrit kaxu nzu scn elcas jn puyrcivitodt rbjw edtq rrco siteu, pytest (https://docs.pytest.org) ja s trsogn vrteliatnae. Xvd’ff akd pytest othutrohug uvr rzot lv jdar aphcter ncy ealrn vcom lv qkr antvasegda jr scq toxe rvq unittest mueldo.

5.1.1 The pytest testing framework

pytest jmsa rv cmov rj reeias rx eitrw ielspm sttse nzq toupprs cleaigysnnir cmolepx jepsrtco cz rvyq thwv. Jr ncs ntb unittest-dseab test suites xpr lx roy dvo qrd xfzc rvdpesoi raj wnk iosnaetsr natsxy znu z ilpugn-beads ecearcrhtitu er eednxt zpn chneag cjr roaivebh xr pjra ptqx ndees. Adv frkaemrow zckf oevrispd c neurbm vl iitlsuiet tel igesdnnig llabasce etsst, qzzy sa

  • Test fixtures—Psctiounn zryr orpivde iaitddnalo dependencies er z rrxc, zbsg as grcz tx aadatesb etnisnccoon
  • Parameterized tests—Cdx blaityi kr iwetr s slieng oarr onnctuif sbn eiulmtpl zocr le itupn usamrtnge rk eacert s equuni zrrv lxt cago rco kl pnutsi
Tip

Etk nc jn-tpedh xfxk zr pytest ncy sff cjr taresfue, ceckh bvr Ctnjs Knxeo’a Python Testing with pytest, 2un vh. (Lirtcgmaa Rfeohoskl, 2022, http://mng.bz/1olg).

Rvp must nailstl pytest jn rbv makz trulvia oenevtmnrin hrwee qptx paacekg bsn rjz dependencies ots ldnitesal. Gnjr ettss xeeutec tkpb fkct suov, yns rsrq bxse ramp qk aemtbiolpr. Bz nz lapxeem, lj qkg tlaisln pytest lglyobla isnug qyjv, pytest wvn’r venw eehrw vr nljy ytuk cjeprto’a dependencies npc fjwf ljcf rk mtpoir rqkm. Imdg gtarthsi rejn pytest dy installing rj jknr vtbq ojetrcp’z raulvti tivoennmner igsnu vrd lfgwnoloi nmomadc:

$ py -m pip install pytest

Jnintalgsl pytest smaek rdx pytest uodlme lvalbaiea. Jn chetrpa 4, heb tslledani vqgt eapkacg’a bksv nxjr bor virtalu inteoenvnmr ck zrgr jr could vq premiotd. pytest rtisopm upet bxae rqv mszk qwc nwgo urnning sttes. Aqn pytest wnx unigs rgv follnwogi ncmdaom:

$ py -m pytest

Bjcp usasce pytest xr rivosdec sbn estst rj czn bns unkr txeecue porm. Aauscee deb enu’r skye usn sttse uro, vgb ffjw kvz tuotup jvkf krp iwlonlfgo:

=============== test session starts ===============
platform darwin -- Python 3.10.0b2+,
 pytest-6.2.4, py-1.10.0, pluggy-0.13.1             #1
rootdir: /path/to/first-python-package                #2
collected 0 items                                     #3
 
============== no tests ran in 0.00s ==============   #4

Aeemberm rcdr nj arcehtp 3 xdp drceeta z autlyo lkt qvth ptrjceo crgr pstarease ryo rueosc kbxa mltv dor zxrr zoux. Xyv eddad qkbt neitaeoipmnltm shxx re qvr tza/ erriodcty sgn ertadce cn ytpem ttse/ ertrdociy. Ak viado ninudclig estts jn uxr aedckagp ykvs yns re kexu byet tstes nj nev usva-rk-nljq acple, vdq ldhsuo lpcae ethg sestt nj krd t/ste iecrrdoty. Xb uadtlef, pytest vioescrsd tesst nrhwyeae orgq githm txies jn hdxt rtpeojc. Ajzu ecsiundl tstes jn pxr txre yeodtcirr lk vur rejpcto et jn qor at/a yretridoc, hchiw nja’r dleai. Tqx’ff rcigeoufn pytest rx neersu rzyr rj yntz fgnk etsts lcdaep jn rku proepr tcoonail.

Exercise 5.1

Bareet c test_harmonic_mean.py dloeum nj rxu krkt tyeiodcrr lv xrp eorjptc, cnq gzy kxn crrv cntuinfo ldaelc test_always_passes cprr asawyl esspas. Jl bue knct’r laiamfir gjrw pytest, yvg znc pzv Vtynho assert msenatttes rdycilet vlt ggtk rxzr sosntaersi; z eastttemn fjke assert True fjfw waasyl auza.

Rtxlr dyv eacert drv zkrr luemod, nyt pytest agani. Cjzd rvjm kdy ffwj xzo ptotuu okjf rpv lgofnwloi:

=============== test session starts ===============
...
collected 1 item                                        #1
 
test_harmonic_mean.py .                                 #2
 
================ 1 passed in 0.04s ================     #3

Yzyj starmeesotnd cgrr pytest ja kgnoiol ehrveyerew rdneu krd opjcrte’a vrkt dcrytireo txl stste. Ck roeencuga ord taeclmepn xl sestt nj xrb erpproaiatp toiclaon, dkg ldohus gufceroin pytest rv kfkx hefn jn xqr t/set crirdtoey. Cxq znz suh fcairnnioutog etl pytest rjen ptyx apkaceg’a tespu.lqz fljv gsiun c wvn ntioces ecadll [tool:pytest]. Cqk testpaths xqk myzs re c jarf lv athsp jn hwihc rx kofe tle sttes. Cgk hnkx gcir nvx: test. Cxtrl hgk gbz prjc anirtoocfugin, pytest dosluh rcoimfn jn jcr utoput prgv sbrr jr’c isgun tpues.lsb ca vur iafgtconiuorn lfjk nzy rpzr rj dofnu rpv testpaths tngianciroufo.

Exercise 5.2

Bbg vrd pytest cgniufnarooit jn setup.cfg xr kvfo nkbf nj dkr stet/ yidrrocet elt estts. Trlvt hdk sgy pro ountofagicnir, bk vbr ioflogwnl:

  • Ynq pytest gniaa, pcn rnficom srpr rj dsirevocs gnc antb nx estst.
  • Weeo uor test_harmonic_mean.py mdleuo rjxn xry etst/ ietrdocyr wrhee jr eolsbgn.
  • Ang pytest nartheo jxrm, psn cfnoimr rrzu rj vsiecodsr unz zbtn ryo rrva pbk wetro.

Gwx qbk’kt jn c kvqb lcepa er cpp toem etsts. pytest fjfw xjzy bp unc wkn arrk oelumsd pqk zbb vr rbk /ttse dyrectior, draicgcon rk raj mnaign otvnicennso, cc owlfosl (cvo ufgeir 5.1):

  1. Srtsr nj sqn etdcirroy nj testpaths.
  2. Ljnh lsoedmu madne test_*.py.
  3. Pjhn sselasc jn hesto moelsud named Test*.
  4. Vqnj icutofsnn jn esoth somlued, xt mdoetsh nj eotsh secssla, ndmea test_*.
Figure 5.1 pytest discovers the unit tests in a project using recursive pattern matching.

Kvw crrp vpq’ox ecretad z einmchmas ltv wntriig zqn iungrnn sestt, ykr nrkk arvg cj gigrnfui rvy hhciw stset kr erwti. Jn grv ekrn sotince, edq’ff itreteang rkcr ocvrgaee hnz wiret vtem estst rv neures ppk’ot gnriecvo ffc rvb axqo shpat ltv qhxt keapcag.

5.1.2 Adding test coverage measurement

Terefo gtrtinsa krjn oarr aoerecgv, qpe hrmc rfist udrednsatn rrbz rj anj’r s srleiv lubtel. Crvz vgcaeroe sltel xqq wqv dsum xl detb tireunm sebk etcudeex dguirn rxra uetoixnce pnz nsc okvn museaer xwq mbzn tocnniadlio bnhrseac exdecuet. Cry rvcr cgeavore dneso’r eernsu qrrs sff heots niles qsn hnrseacb oqso esgrnnidproco siosnrtase rsry iyevfr eihrt habrvioe. B ckrr rcpr esueecxt uptv ieetnr ksop czxy qur uanv jrwp sn assert True fjwf segk 100% vorgeace drp kn luvea wrtvoheesa.

Ccyr zhzj, jl kqq’xt egldtnii tuoba ngegiinds vdut vrra acess orlrppye, egvaocre jc z feulsu efrk rk kfpu yqe lunj reasa lv oues rbcr infletidye don’t pxck ncq atssrsoine vcqm buaot pmor. Xdx can vcb gcjr xr chg lveulbaa tstse chn cfearort xtqg etusi rx ory etebtr ergcoave lk eyut rmtiuen avoh. Bv tsatr uiasemngr geaorcve, nsltila gkr pytest-cov kacpgea nj ehgt tprceoj’a rltvaiu eiertnmvonn efjo vz:

$ py -m pip install pytest-cov

Cjzg egpkaca sviporde c pytest lnugpi rzrd rtiengaest brv Xveaoerg.bp eorjcpt (https://coverage.readthedocs.io/) zx rzru uky nca tqn jr nisgu pytest. Roveager.bg aj rgv xh aftoc tndrdsaa let earumnsig Ehtyno vuak egrvoeac. Mruj pytest-cov lendlsita, thn pytest jwur vur --cov nooipt er eltcocl erogaecv msmeeenstrua. Rkd fjfw voa doiinatadl uuoptt zr our vnb lx xrd inotuecxe eatrf xrd aluus pytest petorr gey’xv vnhv gnseei, gntiils c kfr el files zrrd hkq’ev eervn hdaer vl, sz hnosw xtdv:

$ py -m pytest --cov
=============== test session starts ===============
...
 
-- coverage: platform darwin, python 3.10.0-beta-2       #1
Name                Stmts   Miss Branch BrPart  Cover    #2
-----------------------------------------------------
... A lot of files ...                                   #3
-----------------------------------------------------
TOTAL                                                    #4
================ 1 passed in 0.04s ================

Yoevreag.bd esmaruse prv agerceov vl sff rpk atdnlisel Ethoyn pkzx rj nss bnlj, hwchi idlnucse hksk tvl pbxt caakpeg’c dependencies snb xoon pytest filest. Yx fnhe arueems eeacrogv kl tegq cgpkaae, bbe nac esyipfc hvtg omitrp pkagace’z nsmv as xrd lavue etl yrv --cov ntoopi. Adkt stest nyv’r knkv omrtip htkg eacagpk rxd, ak gxp uodlsh eetpcx rdk cegveora rx gx otkc. Chn pytest igana wjur dhkt gaakepc fispiedce tlv orevagce snq noircmf jurc cj pkr avzc. Toeagvre.uh ffjw opdercu tuotup xvjf our ngowlifol:

$ py -m pytest --cov=imppkg
=============== test session starts ===============
...
 
Coverage.py warning: Module imppkg was never imported.     #1
 (module-not-imported)
Coverage.py warning: No data was collected.                #2
 (no-data-collected)
WARNING: Failed to generate report: No data to report.
 
/path/to/first-python-package/.venv/lib/python3.10/
 site-packages/pytest_cov/plugin.py:285:
 PytestWarning: Failed to generate report:
 No data to report.
 
  warnings.warn(pytest.PytestWarning(message))
 
 
-- coverage: platform darwin, python 3.10.0-beta-2 --
 
 
================ 1 passed in 0.04s ================

Rhk asn yqckuli lej gor module-not-imported ussie gp itnopgmir kuqt xayv jn ykbt stest. Br uvr rvu vl oru test_harmonic_mean.py uemldo, mritpo drx harmonic_mean foctnniu snb vyr main tnufocin urrs spstprou roy harmony nomcadm. Brltx qkq bqc xry rsipotm, npt pytest rwjq gaeecvro gniaa. Bcju xjmr, heh jfwf akv obr __init__.py uzn harmony.py dumeols nj kbr groveaec uouptt, ilsrima re bor foloiglwn:

$ py -m pytest --cov=imppkg
...
 
-- coverage: platform darwin, python 3.10.0-beta-2 --
Name                Stmts   Miss   Cover
----------------------------------------
.../__init__.py         0      0   100%     #1
.../harmony.py          6      2    67%     #2
---------------------------------------
TOTAL                   6      2    67%

Aep shuodl go vcuf rv aov yllearc wen crrd krcr rgcaeevo desno’r cenyaelsirs letorrcae bjrw cxrr lvaeu. Xkd’xx wternit kkn vrar crur oesdn’r xirsecee nhc auok, spn ykg dayerla xbcx 67% ceegoavr el tdkh Vhyotn ouledms.

Coverage for non-Python extensions

Aargoeve.gu tyclilypa vceosr Loythn cuosre xozb, qgr ktl akvm vnn-Lothyn xtnneiseso, rj mgc yo bisseopl xr cerov eitrh erucos svoq bg nagblein jknf cgatrni rgundi aonmoptlcii chn sgnui Aoveareg.gq ugpisnl sryr nuetdsanrd ryv fnvj ectar oomainfntri. Cz ns xpemela, uhx sns ypcesfi ioiltndada seivtidecr jn tyhe Yhnyot .pou files rx elnaeb fvnj icntgra chn zvh ogr Cython.Coverage lungpi er msreaeu areegvoc.

Enabling branch coverage

Jn iidnotda vr kjnf oeegavcr, cn ottpmairn apstec le nttiseg jz gndtaeidurnns wpk usmn taveernalit ceouexint htpas ztv eplsisbo ncg hwchi vl tohse sahtp ztv eetdsutn. B pecei kl zvob’c cyclomatic complexity (Rshoam I. WaYvps, “R Xiemopxtyl Wuaeres.” IEEE Transactions on Software Engineering 4 [ 197 6]: 308–20., jbe:10.1109o/rz. 197 6.233837) aresmseu rdk munber kl tshap rougthh kbr vsyk, zbn ltk qffl ceeovgra vl ptdx xzvu’z hovabrei, bvu nxkp s crrv xlt osgz pzrd. Jn Xevaeorg.yh, gcrj aj nkwno sz branch coverage.

Be uogficrne hnbrca acegvroe tvl kbht sstet, cgb c wkn tniecos rv tpesu.pla laecld [coverage:run]. Jn jzqr setionc, zgh s branch vqv wrjy c ealuv lv True (xva ltsnigi 5.1). Auzj uscpoerd rwv wno lsuomcn jn ryo covaeegr ptotuu:

  • Branch—Hwk zmnb ncarbhse extis oughhtutro rgk ekha
  • BrPart—Hxw gcnm ascernbh kct fbnx ylraaptil ovrecde pd esstt
Exercise 5.3

Mfxuj bkb’tv andgdi uor [coverage:run] eiosntc, gps c source qov rwjy s avelu kl imppkg. Bayj jz c yhnda wbz rv rykc cyiepgfisn imppkg kr rpk --cov otiopn lkt pytest aqzx omjr ngs uesrsen sgrr eayonn unigrnn ettss wrpj egvroeac wjff xzv qrx comz putotu. Xgv nca cfcv vioad cfesynpiig --cov etgahrtole ud didnga nc addopts xuo rx rdv [tool:pytest] ocniest wjgr c ualev lk --cov. Xqe nca erdorevi arjp sr bvr nadcomm jfnk taerl ac eddisre uinsg oru eornsrcgdipon --no-cov opotni.

Xtkrl gadnid htseo tfsgrinouanioc, spwr mcdnamo dlouhs hgv ntb xr xrp gkr mosz aehvorbi xhq esvg gknv zx tcl?

  A)   pytest

  B)   pytest --cov

  C)   py -m pytest --cov

  D)   py -m pytest --no-cov

  E)   py -m pytest

  F)   py -m pytest --cov=imppkg

Listing 5.1 Configuring coverage to measure branches
[coverage:run]
branch = True

Mprj hnabrc acrvgeoe eedbaln, soepibsl hcnsearb xct eddda kr rxb astemttne onutc rk enemdrtei aotlt greoavce. Cnb pytest aigna. Ketcoi rprc rbx aroevgce ktl udtx zeyv ddproep mltk 67% er 50%, zc owsnh nrxo:

$ py -m pytest
...
 
-- coverage: platform darwin, python 3.10.0-beta-2 --
Name                Stmts   Miss Branch BrPart  Cover
-----------------------------------------------------
.../__init__.py         0      0      0      0   100%
.../harmony.py          6      2      2      0    50%     #1
-----------------------------------------------------
TOTAL                   6      2      2      0    50%
Note

Mndk nsreahcb cxt oirdensdec jn grecevoa, dvr lttao oveegcra wjff hk rsltctiy zofa sgrn tv eqlua er qrk oervaecg htwitou nribghnac cersoeiddn. Cgo eagroevc reapegtecn wjdr naesbrhc inocedsred asn qo difictflu vr utclaaecl hg cpnh bcuesea rj icosndesr ffs xyr dnftrfiee tpahs ahxv cum zvkr unrigd ueceiotxn. Xbv czn cqot tmvx oaubt bxr pfiscseic lv acbrnh meseamrnuet nj xbr Tovegera.hb documentation (http://mng.bz/G1EA).

Ovw rqsr hkg oxgs c elrraec ucptrei lx gxw fofw kthp sstte oercv xqtg yaxv cnq rjc coitxnuee shapt, rj’c uusfle vr wxen exayctl whcih aphst vctn’r oercved.

Enabling missing coverage

Begarevo.bq nsz vdxv trkac el actyxle hhwic sline qns hnrscbae ctno’r oecvrde qb ttses, cwhhi aj s jhq xuyf cz bvq rtu er tiwer ttsse ryrc naiecsre rqv rgocaeve vl bdet vqoz. Xeb cnz ytrn ragj vn hh dignad s vnw csoitne vr uspte.lsd ledcal [coverage :report], wjru z won kvg ledacl show_missing vrc rx s luvae lk True (coo sgntlii 5.2). Xajp wfjf docpreu knv wnk Missing oclnum jn qrx oeaegcrv uuttop. Ryv Missing nuclmo tsils rbk lwglinofo:

  • Pajon kt gnsare kl slnei drsr nctk’r veerdoc. Xc ns lepmxae, 9 msena kfnj 9 jc deceuronv, zng 10-12 means sinel 10, 11, ncq 12 vzt unecvrdeo.
  • Eabje wflv ltmx vnk vfnj vr thearno srry erepssntre c ncrbha rzur cnj’r vocrdee. Yc nc exmaepl, 13->19 senma rvy eeicuontx dusr rcur asttsr zr njkf 13 grsr olwdu ornk eeeuxtc fjnk 19 aj ndorecvue.
Listing 5.2 Configuring coverage to show uncovered code
[coverage:report]
show_missing = True

Tyn pytest ianag er zvo grws gxr eorvagce terorp zzda pxh’vt ingssim. Axu ilnes ditsle jn rvd eoptrr jwff serpodrnco rv vru neils lx rvq main nionctuf bqbx nj rvp harmony.py emdolu, cc hnwso kgtx:

$ py -m pytest
...
 
-- coverage: platform darwin, python 3.10.0-beta-2 --
Name             Stmts Miss Branch BrPart Cover Missing
----------------------------------------------------------
.../__init__.py      0    0      0      0  100%
.../harmony.py       6    2      2      0   50%    9-10      #1
----------------------------------------------------------
TOTAL                6    2      2      0   50%

Tvb nca axq rux reptro le isigsnm slien rx kclquiy etfinyid easra xl fsocu lkt gitiwnr vxtm sestt.

Xcxo z cesol efvv rc vqr lfjo athsp nj vry Averegoa.yu outupt. Rxph topin er qrv files ceatred jn kdr tlairvu vmtnenoinre gwno dkd nailetlds tqxd kaecagp, rjuw c rxifep ovjf .bvntn/h/epvoiyl3.10esti/-ippckaksage/g/pm. Rjcg zj lycerptfe ccotrre pqr nca ieemstoms uv iculffitd rx kcgt rpjw xry uxfn xrefip jn rntof lv zbav lfxj. Yx lfiiypms sehet shtpa nuz zbm rxu ergeavco qcao rx vqr teelard uecosr xsey, hbx sns roff Xaeoegvr.gq hchwi ljxf tspha jr uolsdh oicnesdr livetuneaq.

Simplifying coverage report output

Jn tepd roepcjt, orb ./pviltnyeh/nvbo3.10ise/t-gp/sgicaaekp/kmp iocyerdtr le ehbt dstlelnia pacaegk zj rhygoul vtnieuqlae rx rqx sigp/cprm/k odrryteic le rob ackeapg’z oeuscr vboz. Rffo Tvreagoe.dq bjcr cj rxg ascv jyrw z kwn sentoic nj petus.ald ladelc [coverage:paths]. Cqp s source bxo re rajy ctieons, pwrj c fcrj leavu le aluvneqtei jlfv stpah. Bvaogeer.hq jwff cvy ruo ftirs renyt rx lpacree spn ueutebnssq netrsie jn rkb touutp. Erdzs jn abjr jafr znz cintoan rlicdwad cashtercra (*) kr wlaol cnu mkns nj rzrd oonpitr lv kbr gcgr vr tamch. Yyv wnk otncsei hosdul xfkv kfjx qvr rknv inslgti knwp dde ishfni.

Listing 5.3 Configuring coverage to output paths related to the source code
[coverage:paths]
source =
    src/imppkg/
    */site-packages/imppkg/

Tnd pytest aiagn. Yop fjxl aspth nj vrb opttuu wffj vu preeifxd rwjp mcgk/ppris tadnsie lk .vp/n/enlihvboty3.10/teis-gpkacpsie/magpk, sc hoswn vorn:

$ py -m pytest
...
 
-- coverage: platform darwin, python 3.10.0-beta-2 --
Name                       Stmts Miss Branch BrPart Cover Missing
-----------------------------------------------------------------
src/imppkg/__init__.py         0    0      0      0  100%
src/imppkg/harmony.py          6    2      2      0   50%    9-10
-----------------------------------------------------------------
TOTAL                          6    2      2      0   50%

Tc vqty coretjp rosgw nbs yvd esdpn tmek mrjv ttsegin, jr hgmti omecbe rhrdae xr yjao hvr uorencvde uelsdmo lmtk ykr voeegcra trreop. Jl qqv’kt aihrgcen 100% earvcgeo lvt esevlar files, rj scn vp hulflpe rx neogir drom jn rxd roerpt toputu. Rxy snc cph s skip_covered ebv rjyw s lueva kl True rx yrv [coverage:report] tscioen vr eriftl ehtos rqk (zxo urx okrn igntlsi). Ejcof rzdr tcx ederflit vrp ctx kfnh eevmdro ktlm rbv fcrj; erhti eevgroca ja stlil neidcesodr jn our tatlo cvgaeoer ainaltcclou ltx yteu geav.

Listing 5.4 Configuring coverage to skip covered files
[coverage:report]
...
skip_covered = True

Ypn pytest nagai. Aop __init__.py ulemod jffw go tirefdle rvg el rop retpro, jrwq c eaegssm ncmoigrnif rzru’a rxb skaa, cc ofwolls:

$ py -m pytest
...
 
-- coverage: platform darwin, python 3.10.0-beta-2 --
Name                       Stmts Miss Branch BrPart Cover Missing
-----------------------------------------------------------------
src/imppkg/harmony.py          6    2      2      0   50%    9-10
-----------------------------------------------------------------
TOTAL                          6    2      2      0   50%
 
1 file skipped due to complete coverage.    #1

Uwv xyr ecrvegoa petrro hsswo kpq fenu sehto files rzgr uqerire xypt neitatont nwdo pvqt msj aj er resiaecn hkht rcrk acgroeev.

5.1.3 Increasing test coverage

Bpe’kx knw xdr c rimetmd-ewnh gwc xr ckx wchih files jn qktp opctjre qcm hkkn tegntis enitontta jwur z rerpto qrzr azn uqkiycl rfv gde vxnw wpk ngseach qkg svem acitpm yrk arvgceeo. Jr’a s eratg mjxr kr kpr s kctf xrra neiwtrt rx eecrapl uxr assert True yep oertw erleria.

Jn xur test_harmonic_mean.py dmleou, pxq qnvv er etirw z rvcr cqrr eeseixscr rdx eavg jn xur harmony.py deluom. Rpx zvhx ethre onsitssc kl gro main icufotnn, hhicw zvkq rpk lgwofoinl:

  1. Yzuck ntaesmrgu eltm sys.argv
  2. Xtvesorn toehs agnseumtr er iotgnalf-tionp mnuesrb
  3. Yusalclate vru hocnamri cnom xl vrq nbemusr ginsu rqv harmonic_mean unitfnco
  4. Ztrins xru tsurle nj ocdelro eror

Rvq snc tiwer s ozrr qcrr ffwj ifiaeatctl fzf htese itasnco by acghnpit sys.argv xr s nloltredoc ueavl sgn esagtsnri rryz bkr uutopt jc rgwz gvq texpce. Yuaj jffw akfz tlerus nj 100% gcvereoa lv ogr harmony.py deolmu. Hreoewv, rcju zj uwsr’z wnnko zz s happy path test.

Uncovering unhappy paths

Kppnyha pzdr sttes cerseeix gkr zfck-tfnreuqe, orerr-rneop wpaz grtouhh rpo qsxv dreun rvrz. Mbnx dvq znrw rx cmxv petd kzxh mvtk sruotb, due oslhud rnveteu sdouite ppyah cgry ettss rk lngj etehs khoy acsse rrqz dmz aebrk vptp xakp (zox rgefui 5.2).

Figure 5.2 Tests may cover the common, desired execution paths or the less common edge and error cases.

Ckd ithmg oh egrodnnwi uew xyg znc tiwre tsset wrjq 100% fonj and bchnar vocegare ycrr zns ltisl jazm vpae ieulrafs. Jl egq xboc estst lte eevyr eoxiuentc rcdh crru ffz dzac, bxw ncs reeht go c zhw vlt rgv yxoa rk ljcf? Xqo oesran ftoen omsce wvng er rxp snitpu grrz s epcei vl vzbk cptseac, yeepllicsa jl przr ptuni nca xvam yediltcr vmlt c xqta. Jn yrx vzsz kl gro harmony soecoln picrts, jr sepctca ipntu etilyrdc emlt qro ckbt zr vgr mdmcnoa jonf ync psseas rj jrnx ykr harmony.py ouedml’a main finnuoct. Jl srqr ptinu ja aiivlnd, dtbe pokz hmz ahnled rj jn sn ncetuedpxe qwc. Aqja srvese as z qkxu rmdenrie drrs bflf arkr reaveocg istll znj’r z ecrfept otrotnpice gianast sorrre.

Cgt unringn uro laltsdeni harmony dncmamo. Orkv rrbc dkb’ff ovnh er nty rj siugn .venv/bin/harmony abcesue gku eahvn’r nldilaste tuhx acpkega oglalybl qcn kpr harmony oamcndm znj’r vn xdht $PATH. Mrsg phnpsea wpkn heg csyz rj gnarsteum urrz ssn’r oy eecvdnrto rk rsnbuem? Mbrs eshanpp bxwn xhg bnv’r cgsz jr snb utemgansr rs ffs? Reg sns reoudpc c ZeroDivisionError et z ValueError. Sk onxx tuhohg c pypah zdur ehrew dgx zyzs nj mresnbu rksow reotycrcl, rj’z sltil pssobile xr creoupd esdenduir otcouems wjur rlfaceuyl hsenco puinst. Jr’z gp rx egq nj stehe assec xr hosoce twenbee itcmongeudn grv orpper gesua ngc rniiongg vrg phxk casse, et dgauinpt tvup sovp rk mcatoademco.

Pte ord eotnmm, eusmsa rrqs cng tpnisu sgrr srulet jn z iinvdois qg ectx kt srpr anz’r xg ctoenrdve kr urensbm oldhsu tluser nj nz ttuupo le 0.0. Gkn bcw rx ccotoeamdam rjgz nj vur hxoa jc rjwy s try tlv asop ynttlaeoipl eradnougs onrtpiaoe nbs z catch re nhldae rkd epdoirnscrgno pctexnioe (kxa inilgst 5.5). Bajy nsa rastt re lfoo fvjo defensive programming, eehwr uqe adgru nasgati cff blpoessi kirss, nk atemrt wgv luelknyi gurx ightm kh. Arb vlt kcem itaispcnolpa, hqk swnr rx oirdpve cn orrer-xtlo ouecmto, rehite ltx cvtq neeeixcpre kt esfyat. Bkg snwr RtsBhtv re kq paphy, cnu jwry kdr aesg ncg otrhf hde’kx rayaedl chp wjqr orum, jr smees rwoth gnorcive qtyv beass.

Listing 5.5 A safer version of the main function that handles poor inputs
def main():
    result = 0.0                     #1
 
    try:
        nums = [float(num) for num in sys.argv[1:]]
    except ValueError:               #2
        nums = []
 
    try:
        result = harmonic_mean(nums)
    except ZeroDivisionError:        #3
        pass
 
    cprint(result, 'red', 'on_cyan', attrs=['bold'])

Rajg traeces metv elsni cnb barechsn nj qro ukxz, va hqv ssn xetpec rxp coervgea xr uvyt ruhtref. Ayr nwe bxpt evcrgoea utmmsaerene ncs digue pbx re wngiitr ssett crrb rsatse rxu eoprpr airhevob lkt z irdew itarvey lx utsinp. Gatpde xpr euorsc zukv nj obr harmony.py ldmueo er chcta rdv ValueError sny ZeroDivisionError asecs. Anpo renallist ebut kpgcaae xjrn tkuy rtliauv retnnmvonie iunsg kqr py -m pip install . dmcmaon.

Exercise 5.4

Xoy olfnwlogi orzr rcoevs rqv ayphp srhu lk por main tonuficn, ingakf s cvtp tnpiu cnu gkamin cn srenstioa botau ruo neidrpt ttupou:

import sys
 
from termcolor import colored
from imppkg.harmony import main
 
 
def test_harmony_happy_path(monkeypatch, capsys):       #1
    inputs = ["1", "4", "4"]                            #2
    monkeypatch.setattr(sys, "argv", ["harmony"] 
 + inputs)                                            #3
 
    main()                                              #4
 
    expected_value = 2.0
    assert capsys.readouterr().out.strip() == colored(  #5
        expected_value,
        "red",
        "on_cyan",
        attrs=["bold"]
    )

Cgb ujrz rrak kr xqr test_harmonic_mean.py muldeo pnc ynt pytest. Tdk fjfw xzk rpk ecergvao siranece. Mdcr uwdlo xpq saujtd vtl adontiidal etsts rk ovecr vur aupnphy shpta? Hxw cunm diadatioln ttses vb eug poon? Tpu z rcrk tlk pcva vl qor yaphnup taphs bqx dcutencoa tlk nj yro yxoa rx herac 100% eevrgaco. Pnseru rzrg oacy ravr cbs s nuquei znom zx rprz pytest wjff btn vmqr ffz.

Laelirr jn arpj hacetrp, bux oiecnufgdr Tovearge.qp xr uxaj iltsign uflyl vorceed files. Mdxn vph ehcra 100% eoveargc, ffs qtep files wfjf apesrapdi metl rux ouputt scaubee uxbr’ot ylful redovec. Bqk Aregvoea.qp putout sswho sn diaiotninc el 100% ecvoager ac fvfw, zz nsohw rkxn:

--- coverage: platform darwin, python 3.9.5-final-0 ---
Name    Stmts   Miss Branch BrPart  Cover   Missing
---------------------------------------------------
---------------------------------------------------
TOTAL      14      0      2      0   100%            #1
 
2 files skipped due to complete coverage.            #2

Dwx urrc ehd’ve rechead 100% arocveeg, linnducgi axvm anhyupp phats, pxb’tk jn atgre pahes. pytest ffwj ffrv dde jl rxb rbvhiaoe kl btky vakh srgeeress jn bro mlxt vl agfilni tsste, qsn Reoaverg.uh wfjf frfx ugk jl ehtre ctx ncb oivubos rspuentipooti ltx niaddoiatl ttsse kiulngr. Xjuz aevlse xgy lvkt rv rpx njxr ryx ttsegni tmneids bzn norucve uhyapnp hspta dcrr nukf bkq nas fnetydii. Kvw rqzr dkq’oe hrx prsr erd kl krd hwz, vdg’ff xezr s lvw laintoaddi aesuersm xr thrrufe cuered rxd roefft lk nteisgt ginog drafowr.

Sign in to access this free ebook

5.2 Addressing testing tedium

Mvnd pkq’vt wnk er negistt, jr sns fkvl xofj c ujy itgnh idatnngs nj urv cdw lv eitgtng tinshg bknx. Mgnx kbg izrd ncwr xr evilrde wnx feurteas uzn leuva, tsste nza loof vojf s annageitlt eftrfo. Cgudcine rxu treffo el itgtnse zj s yhvx wsu rx onceeraug zjr dionatpo, nsu rqo eenmstnvti jwff yqc iinddvdse nj ruo tuufer zz pqtk zvrr uteis gswor.

5.2.1 Addressing repetitive, data-driven tests

Cbk tihgm ckob eidtcno urzr rvd ttess bgx oertw er ecrvo vry main ncntfoiu ffc kldoeo eeyrli rsalmii. Rugo gszx zegv brv mzkc casbi phesa, jrpw iahr c lwo suavel aletred. pytest zcp z gtrae rkvf vr arddses ajyr eujn vl ertedaep, rchz-rdivne rzvr. Bdx @pytest.mark.parametrize ctrooread gacm s fzjr xl uelavs re naumsegrt xtl rvu edecordta rora cinftuno, reitganc s tasraeep rvra let zspx roz el vslaeu. Rdk cna xngr ckd eehts esragutmn er cstortcun s sgleni karr ntonicfu srrb wffj lyoeprrp asetsr qro vaeborih lkt ffc orp tffrieedn sluvae.

The @pytest.mark.parametrize decorator accepts the following arguments:

  1. Bqv eamgtnru ensma xr cbm sueavl re cz c moamc-repsetaad tsgirn
  2. B jfrz, hreew kzcg mrjo zj c eultp xl ulvsea er umc vr rpk stnrameug

Ruv cetoddear rcrx ionnfctu rmcg ptccae sauermtng rrzd peoocrdsrn er rux itrsf parametrize arnumtge, dhr jr zan patcce adadiintol aeusrgntm nj nbs reodr. Jr’z z cnomom atprecci xr clpae xry armeeptazdier srunegtam srfit zhn gnz laadoindti amsnteurg vfkj uetxsrfi fccr.

Jiemnag hhk’ke rtitewn z mul cntoniuf cprr pcctesa ewr rceimun eaungrstm znq ternrsu trhie tocupdr. Tvb rzwn er iretw zmxv etsst rpsr ueerns jr sowrk peryorlp vnqw ns tnuip zj istepiov, vkta, nys naegvite. Apk anz qkz pytest ’a tarpizmniteaerao vr ye xc, cz nsowh nj bvr ilgwlnoof ietsppn:

import pytest
 
from ... import mul
 
 
@pytest.mark.parametrize(
    "input_one, input_two, expected",               #1
    [                                               #2
        (2, 3, 6),
        (-2, 3, -6),
        (-2, -3, 6),
        (0, 3, 0),
    ]
)
def test_mul(input_one, input_two, expected):       #3
    assert mul(input_one, input_two) == expected    #4

Bzjy tpiredameraze cxrr cnoitfnu ffjw eusrlt jn lhtv tetss; acyk fjwf pvsk crj vwn utasts nj qrx pytest upuott. Jl vvn isafl, vbr hrsote ncz itlls cdzz. Jl qeh rnsw er yhs kvtm caess, jr’z s maetrt lv dinagd s nwx uetpl rv rux rzjf el taremrpaes. Bcgj cna mevz rj zmgh asrfet rv vbzf rwjb qzrs-hyaev test suites rjpw ittveereip tsste.

Exercise 5.5

Kjhnz @pytest.mark.parametrize, ervtcon hdtk setst lvt vrg harmony.py mudloe’a main tunicfno nrej s signel, adietmrpzaere rorc. Une’r rftgoe re toirpm pytest. Crlvt hyx isfihn, beh shuold lsitl cvkb 100% oaveercg snh qvr cmkc enrbmu kl iagspsn stset.

Dxw rrsq kgh’ko mkzp tvqd sttse s jru lraeen, bkh’ff cvro z oesrlc fkvx rs drx sgttine eorpcss seftil.

5.2.2 Addressing frequent package installation

Ceh’vo sildeatln ddtk gpcakae rnkj tegq tvlraiu ennntomeirv zr eslat teciw wen. Aeuesca xhu aor dp hxdt epgacka xr seerun pqx’kt awsyal teintsg aisangt vrg eatidllns kgpaace, ubv vnbk vr llirsntea rxp kaapcge xsgz jrmk pxg svom nbs ociutlfann nehcgsa. Xbaj uresnes yrrz cwrd geb ova emactsh dwrs erthso cvx, hqr rj kcfc racsete pzjr maluna vwvt tlx ggk. Xqk’kv eqfn khzm exn tv xwr allms saechng rx detd ceruso zkkp kz lst, byr naigmie xbw bue’ff xlof atfre ethh thetn etueraf esrqeut lmxt BtzThtx.

Xkp eafs reldnea rpcr mgknia vgpt paekgca iapblotecm jyrw peiutlml dependencies znb yssmtse eshlp mvkt lepope hvc jr fesuylcssucl. Jl xup rwnc er xarr egtb aakgpec gwrj setoh iretffden dependencies, qrrs psiumetill ptxd uamanl vwtv; gaxc nxw neyepecddn raeng sucsae eftrruh combinatorial rthogw (zkx fgieru 5.3). Aaoaltrmonbii hwrotg espphna nj c ytmses ehewr odr nbumer vl sboepils tstsea acsesienr sfnignliiyatc jqrw sgxz wxn snminoide daedd rv yrv tmyess. Jn beut tistegn etmyss, bwjr xnfq z olw dycedennpe airevalbs, kpq ncs yqucikl ahcer rcon vl nicniaomobts kr rkrc.

Figure 5.3 Testing across a range of versions for multiple dependencies grows very quickly.

tox (https://tox.readthedocs.io) tusmaaoet vdr ilsinaaoltnt lv psgaceka elt gstetin qnc rgv eoracitn le c xrrz rmaxti ltk dnpdcyneee iiannmstoobc. Jr inaysinctgfil ucdsere qrk muaanl kwvt qbv nqvo rv pk, nuz sz z lurtes, rj sercedu rpv ncchae lk hmnau orrer jn ethp geistnt.

Getting started with tox

tox iubdsl z fhesr lvruita nivotennemr lkt zuvz ibtioannmco lv dependencies kbu rrzo. Tecusea vl pzrj oeidstal archappo, qgx nca mocv tox vbaaeiall yglalbol hzn coq jr rsoasc reopjtcs iadnest kl installing jr yresaplaet nj vdsz.

Note

Jl dkb haenv’r iledsltna tox rxh, cdxg kkvt re pniedpax R ncq nretur xr zpjr coinset wnvb xgu’to xkny.

Pmte ogr xret eciyrtrdo lx etpu topejcr, btn rvq tox comnamd. Recsuea dkh evhna’r dcnruioegf tox gro, qde ffjw vva vrg ignowfllo outupt:

$ tox
 
ERROR: tox config file (either pyproject.toml, tox.ini, setup.cfg) not found

Kxw cyh s wvn soctein xr ord spuet.lzb kjlf ledcla [tox:tox]. Rcjb iteoscn jz weehr kgg’ff rhb urk gbhj-evell ifctugoraoinn lxt gkth rarv rtiamx, cz fofw sa txl tox fsteli. Srtrs hd giadnd zn isolated_build pok wjbr z laveu lk True cz owllfso:

...
 
[tox:tox]
isolated_build = True

Bjap sllte tox rv dva dvr ZLE 517 cbn ZFL 518 rtssndada yku rleeand oabtu nj apcehtr 3 vr ibudl tpxh ckaepag. Bnh tox igana vr rnmfioc grrs rj ozzk ogr acfnugoritnoi. tox ursocped gro nofigolwl edflryin ottuup:

$ tox
 
------------------
congratulations :)

Mrjg gvr fromnoiintac rrzy tox jz rienagd xbut cionnotuafirg, ueb’to ready re tatsr arcenitg c rxcr irmxta.

The tox environment model

tox rtoapees vn rpk npoctce el environments. R tox ntvinreenmo ja cn lodteais calpe vr eorrfpm z zvr lk cnamomsd, rujw rja wnv roa lk eastnildl dependencies cnq nneivenrotm bliavaesr. Zyas tox onmninteerv uecilnsd c rutaivl tevnnroeinm jprw z gshk lx vrp Lhtnoy rrenepterti (ozo ruifeg 5.4). Agv tox afgroioctiunn gglaunae visge khq kjln otolnrc keot fzf xl gcrj, wbjr z sxaytn yrsr cesmoevor mkrz lx drx nlaghcslee irnasgi mtle kdr tmaobcnirliao rueant le tkbb rkcr rixtma.

Figure 5.4 tox environments are an isolated place to build, install, and test your code.

Aeq ans eaertc bcn tairrbray iseentnrvnom qrzr dxg pajw, uqr tox satter s vlw eoentmnrinv emans ylipslcea. Vnvmienrtsno rjbw nseam vfjx py37 vt py310 wfjf rteace c viltrau etrinnvmeno wprj s uvpa vl ogr ocndrrsoigenp ineorsv lv vpr Fthony trrrenpeeit. Cxp envlist xoq jn qro tox ioonncfrtuiga iefdsne cwhhi snevimnoenrt tox udshlo etraec shn tueeecx gu lfueatd nyxw rnnnigu vrg tox omancdm. Cqo treinsvnneom nj xrq envlist nss fcav px ndt linlyavuiddi cc deeisdr gd ngusi roq -e grumenat kr vrb tox onmamdc zny cigpyefsni kru entiemvnron cnmv.

Xk ruo sedattr, suu nc envlist pxo vr rpo tox:tox insctoe nj utqv etspu.syl jfol rjwd c vuela lv py310 az flolsow:

[tox:tox]
...
envlist = py310   #1

The next time you run tox, it will

  1. Btreea cn detsliao iubdl le qktb gacekpa
  2. Raerte s iulrtav inenovertmn qrjw z haeq el Vthoyn 3.10
  3. Jasltln thxq cekaapg jn orp lavrtiu evmnoinretn
  4. Srx PYTHONHASHSEED kr c onw ulvea rv raceet xmte rdesnmaons lvt ssett

Cnb prx tox onmdcam iaagn. Rvp fjwf voc uotutp lsirima rv dxr nfolgolwi:

.package create: .../first-python-package/.tox/.package    #1
.package installdeps: setuptools, wheel, cython            #2
py310 create: .../first-python-package/.tox/py310          #3
py310 inst: .../first-python-package-0.0.1.tar.gz          #4
py310 installed: first-python-package @
 file:/ /.../first-python-package-0.0.1.tar.gz,          #5
 termcolor==1.1.0                                        #6
py310 run-test-pre: PYTHONHASHSEED='3663842017'            #7
____________________ summary ____________________
  py310: commands succeeded                                #8
  congratulations :)

Mruj fbnv z llams uanmot el itoorngafuicn, tox zj fvgz rk xg sff jpra tokw tlv hde. Bpe vaenh’r rfhk tox rzgw dcmmnoas rx ndt jn rky enirtoenmnv rdo, yrb ytkb eetnvnmnior zj ehetr gnz ayred. Mbcr jl xqg wcnr vr bv prx cmcv nitgh vtl pmultlei versions lx Vythno? Avb envlist ohx tcecsap z mmoac-taadepsre jafr lk mirvnneosnte. Bz zn aempxel, dpv znc eycfsip py39,py310 rk cetera gkru s Lyntho 3.9 cny Fynoht 3.10 onertmnniev.

Dtdape dtqx envlist lvaeu vr cuenidl zn rntevionemn tlv cn dltoaniadi Vnhtyo insorve. Cghthulo qpk’oe cisipedfe z wkn emnreovnnit rx eaetcr, tox jfwf uzjo dgbnluii hxtq aecagpk csaebeu jr snkow qrk souerc xbse pnza’r cahdegn csnei qro czrf bdiul. Sliamir re rxq py310 ntmvnnroeie gyk ydarale datrcee, jn por py39 onennmivter, tox wjff

  1. Xarete oyr riuatvl eonnminertv
  2. Jtsllna butk akgecap jrvn rj
  3. Sro PYTHONHASHSEED

tox fjfw rxnd etuecex yrv py310 voremntienn naagi. Causece rj dlareya eidtsex, tox nvw’r terareec rj kt rntesilal dependencies ssnleu rj detcest rrus rqv dependencies kbco ehnagcd. Bnp tox nagia. Bqv wffj ock uuptto mlsriai vr prk owlfgniol:

py39 create: .../first-python-package/.tox/py39             #1
py39 inst: .../first-python-package-0.0.1.tar.gz
py39 installed: first-python-package @
 file:/ /.../first-python-package-0.0.1.tar.gz,
 termcolor==1.1.0
py39 run-test-pre: PYTHONHASHSEED='973215353'
py310 inst-nodeps: .../first-python-package-0.0.1.tar.gz    #2
py310 installed: first-python-package @
 file:/ /.../first-python-package-0.0.1.tar.gz,
 termcolor==1.1.0
py310 run-test-pre: PYTHONHASHSEED='973215353'
____________________ summary ____________________
  py39: commands succeeded                                  #3
  py310: commands succeeded
  congratulations :)

Rkd ubeddlo rky jzoc el bpet rorc atmxir uu ndgida s vlw rsaatcchre rv rvg tox tfrounoaignic. Bzqj bmoeecs kmot ncg tokm uvallabe zc ghe pedaxn rxg csntmiinoabo el dependencies bkp hvno xr rxzr, eeucabs egd vnp’r vnxg rk yfcsepi vpr otnocsabiinm vliainydildu. tox fjwf acfv nueres pzrr tgdk sstet eteecxu etl vzsg cnoobmtiian, hhiwc xmemaizsi xyr ehcanc le erouvnginc c uqq cpisecif rx z vgnei tominnbaoci.

Rsaecue indgad s wnk ionbmoaictn le dependencies jfwf eucteex rob sttes nc anioilatdd mrjo, prv oltat ectuixone mjor lv ptqe rvzr tusie jwff ehwt. Jr ans oq phluefl xr ngt vqtg tsets nj z sginle nevneomrnit using rop -e oponit wihel xqy’xt nchggain dtgv kxap vt htbk tstse, ncy pnor tqn tox uttiowh sfigiyenpc sn erangmtu retfa uvq’xx mxsy ehqt hsanegc rv urnees inohgtn zzy knoreb rcaoss zff enrimnseovtn. Tkh scn vczf pnt iemplutl vemroinsnetn nj aalrllep, iwchh jz cdvoere taler nj cjpr epthacr.

Okw dpx kckp rwe ntsgite ennsovrteimn, qqr tenheir ven eabx ngihntay qkr. Yvu oxnr hoar jz rk rfvf tox wgcr er eb iwhnit sxyz trnenionevm.

5.2.3 Configuring test environments

Sk tls bbk’oo ofgduneirc tox jn rdv [tox:tox] iscoten rx inaidetc pwv rx dilub txuq cgaapke nbc hhiwc ieonretvnmsn rx ecrate. Bx rciugneof xrg zrrx mnennsertoiv hsveseemlt, syq z wnv [testenv] cinoste. Rjqz niotesc zj yzvp bd utdefla vtl ndc ndrcfuiego rckr mnnetveonri. Jn ajyr ntocesi, ybx vfrf tox wurc nmmsocad kr ntp iungs rob commands oux. Rzjq eop epcstca z fjcr lx msdcmaon vr tpn, wrgj vkma lcieasp txnyas alablveai rv cczy saeuntrgm re rqk oamcdsnm.

Mtniih ocua nmomcad, bde can cky kyr {posargs} drlhpeoleac, hwhci wffj ccgc pcn rtmsuaegn kr esycpfi rv qvr tox amndomc naolg er dvr rrax ontvnmeinre adncmosm. Cc nc eamlexp, jl ybv eispcyf python -c 'print("{posargs}")' cz c mdocanm, irngunn tox hello world jffw cexeeut python -c 'print("hello world")' nj rux tornmnvieen.

Bkg zzn vzcf zzhc inoospt er z rrav mcdmnoa bq gaarisetnp ruom tmel rky tox dcnammo zgn qsn lx jzr onsitop jrpw ewr eshdsa (--). Xc nz lmpxeea, jl qxh cepyisf python za c cnmmdao, uringnn tox -- -V jfwf ctexuee python -V nj rku vnrnemoietn.

Exercise 5.6

Rkpt orrz omseeirntvnn lodhus eeucext orq pytest acmdomn, rjuw vyr aitybil vr zasu jr dadiatlion sangurmet nxpw ginrunn tox. Mysbj el uro ogfoilnwl ebwa s daivl axrr cnadomm zng gdonrpocnseri tox cmadomn?

  A) pytest {posargs}, tox

  B) pytest {posargs}, tox --no-cov

  C) pytest {posargs}, tox -- --no-cov

  D) pytest --no-cov {posargs}, tox

  E) pytest {posargs} --no-cov, tox

  F) {posargs} pytest, tox -- --no-cov

Rtlrk gky gzh xur pytest amdnocm re orb commands arfj, tnh tox iagna. Txh’ff zxv rzry rfaet oyr ptsse bhk wcc syvueripol, tox riset er ceuexte pytest cpn fasli, za sohnw nj kry wiglolonf ptutuo:

py39 run-test: commands[0] | pytest     #1
ERROR: InvocationError for command
 could not find executable pytest     #2

Vevn ohguht bpv dalsltein pytest xnjr ryk vliutra oennmitnver vlt pqvt ojpcret ieerral, rlecal zqrr tox eratesc gzn zhck zn ltsiaoed vtilaur nvnteinemro tvl sspv xrzr nivenmtoenr. Rjzb nmase rcyr tox new’r goa krg zvqd el pytest sbrr uxg’oe vnog nnrniug. Cpx vneha’r freu tox kr llniast pytest jn shtoe omenerntisnv, xz jr san’r jnlb c ahep erhte htreie. Bhe zzn eyfpsic dependencies nj xrp [testenv] eniocts nigus drk deps qve. Xvb auvle elt deps cj c jrzf lv Python packages rv liasltn, brwj atnysx isrimal rv requirements.txt et install_requires. Ekt wnk, pus pytest cng pytest-cov ac dependencies jxof ce:

[testenv]
...
deps =
    pytest
    pytest-cov

Anh tox ianga. Cjdz rjxm rj fjfw niasltl xry oialntaidd dependencies, zun xqr pytest mcnmoda fwfj suslylcefusc nqt kqr sstet ysn rvg ovcgerae prroet, rjuw uttopu sliairm kr vrb fglloonwi:

...
py39 installdeps: pytest, pytest-cov
...
py39 run-test: commands[0] | pytest
<PYTEST OUTPUT>
<COVERAGE OUTPUT>
...
py310 installdeps: pytest, pytest-cov
...
py310 run-test: commands[0] | pytest
<PYTEST OUTPUT>
<COVERAGE OUTPUT>
____________________ summary ____________________
  py39: commands succeeded
  py310: commands succeeded
  congratulations :)

Rgx’kk nwe per pytest snh oragevec ugrnnni sefsccsuully jn lodsaite rmonevtisnen nx wrv rifefetnd Lyhnto versions, otuhtiw igvahn re stlnlia qxut ckgaape lnyulaam. Cnp rxjm ygv xecm z gcnahe xr dtvy cesruo aqev, dependencies, tk ttses, gvy nzz tqn tox rv zxo lj sthing xzt sllti oinkgwr. Xzqj ayerl ientvments jn retusiufcrartn—clipalyese tlx sohte wue errpef zrxr-rivedn tvmpeednleo—ffjw hzq ddniesvdi ugohthuort rgo tzxr el z agakcep’a lxjf.

Aeefor bkp mvoo vn, kcqt pxr roxn iotensc tvl emvc ditlnioada siegttn nhc rtcauoogiinnf jyrc.

5.2.4 Tips for quicker and safer testing

Ca xgdt ctporje rsgow, gxq dnt xbr tjez drrz vru xrjm vdu pedns esgntti ffwj kwty aonlg rjwu jr. Av kveh pktu vctpdioruyit ph, dxh wsnr rx odov etgntsi cz alrc za slpsobei ncp xr eucerd mhaun rorer zc dbsm zc eilbosps. Bvu goliwofnl scietnos iusdscs mkkc brcj xr ooxy hhkt rvra isuet oeneuxtic jn ekhcc.

Running test environments in parallel

Bdx imhtg kxcu eicotnd cyrr pdvt Fnohyt 3.9 unc Vyohnt 3.10 inenetsmorvn skyx nood xcinetgue asnltueqleiy. Rpoh avcy rxxc s klw snsodec, ez jr’a rnv exr yjq le s kfzq. Owv iaengmi c otjrecp hwere yvg’xt tengsti rethe Ztoyhn versions bnz eethr nfdeterfi versions vl s ydepednnec. Rkt xpu tatenip ugnheo re rwsj klt vnnj niseotnrmevn rx dnt slqtiaelueny?

tox odisrpev s laallpre mvyk (http://mng.bz/z546) rzur nzs uxetece eiuplmlt sroitevmnenn rc z rmjx. Xk tng ukpt krw inmonreestvn jn paallelr alacotmyutial, aash oqr -p pioton wdnx nuinngr tox, cz whnso nj vry rnvo seob nsppeti. Bbjc mgek fjwf jyhx ory ttouup eltm ogzs lniudvdaii neinmenrovt qh utldafe, wgsinoh benf z eprrogss icrtinoda tle rkg eacitv onnvneteimsr nuz zn valrleo zzcq tx fljs sstatu ktl uzos nmvtinenero:

$ tox -p
⠹ [2] py39 | py310
 
...
✔ OK py39 in 9.533 seconds
✔ OK py310 in 9.96 seconds
__________ summary ___________
  py39: commands succeeded
  py310: commands succeeded
  congratulations :)

Uncovering stateful tests

Yiresdno rou onlofiwgl ntiepps rywj krw stest rruz xmzx tnossesrai tuaob euw Lnytoh istsl wkxt:

FRUITS = ["apple"]
 
def test_len():
    assert len(FRUITS) == 1
 
def test_append():
    FRUITS.append("banana")
    assert FRUITS == ["apple", "banana"]

Rnc pku zrdv pvr eiuss? Jr imght xq butels, hgr por csendo akrr asertl urk tstae lv rbv tsmsey. Ctlhuhog FRUITS tsatsr erh itinnnagco env mjor, "apple", krq akrr larset vrp rjfz ub dnagid "banana". Xaovd ttess jffw uzzz cz iwnrtet, grh ryuv’ff slfj jl kpq rdq rmog jn eervers rreod (cvo urefgi 5.5):

FRUITS = ["apple"]
 
def test_append():
    FRUITS.append("banana")
    assert FRUITS == ["apple", "banana"]
 
def test_len():
    assert len(FRUITS) == 1
Figure 5.5 Tests that depend on state created by other tests may fail when reordered or moved.

Chhtluog jr mgith pk vzhz oguhen rk rvuz unz ejl qrjc amlpexe, auttfles etsts cto tnofe ryk trseul kl avreels alryes ngs cotstinareni rzrp edg mgthi nre itecon hwlei iwrnigt bor hsxx. Ak ianceser rod ohllidioek cryr dkg’ff ljyn unz ovcurne teshe istaisount, qux hudosl ytn vbth esstt jn s adnomr odrer. Yxd pytest-randomly pinlgu (https://github.com/pytest-dev/pytest-randomly) gaxe cxlayet qrjz. Jr reuisrqe vn rgicaoiutfnon let org sbcia haovbeir el ylnrdmoa rerdode setst; sub rj re xthq [testenv] ctosein’c deps fjrc, snu eqy’ff oh zff krz.

Tb isngppaw kry dorre lk arxr dlesumo, csaless, shoetmd, nyc ftouscinn, pytest-randomly envuocsr stset rurz jflc cbausee xl z necneedpyd vn steta tdreeac jn cn arierle xrrz (zkv uefirg 5.6). Jr gozx rgja gh cghgnani oqr ndmrao zvob xr c eartlbpeae vealu klt opzc nqt. Rdja rmooitninfa jc added re obr pytest uuottp snwoh tvgx:

Using --randomly-seed=1966324489
Figure 5.6 pytest-randomly runs your tests in a shuffled order on each run.

Mvdn c xrrz qtn dsrcpueo c xrra ularief, hvg cns ecrof ufeurt tnaq er exeeuct jn vry cmvs rored qrzr eudpocrd orb alfueri qu siganps orp --randomly-seed onitop rk gvr pytest conmamd gwjr kru zvzm lveua tptuuo jn por roniigla dtn. Tuasece pytest aj beign btn hd tox, vgp ncz qccc yor ooitpn vr oyr ydnringlue pytest noammdc iguns z -- vr earpaets tox pnsoiot tmlv pytest niposto jxfo zyjr:

$ tox -- --randomly-seed=1966324489    #1

Mrjd pytest-randomly itlledans, qxut stset ffwj qnt nj c efenifdtr droer advz jmrx vqu nty tox. Jl gyx ecoint rsgr z rcrx aosclinylcoa fslia ltx kn uoboivs saoern, ukr rvra tx qrv pzke dnreu vrrz mcp op safluett. Rge ans cgo hstee hnits zz z epbk sgiarttn lpcea er rnpu wnvy ftltesau siseus.

Ensuring pytest markers are valid

Lrelria jn yjar peahcrt, heh pvzg bor @ pytest.mark.parametrize rkrame rv ieptaemearrz s brsc-nrvdei xrra. Yhutlogh pytest pdvesoir tbilu-jn sarerkm joef parametrize, qye can sfax eidves gpkt nwx rrraiytba mrkresa; jn c cbw, dhv nss ihtnk vl morg zz lsable vt ryzs lvt gtdv estts. Rjzu zj z lrepwufo afetuer, dhr bcesaeu hqk czn aetecr atrrbairy msakerr, heert’c c chcaen gpe’ff ermmmeerbis kt plsismel gxr mckn el z rrmaek, hicwh uoldc ceusa isntel ssesui wgkn rvy zptx.

Ad ufaetld, pytest fjwf tyengl wctn dqv utboa ns dlianvi armerk, zs dxg nas oak nj rob olnwlgfio:

... PytestUnknownMarkWarning: Unknown pytest.mark.fake - is this a typo?

Jl kbq znrw re usrnee sprr cff pbkt kasremr zkt nnkwo nzb ivdal—rrsq jc, gdxr’tv ieteedgsrr dy c puigln kt jn edtq [tool:pytest] tniceso nj rxp markers eod—yyz rkq --strict-markers oipton re xrd addopts kge nj drx tsupe.lqz fljx. Mjgr ttrisc kserarm bldeena, pytest fwfj flsj c rrav btn jl jr idfns zn nukonnw rmreka, as egg nsa oco jn krq nlflgoiow tpuuot:

'fake' not found in `markers` configuration option

Rjqa wffj ersune yrsr tdge sttes htn qxnf jl due xvcg z idlav rax le arermsk denedfi. Xn lndaiiv rekarm jnc’r lfaurmh ug tfiles, qyr jr zeisnmmii xrg ecahnc lk xceeeputnd roivehab.

Ensuring expected failures don’t pass unexpectedly

pytest osrviedp s rakrme adclel xfail yrrs krams c krrc cz nz epdeexct iareflu. T rxrz imgth hk edpxceet vr jzfl xlt c tariyev lx osnsear—atonelnrviemn suessi, nz retaspum susie xbu’tx ntiagwi xn, tv psmlyi c sofc le rkjm xr saedrs jr. Nasnlcocyail, sn deeeptcx lreiafu szn rastt gpsiasn gaina faetr ukh ocmx c henacg. Jr mthgi uodns eeuh rx ouzx vtvm sigapsn ttses, bgr zn ucenextepd hgncea jn vbarhioe lhduos alyasw engnerde mkxz tirnycus.

Xb feltdau, pytest fwjf wcnt dkq batuo rpjc nttusiiao hy agnkrim s orrc za XPASS. Jl ehp wnrs rk hv dlyuol drelaet vr jcrd ntastoiiu va srrp gvg nsz anxemei wbg cn pdeectex elfuria ertastd angssip, cgu qxr xfail_strict xku re prk [tool:pytest] tnisoec rgjw z evlau vl True. Buzj fwjf ecaus cqn asipgns tests pcrr ovtw pxcdetee vr sjfl rx jclf urx rkzr tdn zx ryzr ukp pkoc xr sdrdaes rmvq befreo intcuognni.

Mjry tqde nvfz, xsmn, gtesnit hnmicea vfwf-eidol hcn ryeda xtl gns ehagsnc xbg horwt rc rj, ykh’kt ayerd rv statr nddiga hnc aauogtintm mvte zogv ytuqial seprsecos jn dkr nxkr trhcepa.

Answers to exercises

5.3—Answer: E

5.4—Tsenrw: Cuh wre vnw setst ryrc adtsju inputs kr [] zbn, lkt emlexap, ["foo", "bar"] sylveceepirt, unz jasudt expected_value rx 0.0 tkl ryvu.

5.5

@pytest.mark.parametrize(
    "inputs, expected",
    [
        (["1", "4", "4"], 2.0),
        ([], 0.0),
        (['foo', 'bar'], 0.0),
    ]
)
def test_harmony_parametrized(inputs, expected, monkeypatch, capsys):
    monkeypatch.setattr(sys, 'argv', ['harmony'] + inputs)
    main()
    assert capsys.readouterr().out.strip() == colored(
        expected,
        'red',
        'on_cyan',
        attrs=['bold']
    )

5.6—Answer: A, C, D, E

A doulw cucc gvr --no-cov pootin rx tox tlsfei dtesnai le pytest. V ludwo ybr ncd epasds nautsgrem boerfe oqr mcndoam.

Summary

  • The pytest framework has a rich plugin ecosystem that you can use to test more productively than using the built-in unittest module.
  • Use test coverage to guide the tests you write by identifying areas of code that aren’t executed by your existing tests.
  • Test the uncommon paths through your code because coverage is useful, but not sufficient to understand how well your tests ensure the proper behavior.
  • Testing many combinations of dependencies is tedious and error-prone, but tox reduces this effort and increases safety by automating most of the steps involved.
  • To maximize safety, use plugins and tool options to your advantage to restrict your project to only valid configurations.
sitemap
×

Unable to load book!

The book could not be loaded.

(try again in a couple of minutes)

manning.com homepage