A Brainstorm Before Starting

  |   Source

A Brainstorm before starting

Taken from: Julien Danjou's Hacker's Guide To Python

In [2]:
import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
In [1]:
import sys

sys.builtin_module_names
Out[1]:
('_ast',
 '_bisect',
 '_codecs',
 '_codecs_cn',
 '_codecs_hk',
 '_codecs_iso2022',
 '_codecs_jp',
 '_codecs_kr',
 '_codecs_tw',
 '_collections',
 '_csv',
 '_datetime',
 '_functools',
 '_heapq',
 '_imp',
 '_io',
 '_json',
 '_locale',
 '_lsprof',
 '_md5',
 '_multibytecodec',
 '_pickle',
 '_random',
 '_sha1',
 '_sha256',
 '_sha512',
 '_sre',
 '_string',
 '_struct',
 '_symtable',
 '_thread',
 '_warnings',
 '_weakref',
 '_winapi',
 'array',
 'atexit',
 'audioop',
 'binascii',
 'builtins',
 'cmath',
 'errno',
 'faulthandler',
 'gc',
 'itertools',
 'marshal',
 'math',
 'mmap',
 'msvcrt',
 'nt',
 'operator',
 'parser',
 'signal',
 'sys',
 'time',
 'winreg',
 'xxsubtype',
 'zipimport',
 'zlib')
In [3]:
from itertools import permutations

print(sum(1 for _ in permutations([1, 2, 3, 4, 5])))
120

"Whenever you are about to write our own function to handle a simple task, please stop and look through the standard library first."

"There’s a few of the standard modules that you should definitely know about:

  • operator: provides functions implementing the basic Python operators which you can use instead of having t owrite your own lambda expressions
  • os: provides access to basic OS functions.
  • random: provides functions for generating pseudo-random numbers.
  • re: provides regular expression functionalities.
  • functools
  • iterators
  • uuid: allows to autogenerate UUIDs(Universall Unique Identifiers)."

Fixtures

"In unit testing, fixtures represent components that are set up before a test, and cleaned up ater the test is done. It’s usually a good idea to build a special kind of component for them."

Basics

In [ ]:
## Unittest setUp() and tearDown(). Running code before and after unittest

import unittest

class TestMe(unittest.TestCase): 
    def setUp(self): 
        self.list = [1, 2, 3]
    
    def test_length(self): 
        self.list.append(4) 
        self.assertEqual(len(self.list), 4)
    
    def test_has_one(self): 
        self.assertEqual(len(self.list), 3) 
        self.assertIn(1, self.list)

fixtures module

"The fixtures module provides a few built-in fixtures, like fixtures.Environment Variable – useful for adding or changing a variable in os.environ that will be reset upon test exit."

In [9]:
import fixtures # not part of the standard library
import os

class TestEnviron(fixtures.TestWithFixtures): # inherits from unittest TestCase
    def test_environ(self): 
        fixture = self.useFixture(fixtures.EnvironmentVariable("FOOBAR", "42")) 
        self.assertEqual(os.environ.get("FOOBAR"), "42")
    def test_environ_no_fixture(self): 
        self.assertEqual(os.environ.get("FOOBAR"), None)

Mocking

"Mock objects are simulated objects that mimic the behaviour of real application objects, but in particular and controlled ways. These are especially useful in creating environments that describe precisely the conditions for which you would like to testcode"

In [11]:
from unittest import mock 

m = mock.Mock() 
m.some_method.return_value = 42 

print(m.some_method())

def print_hello():
    print("hello world!") 
    return 43 

m.some_method.side_effect = print_hello 

print(m.some_method())

m.some_method.call_count  # stats on method calling
42
hello world!
43
Out[11]:
2

Example: Using mock.patch to test a set of behaviour

In [ ]:
import requests 
import unittest 
import mock

class WhereIsPythonError(Exception): 
    pass

def is_python_still_a_programming_language(): 
    try: 
        r = requests.get("http://python.org") 
    except IOError: 
        pass 
    else: 
        if r.status_code == 200: 
            return 'Python is a programming language' in r.content 
    raise WhereIsPythonError("Something bad happened")
        
def get_fake_get(status_code, content): 
    m = mock.Mock() 
    m.status_code = status_code 
    m.content = content 
    def fake_get(url): 
        return m 
    return fake_get

def raise_get(url): 
    raise IOError("Unable to fetch url %s" % url)
    
class TestPython(unittest.TestCase): 
    @mock.patch('requests.get', get_fake_get(200, 'Python is a programming language for sure')) 
    def test_python_is(self): 
        self.assertTrue(is_python_still_a_programming_language())
        
    @mock.patch('requests.get', get_fake_get( 200, 'Python is no more a programming language')) 
    def test_python_is_not(self): 
        self.assertFalse(is_python_still_a_programming_language())

    @mock.patch('requests.get', get_fake_get( 404, 'Whatever')) 
    def test_bad_status_code(self): 
        self.assertRaises(WhereIsPythonError, is_python_still_a_programming_language)

    @mock.patch('requests.get', raise_get) 
    def test_ioerror(self): 
        self.assertRaises(WhereIsPythonError, is_python_still_a_programming_language)

Scenarios

"This package provides an easy way to run a class test against a different set of scenarios generated at run-time." Same example as in Mocking, the test runs three times because three scenarios are defined:

In [ ]:
import mock 
import requests 
import testscenarios

class WhereIsPythonError(Exception): 
    pass

def is_python_still_a_programming_language(): 
    r = requests.get("http://python.org") 
    if r.status_code == 200: 
        return 'Python is a programming language' in r.content 
    raise WhereIsPythonError("Something bad happened")

def get_fake_get(status_code, content): 
    m = mock.Mock() 
    m.status_code = status_code
    m.content = content 
    def fake_get(url): 
        return m 
    return fake_get

class TestPythonErrorCode(testscenarios.TestWithScenarios): 
    scenarios = [ ('Not found', dict(status=404)), 
                  ('Client error', dict(status=400)), 
                  ('Server error', dict(status=500)), ]
    
    def test_python_status_code_handling(self): 
        with mock.patch('requests.get', get_fake_get( self.status, 'Python is a programming language for sure')): 
            self.assertRaises(WhereIsPythonError, is_python_still_a_programming_language)

Profiling

In [7]:
import cProfile
import time

def print_hello():
    def print_it():
        for _ in range(0, 5):
            time.sleep(0.2)
        print('here')
    print("hello world!") 
    print_it()
    return 43 


cProfile.run('print_hello()')
hello world!
here
         501 function calls (497 primitive calls) in 1.005 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    1.005    1.005 <ipython-input-7-04c863fb3a13>:4(print_hello)
        1    0.000    0.000    1.004    1.004 <ipython-input-7-04c863fb3a13>:5(print_it)
        1    0.000    0.000    1.005    1.005 <string>:1(<module>)
        4    0.000    0.000    0.002    0.001 __init__.py:274(dumps)
        1    0.000    0.000    0.000    0.000 attrsettr.py:35(__getattr__)
        1    0.000    0.000    0.000    0.000 cp1252.py:11(encode)
        4    0.000    0.000    0.000    0.000 encoder.py:120(__init__)
        4    0.000    0.000    0.002    0.000 encoder.py:248(encode)
        4    0.000    0.000    0.000    0.000 encoder.py:277(iterencode)
        4    0.000    0.000    0.000    0.000 encoder.py:380(_make_iterencode)
       26    0.000    0.000    0.000    0.000 encoder.py:41(encode_basestring)
        1    0.000    0.000    0.000    0.000 encoder.py:51(replace)
       60    0.000    0.000    0.001    0.000 encoder.py:513(_iterencode_dict)
    64/60    0.000    0.000    0.002    0.000 encoder.py:600(_iterencode)
        1    0.000    0.000    0.000    0.000 hmac.py:114(hexdigest)
        4    0.000    0.000    0.000    0.000 hmac.py:75(update)
        1    0.000    0.000    0.000    0.000 hmac.py:82(copy)
        1    0.000    0.000    0.000    0.000 hmac.py:95(_current)
        5    0.000    0.000    0.000    0.000 iostream.py:102(_check_mp_mode)
        1    0.000    0.000    0.000    0.000 iostream.py:123(_flush_from_subprocesses)
        1    0.000    0.000    0.003    0.003 iostream.py:151(flush)
        4    0.000    0.000    0.003    0.001 iostream.py:207(write)
        1    0.000    0.000    0.000    0.000 iostream.py:238(_flush_buffer)
        1    0.000    0.000    0.000    0.000 iostream.py:247(_new_buffer)
        6    0.000    0.000    0.000    0.000 iostream.py:93(_is_master_process)
        1    0.000    0.000    0.000    0.000 iostream.py:96(_is_master_thread)
        4    0.000    0.000    0.002    0.001 jsonapi.py:31(dumps)
        1    0.000    0.000    0.000    0.000 jsonutil.py:102(date_default)
        1    0.000    0.000    0.000    0.000 poll.py:77(poll)
        1    0.000    0.000    0.000    0.000 py3compat.py:18(encode)
        1    0.000    0.000    0.000    0.000 session.py:195(msg_header)
        1    0.000    0.000    0.000    0.000 session.py:200(extract_header)
        1    0.000    0.000    0.000    0.000 session.py:441(msg_id)
        1    0.000    0.000    0.000    0.000 session.py:493(msg_header)
        1    0.000    0.000    0.000    0.000 session.py:496(msg)
        1    0.000    0.000    0.000    0.000 session.py:515(sign)
        1    0.000    0.000    0.002    0.002 session.py:530(serialize)
        1    0.000    0.000    0.003    0.003 session.py:589(send)
        1    0.000    0.000    0.000    0.000 session.py:655(<listcomp>)
        4    0.000    0.000    0.002    0.001 session.py:84(<lambda>)
        1    0.000    0.000    0.000    0.000 socket.py:289(send_multipart)
        1    0.000    0.000    0.000    0.000 threading.py:1055(ident)
        1    0.000    0.000    0.000    0.000 threading.py:1204(current_thread)
       13    0.000    0.000    0.000    0.000 traitlets.py:395(__get__)
        1    0.000    0.000    0.000    0.000 uuid.py:104(__init__)
        1    0.000    0.000    0.000    0.000 uuid.py:230(__str__)
        1    0.000    0.000    0.000    0.000 uuid.py:563(uuid4)
        1    0.000    0.000    0.000    0.000 {built-in method __new__ of type object at 0x000000001E350250}
        1    0.000    0.000    0.000    0.000 {built-in method charmap_encode}
        1    0.000    0.000    1.005    1.005 {built-in method exec}
        1    0.000    0.000    0.000    0.000 {built-in method get_ident}
        7    0.000    0.000    0.000    0.000 {built-in method getattr}
        7    0.000    0.000    0.000    0.000 {built-in method getpid}
        1    0.000    0.000    0.000    0.000 {built-in method hasattr}
        4    0.000    0.000    0.000    0.000 {built-in method id}
      135    0.000    0.000    0.000    0.000 {built-in method isinstance}
        8    0.000    0.000    0.000    0.000 {built-in method len}
        1    0.000    0.000    0.000    0.000 {built-in method locals}
        1    0.000    0.000    0.000    0.000 {built-in method max}
        1    0.000    0.000    0.000    0.000 {built-in method now}
        2    0.000    0.000    0.003    0.002 {built-in method print}
        5    1.001    0.200    1.001    0.200 {built-in method sleep}
        4    0.000    0.000    0.000    0.000 {built-in method time}
        1    0.000    0.000    0.000    0.000 {built-in method urandom}
        3    0.000    0.000    0.000    0.000 {method 'append' of 'list' objects}
        4    0.000    0.000    0.000    0.000 {method 'clear' of 'dict' objects}
        1    0.000    0.000    0.000    0.000 {method 'close' of '_io.StringIO' objects}
        3    0.000    0.000    0.000    0.000 {method 'copy' of '_hashlib.HASH' objects}
        1    0.000    0.000    0.000    0.000 {method 'copy' of 'dict' objects}
        1    0.000    0.000    0.000    0.000 {method 'count' of 'list' objects}
        1    0.000    0.000    0.000    0.000 {method 'digest' of '_hashlib.HASH' objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        5    0.000    0.000    0.000    0.000 {method 'encode' of 'str' objects}
        2    0.000    0.000    0.000    0.000 {method 'extend' of 'list' objects}
        1    0.000    0.000    0.000    0.000 {method 'get' of 'dict' objects}
        1    0.000    0.000    0.000    0.000 {method 'getvalue' of '_io.StringIO' objects}
        1    0.000    0.000    0.000    0.000 {method 'group' of '_sre.SRE_Match' objects}
        1    0.000    0.000    0.000    0.000 {method 'hexdigest' of '_hashlib.HASH' objects}
        1    0.000    0.000    0.000    0.000 {method 'isoformat' of 'datetime.datetime' objects}
        3    0.001    0.000    0.001    0.000 {method 'items' of 'dict' objects}
        4    0.000    0.000    0.000    0.000 {method 'join' of 'str' objects}
        7    0.000    0.000    0.000    0.000 {method 'send' of 'zmq.backend.cython.socket.Socket' objects}
       26    0.000    0.000    0.000    0.000 {method 'sub' of '_sre.SRE_Pattern' objects}
        5    0.000    0.000    0.000    0.000 {method 'update' of '_hashlib.HASH' objects}
        1    0.000    0.000    0.000    0.000 {method 'upper' of 'str' objects}
        4    0.000    0.000    0.000    0.000 {method 'write' of '_io.StringIO' objects}
        1    0.000    0.000    0.000    0.000 {zmq.backend.cython._poll.zmq_poll}


Disassemble

In [8]:
import dis

dis.dis(lambda x, y: sqrt(x ** 2 + y ** 2))
  3           0 LOAD_GLOBAL              0 (sqrt) 
              3 LOAD_FAST                0 (x) 
              6 LOAD_CONST               1 (2) 
              9 BINARY_POWER         
             10 LOAD_FAST                1 (y) 
             13 LOAD_CONST               1 (2) 
             16 BINARY_POWER         
             17 BINARY_ADD           
             18 CALL_FUNCTION            1 (1 positional, 0 keyword pair) 
             21 RETURN_VALUE         

Memory allocation

A variable and its name have the same address in RAM.

In [11]:
a = 2
print(id(2))
id(a)
507103136
Out[11]:
507103136
In [ ]:
a = 2
id(a)
>> 507098816

a = a+1
id(a)
>>> 507098848
id(3)
>>> 507098848  # 3 and the a+1=2 have the same address

b = 2
id(b)
>>> 507098816  # b has value 2, so it has the same address as a

### http://www.programiz.com/python-programming/namespace

This dynamic nature of name binding makes Python powerful.