1. The state of Python profilers in two words

    Use cProfile.

    This is for people who still wonder what they should use every time.

    • cProfile was added in Python 2.5 after much discussion about the downsides of existing profilers.
    • hotshot does not time C calls. If your code does a lot of work with built-in methods, like sorting lists, hotshot won’t report it. It has an option to record line events—how much time is spent on each line of code in your source file; however, this information is lost when the log is loaded into pstats.Stats, Python’s class for printing profile data.
    • hotshot generates huge log files that take about as long to load as the profile took to run. That eleven minute profiling session in my previous post generated a 625M log file with hotshot. cProfile’s was 52K and more useful.

    Use cProfile like this:

    If you just want to print the results immediately:

    import cProfile
    cProfile.run("test_function(a, b, c)", sort=1)

    sort=1 will sort by the “tottime” column, which is normally the most sensible way to find out what your code is spending time on.

    If your statement contains free variables (references names that are defined in your interpreter but not in the string to execute), use runctx:

    cProfile.runctx("test_function(a, b, c)", globals(), locals())

    Use the filename argument to log results to a file:

    cProfile.run("test_function()", filename="test_function.cprof")

    And load saved results like so:

    import pstats
    stats = pstats.Stats("test_function.cprof")

    You will almost always want to manipulate and trim down the output like so (the top 20 items by tottime, in this case):