Ipython Magic Executing Previous Line Again

October 12, 2016

28 Jupyter Notebook Tips, Tricks, and Shortcuts

Jupyter Notebook

Jupyter notebook, formerly known as the IPython notebook, is a flexible tool that helps you create readable analyses, as yous tin keep code, images, comments, formulae and plots together. In this post, nosotros've collected some of the top Jupyter notebook tips to quickly turn yous into a Jupyter ability user!

(This post is based on a postal service that originally appeared on Alex Rogozhnikov's web log, 'Brilliantly Wrong'. We accept expanded the post and will continue to do so over time — if you accept a suggestion delight let usa know. Thanks to Alex for graciously letting u.s.a. republish his piece of work here.)

Jupyter is quite extensible, supports many programming languages and is hands hosted on your estimator or on near whatever server — y'all only need to have ssh or http access. All-time of all, it'southward completely free. Now let'south dive in to our list of 28 (and counting!) Jupyter notebook tips!

The Jupyter interface.

Project Jupyter was born out of the IPython project equally the project evolved to become a notebook that could back up multiple languages – hence its historical name as the IPython notebook. The proper noun Jupyter is an indirect acronyum of the iii core languages it was designed for: JUlia, PYThon, and R and is inspired past the planet Jupiter.

When working with Python in Jupyter, the IPython kernel is used, which gives united states some handy access to IPython features from within our Jupyter notebooks (more on that afterward!)

Nosotros're going to evidence you 28 tips and tricks to make your life working with Jupyter easier.

Learn Data Skills

Get that side by side raise or to switch to a career in data science past learning data skills.

Sign up for a free account and endeavor our interactive courses in Python, R, SQL, and more!

1. Keyboard Shortcuts

As whatever power user knows, keyboard shortcuts will save y'all lots of time. Jupyter stores a list of keybord shortcuts under the menu at the top: Assistance > Keyboard Shortcuts, or by pressing H in command fashion (more on that later). It's worth checking this each time you update Jupyter, as more shortcuts are added all the fourth dimension.

Another mode to admission keyboard shortcuts, and a handy style to learn them is to use the command palette: Cmd + Shift + P (or Ctrl + Shift + P on Linux and Windows). This dialog box helps you lot run any command by name – useful if you don't know the keyboard shortcut for an action or if what y'all want to practise does not take a keyboard shortcut. The functionality is similar to Spotlight search on a Mac, and once you start using it you'll wonder how you lived without it!

The command palette.

Some of my favorites:

• Esc will take you into command way where you lot can navigate around your notebook with arrow keys.
• While in command mode:
  • A to insert a new cell in a higher place the current cell, B to insert a new cell below.
  • Thousand to modify the electric current cell to Markdown, Y to change it back to code
  • D + D (press the primal twice) to delete the current cell
• Enter will have you lot from command mode dorsum into edit fashion for the given cell.
• Shift + Tab will show you the Docstring (documentation) for the the object you take merely typed in a lawmaking cell – you lot tin can keep pressing this short cut to cycle through a few modes of documentation.
• Ctrl + Shift + - will carve up the current cell into two from where your cursor is.
• Esc + F Discover and supersede on your lawmaking but not the outputs.
• Esc + O Toggle cell output.
• Select Multiple Cells:
  • Shift + J or Shift + Down selects the side by side sell in a downwardly direction. Yous can also select sells in an up direction by using Shift + Grand or Shift + Up.
  • In one case cells are selected, y'all can and so delete / copy / cut / paste / run them as a batch. This is helpful when you need to move parts of a notebook.
  • You lot tin also utilise Shift + Grand to merge multiple cells.

Merging multiple cells.

2. Pretty Display of Variables

The first part of this is pretty widely known. Past finishing a Jupyter cell with the name of a variable or unassigned output of a statement, Jupyter will display that variable without the need for a print argument. This is especially useful when dealing with Pandas DataFrames, as the output is neatly formatted into a table.

What is known less, is that yous tin can modify a change the ast_note_interactivity kernel option to brand Jupyter practise this for any variable or statement on its own line, so you can come across the value of multiple statements at once.

                          from              IPython.core.interactiveshell              import              InteractiveShell InteractiveShell.ast_node_interactivity              =              "all"                      

            from pydataset import information quakes = data('quakes') quakes.caput() quakes.tail()                      

lat	long	depth	magazine	stations
1	-20.42	181.62	562	4.8	41
2	-twenty.62	181.03	650	four.two	fifteen
iii	-26.00	184.10	42	5.4	43
4	-17.97	181.66	626	4.1	19
5	-20.42	181.96	649	4.0	eleven

lat	long	depth	mag	stations
996	-25.93	179.54	470	four.iv	22
997	-12.28	167.06	248	four.7	35
998	-20.13	184.20	244	4.5	34
999	-17.40	187.80	40	4.5	14
chiliad	-21.59	170.56	165	half dozen.0	119

If y'all want to gear up this behaviour for all instances of Jupyter (Notebook and Console), simply create a file ~/.ipython/profile_default/ipython_config.py with the lines below.

            c              =              get_config(              )              # Run all nodes interactively              c.InteractiveShell.ast_node_interactivity              =              "all"                      

3. Easy links to documentation

Within the Help menu y'all'll notice handy links to the online documentation for common libraries including NumPy, Pandas, SciPy and Matplotlib.

Don't forget also that by prepending a library, method or variable with ?, y'all tin access the Docstring for quick reference on syntax.

            ?str.supercede(              )                      

            Docstring: S.supplant(quondam, new[, count]) -> str  Return a copy of South with all occurrences of substring old replaced by new.  If the optional argument count is given, only the first count occurrences are replaced. Type:      method_descriptor                      

4. Plotting in notebooks

There are many options for generating plots in your notebooks.

• matplotlib (the de-facto standard), activated with %matplotlib inline – Here'south a Dataquest Matplotlib Tutorial.
• %matplotlib notebook provides interactivity but can be a trivial slow, since rendering is washed server-side.
• Seaborn is congenital over Matplotlib and makes building more than bonny plots easier. But by importing Seaborn, your matplotlib plots are fabricated 'prettier' without whatsoever code modification.
• mpld3 provides culling renderer (using d3) for matplotlib code. Quite nice, though incomplete.
• bokeh is a meliorate option for building interactive plots.
• plot.ly can generate dainty plots – this used to be a paid service only but was recently open up sourced.
• Altair is a relatively new declarative visualization library for Python. Information technology's piece of cake to employ and makes great looking plots, notwithstanding the power to customize those plots is not well-nigh every bit powerful equally in Matplotlib.

The Jupyter interface.

5. IPython Magic Commands

The %matplotlib inline you saw in a higher place was an example of a IPython Magic command. Beingness based on the IPython kernel, Jupyter has access to all the Magics from the IPython kernel, and they can make your life a lot easier!

                          # This will list all magic commands                        
                          %lsmagic                      

            Bachelor line magics:
%alias %alias_magic %autocall %automagic %autosave %bookmark %cat %cd %clear %colors %config %connect_info %cp %debug %dhist %dirs %doctest_mode %ed %edit %env %gui %hist %history %killbgscripts %ldir %less %lf %lk %ll %load %load_ext %loadpy %logoff %logon %logstart %logstate %logstop %ls %lsmagic %60 %macro %magic %man %matplotlib %mkdir %more than %mv %notebook %page %pastebin %pdb %pdef %pdoc %pfile %pinfo %pinfo2 %popd %pprint %precision %contour %prun %psearch %psource %pushd %pwd %pycat %pylab %qtconsole %quickref %recall %rehashx %reload_ext %rep %rerun %reset %reset_selective %rm %rmdir %run %save %sc %set_env %shop %sx %organization %tb %time %timeit %unalias %unload_ext %who %who_ls %whos %xdel %xmode              
Available cell magics:%%! %%HTML %%SVG %%bash %%capture %%debug %%file %%html %%javascript %%js %%latex %%perl %%prun %%pypy %%python %%python2 %%python3 %%ruby %%script %%sh %%svg %%sx %%system %%time %%timeit %%writefile              
Automagic is ON, % prefix IS Non needed for line magics.          

I recommend browsing the documentation for all IPython Magic commands every bit you'll no doubt find some that work for yous. A few of my favorites are below:

half dozen. IPython Magic – %env: Set up Environment Variables

You tin can manage environment variables of your notebook without restarting the jupyter server process. Some libraries (similar theano) use environment variables to control behavior, %env is the nearly convenient style.

                          # Running %env without whatever arguments              
              # lists all environment variables              # The line below sets the environment              
              # variable              
%env OMP_NUM_THREADS%env OMP_NUM_THREADS=4          

            env: OMP_NUM_THREADS=4          

seven. IPython Magic – %run: Execute python lawmaking

%run tin execute python code from .py files – this is well-documented behavior. Lesser known is the fact that it tin can also execute other jupyter notebooks, which can quite useful.

Notation that using %run is not the same every bit importing a python module.

                          # this will execute and show the output from              
              # all lawmaking cells of the specified notebook              
%run ./two-histograms.ipynb          

8. IPython Magic – %load: Insert the code from an external script

This will replace the contents of the jail cell with an external script. You lot can either employ a file on your computer equally a source, or alternatively a URL.

                          # Before Running              
%load ./hello_world.py          

                          # Afterward Running              
              # %load ./hello_world.py              
if __name__ == "__main__":
              print("Hello World!")          

            Hello Earth!          

9. IPython Magic – %store: Pass variables between notebooks.

The %store command lets y'all pass variables between 2 different notebooks.

            data = 'this is the string I want to pass to dissimilar notebook'
%shop data
del data              # This has deleted the variable                      

            Stored 'data' (str)          

At present, in a new notebook…

            %store -r data
print(data)          

            this is the string I want to pass to unlike notebook          

x. IPython Magic – %who: List all variables of global telescopic.

The %who command without any arguments will list all variables that existing in the global scope. Passing a parameter like str will listing merely variables of that blazon.

            one = "for the money"
ii = "for the evidence"
three = "to go fix now get true cat go"
%who str          

            one 3 two          

11. IPython Magic – Timing

There are 2 IPython Magic commands that are useful for timing – %%time and %timeit. These are especially handy when you have some slow lawmaking and y'all're trying to indentify where the consequence is.

%%time volition give you lot data about a unmarried run of the lawmaking in your jail cell.

            %%time
import time
for _ in range(1000):
              time.sleep(0.01)              # sleep for 0.01 seconds                      

            CPU times: user 21.5 ms, sys: 14.8 ms, full: 36.iii ms Wall time: xi.6 s          

%%timeit uses the Python timeit module which runs a statement 100,000 times (past default) so provides the mean of the fastest three times.

            import numpy
%timeit numpy.random.normal(size=100)          

            The slowest run took 7.29 times longer than the fastest. This could mean that an intermediate result is being buried.
100000 loops, all-time of 3: 5.5 µs per loop          

12. IPython Magic – %%writefile and %pycat: Export the contents of a cell/Show the contents of an external script

Using the %%writefile magic saves the contents of that cell to an external file. %pycat does the reverse, and shows you lot (in a popup) the syntax highlighted contents of an external file.

            %%writefile pythoncode.py              
import numpy
def append_if_not_exists(arr, ten):
              if x non in arr:
              arr.append(ten)def some_useless_slow_function():
              arr = list()
              for i in range(10000):
              x = numpy.random.randint(0, 10000)
              append_if_not_exists(arr, x)          

            Writing pythoncode.py          

            %pycat pythoncode.py          

            import numpy
def append_if_not_exists(arr, ten):
              if ten not in arr:
              arr.suspend(10)def some_useless_slow_function():
              arr = list()
              for i in range(10000):
              x = numpy.random.randint(0, 10000)
              append_if_not_exists(arr, x)          

13. IPython Magic – %prun: Show how much time your program spent in each function.

Using `%prun statement_name` will requite yous an ordered table showing you the number of times each internal function was called within the statement, the time each call took likewise as the cumulative time of all runs of the function.

            %prun some_useless_slow_function()          

            26324 function calls in 0.556 seconds              
Ordered by: internal fourth dimension              
ncalls tottime percall cumtime percall filename:lineno(part)
10000 0.527 0.000 0.528 0.000 :2(append_if_not_exists)
10000 0.022 0.000 0.022 0.000 {method 'randint' of 'mtrand.RandomState' objects}
1 0.006 0.006 0.556 0.556 :6(some_useless_slow_function)
6320 0.001 0.000 0.001 0.000 {method 'append' of 'listing' objects}
1 0.000 0.000 0.556 0.556 :i()
one 0.000 0.000 0.556 0.556 {congenital-in method exec}
one 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}          

14. IPython Magic – Debugging with %pdb

Jupyter has own interface for The Python Debugger (pdb). This makes it possible to go inside the function and investigate what happens there.

Yous tin can view a listing of accepted commands for pdb here.

            %pdb              
def pick_and_take():
              picked = numpy.random.randint(0, 1000)
              raise NotImplementedError()
pick_and_take()          

            Automatic pdb calling has been turned ON          

            --------------------------------------------------------------------
NotImplementedError Traceback (almost contempo phone call final)
in ()
5 enhance NotImplementedError()
6
----> 7 pick_and_take()
in pick_and_take()
3 def pick_and_take():
4 picked = numpy.random.randint(0, thou)
----> 5 heighten NotImplementedError()
half dozen
vii pick_and_take()
NotImplementedError:          

            > (5)pick_and_take()
three def pick_and_take():
4 picked = numpy.random.randint(0, 1000)
----> 5 enhance NotImplementedError()
half dozen
7 pick_and_take()          

            ipdb>          

xv. IPython Magic – High-resolution plot outputs for Retina notebooks

One line of IPython magic will requite yous double resolution plot output for Retina screens, such as the more recent Macbooks. Note: the example beneath won't return on non-retina screens

            10 = range(grand)
y = [i ** 2 for i in 10]
plt.plot(x,y)
plt.bear witness();          

            %config InlineBackend.figure_format ='retina'
plt.plot(x,y)
plt.bear witness();          

16. Suppress the output of a terminal function.

Sometimes it's handy to suppress the output of the function on a concluding line, for example when plotting. To do this, y'all merely add a semicolon at the end.

            %matplotlib inline
from matplotlib import pyplot as plt
import numpyx = numpy.linspace(0, 1, 1000)**ane.5          

                          # Hither you get the output of the office              
plt.hist(x)          

            (assortment([ 216., 126., 106., 95., 87., 81., 77., 73., 71., 68.]), array([ 0. , 0.1, 0.2, 0.3, 0.4, 0.v, 0.6, 0.vii, 0.viii, 0.nine, 1. ]),
<a list of 10 Patch objects>)          

                          # By adding a semicolon at the terminate, the output is suppressed.plt.hist(x);                      

17. Executing Shell Commands

It'due south piece of cake to execute a beat command from inside your notebook. You can use this to cheque what datasets are in available in your working binder:

            !ls *.csv          

            nba_2016.csv titanic.csv pixar_movies.csv whitehouse_employees.csv          

Or to check and manage packages.

            !pip install numpy !pip list | grep pandas          

            Requirement already satisfied (apply --upgrade to upgrade): numpy in /Library/Frameworks/Python.framework/Versions/three.4/lib/python3.iv/site-packages pandas (0.18.1)          

18. Using LaTeX for forumlas

When yous write LaTeX in a Markdown prison cell, it will exist rendered as a formula using MathJax.

This:

            $P(A \mid B) = \frac{P(B \mid A)P(A)}{P(B)}$          

Becomes this:

Markdown is an important role of notebooks, and then don't forget to use its expressiveness!

19. Run code from a dissimilar kernel in a notebook

If y'all desire to, you can combine code from multiple kernels into one notebook.

Just use IPython Magics with the name of your kernel at the offset of each cell that y'all want to use that Kernel for:

• %%bash
• %%HTML
• %%python2
• %%python3
• %%ruby
• %%perl

            %%bash
for i in {1..5}
practise echo "i is $i"
done          

            i is 1
i is 2
i is 3
i is 4
i is 5          

20. Install other kernels for Jupyter

One of the squeamish features nearly Jupyter is ability to run kernels for different languages. As an example, hither is how to get and R kernel running.

Easy Option: Installing the R Kernel Using Anaconda

If you used Anaconda to fix up your surround, getting R working is extremely easy. Only run the below in your terminal:

            conda install -c r r-essentials          

Less Piece of cake Option: Installing the R Kernel Manually

If you are not using Anaconda, the process is a little more complex. Firstly, yous'll demand to install R from CRAN if yous haven't already.

Once that's washed, fire up an R console and run the following:

            install.packages(c('repr', 'IRdisplay', 'crayon', 'pbdZMQ', 'devtools'))
devtools::install_github('IRkernel/IRkernel')
IRkernel::installspec()              # to annals the kernel in the current R installation                      

21. Running R and Python in the same notebook.

The best solution to this is to install rpy2 (requires a working version of R likewise), which tin can be easily done with pip:

            pip install rpy2          

Y'all tin and so use the ii languages together, and even pass variables inbetween:

            %load_ext rpy2.ipython          

            %R require(ggplot2)          

            assortment([one], dtype=int32)          

            import pandas as pd df = pd.DataFrame({
'Letter': ['a', 'a', 'a', 'b', 'b', 'b', 'c', 'c', 'c'],
'X': [4, 3, 5, two, 1, seven, seven, v, nine],
'Y': [0, 4, 3, 6, 7, 10, 11, 9, 13],
'Z': [ane, 2, 3, 1, ii, 3, 1, 2, 3]
})          

            %%R -i df ggplot(data = df) + geom_point(aes(x = X, y= Y, color = Letter, size = Z))          

Case courtesy Revolutions Blog

22. Writing functions in other languages

Sometimes the speed of numpy is non enough and I demand to write some fast lawmaking.
In principle, you tin can compile part in the dynamic library and write python wrappers…

But it is much meliorate when this wearisome function is done for you, correct?

Yous tin write functions in cython or fortran and utilise those direct from python code.

Beginning you lot'll need to install:

            !pip install cython fortran-magic          

            %load_ext Cython          

            %%cython
def myltiply_by_2(float x):
return 2.0 * ten          

            myltiply_by_2(23.)          

Personally I prefer to utilize fortran, which I plant very user-friendly for writing number-crunching functions. More details of usage can be found here.

            %load_ext fortranmagic          

            %%fortran subroutine compute_fortran(ten, y, z)
real, intent(in) :: 10(:), y(:)
real, intent(out) :: z(size(x, i))
z = sin(x + y)
finish subroutine compute_fortran          

            compute_fortran([i, 2, 3], [4, v, vi])          

There are also different jitter systems which tin can speed up your python code. More examples can be found here.

23. Multicursor support

Jupyter supports mutiple cursors, similar to Sublime Text. Simply click and elevate your mouse while holding downward Alt.

Multicursor support.

24. Jupyter-contrib extensions

Jupyter-contrib extensions is a family unit of extensions which requite Jupyter a lot more than functionality, including e.g. jupyter spell-checker and code-formatter.

The following commands will install the extensions, also every bit a menu based configurator that will assist y'all browse and enable the extensions from the main Jupyter notebook screen.

            !pip install https://github.com/ipython-contrib/jupyter_contrib_nbextensions/tarball/primary !pip install jupyter_nbextensions_configurator !jupyter contrib nbextension install --user !jupyter nbextensions_configurator enable --user          

The nbextension configurator.

25. Create a presentation from a Jupyter notebook.

Damian Avila's Rising allows y'all to create a powerpoint style presentation from an existing notebook.

You can install Rising using conda:

            conda install -c damianavila82 rise          

Or alternatively pip:

            pip install RISE          

And and so run the following lawmaking to install and enable the extension:

            jupyter-nbextension install ascent --py --sys-prefix jupyter-nbextension enable ascent --py --sys-prefix          

26. The Jupyter output organisation

Notebooks are displayed every bit HTML and the cell output tin can be HTML, so you can render virtually anything: video/sound/images.

In this case I scan the folder with images in my repository and evidence thumbnails of the first v:

            import os
from IPython.display import display, Image names = [f for f in bone.listdir('../images/ml_demonstrations/') if f.endswith('.png')]
for name in names[:5]:
              display(Image('../images/ml_demonstrations/' + name, width=100))          

Nosotros tin can create the same list with a bash command, because magics and bash calls return python variables:

            names = !ls ../images/ml_demonstrations/*.png names[:5]          

            ['../images/ml_demonstrations/colah_embeddings.png',
'../images/ml_demonstrations/convnetjs.png',
'../images/ml_demonstrations/decision_tree.png',
'../images/ml_demonstrations/decision_tree_in_course.png',
'../images/ml_demonstrations/dream_mnist.png']          

27. Big data analysis

A number of solutions are available for querying/processing large data samples:

• ipyparallel (formerly ipython cluster) is a good option for unproblematic map-reduce operations in python. We use it in rep to railroad train many machine learning models in parallel
• pyspark
• spark-sql magic %%sql

28. Sharing notebooks

The easiest style to share your notebook is simply using the notebook file (.ipynb), only for those who don't use Jupyter, y'all take a few options:

• Convert notebooks to html files using the File > Download as > HTML Bill of fare choice.
• Upload your .ipynb file to Google Colab.
• Share your notebook file with gists or on github, both of which render the notebooks. See this example.
  • If y'all upload your notebook to a github repository, yous can use the handy mybinder service to allow someone one-half an hour of interactive Jupyter access to your repository.
• Setup your own system with jupyterhub, this is very handy when you organize mini-grade or workshop and don't accept time to care about students machines.
• Store your notebook e.yard. in dropbox and put the link to nbviewer. nbviewer will render the notebook from whichever source y'all host it.
• Utilize the File > Download as > PDF menu to save your notebook every bit a PDF. If you lot're going this route, I highly recommend reading Julius Schulz'southward first-class commodity Making publication ready Python notebooks.
• Create a weblog using Pelican from your Jupyter notebooks.

What are your favorites?

Let me know what your favorite Jupyter notebook tips are.

At Dataquest, our interactive guided projects use Jupyter notebooks to edifice information scientific discipline projects and go a job in information. If you're interested, you tin can sign upwardly and practise our first module for gratuitous.

I also recommend the links beneath for further reading:

• IPython born magics
• Prissy interactive presentation about jupyter by Ben Zaitlen
• Advanced notebooks part 1: magics
  and part two: widgets
• Profiling in python with jupyter
• 4 means to extend notebooks
• IPython notebook tricks
• Jupyter vs Zeppelin for big information
• Making publication fix Python notebooks.

hernandezhicandre.blogspot.com

Source: https://www.dataquest.io/blog/jupyter-notebook-tips-tricks-shortcuts/

Share this post