Saturday, February 9, 2019

Line graphs in JupyterLab

Recently I've been doing quite a bit of work on climate solution models, using Jupyter Notebook to interact with the backend model. The Jupyter Notebook is a way to allow front-end code running in a browser to interact with backend code running as kernels managed by Jupyter. Though Jupyter evolved from iPython, there are now kernels available for many languages including Java, R, and even C++. All of my work with Jupyter is in Python, however.

I spent time investigating various graphing and charting packages which support interactive use in a JupyterLab notebook. The demo code for the different charting packages is available on github for reference. Note that github will render this file in a way which looks nice, but it is not actually running Jupyter and the graphs are just static snapshots.

The notebook can be run for keepsies using mybinder.org, which creates a container to run user code. Clicking on the button below will open Jupyter in a new browser window, albeit after a perhaps lengthy pause while initializing a container to run it.


At launch an error dialog may appear, about jupiterlab_pyviz. I think that is a false alarm as jupiterlab_pyviz is installed in the container and the Altair-viz chart does work, but I haven't figured out how to suppress the popup.


For those who have not used Jupyter before: each numbered block of code in the Notebook is a cell, and can be run using the right-pointing triangle button at the top. When the Notebook is first opened there will likely be no graphs displayed. Clicking the run button in each cell (and twice in the matplotlib cell to get it to render) will run each one and allow the graphs to be interacted with.



Matplotlib + ipympl

matplotlib is one of the oldest plotting packages for Python still in active development. Its primary design point is static graphs, but it does provide interactive features with a suitable renderer. In addition to Jupyter notebooks matplotlib supports renderers for many GUI environments like PyTk and Qt.

Instructions on the web will generally say to include a "%matplotlib inline" statement for use in Jupyter notebooks, but inline graphs are completely static. Instead, this example uses ipympl and "%matplotlib ipympl" which allows interactive features to work. A toolbar at the bottom allows panning and zooming.

However, matplotlib's interactive features are quite limited. Only one chart at a time can be interactive, and the user must click a power button in the upper right to deactivate it before another can be activated. The overall experience isn't great, matplotlib is much stronger when producing purely static graphs.


 


Bokeh

Bokeh is a charting and dashboarding package. Though it supports use within Jupyter, bokeh appears to be mainly aimed at creation of dashboards and larger collections of visualizations. For example it doesn't always cooperate with Jupyter widgets, preferring bokeh server's layout functions.

Bokeh's interactive features include a tooltip showing exact values when hovering over a datapoint, plus zoom and panning within the chart.


 


hvplot

hvplot is built atop bokeh and looks quite similar in operation, with tooltips and pan-and-zoom. hvplot's programming interface is integrated with Pandas, adding an hvplot() method to the dataframe.


 


Beaker/X

Beaker/Xs a collection of extensions for Jupyter Notebook, spanning the gamut from computation kernels to a graphing package. Its interactive features include tooltips on hover plus panning and zooming, with double-click to return to the original zoom level.

The charts in Bokeh are implemented in Groovy. The Python APIs are a direct transfer of the Groovy APIs. They aren't Python bindings in a Pythonic interface, Beaker/X provides a mechanism for different language kernels to call into each other. This did not make it very easy to code for in Python, for example I could not figure out how to set up a Tooltips object and had to leave it with the default.


 


bqplot

bqplot was released by Bloomberg, and is a charting package used in producing various Bloomberg media properties. It is a thin Python layer which passes through most commands to D3.js for implementation.

bqplot has ways to support both tooltips on hover and zoom-and-pan features, but I was not able to get both to work at the same time. The video shows a quick edit and re-running the cell in order to demonstrate one and the other. There may well be a way to do so, but I didn't figure it out.

Additionally, the tooltips for Line charts are currently not able to retrieve data values and always print NaN.


 


Altair-viz

Altair is somewhat different from the other packages described here in that it is a declarative API, not imperative. That is, instead of a series of operations to produce a graph, Altair instead provides a way to declare what elements of data should be graphed and how to transform the data while doing so. In that sense, Altair aspires to be more like SQL for fetching data or Prolog for transforming data than it does to be a more traditional API.

Altair-Viz emits a Vega-Lite description of the desired graphic, which is expanded to Vega, which ultimately sits atop D3.js in the browser. JupyterLab, an early access release for the next major Jupyter version, supports Vega-Lite and Vega natively. Altair does not require installation of a Jupyter extension to work, the only charting package in this post which does not need one.

For interactive use, Altair charts support tooltips on hover and pan-and-zoom of the graph.


 


plot.ly

Plot.ly is a plotting-as-a-service provider. The main service is hosted online, but the basic functionality of the plot.ly implementation is also available as an open source package which is run locally.

plot.ly charts support tooltips and zoom-and-pan, as well as a mode which prints the X/Y coordinates as the pointer moves over the chart.


 


My intention at this point is to focus on Altair-Viz as it is working well and shows promise for future capabilities, with Jupyterlab having native support for the Vega graphics grammar.

Tuesday, October 16, 2018

Carbon Capture: BECCS

BECCS is an acronym for Bio Energy with Carbon Capture and Storage. It uses plant material in a pyrolysis process to produce electricity. As discussed in the earlier post about biochar, the pyrolysis process produces three outputs:

  • a carbon-rich gas called syngas which is flammable, and contains about half the energy density of natural gas.
  • the solid char, a charcoal which has a much higher concentration of carbon than the original plant material.
  • a thick tar referred to as bio-oil, which is much higher in oxygen than petroleum but otherwise similar.

BECCS is a commercial operation to pyrolyze organic material at scale, usually by growing trees specifically for the purpose.

  • generate electricity by burning the syngas
  • use the char to keep the carbon it holds sequestered for a significant length of time. Though this might involve burial deep underground, char is also useful as a soil additive and takes many years to biodegrade. We could handle a substantial amount of carbon returning to the environment at a long enough cadence.
  • the bio-oil currently has little commercial use but has great potential, as it could displace petroleum in a number of chemical processes.

Because the feedstock for BECCS is newly grown vegetative material, it is strictly carbon neutral. If the char keeps carbon out of the atmosphere for a lengthy period of time, BECCS becomes carbon negative and draws down carbon from the environment while providing revenue via power generation to fund its own operation.

BECCS gets a substantial amount of attention because it is already operating at a substantial scale, removing hundreds of kilotons of carbon dioxide from the atmosphere each year. This is a few orders of magnitude off from where we need to get, but is proof that the process works.

The existing BECCS installations capture byproducts produced in existing agricultural processes, like fermenting corn for ethanol production. An analysis of geo data in 2018 estimated that BECCS could draw down approximately 100 megatons of carbon dioxide per year by 2020 using available land area.

Thursday, October 11, 2018

Carbon Capture: Enhanced Weathering

Chemical weathering is the process by which various types of rock is broken down by exposure to water, oxygen, and/or carbon dioxide. For our purposes, the most relevant forms of weathering involve uptake of carbon dioxide. CO2 dissolved in rainwater forms carbonic acid, which is quite mild as acids go but sufficient over time to dissolve minerals from rock. Calcium and silicon exposed to carbonic acid will form HCO3 bicarbonate, and release calcium and silicates.

Occurring naturally, this chemical reaction takes place gradually over millions of years. Most of the bicarbonate thus produced eventually washes out to the ocean, where various organisms like coral pull carbon and dissolved calcium out of the water to make shells. The rest of the bicarbonate gradually settles into the deep ocean and eventually adds to the limestone at the ocean floor.

Enhanced weathering is a plan by which humans can accelerate this process, by grinding the appropriate types of rock into particles to maximize surface area and spreading them over an area to take up CO2. There are a number of options.

  • Bicarbonate, calcium, and magnesium at appropriate concentrations are beneficial to soil health, especially tropical soils which tend to be depleted in these minerals. Spreading powdered olivine over one third of tropical agricultural land could pull between 30 and 300 parts per million of carbon dioxide out of the atmosphere.
     
    There is a large range in that number because we just don't know enough about how these processes work at scale. Perhaps fortunately, we also don't have the capacity to quickly seed such a large fraction of the planet's land area. Over time, the results of the earliest years of effort can be measured to guide future plans.

  • Though tropical land is ideal, using olivine as a soil additive in agricultural land elsewhere would still have an effect.
     
    The term "electrogeochemistry" has been coined to refer to enhanced weathering done at large scale.

  • Mine tailings are the heaps of excess rock discarded from mining operations after the valuable minerals have been extracted. The tailings generally contain large amounts of the types of rock which will absorb CO2 as they weather, and in fact do rapidly form a shell of carbonate at the surface of the pile. If mining regulations are made to require the tailings be ground more finely and appropriately distributed, they can be effective in pulling carbon dioxide from the atmosphere.
     
    Mine tailings also tend to contain trace amounts of substances which can be harmful, like mercury. Processes such as those developed by Advanced Materials Processing, Inc to remove harmful substances from tailings would be necessary.

Wednesday, September 19, 2018

MacOS Preview.app has a Signature Tool

When I receive a PDF file to be signed and returned I have generally printed it out to sign and scan back in... like an animal, as it turns out. On a MacOS system there is a convenient way to add a signature to a PDF file without needing to print it, using only the Preview.app which comes with the system.

In the toolbar is a squiggly icon with a drop down menu:

Clicking it allows one to create a signature by either signing with a finger on the trackpad, or writing a signature on a piece of paper for the camera to scan in. The camera option does a good job of edge detection to extract only the writing and not shadows on the paper.

The resulting signature can then be added to the document and dragged to the right spot.

Saturday, September 15, 2018

The Arduino before the Arduino: Parallax Basic Stamp

I recently had cause to dig down through the layers of strata which have accumulated in my electronics bin. In one of the lower layers I found this bit of forgotten kit: the Parallax Basic Stamp II. This was the Arduino before there was an Arduino, a tiny microprocessor aimed at being simple for hobbyist and low-volume commercial use.

The Basic Stamp line is still sold today, though with designs developed over a decade ago. The devices have enough of a market to remain in production, but are otherwise moribund. The past tense will be used in this blog post.

The Basic Stamp line dates back to the early 1990s. The Basic Stamp II shown here was introduced in 1995. It used a PIC microcontroller, an 8 bit microprocessor line which has been used in deeply embedded applications for decades and is still developed today. The PIC family is a product from Microchip Technology, the same company which now supplies the AVR chips used in the Arduino after acquiring Atmel in 2016.

The PIC contained several KBytes of flash, which held a BASIC interpreter called PBASIC. An external EEPROM on the BS2 board contained the bytecode compiled user BASIC code. Though it may seem an odd choice now, in the early 1990s the choice of BASIC made sense: the modern Internet and the Tech industry did not exist, with the concordant increase in the number of people comfortable with developing software. BASIC could leverage familiarity with Microsoft GW-BASIC and QBASIC on the PC, as MS-DOS and Windows computers of this time period all shipped with BASIC. Additionally, Parallax could tap into the experience of the hobbyist community from the Apple II and Atari/Commodore/etc.


' PBASIC code for the Basic Stamp
LED         PIN 5
Button      PIN 6    ' the BS2 had 16 pins
ButtonVal   VAR Bit  ' space is precious, 1 *bit* storage
LedDuration CON 500  ' a constant

' Init code
OUTPUT LED
INPUT  Button

DO
 ButtonVal = Button                 ' Read button input pin
 FREQOUT LED,LedDuration,ButtonVal  ' PWM output to flicker LED
 PAUSE 200                          ' in milliseconds
LOOP

PBASIC supported a single thread of operation, the BASIC Stamp supported neither interrupts nor threads. Applications needing these functions would generally use a PIC chip without the BASIC interpreter on top. Later Stamp versions added a limited ability to poll pins in between each BASIC statement and take action. This seemed aimed at industrial control users of the stamps, for example Disney used BASIC Stamps in several theme park rides designed during this time frame.

A key piece of the Arduino and Raspberry Pi ecosystems is the variety of expansion kits, or "shields," which connect to the microprocessor to add capabilities and interface with the external world. The ecosystem of the BASIC Stamp was much more limited, suppliers like Adafruit were not in evidence because the low volume PCB design and contract manufacturing industry mostly didn't exist. Parallax produced some interesting kits of its own like an early autonomous wheeled robot. For the most part though, hobbyists of this era had to be comfortable with wire-wrapping.