A Shot of Noise

Well, my epic battle with the dissertation dragon is nearing a close - I defended my thesis monday morning (at 8:30am!), and am now nearly free from the PhD coils. As the start-time of the official defense was a bit on the early side for most of my colleagues, I've decided to have an un-official mini-defense this Saturday at my house in SF. The start time of this second event is a much more civilized 8:30 PM! As of now, the plan consists of three sub-goals:

1) abridged presentation on the roof (with boringness filtered out), starting once the sky is dark enough to see the projector (inside in case of thunderstorms)

2) home-brewed beer (not much filtered out), and also factory-brewed beer, and snacks

3) stencils of microscopes / zebrafish, applied to whatever you see fit (bring some old garb; together we can spread word of the Light Field Microscope across the land!)

Here is the title of my dissertation: "Imaging 3D visually-evoked calcium dynamics in larval zebrafish optic tectum using light field microscopy". By contrast, here is the title of Saturday's event: "Biology looks better in 4D!". I'll also throw in a blurb or two about my plans for the next few months, which include driving to Mongolia (for charity!) and also driving to Baja (for Science!).

No need to bring anything, other than some clothes if you want to do some science-fashion fusion experiments. Bring anyone you like. Hope to see you Saturday!

[PS: Did you know: the background of this image is actually a representation of the light field captured from a golgi-stained rat brain? The foreground is St. Michael slaying the dragon. I especially like how the saint is fully neck-stomping that dragon...]

This is a little off the topic of light field microscopy, but I've been reading a bunch of PDFs lately trying to write the intro to my thesis. I got Papers, which is a very nice PDF organizer for Mac, and set it to open articles with Skim, an equally nice PDF reader. The problem is - and I should say I've found this problem in Skim, Acrobat, Preview... any reader I've tried - highlighting multiple-line passages in columned text is frickin' impossible! Basically, what happens is this:

which is truly annoying. Here is an example of what we are looking for:

and this is achievable with Skim, one must just master a sneaky key stroke. Okay, so use the highlight tool (Command-Control-5) to highlight single lines, then hold down Shift and click directly on each of the high-lit lines you just selected. Bamn! One big highlight in your highlights list.

Nights!

When I first attempted delivering visual stimulation to fish during light-field recording, I used the LCD screen left over from the successful project of my predecessor Cris Niell. This system was designed for use with the two photon microscope, which uses an infrared wavelength of light to excite fluorescence in the fish brain, and is thus invisible to the fish. In a dark chamber, the small amount of light put out by the LCD was sufficient to interest the baby fish. The light-field, however, is a one-photon technique, and thus the excitation light is quite visible to the fish as a massive blue beam, not unlike something out of Close Encounters of the Third Kind, I imagine.

Against this background, the LCD could not produce enough of a visual stimulus to engage the fish tectum, or at least that was how I explained my inability to record a visually-evoked calcium signal. I did see such signals if I coupled a red laser with an optic fiber and placed it right up near the fish retina. So, how to generate visual patterns with laser-like intensity and contrast, but in a 2D manner?

Uploaded with plasq's Skitch!

Wandering around my favorite electronic component website one day, I saw that they had begun carrying tiny OLED screens for only $71 (now reduced to like $50!). OLED stands for Organic Light Emitting Diode, which means this little screen is basically an array of tiny little individually-addressable LEDs. As you may know from recent camping trips or encounters with upwardly-mobile folks driving German automobiles, LEDs can be really bright! In addition, when an OLED pixel is off, it is truly off - no power runs through it, and so no light is emitted. This is in contrast to LCDs, where each pixel filters the output of a bright white backlight - even when a pixel is filtering as hard as it can (i.e. it is set to black), some light pokes through. What this means for me and my fish is the OLED creates a beautiful, bright, high-contrast image, beautiful enough to engage the neurons of the optic tectum, providing me lovely visually-evoked calcium signals, which the light-field records in 3D. Here are the details on the stimulation setup.

Uploaded with plasq's Skitch!

Hardware: I’ve got the OLED display, one µOLED-96-G1 from 4D Displays (out of Australia, I believe), purchased from SparkFun for ~$71. It is 0.96” across, and will just fit under the microscope objective. I made a chamber using an old Pomona Electronics project box. It’s black plastic, fairly shiny, but I sanded the inside to induce more photon scattering / absorption. The idea is to basically build a movie theater for the fish, with stunning contrast and acoustics (well...). I milled a hole just the size of the screen active area (20mm x 14mm), and then glued a cover slip to the outer edge (for waterproofage). Then screw the OLED to the chamber so it presses against the coverslip. Click the image below for a video showing the chamber setup rotating (is this video itself a type of light-field, capturing multiple angles at multiple spatial positions, except with every view recorded at a separate time point rather than all at once? I think so!):

(click for 3D view)

Software: This particular OLED can be controlled in two ways. Well three, kinda.

1. Real-time Drawing: The included GPU (I don’t think it has the parallel processing power expected of typical desktop GPUs) can draw graphical primitives directly to the screen, based on byte input arriving over the serial port. So, sending something like “0x43 0x20 0x30 0x04 0x61 0x33” (not sure why the ‘0x’ - some kind of byte type designator I guess) will draw a circle centered at (20, 30) with a radius of 4 pixels and a whitish color (that’s the last two bytes, who’s encoding I still don’t quite understand). The display can also produce single pixels, rectangles, and lines, as well as text. I ported Oscar’s Arduino OLED library over to Processing (I’m not gonna lie, it was easy), so you can use that elegant language to script your real-time OLED antics, or make the screen react to any input device which Processing can understand (i.e. pretty much any input device - get your Wii on!). If you have a serial port on your computer, you’re a very lucky person - I haven’t seen one of those in years. I convinced a MacBook Amateur to speak with the OLED via a USB-UART convertor CP2102 (driver or driver). You can view a screencast of the stimulation software I wrote here, and you can download the Processing code here. Make sure you add the OLED library (oled160drv.pde) to your Processing sketch file, like this.

(click for screencast)

2. Hard-coded Drawing: If you intend to use the screen in embedded applications, or don’t want to tie up your serial port, or want fast hardware triggering, or... (other ideas appreciated), you can hard-code your animations into the memory of a microcontroller. A Spaniard named Oscar wrote a hardware library to allow the famous Arduino to talk with this family of OLED screens, and someone named Jenny wrote a nice tutorial on the subject. I may end up using this method, once I’ve got some nice animation code worked out, as it will allow me to trigger the animation in hardware through the Arduino’s digital input channels, driven from MatLab (via the DAQ board, or parallel port), which is the software environment currently orchestrating my camera frame / visual stimulus symphony. I’m hoping this will allow for more precise triggering than attempting to thread my stimulation routines through the Windows concurrency minefield, but I haven’t actually tried it yet (and super-precise timing is not crucial for my project, running as it does at 4 fps). The Arduino has a total of 14 DIOs (that’s Digital Input / Output) ports, one of which is used to reset the OLED, leaving 13 bits to specify animations. That should be 2^13 possibilities; I probably need 2^4. In addition, this microcontroller option will make the whole stimulation system oh so portable (if not so easily re-programable) - maybe I can even take it to Mexico.

3. Reading From a Flash Card: Okay, if you weren’t already sitting down & freaking out, please now take a seat and prepare to _freak_ _it_ _out_! This tiny OLED contraption also comes with an SD card slot (well, micro SD, but don’t let that dampen your enthusiasm, a 1 gig card with adapters to mini and regular SD card sizes will run you $10 at a physcial store, like Frys)! You can tell the OLED GPU, via serial commands, to read images or movies (AVIs, uncompressed I think) off this memory (up to 2 gb worth, or half a season of The Deadliest Catch - that’s a lot of stimulation for a juvenile zebrafish!). These commands are kind of annoying, because you must use 4D Systems crazy sector mapping program to find the memory addresses of the images / movies you load onto the card, but it is never-the-less quite cool. I might use this to display natural stimuli to the fish, which is something I’ve always wanted to try. After all, these little guys didn’t evolve to chase bright blue perfect circles through the darkness.

Lately I've been working with my home-piece and statistic-mystic Logan Grosenick on using light-field data to predict the stimulus which the fish viewed during imaging. Basically, we are running something called independent component analysis (aka ICA - check the sweet sound demo) to break the 4D light-field dataset into little uncorrelated chunks of voxels (within each chunk, the voxels are quite correlated), and then we search for a combo of these chunks which best recreates the stimulus witnessed by the fish. Well, we aren't doing any searching with our human selves; this technique belongs to the scientific family of "machine learning", and thus machines (8-core Mac Pros with hella RAM) are the ones with fine-toothed combs. And what a job they do! I made a little Processing applet to compare the estimated stimulus location with the true stimulus location; click here for that.

Also, I had the Processing program overlay all of the estimated & true stimuli locations onto the same image, but with a low degree of opacity. This means that any intense blocks of color you see in the image above represent areas which were often predicted to be home to the stimulus. I really like the mosaic effect; this is an artifact of up-scaling the stimuli from the tiny fish TV (96 x 64 pixels) to a more humane 576 x 400. And thus we have something of a fishy arabesque!

When I first started generating large numbers of tiff stacks representing 3D volumes at sequential time points, I quickly realized I needed a way of exploring a 4D dataset. Basically I wanted to see a trace of fluorescence intensity versus time for a definable region of interest (ROI), and I wanted to be able to reposition the ROI and watch the trace update in real-time. I tried all kinds of currently available software, commercial and opensource, but nothing I found delivered satisfactory 4D interaction. So I wrote a program in my favorite language, Processing, which loads up a bunch of images representing many focal slices at many time points and allows the user to smoothly explore intensity waveforms across the time or Z dimensions. If you want to give it a whirl yourself, click here. More recently I made a video to demonstrate the software:

Another tigriopus light field for you, this one showing 9 consecutive captures of an actively-digesting shrimp. This is shows just the red channel; we can't record color and time simultaneously with the current hardware, as our camera only sees one color at a time. In the future, it should be possible to split the light three ways (into red, green and blue channels) and use three cameras to simultaneously record each color.

Tigriopus Gut Action from Todd Anderson on Vimeo.

Higher resolution: tigri gut

Tags:

• biology
• invertebrate
• marine
• timelapse

About a year and a half ago, I took a early prototype of the light field microscope to the Hopkins Marine Station in Pacific Grove, California. This is the marine biology wing of Stanford's bio department, and contains the lab of one of my thesis advisers, Stuart Thompson. I was there to help teach course on experimental neuroscience, as well as to see what interesting vistas the light field could provide. So Stuart, Liz Doyle (an under-grad enrolled in the course) and I set off to cram as many strange sea creatures as possible beneath the scope. One particularly interesting specimen turned out to be this halophilic (salt-loving) shrimp, Tigriopus californicus. These guys chillax in the salty hot tubs of the splash zone, where sun-baked tidepools grow ever more saline as the splashed ocean water evaporates. For this video, we placed an individual in agarose gel, which kept it from moving out from under the microscope while still allowing it to breathe.

Higher resolution (h264, 3.9 mb): tigriopus

The light field does not capture color information on its own; for every image acquired, we capture the light intensity of any wavelength (color) along many angles emanating from many points. If we filter the input to the light field imager, such that only light of a certain color can pass, we can record angular & spatial information for just that color. For the video above, we use three different filters to capture three separate images, and then "merged" these images into a single color image using a free program called ImageJ. To our delight & satisfaction, the light field rendering software (written by Zhengyun Zhang in the CS department) accepted this merged image with nary a hiccup. It so happens that proteins inside the Tigriopus fluoresce at different intensities depending on what color of light is used for excitation. Proteins fluorescing in the blue channel seem to be localized to the central core of the animal, while green fluorescers appear throughout the exoskeleton, and red ones in certain internal structures (digestive tract?).

Higher resolution (xvid, 3.4 mb): tigriopus egg sac

Here we have another tigriopus, this one expecting a bunch of baby tigris to pop out of that giant green egg sac at any moment!

This guy shows you how:

Of course, this assumes you have on hand a nice Mamiya camera with a digital back, which costs about $10,000 - a figure oddly similar to the price of the Retiga 4000r we use for lightfield microscopy.

Panning Around a Standard Deviation Projection Through a Lightfield
From Todd Anderson on Vimeo.

Here we have the result of a standard deviation projection across 160 lightfield images recorded at 4 fps from a larval zebrafish optic tectum injected with Oregon Green BAPTA-1 calcium sensitive dye. The fish was watching a small blue dot move left and right, which should cause a certain population of neural cells to fire, flooding these cells with calcium and thus increasing their fluorescence. The SD projection technique is intended to highlight those voxels which exhibit the most fluorescence change over time - i.e. those neurons whose per-frame intensity is most often dissimilar to the mean. The SD projection is fairly noisy, so I decided to also try projecting for the median across time. This gives you a very low-noise volume, but is fairly boring in comparison with the punctuated salience of the SD volume. So, I tried multiplying the two projections, which I will admit to thinking would retain the noise and lose the salience; instead, the resulting volume seems to include the best properties of both projections - low noise with high salience!

Median Projected Volume Lightfield Time Projections Multiplied and Panned! From Todd Anderson on Vimeo.

Median x SD Volume Median Time Projection Lightfield From Todd Anderson on Vimeo.

Closer inspection of the raw lightfield data (crops of which are shown below) reveals that the majority of the scrubbed noise was removed from the very high-angle light (i.e. those pixels near the edge of each lenslet image), which I don't really use in producing these panning movies, but an interesting technique none-the-less.

Median

SD

SD x Median

And finally, here is the focal stack through the median x SD volume:

Lightfield Time Projections Multiplied & Focused!
From Todd Anderson on Vimeo.

I've been trying to rearrange my 4D datasets (volumes over time) into a specially-ordered 2D matrix, such that I might facilitate automatic variable selection and the fitting of regressions. This means I'd like to end up with a very wide & short array, where each row represents a single trial (i.e. stimulus presentation) and each column within the row all the voxels at all time points for that trial. The resulting array is approximately 8 rows by 22 million columns, which is not so easy to visualize, and makes sanity-checks difficult. So, I rearranged the data again, this time substituting each voxel in a single XY plane with the 8 x 20 element matrix representing that voxel's intensity value in 20 frames for each of 8 trials. Now, the wide-angle view of the data indicates average intensity at each voxel in a plane over the entire imaging run, and the zoomed-view indicates the activity across time for every trial. This idea is somewhat analogous to the raw lightfield image, which from far away encodes spatial information about a single plane, and from close-up indicates information about the angular information recorded for each XY position.

Here is the wide view:

and here a zoomed-view of a cropped-out section:

Of course no Shot of Noise post would be complete without a volume-rendered movie, so here you go (not too exciting though, I must admit):

High-res