Light Field Visualizer: I built this Processing applet to allow interactive exploration of 4D light field data. To do this, a series of 25 compressed PNGs representing a synthetic focal stack is generated for each light field camera exposure using an Python script which coordinates the efforts of ImageStack, ImageJ, and the Ubuntu filesystem. Because the program loads all image files into memory up front, it takes some time to initialize, but is very fast once up and running. In the interests of reasonable download speed, this online version is limited to loading 25 frames, though the stand-alone application can load hundreds on my MacBook Amateur. The example data set here comes from a video of a larval zebrafish optic tectum (injected with Oregon Green BAPTA-1 AM calcium-sensitive dye) responding to visual stimulation (a small dot moving in a circle), imaged using a 63x water-immersion objective (0.9 NA) and a lightfield camera. One entire optic tectum is labled, anterior pointing down, lateral side to the right. The arc running along the right & bottom sides of the image is the cell body layer; the dendritic neuropil is center & upper-right. This dataset can be rendered as a 4D (3D + time) movie. Other examples of rendered lightfields may be found here.
GUI Info
Slice View: shows the x/y slice at the current Z-position. ROIs may be created (Option+drag), adjusted (drag within existing ROI or resize handle) and deleted (click on an ROI and then press Backspace (fn + delete on mac laptops) here.
ROI Zoom View: shows a zoomed view of the pixels inside the currently-active ROI. The zoom window will not go below a certain minimum size, so if you make the ROI smaller than this minimum (currently 7 data pixels), a white square will be drawn inside the ROI zoom view to indicate the position of the ROI. The display range for these pixels is adjusted such that the minimum pixel intensity is set to black and the max to white (in other words, the view is normalized). You will notice the display range changing when you adjust the position / size of an ROI - this effect can be disabled by clicking the "Lock Levels" button.
TZ View: shows a grid of rectangles filled with a grey value proportional to the average intensity within the active ROI for all time points at all Z-positions. The x-position of each rectangle indicates the time point at which that average occured, and the y-position indicates the Z-position. Clicking any rectangle will cause the slice view to jump to the corresponding time point and Z-position. This view is intended to give you a quick impression of the activity within the ROI.
I vs Z plot: shows a graph of intensity (left-right axis) versus Z-position (up-down axis). Greater intensity is to the right. The plot is continually normalized to the minimum & maxium intensities present across all the created ROIs, and the min / max values are printed near the bottom of the plot. The width of the indicator bar at the bottom is scaled relative to the difference between the active ROI's max and min intensities, and is intended to give a general impression of the fluorescence change along Z within the active ROI. Dragging within the plot will cause the slice view to jump to the new Z-position at the same time point.
I vs T plot: shows a graph of intensity (up-down axis) versus time point (left-right axis). Greater intensity is up. Dragging within the plot will cause the slice view to jump to the new time point at the same Z-position. The red vertical lines indicate the time points at which a stimulation occured. The other parts of this plot are identical to those of the I vs Z plot.
Z: normalize TZ View over all time points & z-levels ("T & Z")
X: normalize TZ View each z-level along time ("EACH Z")
C: normalize TZ View all time points & z-levels based on the current Z-position ("THIS Z")
More Info on User Input:
Click and drag within the TZ view or the sliders to adjust time and z-slice
horizontal axis changes time (can also be adjusted using the left & right arrow keys, or < & >)
vertical axis changes z-slice (can also be adjusted using the up & down arrow keys)
clicking within an individual slider locks the change to that axis (i.e. either T or Z, but not both at the same time)
Holding option while click-n-draggin' within the image will create a region of interest (ROI). The program automatically plots the average intensity over time and z-slice, and also creates a zoomed and normalized view (with historisis! Inspired by Johannes Kopf's gigapixel image project) of the area within the ROI. Up to four ROIs may be created, and each can be independantly repositioned, which dynamically adjusts the plot & zoom view (the zoom view shows only the active ROI, which is indicated by a brighter border). The span of the box on the lower-right side of each plot indicates the delta F over F (a.k.a. the difference between min and max intensity divided by the min intensity) for the active ROI, while the intensity of the box indicates the minimum intensity value. This "ambient" interface is intended to give you an almost sub-conscious feeling for the fluoresence baseline and change.
The active ROI (last ROI clicked) is highlighted with a bright frame, and may be precisely positioned by holding shift while pressing the arrow keys. It may also be expanded or reduced by pressing - or +, or by click-n-draggin within the small box in the upper-left corner of the ROI (the resize handle). The active ROI may be deleted by pressing Backspace (fn + delete on mac laptops)
Pressing TAB cycles through the existing ROIs.
Pressing P will play through the timepoints, in a continuous loop. ROIs and z-slice may be adjusted while the video plays. Auto-normalization is disabled during play-back, with the level frozen at the values which existed when play-back was started. This is also true when dragging the Z and T sliders, but is not true when using the keyboard to change those parameters.
The current level settings (for normalizing the ROI zoom view by mapping data values to screen drawing values) may be locked at any time by pressing L, or by pressing the "lock levels" button.
Historisis in the normalization effect can be toggled by pressing H, or by pressing the "historisis" button.
Historisis during ROI dragging can be togggled by pressing J
Pressing S or clicking the "subtract" button will toggle between raw slice view and frame subtraction. Upon pressing S, the current frame will be subtracted from all subsequent frames rendered to the screen. The the zoom window will continue to auto-level (aka normalize) the ROI territory (this is pretty janky code at the moment - I really need a normalization routine which does not involve drawing intermediate steps to the screen). Please keep in mind that if you change the z-slice, the subtraction will be showing the difference in z as well as time, which might be confusing (I will probably change the code to automatically use the new z-slice if changed).
I doubt this works through the applet, but if you have a Mac laptop with a Sudden Motion Sensor (sms) accelerometer, you can change the z-slice by pressing A then raising / lowering your laptop. There is also a provision for interacting with the data using a Wii controller - download the source code to try that out.
What might the future include?
Features:
File choosing dialog box
Implemented in off-line version (works in Processing 0135)
ROI image segment automatically auto-levels to stack histogram (instead of slice histogram)
Zoom window on full image, ROI drawn on zoomed image, to make ROI more precise
I did this for the active ROI
Volume of interest: average ROI through multiple slices
Give ROIs a z coordinate
how to indicate?
tick marks on z slider?
stroke intensity (could indicate active ROI with double rect instead of brighter stroke)
Leave notes at ROI / T / Z
Save in text file, place visual indicator on the slice to show that something has been noted?
Sounds a lot like the graphics prof candidate's work, but with image processing
Output plot points to a text file (CSV)
works in offline version
Place all experiments on a single "platonic fish map"
I've had this idea before (see Journler for "software")
New map-tiling library for Processing could really help
Probably would best be done in some other language, but for prototyping, Processing is suitable
Plot T/Z average intensities in 3D, instead of as grey values
Bugs:
Streamline image loading
Load PNGs as 8 bit grayscale (instead of RGB)?
check forum: img.format=ALPHA, or possibly img.format=2;
uncompressed, each slice is 16k (a 8 bit).
16*50*150=~120mb should be do-able, even at 3x for RGB
Sneaky trick: load a seperate PNG into each channel (RGBA), than convert at display time to color(R, R, R)
Java class for reading tiff stacks, a la ImageJ
QuickTime works, but has odd jitter around selecting a frame from the movie
is there a better frame selector command for Processing?
Make small z's negative
Subtraction mode does not account for changes in Z-position
