Scopi is an intelligent camera app for taking VR photos. Using Scopi, users can take photos in stereoscopic pairs. Scopi automatically registers (aligns), crops and stabilizes your pairs for the best viewing experience in a VR viewer. Scopi uses Kotlin and Java code on Android and Swift on iOS. Algorithm code is written in C++. Scopi has over 10,000 users across both platforms.
My own flavors of a vicuna based instruction-following LLM model integrated with a diffusion model for interactive image generation and conversational assistant capabilities all in one. Models are based in PyTorch and are accessible from the web via simple Quart based webserver and UI. Currently hosted for friends and family on my personal GPU server, shoot me a message for access (find me on discord or via email)!
I regularly leverage generative diffusion models in my hobbyist work. For this particular project I fine-tuned a model on a large dataset of self portrait photographs. Employing my initials as a symbolic token, I established a link between the generated images and my personal likeness. Consequently, the model was not only capable of creating media via text descriptions, it could inherently include me in them when prompted, fun stuff!
ScoPy is a Python package for stereo image processing. ScoPy can perform many operations for on stereoscopic photos including registration, cropping, interlacing, disparity map creation, and others. ScoPy utilizes the same methods employed by the Scopi application for Android and iOS (the applications use native C++ functions). ScoPy is closed source at the moment, but I’m happy to add interested contributors to the GitHub project, just shoot me a note.
Coral reefs are biologically diverse and structurally complex ecosystems, which have been severally affected by human actions. Consequently, there is a need for rapid ecological assessment of coral reefs, but current approaches require time consuming manual analysis, either during a dive survey or on images collected during a survey. Reef structural complexity is essential for ecological function but is challenging to measure and often relegated to simple metrics such as rugosity. Recent advances in computer vision and machine learning offer the potential to alleviate some of these limitations. We developed an approach to automatically classify 3D reconstructions of reef sections and assessed the accuracy of this approach. 3D reconstructions of reef sections were generated using commercial Structure-from-Motion software with images extracted from video surveys. To generate a 3D classified map, locations on the 3D reconstruction were mapped back into the original images to extract multiple views of the location. Several approaches were tested to merge information from multiple views of a point into a single classification, all of which used convolutional neural networks to classify or extract features from the images, but differ in the strategy employed for merging information. Approaches to merging information entailed voting, probability averaging, and a learned neural-network layer. All approaches performed similarly achieving overall classification accuracies of ~96% and >90% accuracy on most classes. With this high classification accuracy, these approaches are suitable for many ecological applications.
We propose and compare patch-based CNN and FCNN architectures capable of exploiting multi-view image information to improve the accuracy of classification and semantic segmentation of the input images. We investigate extensions of the conventional FCNN architecture to incorporate stereoscopic input image data and extensions of patch-based CNN architectures to incorporate multi-view input image data. Experimental results show the proposed TwinNet architecture to be the best performing FCNN architecture, performing comparably with its baseline Dilation8 architecture when using just a left-perspective input image, but markedly improving over Dilation8 when using a stereo pair of input images. Likewise, the proposed nViewNet-8 architecture is shown to be the best performing patch-based CNN architecture, outperforming its single-image ResNet152 baseline architecture in terms of classification accuracy.
Two major deep learning methods for semantic segmentation, i.e., patch-based convolutional neural network (CNN) approaches and fully convolutional neural network (FCNN) models, are studied in the context of classification of regions in underwater images of coral reef ecosystems into biologically meaningful categories. For the patch-based CNN approaches, we use image data extracted from underwater video accompanied by individual point-wise ground truth annotations. We show that patch-based CNN methods can outperform a previously proposed approach that uses support vector machine (SVM)-based classifiers in conjunction with texture-based features. We compare the results of five different CNN architectures in our formulation of patch-based CNN methods. The Resnet152 CNN architecture is observed to perform the best on our annotated dataset of underwater coral reef images. We also examine and compare the results of four different FCNN models for semantic segmentation of coral reef images. We develop a tool for fast generation of segmentation maps to serve as ground truth segmentations for our FCNN models. The FCNN architecture Deeplab v2 is observed to yield the best results for semantic segmentation of underwater coral reef images.
DeepSegments is a tool for generating ground-truth segmentations for use in deep learning segmentation models. It provides a simple method for researchers to quickly segment their datasets. Users can choose from two different segmentation methods, SLIC and graph cuts. The program will make label suggestions by propagating user-given labels to unlabeled portions. Label suggestions can be given by an unsupervised k-means algorithm or a user-supplied pre-trained model.
This repository implements a variety of fully convolutional neural networks for semantic segmentation using Keras. Fully convolutional networks make classification predictions at every pixel in an image instead of giving a single class output. This means as output you get both a segmentation map and a classification distribution. This repository has a simple implementation of the original fully convolutional network (fcn) and the network proposed in the paper Multi-Scale Context Aggregation by Dilated Convolutions (dilation8). This clean implementation serves as a great starting place for fully convolutional models and was created as part of a research project on coral reef image data (the displayed image is a segmentation map of a coral reef).
As part of my consulting work for a brewery group I developed a computer vision tool for yeast cell counting and classification. Brew Counts is a tool for automatically counting and classifying yeast cells. Brew Counts makes in-lab yeast analysis fast and easy while improving your data’s accuracy, objectivity, and sample size. It allows users to tightly control pitch rates, track yeast health, and monitor fermentations without spending hours counting circles through a microscope.
