1. ©Elphel, Inc. 2011 Content availble under GNU FDLv1.3
10 years of Open Hardware
Olga Filippova
Elphel, Inc W S. #205
West Valley City, UT 84119
Designing and Manufacturing Hi-resolution, Hi-Frame Rate Cameras
Free Software and Open Hardware
Elphel is an Open Hardware Company; was founded in 2001, located in SLC.
Designing and manufacturing reconfigurable, high-performance network cameras.
In the last couple years we have focused a lot of resources into development of panoramic cameras.
©Elphel, Inc Content availble under GNU FDLv1.3

2. ©Elphel, Inc. 2011 Content availble under GNU FDLv1.3
Hardware: NC353L
Tech specs:
5 Megapixel CMOS Sensor
Exchangeable Lens (c-mount)
75 Megapixels/second Throughput
FPGA with Image pipeline
Embedded Linux computer
(webserver, Ethernet, USB, SATA connections)
Starting the company we had the idea to create a linux-based High-End camera that will be Free software and Open Hardware.
Why Open?
From the beginning we targeted a scientific market and tried to develop cameras that can be easily modified and reconfigured for different purposes while preserving high performance parameters.
Designing for the cell phones users vs designing for scientific research;
We actively pursue the users freedom to modify the camera for their particular applications, with GNU GPL license.
Since a hackable camera is our goal we design for Reconfigurability and user friendliness (easyaness in modifications):
the camera has Linux based computer inside
and FPGA responsible for high-speed image processing
The combination of processor running GNU / Linux and reprogrammable FPGA seemed to be more efficient in compare to using ASIC (Application Specific Integrated Circuit) or DSP (Digital Signal Processor). FPGA is much more flexible and allows to use sensors with different interfaces with the same system board. FPGA code is also Open.
A separate sensor board (allows to modify the camera to use different sensors).
The ability to use different sensors, including the ones that appeared on the market after the camera was developed. This ability comes from making a sensor board separate. For each new image sensor type a new board is designed.
©Elphel, Inc Content availble under GNU FDLv1.3

3. ©Elphel, Inc. 2011 Content availble under GNU FDLv1.3
Applications: Document Scanning
Elphel Model 323
35-mm format Kodak CCD image sensor
11 megapixels resoluion (4008 x 2672)
2 fps framerate
Nikon F-mount
High-performance, reconfigurable and Open Design are the factors that helped us to get our first large order for 11Mpix camera with a Kodak CCD sensor. It used 2 system boards from a regular camera (for a quicker Camera-to-PC upload); additionally the new sensor board was designed for that sensor. So, the modular structure of the hardware helped not to build the whole camera from scratch.
In this project we were able to successfuly compete with other leading manufacturers of digital cameras – that was due to the Open Design, High Performance and flexibility that allowed a quick implementation of such specific things like electronic shutter and fast download – none of the off-the-shelf cameras offered at that time.
©Elphel, Inc Content availble under GNU FDLv1.3

4. ©Elphel, Inc. 2011 Content availble under GNU FDLv1.3
Applications: Panoramic Imaging
and the next project was to provide Elphel camera modules for the high resolution panoramic images.
©Elphel, Inc Content availble under GNU FDLv1.3

5. ©Elphel, Inc. 2011 Content availble under GNU FDLv1.3
Variety of Applications:
SCINI - Antarctic Underwater Exploration Robot
Submersible Capable of under-Ice Navigation and Imaging
Variety of Applications is allowed by Open Design and highly reconfigurable nature of the camera:
One of the applications is SCINI project. It's an underwater robot designed by Moss Landing Marine Lab for observing the life under the Antarctic ice.
In the robot they used 2 cameras – one is set to a higher frame rate and lower resolution for navigation, and another - for taking high resolution photographs.
It is enough to drill a 20 cm. hole in the 6 meter thick ice using hand-held Jiffy ice drills, which allows research teams to survey many areas during their short field season.
Engineers of the lab developed the robot and integrated our open hardware with minimal technical support from our side, because the camera is open for modifications and documentation is available.
Moss Landing Marine Lab
©Elphel, Inc Content availble under GNU FDLv1.3

6. ©Elphel, Inc. 2011 Content availble under GNU FDLv1.3
Applications: NASA Global Hawk UAV
Aerial Near Space Exploration
Another interesting application of Elphel cameras is the photographic imaging of Earth from near-space. Camera was integrated in scientific equipment and installed on NASA unmanned airplane, that flew at 20 km altitude for the duration of 36 hours.
The team used our public mailing list for some basic support during the integration process, but mostly they could perform this task on their own.
©Elphel, Inc Content availble under GNU FDLv1.3

7. ©Elphel, Inc. 2011 Content availble under GNU FDLv1.3
Applications:
Apertus - Open Source Cinema Project
Another application is rather unusual, since the camera is used not for scientific or engineering purposes, but artistic . It is Apertus – Open Cinema project, created by a group of artists, with the goal to develop open hardware cinematographic camera.
They have attached LCD screen for camera control, designed the rod support to tie all the camera hardware components, added a battery pack and attached a viewfinder.
Also they coded their own Open Source user interface designed for filming.
©Elphel, Inc Content availble under GNU FDLv1.3

8. ©Elphel, Inc. 2011 Content availble under GNU FDLv1.3
Boards Overview
10353 System Board
10338 Sensor Board
10369 IO Board
10359 Multi Sensor Board
Camera hardware has a modular structure (along with high performance). These are the main and most commonly used boards:
The system board has a CPU running Linux and we can use it's universality to implement open code done by others as well as our own in the camera, even the processor is not the most common one – it is Etrax, by Axis. The FPGA (Field Programmable Gate Array) is responsible for image processing and compression.
Interface board is for additional interfaces like SATA, USB or Serial port. It makes possible attaching HDD to the camera to record images on it, and USB external devices like GPS. This board also allows precise external synchronization. It is helpful for software development and troubleshooting, as it provides serial console.
Separate Sensor board allows to use different sensors with the same system board.
With the Multiplexer board up to 3 sensors can be attached to the camera. Attaching 2 sensors makes it a stereo camera, for example.
More FPGA programming can be also done on the Multiplexer board, without compromising the FPGA code on the system board.
©Elphel, Inc Content availble under GNU FDLv1.3

9. ©Elphel, Inc. 2011 Content availble under GNU FDLv1.3
Camera Configurations
Basic camera NC353
Stereo
The most common camera set-ups;
New ones: Camera with IMU and GPS
HDD
GEO
©Elphel, Inc Content availble under GNU FDLv1.3

10. ©Elphel, Inc. 2011 Content availble under GNU FDLv1.3
/* source is inside */
Camera is regonfigurable and user/developer friendly.
Controlled through a web-based user interface.
HTML, JavaScript, PHP
C, C++
Verilog HDL
Languages:
Most of our clients are universities, scientific labs and start-up companies – all working on the new technologies, doing experiments. And as their applications can be very different, we are considering that and work on providing an easy access to camera modifications.
Camera is reconfigurable on different levels from PHP to FPGA.
PHP is the most commonly used language by web developers and non-programmers – it allows more users to create their own application without knowledge of system languages.
The camera has a web-based user interface and supports HTML, JavaScript, PHP. The PHP extension allows to write your own API without loosing the functionality as in the case of the universal API.
The deeper modifications are possible including C/C++ programming.
FPGA – is programmed in Verilog HDL and is Open Source . Tools for FPGA programming are not Open Source, unfortunately, but at least they are free.
©Elphel, Inc Content availble under GNU FDLv1.3

11. ©Elphel, Inc. 2011 Content availble under GNU FDLv1.3
Code is on the SourceForge.net
Schematics, PCB layout & Documentation are available on
wiki.elphel.com
All of the Elphel sources are available for download through SourceForge, which we also use as a version control tools, and upload our work in progress.
The documentation with schematics and PCB layout can be found on our wiki site.
©Elphel, Inc Content availble under GNU FDLv1.3 11

12. ©Elphel, Inc. 2011 Content availble under GNU FDLv1.3
Licenses:
GNU GPL V3
GNU Free Documentation License V1.3
CERN Open Hardware License V1.1
The source code is distributed under GNU General Public License, and schematics and PCB layouts are under Free Documentation License.
Hardware and mechanical design are released under CERN Open Hardware License.
©Elphel, Inc Content availble under GNU FDLv1.3

13. ©Elphel, Inc. 2011 Content availble under GNU FDLv1.3
Camera users around the world
Open Reconfigurable Design
High End Product
Open Design and High Performance of our cameras are valued by our clients all around the world. Elphel cameras are present on all 7 continents.
Many of these cameras on the map are in Universities, research labs and other innovative institutions.
©Elphel, Inc Content availble under GNU FDLv1.3

14. ©Elphel, Inc. 2011 Content availble under GNU FDLv1.3
Eyesis - Panoramic Camera
9 Color 5 Mpix sensors
Precise image capture synchronization
GPS Geo-tagged images
Very small parallax
Free Software and Open Hardware
Last year we have developed panoramic camera Eyesis, which uses the same camera modules and a single lens camera:
3 system boards, 3 interface boards, and 9 sensors boards. It is designed to take simultaneously 9 images (8 around and one pointing up) at up to 5 Frames per second. The total resolution is 45 Mpix ( 38 Mpix after stitching).
The camera can continuously take images for 11 hours, recording them to 3 internal Solid state drives.
It can be mounted on the car or on person.
©Elphel, Inc Content availble under GNU FDLv1.3

15. ©Elphel, Inc. 2011 Content availble under GNU FDLv1.3
Interface for Camera Controls
Image Acquisition is controlled from a laptop with web based Graphical User Interface.
©Elphel, Inc Content availble under GNU FDLv1.3

16. ©Elphel, Inc. 2011 Content availble under GNU FDLv1.3
Post-processing Tools
Image extraction and Indexing
Metadata extraction
RAW conversion
Aberration correction
Image enhancement
Panorama stitching - Hugin
Integration with map
Tools for complete pipeline for post-processing captured RAW image-sets are developed under GNU-GPL license by Elphel and by 3rd party allow us to create 360 degree panoramas and integrate them with the Open Street map.
We have also developed the panorama position and orientation editor before moving forward with the IMU data processing.
Panoramas are integrated with Open Street Map
©Elphel, Inc Content availble under GNU FDLv1.3

17. ©Elphel, Inc. 2011 Content availble under GNU FDLv1.3
Current Developments: Eyesis 4π
Full Sphere Panoramic Camera
High Resolution: 120 Mpix, total (64 Mpix - panorama)
Images are synchronized with GPS and IMU – allows presize position and orientation of panoramas
Eyesis 4Pi is the next generation of Panoramic Camera by Elphel. It will be able to capture high-resolution images in full 360 degrees and create 4 Pi spherical panoramas. The first prototypes are being constructed.
24 sensors ensure a uniform resolution over the entire covered area.
It also features an Inertial Measurement Unit (IMU); together with integrated GPS they provide precise position and orientation of the camera.
©Elphel, Inc Content availble under GNU FDLv1.3

18. ©Elphel, Inc. 2011 Content availble under GNU FDLv1.3
Eyesis 4π R&D
Camera Calibration
Aberration and Distortion Correction and Image Sharpenning
Camera as a measurment tool for Photogrammetry
Inertial Measurement Unit (IMU) :
3D Position and Orientation
Compensate for Rolling Shutter Distortion
3D Reconstruction
There are several research projects involved in development of Panoramic camera and related image processing that will allow it to:
1) create very sharp high resolution panoramic images, and 2) create 3-dimensional panoramas.
A lot of it has to do with the precision of camera calibration witch will compensate optical distortions and aberrations of the lenses, and also will make the camera as a measurement instrument for photogrammetry.
Photogrammetry is the practice of determining the geometric properties of objects from photographic images.
©Elphel, Inc Content availble under GNU FDLv1.3

19. ©Elphel, Inc. 2011 Content availble under GNU FDLv1.3
Camera Calibration
optical aberration measurement and correction developed for Eyesis cameras
Camera calibration for aberration correction is done by 1) centering the sensor and Lens with the use of Lens Adjustment Machine (designed by Elphel) and the pattern.
By taking the image of the pattern and knowing the ideal pattern we can determine point spread function of each area of the image and then use it for sharpening areas of the real world picture.
Each Lens has it's own unique aberrations, - the further from the center of the lens the more aberration there usually is. Each lens is calibrated by “looking” at the pattern and comparing the image we are getting from the lens with the “Ideal” image of the pattern. This way we will know the aberrations and distortions of each lens and during post-processing of the real image we can correct them.
Lens is alligned and centered with sensor
©Elphel, Inc Content availble under GNU FDLv1.3

20. ©Elphel, Inc. 2011 Content availble under GNU FDLv1.3
Aberration Correction - Results
With 1/2.5″ 5 megapixel sensor we achieved average sharpness improvement over the image area around 40% compared to the raw images, effectively doubling the resolved pixel count.
This sharpening is not uniform as sharpening in image editing software; it takes into account the individual lens aberration and sharpens areas of the image.
©Elphel, Inc Content availble under GNU FDLv1.3

21. ©Elphel, Inc. 2011 Content availble under GNU FDLv1.3
Photogrammetry
Camera as a measuring tool – needed for 3D reconstruction
Precise Angle Measurment
IMU calibration – for higher presision
To be able to measure distances to the objects (for 3D reconstruction) we need to calibrate camera for distortion.
The new Eyesis 4Pi camera will have 24 camera modules (Lens+Sensor). To be able to calibrate all 24 modules we have designed a special Goniometer machine that rotates 2 axises w/ precise angle.
It will callibrate the camera for:
1) distortion and aberration corrections;
2) provide camera module orientation relative to the whole camera;
3) using the color test chart we can also perform flat field correction.
4)Along with optical calibration we will calibrate the IMU
©Elphel, Inc Content availble under GNU FDLv1.3

22. ©Elphel, Inc. 2011 Content availble under GNU FDLv1.3
Inertial Measurement Unit (IMU):
Position and Orientation of camera for Panorama Sequencing
IMU and GPS integrated with the camera and synchronized with images allow to orient and position panoramas in space.
We have developed the panorama position and orientation editor before moving forward with the IMU data processing, but having IMU data will allow to make it automatically.
©Elphel, Inc Content availble under GNU FDLv1.3

23. ©Elphel, Inc. 2011 Content availble under GNU FDLv1.3
Inertial Measurement Unit (IMU):
Compensate ERS (Electronic Rolling Shutter) distortion (like in the cell phone cameraca), by recording position /orientation of camera several times within each frame
This is the example of the Electronic Rolling Shutter distortion, characteristic for the CMOS sensor.
Everybody probably had experienced this effect when taking picture with the cell phone – images get skewed if the camera was moving or rotating.
This effect will also be compensated in post-processing with the recorded IMU data.
©Elphel, Inc Content availble under GNU FDLv1.3

24. ©Elphel, Inc. 2011 Content availble under GNU FDLv1.3
Optical Measurement of camera position and orientation at 2 shots
Our goal of precise optical and inertial calibration of the Eyesis 4Pi camera is to be able to reconstruct 3-dimensional panoramas.
1 meter below the camera head we place 2 additional sensors to be able to measure the 3D points of the objects, visible by the top and bottom sensors.
This 3D information will help to improve precision of the exact distance and orientation between camera in 2 or more positions;
With this information we can determine distance to the objects within the scene and to be able to reconstruct 3D panorama.
Exact distance between camera shots will also increase the absolute position of the camera at each frame, obtained with IMU.
©Elphel, Inc Content availble under GNU FDLv1.3

25. ©Elphel, Inc. 2011 Content availble under GNU FDLv1.3
HDR with moving camera,
will be possible with textures on 3D mesh
3D panoramas – HDR (High Dynamic Range)
Every other shot is overexposed, so by combining the 2 consecutive shots we can create HDR image once we have the 3D mesh reconstructed.
©Elphel, Inc Content availble under GNU FDLv1.3

26. ©Elphel, Inc. 2011 Content availble under GNU FDLv1.3
Research Projects
Image processing: optical aberration and distortion correction, image enhancement
IMU data processing for rolling shutter correction
IMU -position and orientation
3D Reconstruction
HDR with moving camera
Outline of the research we are doing at Elphel, and the engineering problems we plan to conquer.
©Elphel, Inc Content availble under GNU FDLv1.3

27. Development Blog: http://blog.elphel.com/
More information is available:
www3.elphel.com
Development Blog:
Documentation: wiki.elphel.com
Code: SourceForge.net
Visit our office: 1405 W 2200 S ste. 205, SLC ,UT
For further reference, please visit our web site and blog.
Thank you for your attention.
©Elphel, Inc Content availble under GNU FDLv1.3