Progress: October 2005
From LadypackWiki
| Table of contents |
Wednesday, October 26
Was having a lot of problems getting the ladybug2 to work properly with Linux last week. It worked great with my own personal laptop and the Kouwell 1394b Carbus adapter, but not with any other combination of laptops (Seth's T42p, the new HP dv1000) or firewire adapters (dv1000 builtin 1394a, Unibrain 1394a Cardbus, Kouwell 1394b Cardbus). Asked around on the dc1394-devel mailing list, but it doesn't look like anyone's done this sort of thing before. So I ended up biting the bullet and porting all the capture code back to Windows. At least I know I can legitimately pester Point Grey Research if I have problems there.
Have decided to avoid Matlab like the plague. Began writing a python framework for analyzing data streams collected by the capture programs. Have a working viewer.
TODO:
- get an intersense InertiaCube
- contact Antonio Torralba about his code
- integrate the laser range finder into the capture system
Thursday, October 20, 2005
Need to decide what format to store the logged video in. Either use point grey compressed format, recompress to JPEG, or store uncompressed. Ran a few tests to compare
| method | bytes per frame | amount of video per 500 GB at 15 fps | load time per frame |
|---|---|---|---|
| PGR compressed stream | 300 kb | 30.9 hours | 450 ms |
| JPEG quality 80 | 450 kb | 20 hours | 310 ms |
| JPEG quality 90 | 710 kb | 13 hours | 350 ms |
| JPEG quality 100 | 2.5 MB | 3.6 hours | 550 ms |
| uncompressed | 14 MB | 39 minutes | 80 ms |
load time is the time it takes to read a single frame from disk into memory and convert it into an RGB byte buffer.
I'm not sure how much signal loss there is by recompressing the image to JPEG.
Seems like storing uncompressed video just takes up way too much space.
Might as well stick with storing it as the original PGR compressed stream for now, and deal with the slow load time since it doesn't look like the other methods are much faster while also keeping disk usage to a reasonable level.
later
Charlie came by and he got some photos for me.
 |  |  |
Tuesday, October 18, 2005
Unfortunately, it turns out that the current plans for capturing video won't work at 30 fps because the PCI bus saturates at a theoretical 133 MB/s, and a practical <100 MB/s. Since the camera is already consuming about 71 MB/s, this leaves almost no bandwidth for the hard disk. To achieve 30 fps, I see two options:
- buy an IBM Thinkpad T43, which supports PCI Express. Figure out how to get two firewire controllers onto the laptop, or use the USB-2.0 ports. One firewire controller goes to the Ladybug2, the other (and/or USB ports) go to external hard disks.
- put together a barebones system from parts. Get a motherboard supporting PCI express and setup a RAID-0 configuration of SATA or SCSI hard disks. The Western Digital Raptor WD740 looks like the fastest disk out there for SATA. Haven't looked into SCSI drives yet, but a RAID-0 SCSI system seems like it would cost about $1,000 more than a SATA equivalent.
link to WD740 vs SCSI comparison (http://www.gamepc.com/labs/print_content.asp?id=x15vs740gd&cookie%5Ftest=1)
At this point, it seems like the 2nd option is cheaper and more flexible.
Candidate motherboads on Newegg (http://www.newegg.com/Product/ProductList.asp?Submit=Go&DEPA=0&type=&description=motherboard+1394b&Category=0&minPrice=&maxPrice=&Go.x=0&Go.y=0)
There's also the question of: how badly do we want 30 fps? If we were to settle with 15 fps, we could just buy the G-RAID external RAID system and slap it on top of the existing backpack.
later:
Am having trouble reaching 30 fps with the ladybug2 and libdc1394. Posted a message to libdc1394-dev asking for some help.
Monday, October 17, 2005
Now that I have data to analyze, I need to decide what environment to analyze it in. The two main options that stick out in my mind are to go with Matlab and to use Python and various modules and plugins. There was recently a discussion on csail-related that suggested SciPy (http://www.scipy.org) as a potential candidate. Charlie Kemp seemed to have great success with his own homebrewed codebase.
I need the following in a good data analysis environment:
- visualization and display capabilities
- functions for manipulating large matrices and data structures
- good mathematical and scientific toolsets (signal processing, image processing, etc.)
Matlab provides all of these, but the big problem I have Matlab is that it sucks for programming. The I/O capabilities and programming language itself make it very difficult to work with.
Spent a good part of today reading through SciPy and figuring out how much I can use it. It seems like an aggregation of many different modules, like the Python Imaging Library (PIL), Python Numeric, and some others? Charlie's code uses a lot of similar stuff. PyGame looks promising for visualization and display.
Also am finally starting to really dig into Charlie's code and see what I can use.
Paul from PGR wrote back with details on how to deal with the Format 7 Mode 6 JPEG compressed data from the Ladybug2.
Hi Albert, The engineer who is responsible for the Ladybug SDK looked at this issue today and found that all the information you need is in the compressorheaderinfo.cpp and compressorheaderinfo.h files which are in the folder: C:\Program Files\Point Grey Research\PGR Ladybug\src\ladybugCommon. The function CompressorHeaderInfo::parse() is used to parse the 24 JPEG images in the frame data pointed by pData . Each parsed JPEG image info is stored in array ImageInfo[] m_arInfo. Note that the source code is for Windows and therefore in little endian.
So this means that if I want to process the ladybug images in Linux, I'll have to port that parse() function. Bit of a pain, but better than having to write a windows program to process it and then send it back to the linux machines. Started looking through that code also...
Sunday, October 17, 2005
Took a look at external RAID solutions and it seems like external systems with hardware RAID-0 controllers and a Firewire interface might be sufficient. The only problem here is that the laptops I've been using only have room for one firewire controller and the bandwidth on that controller is already saturated by the Ladybug2. Is there such a thing as a laptop with four PCMCIA ports? Will we saturate the PCI bus with all this data? Some googling turned up (http://www.macworld.com/2005/03/reviews/fw800raid/index.php) a few firewire RAID systems that show promise. Apparently the video editing community is very concerned with having fast hard disks because they need to churn through a ton of data.
Finishing writing a simple capture app and took some real data on tether tonight. Am having a problem with the microphone where the signal-to-noise level is ridiculously low. I suspect this is a problem with the Linux sound drivers because using the MS Windows voice recorder works fine. Hopefully I won't have to deal with this problem when the T42p gets back from the repair shop.
Thursday, October 13, 2005
Spent a couple days trying to code something in Windows and then remembered again that I hate writing applications in windows. Was stuck for a while on this strange problem where the Ladybug2 SDK would refuse to capture video at anything greater than 4 fps. I traced the problem in ladybugcap and it seems that ladybugGetCameraInfo() always returns the maximum bus speed as LADYBUG_S100, which forces the framerate to LADYBUG_FRAMERATE_3_75. If I hardcode LADYBUG_FRAMERATE_30 when calling ladybugStartLockNext, then the image capture doesn't work. I've tested this in two different machines with the same result. Initially suspected that this had something to do with recently upgrading to the Ladybug2 1.2.0.9 alpha SDK, so I tried downgrading to the 1.2.0.5 alpha SDK that came with the Ladybug2. This didn't work. I also suspected that there was something wrong with my code, but this problem also occured with the ladybugcap application that PGR supplied.
I did not end up resolving that problem because the hard disk on the IBM Thinkpad T42p failed last Wednesday and I lost the windows code I was working on. Damn.
That won't be replaced for a bit, so I tried working on Linux again. Initially ran into problems with libdc1394 not working properly with the higher bus speeds of 1394B. This was fixed by upgrading to the most recent version of libdc1394 (2.0.0 pre-5), which works beautifully with the ladybug2. I have two source files that demonstrate the recording and playback capabilities on linux: ladybug2-record-simple.c and ladybug2-playback-simple.c
With libdc1394 2.0.0 pre5, I was able to capture at 15fps full resolution uncompressed. Unfortunately, I was not able to record that fast because the IDE hard disks couldn't keep up. Once the hard disk buffer is full, it falls terribly behind the video capture rate. Sent an email to PGR asking them for information on how to decode the Format7 Mode6/7 JPEG compressed images that the compressor unit can put out. Hopefully the compressed images will be more manageable than the full uncompressed frames.
Each uncompressed frame is 1024 x 4608 pixels. At 8 bits per pixel, this is 4,718,592 bytes per frame. At 15 frames per second, this is 70 MB/s. One minute of uncompressed video is easily 4.2 GB of data. I'm not sure what the data rate for compressed frames at 30 fps is. Either way, it seems that we'll need to find a storage system than can handle writing data that quickly.
In the meantime, I'm adding audio-recording. This is fairly straightforward in Linux - just open /dev/dsp as a file, adjust the sampling settings with ioctl, and read.
note: This wikipedia page (http://en.wikipedia.org/wiki/Advanced_Technology_Attachment) suggests that IDE hard disks aren't up to the task for what I'm trying to do, and also that PCI bus speeds are limited to 133 MB/s. Since Cardbus is just a different form factor for PCI, the firewire card in the laptop is probably using the PCI bus and contending with the hard disk for usage. This may be problematic...
Monday, October 3, 2005
Soldered a cable for the battery pack on Friday, but it didn't seem to work. Had two male 5.5mm x 2.5mm power connectors that seemed like they matched the ladybug battery pack and Compressor unit, but when I connected it up, nothing turned on. Maybe the battery pack couldn't provide enough power? It's a 12-volt lead-acid battery, which pt grey said is enough. It's also possible that I did a bad solder job (never soldered before) but I tested the cable with a multimeter and it seemed like the connections were fine and that there weren't any short circuits. Don't really want to bother much more with batteries for now so I ended up going home and digging up an extension cable and have now strapped the AC adapter to the backpack unit. Will just run on tether for a while...
Now that I have the hardware in a usable state, it's time to work on getting a data collection software system setup. Probably the first thing to worry about it synchronizing the three different data streams that I'll be dealing with (video, audio, laser range). Seth suggested using audiovisual cues like the little black and white movie clapboard to detect when streams match up, but that seems like it would involve a lot of manual effort for each data collection. Instead, am trying to write a custom application in C++ that grabs data off of the input streams and timestamps them as they come in, writing it all to one large file. Spent today learning how to do C++ in windows and familiarizing myself with the Ladybug SDK. Hopefully will have this done by tomorrow (??)
