Tags: arthur+c+clarke, space, geosynchronous, satellite, clarke+belt

Tags: nasa, shuttle, atlantis, sts-122

Video Compression Basics

The process of compressing video is pretty complicated but is a well understood art. Uncompressed video is essentially a stream of images played in a sequence. When played quickly enough this stream of images can fool us into thinking the images we’re seeing are moving. Uncompressed digital video takes up an enormous amount of space.

bits per pixel * pixels per frame * frames per second

Let’s plug some numbers in and see how this works:
Bits per pixel: 24 (that’s one byte for each color of the RGB color space) but to make things simple lets make it 3 bytes
Pixels per frame: 307,200 (that’s 640 * 480 or a standard VGA frame)
Frames per second: 30 (in order to perceive motion the frames need to be displayed at a rate above about 20 per second)

Which brings us to: 3 * 307200 * 30 = 27648000 bytes of data needed every second. That is 26.36 megabytes of data every second for video to make no mention of audio. A one hour video would take up nearly 95 gigabytes!

This is where video compression comes in. In images and video there tends to be not only a lot of redundant information but also information that can be thrown away without people noticing. For instance a JPEG compressed image generates a mathematical representation of a source image throwing away a lot of information in the process but without looking closely you’d probably never realize it. I will let Wikipedia’s article explain the science and process of video compression. With modern compression schemes that example video stream that needed 26 megabytes per second can be reproduced with a tiny fraction of that data rate. This data rate for video is referred to as the bitrate.

Not All Compression is Equal

This is a major sticking point for George in his article. He is basing his entire argument on differing bitrates between different services without taking into account what compression systems are being used. The first he rails about is MPEG-2 which is used in digital video broadcast, DVDs, and Blu-ray and HD-DVD discs. It is fairly extensible and scales upwards well. It performs best at a bitrate above 3Mbps and can scale up to around 38Mbps. The bitrate needed by MPEG-2 to produce good quality video is determined by the type of footage being encoded. For Standard Definition video a simple scene without a lot of motion between frames looks good at 3Mbps while an action scene with a lot of movement or complexity will need around 9Mbps. For HD video the required bitrates are a bit higher, the ATSC broadcast spec in the US offers 19.4Mbps for MPEG-2 video which allows for good quality 1080i video. All this being said, MPEG-2 doesn’t hold a candle to MPEG-4 AVC in terms of video encoding quality. AVC has a whole slew of features designed to make up for deficiencies that cropped up in the MPEG-2 system. Instead of simply throwing more bits at the problem AVC does more work during the compression and decompression phases allowing high quality at far lower bitrates than MPEG-2. This was a specific design goal of AVC. Typically AVC can give the same visual quality as half to a quarter of the bitrate of MPEG-2 which means it can easily deliver good quality SD video at 1-2Mbps and HD video at 4-6Mbps (for 720p). AVC also scales up to around 40Mbps for 1080p Blu-ray content. More bits don’t hurt but they’re not necessary to have good looking video if your codec is smart.

Is it really HD?

The HD standard covers a lot of different options. For broadcast it covers two major resolutions, 1280×720 and 1920×1080. The first is the most common resolution for HD televisions and is the lowest hanging HD fruit. It’s a noticeable improvement over SD resolution. More pixels on screen allows for more visual detail and usually a better movie watching experience. This idea extends to the 1920×1080 resolution which is an even bigger improvement over SD. This resolution is less commonly found in HD televisions but does provide a nice quality increase. Because an engineering specification can never be simple the HD spec allows for a lot of variation around these resolutions. It allows for different framerates for video based on the framerate of the source video. For instance 24fps and 30fps roughly equivalent to film and NTSC broadcast framerates.

There’s nothing about a particular bitrate or video codec that defines HD, it’s all about the resolution. The HD content from iTunes and Xbox 360 Marketplace is little different from over the air or cable HD broadcasts in terms of resolution. Both of these services offer 720p content because the files are far smaller than 1080p files. Even with AVC or VC-1 encoding a 1080p video requires a pretty high bitrate and thus results in large files which in turn take a long time to download. The average speed for broadband connections in the US is only 1.9Mbps which makes for long waits for even low bitrate SD video to say nothing of HD.

HD Content

HD content from Apple and Microsoft is not really on par with high quality Blu-ray discs in terms of absolute visual fidelity. Blu-ray video is 1080p and because of the disc medium offers obscenely high bitrates, even for AVC and VC-1. Even for advanced codecs like these having a higher bitrate overhead helps with the visual quality. At the same time they aren’t somehow inferior to their high bitrate MPEG-2 counterparts, these advanced codecs were designed to provide the same visual quality at lower bitrates specifically to be able to transmit HD video over common broadband connections.

Since upgrading my AppleTV I’ve rented and watched a couple of HD movies. These movies looked great on my TV, even in high action scenes. Blu-ray versions of these might have looked a bit better but they’re on par with HD content from places like Apple’s HD Gallery. I was really impressed with the quality considering I only needed to buffer the video for a few minutes before it was available to watch. The previews don’t come close to the quality of the actual movies but you’re getting to watch those for free so I honestly don’t expect phenomenal quality from them.

I think there’s a bit of HD hype burnout on the interwebs. People have been let down by badly mastered Blu-ray and HD-DVD movies and cable and satellite providers cutting corners when it comes to encoding their content. When a content provider with a fixed sized pipe like a cable or satellite company says they’re giving you “more channels!” it means they’re clamping down the bitrate of their content to fit more into the pipe without caring about the quality. Unlike AppleTVs and Xbox 360s most cable and satellite providers don’t buffer the video locally so if they only have a 5Mbps pipe they have to squeeze the video into that 5Mbps even if to get the best quality that particular video ought to be encoded at 6Mbps. The ITU didn’t convene the VCEG to work on AVC because they had nothing better to do, they wanted a more advanced codec than existed in the form of MPEG-2.

Tags: h.264, mpeg-4, mpeg-2, compression

Musicians have long been screwed over by record labels. David Byrne wrote a nice companion piece to the Wired interview which gives some gory details on the depth of artist screwing. If you buy a $15.99 CD from Best Buy the artist only receives a 10% royalty (on paper*). They make a larger percentage on the sale of the same album on iTunes for $9.99 but a larger percent of a smaller number is still a smaller number. To understand how royalties work for recorded music HowStuffWorks has a very good explanation. It’s worth taking a few minutes to break your brain trying to figure it out. I put an asterisk next to “royalty” because artists don’t really make that full 10% that Byrne’s breakdown suggests. As outlined on this page onerous record contracts often leave artists owing the record labels money. Reading all of this it seems unthinkable that a musician would ever want to sign a contract with a big label.

This brings me to the aforementioned buzzwords. Some, obviously not all but some artists are learning the true meaning of The Long Tail and Social Networking. I don’t however use those term specifically in their technological contexts. Over the summer I followed The Format up and down the state while they were here on tour. They were touring with one of their own social networks. They didn’t pack the population of MySpace on a tour bus, only a few bands they were friends with that they wanted to tour with. They were touring with Reuben’s Accomplice, Steel Train, The Honorary Title, and Limbeck. Any fan of The Format that came to one of the shows was exposed to the other four (awesome) bands on the tour. Even if they didn’t fall in love with Robb and Patrick they at least got to see them play and might have bought a CD or poster. The Format like many other bands used their personal network of friends to help them put together a successful tour. If The Format had toured by themselves or with only unknown local acts opening they probably would not have had the turnout they did. From the size of the crowds at each show I think they pulled off a pretty impressive feat. The power of their social network amplified the attraction of the shows and drew in bigger crowds than any of those bands alone might have.

There’s also the effect of the Long Tail as described by Chris Anderson. Amazon allows for a market effect not seen in more traditional brick and mortar stores. Because they have a virtually unlimited stock of items and some context about their customers they can sell hojillions of low volume items and make a lot of money doing so. If you buy a popular book about a subject Amazon can recommend a less popular or more obscure book on the same topic. To apply this to touring bands; a few months ago I saw Beirut in concert. Opening for them was a “band” called Alaska in Winter. It turns out I really dug them and bought their record in the lobby. Simply having them open for Beirut was like Amazon suggesting I buy “The Elegant Universe” because I bought “A Brief History of Time”. This also holds for my example of The Format. If I had never heard of Limbeck but I saw them play with The Format I’d likely buy their CD and see them in a headlining show. I think more smaller groups not tied to bigger labels are going to try to market themselves collectively like this in the coming years. Instead of trying to make it on their own they might band (ha!) together with similar acts they found on Last.FM, Facebook, or even P2P networks and try to fill up venues. They’ll also place more importance of linking to each other in more meaningful ways that record store taxonomies. These sort of links are what allow a long tail economy to build up.

Any sort of music industry that survives this decade will likely be built more out of networks of artists and promoters than loan sharks. I think the music industry is going to move back to a model where artists directly interact with their fans through performances and simple networking. Before gramophone recording allowed a single performance to be reproduced thousands of times musicians had to perform to be heard and to make a living. Now that recording technology is ubiquitous and inexpensive I think the pendulum is going to swing back that direction. Recordings have lost monetary value because their distribution cost is next to nothing and their production costs get lower every time some kid buys a copy of Logic and builds a studio in her garage. People might pay a few bucks to listen to a good recording on their iPod but that’s not going to be a way for musicians to make real money, even if they keep 90% of the revenue.

Tags: music, mafiaa, radiohead, the+format, netwerk, itunes, network+effect

Tags: christmas, holidays, festivus, yule

Tags: atlantis, space+shuttle, sts-122, ISS

3GPP and the iPhone

The first set of files I tried were 3GP files generated with QuickTime Player. For video codecs the iPhone support MPEG-4 part 2 and part 10 (SP/ASP and AVC). Interestingly it does not support h.263 which is an option with 3GP video content. This is pretty interesting because MPEG-4 SP and h.263+ are very similar encoders. I imagine this might have something to do with hardware decoders on the iPhone (like the iPods) but I couldn’t say for sure. On the audio side the iPhone easily handles AAC-LC and AMR-NB. It does not support QCELP for 3GPP2 files though I would expect that considering it is a Qualcomm audio encoder typically handled on mobiles with Qualcomm basebands.

So what does this all mean? A large number of 3GP files found out on the internet will work very well on the iPhone. I was a bit surprised by this at first because I thought 3GP support might be locked down to what QuickTime offers as the iPhone’s cellular encoding preset, H.264 video with AAC-LC audio. Support for a wider range of 3GP files is nice because they tend to offer decent quality at low bitrates. As more mobile devices start supporting MPEG-4 part 10 (H.264/AVC) I’m sure it will become more prevalent on the web but at least those of us toting iPhone’s won’t be left with nothing to watch.

Reference Movies

An almost forgotten QuickTime technology from way back in the day are reference movies. These are QuickTime files that contain the same sort of track description atoms a normal movie might have but point to track data that is external to the file itself. Reference movies can point to files on a local disk or URLs anywhere on the internet. One of the more useful aspect of QuickTime’s reference files are its movie selection rules. When creating a reference movie you can set rules for each movie file referenced, a low bitrate movie can be used if the player is on a slow network connection while a higher bitrate movie will be selected if more bandwidth is available. Reference movies like normal QuickTime movies can also define language alternates, a movie can refer to an English and French audio track and depending on the system language selected when the movie is opened the appropriate audio track will be opened.

Recently the tool used to make reference movies MakeRefMovie was revamped to version 2. This Cocoa application has the same functionality of the old QTML version but is actively maintained and up to date. MakeRefMovie is designed to build a reference movie out of media files that have already been generated. You can encode an iPhone and AppleTV version of a movie file and set the appropriate playback rules with MakeRefMovie. As long as the references are available as valid URLs the reference movie will work properly. If embedded in a web page the QuickTime plug-in will determine the appropriate media reference to load based on the movie’s rules.

This is especially useful on the iPhone. A mobile-friendly website can embed a ref movie pointing to a high quality H.264 reference for playback on a desktop machine and a low bitrate 3GP file for an iPhone. This is what QuickTime Player’s “Export for Web” feature does. It encodes up to three versions of a movie, one for desktops, one for an iPhone on WiFi, and one for an iPhone on EDGE. When the reference movie is opened whatever device is playing it back will pick the appropriate movie to open based on its current configuration or bandwidth available.

Tags: iphone, ipod, encoding, quicktime, ffmpeg, h264, x264, aac, mp3, lame

Dynamic range compression is a technique used in audio engineering. Dynamic range is the difference between two values, in the case of audio the difference between the quietest sounds and loudest ones. A compressor can be given a target threshold and any sound louder than that threshold will be adjusted to to be below it. As the Wikipedia article explains it’s like an automatic volume control. Compression is used in music production as a way to bring the level of quiet instruments closer to the level of louder ones so they can be heard. Moving the quiet and loud sounds closing together makes them easier for our ears to distinguish, otherwise a loud sound will completely mask a quiet sound. If you’re walking down a busy street you can’t hear your footsteps because the street sounds are so much louder they drown out the quieter sounds.

Dynamic range compression is also the bane of music listeners everywhere because it is also used to make music sound louder. Carefully applied compression can give vocals a needed boost while keeping the fidelity of the rest of the instruments intact. Often however it is used on existing tracks to pull the quiet and loud sounds closer together boosting the overall perceived volume of the song. This is done to make it stand out when broadcast on the radio (which will further compress the song and make it sound even louder). The side effect is the range between quiet parts of a song and loud parts almost ceases to exist and instruments with a lot of acoustic energy like drums lose a lot of their power and punch. Overly compressed music is what makes everything on the radio sound so similar and so bad. Don’t just take my word for it.

Dynamic range compression is entirely different though from data compression used to encode MP3 and AAC files on a computer. MP3 and AAC are lossy compression formats which means data in the original file is thrown out of the output file to shrink it down. The data is discarded based on a perceptual model, parts of a sound we can’t hear don’t need to be saved in the output. Here’s a link to a breakdown of the process an MP3 encoder goes through to shrink down an audio file. AAC encoding follows a similar methodology. The goal of lossy data compression is to throw out data to make files smaller for transport or storage without you being aware upon listening that anything is missing. A well encoded MP3 file will do that. Of course some MP3 and AAC files will not sound as good as the originals due to problems with the encoding method or the output bitrate is simply not high enough to properly encode the music.

Modern records don’t sound bad because they’re MP3 or AAC files, they sound bad because the engineering on them sucks. Producers and A&R people want songs to sound as loud as possible so they stand out on the radio and in people’s cars when they listen to the CD. The CD audio specification supports a dynamic range of about 90db and most pop records only use about 5db of that. Dynamic range compression is what makes modern pop music sound really bad, not the fact you downloaded it off BitTorrent or iTunes in a data compression format.

Tags: audio+compression, dynamic+range, mp3, aac, twit

First a little history. Back in the 80s and early 90s Microsoft was not the dominant producer of productivity software used in businesses. The word processor market of the late 80s was practically owned by WordPerfect and the spreadsheet market by Lotus 123. Each of these applications had their own file formats that only they used and understood. In the early 90s Microsoft was looking to take their lunch money away with Word and Excel. Joel Spolsky posted an informative entry on his blog about the process of taking market share away from an entrenched competitor. He knows the story first hand because he was working on Excel during the period when it was struggling to wrest the spreadsheet market away from Lotus 123. One of the important parts of Microsoft’s strategy was to not only read WordPerfect and Lotus 123’s documents but also be able to write them out as competently as they could.

The ability to write out a competitors file formats meant that businesses were no longer tied to that vendor in their future purchases. If they could upgrade half of their company’s computers with Microsoft’s products and still maintain compatibility with their older systems with WordPerfect and Lotus 123 a huge barrier of entry would be removed for Microsoft. That is pretty much what happened. Proprietary formats are the key to vendor lock-in, a lesson Microsoft learned very well. With Office 97 Microsoft published specifications for their new .doc and .xls file formats. Ostensibly anyone could implement these specifications and have 100% interoperability with Microsoft Office files. Unfortunately for Microsoft’s competition the specifications were not complete. A good deal of the specifications referenced non-public or outright proprietary information that Microsoft would not share. While a third party could implement a Word file importer according to the specification a lot of reverse engineering was needed to actually be able to display and work with it properly. Even today, a decade later Word and Excel importers are nowhere near perfect in their handling of documented saved in Office.

In a repeat performance, Microsoft’s OOXML specification contains references to non-public and proprietary information. To fully conform to the OOXML specification an importer would need to not only fully understand the XML portions of the document but also fully implement an Office 97 file importer as OOXML documents can contain embedded Office 97 data as big binary blobs. OOXML is essentially XML wrapped around Microsoft’s old file formats, even with the specification in hand it is nearly impossible to fully implement. Submitting the specification for Ecma and ISO certification is a pretty cheap move on Microsoft’s part. They didn’t even bother persuing such a course until the truly open ODF specification was submitted and approved for standardization. Now they’ve even begun to bribe groups for their ISO ratification votes.

Microsoft hates the idea of ODF gaining popularity because it would help forment a rebellion against Office. As it stands businesses, governments, and pretty much anyone using Office needs to keep on the absolute latest version of the suite to maintain full compatibility with everyone else. This is an absolute cash cow for Microsoft and a money sink for their customers. Besides the price of Office licenses there’s also millions upon millions of documents locked up in a proprietary wrapper. A business might own the data contained in a Word file but they’ll need Microsoft’s blessing (in the form of a software license) in order to get at it properly. ODF has none of these restrictions and even goes a bit further by being a true XML language which allows it to be easily parsed and translated into other formats. Going from an ODF document to DocBook or LaTeX is not terribly difficult, and dead simple if you have an existing XSLT. Microsoft is not and will not be so accomodating.

The argument of ODF versus OOXML comes down to who owns data, the creator or the company that makes the word processor. Microsoft’s plan for the world is to have everyone pay them for the privilege of accessing their data. Proprietary document formats are a huge portion of that plan. If a superior solution comes along (ODF) Microsoft will do everything in their power to snuff it out of existance. The last thing they want is all the companies they stepped on and squashed to turn the tables and do the same thing to them.

Tags: odf, openoffice, oasis, no+ooxml, ooxml, iso, ecma