AVCHD Backup Strategy

Back in December I purchased a new video camera–the High Definition Canon Vixia HF100. Naturally, I did a lot of research before purchasing it as I wasn’t very familiar with the AVCHD format–a high definition H.264 codec that is very efficient and is able to store a lot of video data using a minimal amount of memory. The codec does this by only recording changes in the images rather than full frames. I don’t know the intricate details of the technology, however, it is quite impressive how good the quality is for such an inexpensive camera. You can see the quality this camera can produce by going to Vimeo.com and searching for the tag HF100. So if the video is so great at such a decent price, what’s the catch?

Simply put, the catch is that there are no applications available currently that can edit the AVCHD format natively on the Mac. The Canon camera, and other AVCHD cameras like it, create files that use the .mts file extension. Again, I’m not familiar with all of the technical details, however, I do know the only way to work with the files on the Macintosh is to convert them to some other format. iMovie 08/09 both handle the AVCHD format by importing the .mts files and converting them to the Apple Intermediate Codec (AIC).

iMovie provides several different options for importing the video data, however, if you decide to go with the “Full HD” option when importing, which I recommend if you want to have high quality output, your files will be GINORMOUS! Mine run in the neighborhood of about 1GB per minute of video. That’s huge!

Back in the days of tape, yes probably most people are still using tape, you could simply fill a tape with video and then put a date on it and add it to your physical media archive on a shelf somewhere. This works well as the shelf life for tape, while not infinite, is really quite good. This is exactly how I’ve done archiving in the past (and when I still use my old DV camera for various tasks). The question for me has become, how can I archive such massive amounts of data as efficiently and with some level of confidence that the data won’t get lost. Here are the the various issues I considered in coming up with my backup strategy:

  • Hard Drive: This is a viable option for sure as disks keep getting cheaper, however, I have heard enough stories of failed hard drives that this one makes me a bit nervous. I can see using it as a secondary backup strategy, but I don’t trust hard drives enough. RAID seems like a reasonable solution, but that requires more money as you need redundancy which requires multiple disks.
  • Burning AIC Format to DVD: This strategy would just be too much work. Remember that you can only hold 4.7GB on a single layer DVD-R/DVD+R. This means that you would be able to store less that 5 minutes of video on a single disk. While I worked hard to force myself to record short clips and keep things maintainable at the clip level, manually splitting all of the clips up in order to only get 5 minutes of video per disc seems just unmanageable.
  • Burning AVCHD Format to DVD: The shelf life of optical media is also questionable, however, this is the strategy I am pursuing at the moment. Ideally solid state drives will get much bigger and cheaper at which point I will move everything over. For now, however, burning to DVD is the strategy I’m going to use

So I’ve decided that burning the original files to DVD is the best way to go for now as the shelf life of optical media is reasonable enough that my data should keep in the near term. And I’ve chosen to archive the video in its compressed AVCHD format. Here, however, is where the strategy gets interesting. I want to have a way to get the clips off of the DVD without too much hassle while using iMovie. Your strategy may differ if you use some other editing software. Here is what I mean.

iMovie automatically detects the existence of AVCHD movie clips when you either plug in your camera or mount the SD card with a card reader and loads the import utility. What I have done is taken the data from the SD card; copied it to my hard drive; and then made a disk image (.dmg) file out of it. When this disk image is mounted while iMovie is open, it will automatically detect that the image contains the AVCHD video files and load the import utility just as if it were physical media being mounted. Pretty cool eh? So, yes there is a bit of work on the front end, however, when you burn these disk images to a DVD, you can just mount them directly when you put the DVD in at some point in the future when you are ready to import and start working with the clips. So here are the steps you need to get this done.

  1. Plug in your camera or mount the SD card with a card reader. Then run iMovie
  2. When iMovie runs, it will automatically detect your camera or SD card and will load the import utility. Click the “Archive All…” button at the bottom of the window. Choose your Desktop as the save location (you’ll see why shortly) and give the file a name and iMovie will back up the data to your hard drive.
    Archive All...
    Archive All...
  3. The backup process will take a little while depending upon how much data you are saving. Once it has finished, you need to open Terminal.app, which is located in /Applications/Utilities.
  4. Now we need to create the disk image and will do so using a command line utility called hdiutil. You need to change directories to your Desktop. To do so, at the command prompt in Terminal type cd Desktop and press Enter. You are now in the Desktop folder where we saved the archive.
  5. To save the archive folder as a disk image, type the following command followed by the enter key: hdiutil create -srcdir ArchiveFolder AVCHDImage.dmg. This command will take a little while depending upon how much data your are archiving. You will know the process has finished when the command prompt returns. You should expect it to take a while if you are saving several gigabytes worth of data so be patient as the progress that hdiutil shows isn’t very informative. It is, however, working.

    Note: ArchiveFolder is the name of the folder you saved when you had iMovie create the file on your Desktop. This command will not work if you specified spaces in your folder name. If you do use spaces, you will need to escape them. This means that if you created a folder called ‘Archive Folder’ for example, your command would need to be: hdiutil create -srcdir Archive\ Folder AVCHDImage.dmg. Notice the backslash before the space. Add this backslash before any spaces in your archive folder’s name.

  6. Once hdiutil has finished creating the disk image, you can now burn it to a DVD. Insert a blank DVD and open a new burn folder on the desktop or somewhere handy. Then drag the disk image to the burn folder and click burn.
  7. Once your burn is finished, insert the DVD back into your drive (if it was ejected) and let it mount on the desktop. Open the disk and double click the .dmg file. This will mount the image. You can click Skip during the verification process if you want to have it simply mount without verification. I usually skip it myself as I’m impatient.
  8. Open iMovie again. iMovie should detect the disk image and automatically load the import utility. If it doesn’t then you may have done something wrong. Shoot me specific questions in the comments if you need help here.

There is one caveat I should mention to using this technique. I suggest you keep your data size on your SD card to 4GB or so. In fact I use a 4GB card for precisely this reason–it fits well with my backup-to-DVD strategy since a single layer DVD only holds 4.7GB. If you have a hard drive based camera, you’ll have to keep an eye on the size of the data in order to ensure that your data will fit on a single DVD, but this shouldn’t be too difficult. I have a 16GB SD card for those cases where I need to capture longer shots like when capturing a long presentation, but most of what I do allows me to keep smaller files and therefore reasonable backup sizes.

So that’s it. It’s not a perfect strategy as the shelf life for DVDs is a little disconcerting, however, I have every confidence that solid state memory as it becomes more and more ubiquitous will be the best backup/archival methodology once the drives get bigger and the costs continue to go down. In the mean time, this strategy seems to be the best one for backing AVCHD video. If you have other strategies that have worked for you, share them in the comments. I would love to hear your ideas.

Good luck.

BecomeAnXcoder, And Give, Please!!

This was posted on cocoalab back in October and I’m just now seeing it, but this online book BecomeAnXCoder looks fantastic! I will start pointing people who are interested in Mac OSX programming who have no programming experience there from now on. I hope these guys can continue to produce high-quality content. Which appears debatable–not because they don’t have the ability to create quality content–clearly they do, but because they seem to be having trouble supporting it.

I noticed when I visited the site this note in bold at the top:

Please Donate! Our bandwidth is at record levels: 29 GB in March, and donations do not even come close to covering our costs. If you appreciate our work, please take a minute to send a donation.

I do appreciate their work myself, but it’s mainly because I would send more people, new programmers that is, to them–which in turn is what is giving them the problem of using too much bandwidth in the first place.

To the folks over at Cocoa Lab, this statement really makes you sound desperate. I hope you can get the support you need, but when you give away content for free, a statement like this suggests that maybe your ultimate intention wasn’t to give it away, but rather to give it first so that people may donate. If this is a business model for you, it looks like it’s not working out the way you expected, so, with the deepest sincerity, I thought I might offer a few possible solutions to your predicament.

  • Talk to a publisher. This is really good content that you could probably sell. Show it to a publisher and see what they think. I’ve heard of publishers that *may* be accepting proposals from Mac Programming authors
  • Get cheaper webhosting. There is this company called 1and1 that provides shared hosting (which would work fine for the content you are providing) and give you 2.5TB, yes that was TERABYTES!!, of transfer bandwidth per month for the affordable price of $9.99 per month. You can sign up at 1and1 now. And yes that link is my affiliate link. I will get a kickback if you sign up!! That’s not a sales pitch–just a disclaimer.
  • Convert the site to a blog site. If you make your site into a blog rather than a book and continue to add new content, the benefit to the Cocoa development community continues to grow. It’s great with the content that’s there now, and to those who haven’t checked it out yet there is a *ton* of content, but keep it coming.

I write this not to chastise, but to be helpful. I hate to see good content producers go un-rewarded for their efforts. The solution, however, is either to change your approach, or change your expectations. Best regards to the BecomeAnXCoder writers. Keep up the great work.

Scott Stevenson on Writing Copy For Product Pages

Scott Stevenson, webmaster and author for his Cocoa development site Cocoa Dev Central and weblog Theocacao, has written a new post on writing copy for product pages. The ‘keep it simple’ meme is the gist, but read what he has to say.

This is quite interesting to me as I have a product release that is imminent. Well, maybe not imminent. The code is almost done. Now I need to write some copy, figure out the PayPal IPN, incorporate Sparkle… etc. etc. Ok. Nevermind. It’s not imminent.

Anyhow, thanks for the post Scott. This is good stuff.

Core Animation Tutorial: Dashboard Effect

I just finished another Core Animation tutorial and posted it at CIMGF over the weekend. This was a pretty fun one. I wanted to duplicate the effect of the Dashboard widgets flying in and out from off screen and this is what I came up with. Take a look and let me know what you think. Core Animation Tutorial: Dashbaord Effect.

Zarra Studios releases iWeb Buddy, the post-processor for iWeb

iWeb Buddy Now AvailableColorado Springs, Colorado – Zarra Studios LLC announced today the release of iWeb Buddy. iWeb Buddy is a post processor application for iWeb that is designed to bring additional functionality to iWeb that professionals and “pro-sumers” have come to need on their web sites.

iWeb Buddy allows you to have multiple domain files with iWeb so that you can keep your sites separated into their own files. iWeb Buddy also post processes web pages created in iWeb; allowing the user to add Google Analytics, Social Bookmarking, Haloscan comments and custom HTML code to your website automatically.

In addition, iWeb Buddy remembers your settings, account numbers, etc. making it easy to update your website whenever you make a change in iWeb.

With iWeb Buddy, a user can “monetize” their websites that they have created in iWeb by being able to track the traffic on their sites, add custom html code for shopping carts and more.

Explains Marcus Zarra, owner of Zarra Studios, “iWeb is a great website design application. It is geared for the consumer but it is so close to being truly viable for professionals to use but it is missing a few key features – iWeb Buddy brings those features to iWeb.”

* Use multiple “domain” files with iWeb
* Add Google Analytics to your website
* Redirect your RSS feeds to sites like Feedburner
* Fix the RSS links on your iWeb blog pages
* Add Social Bookmarking to your blog posts
* Add Haloscan comments to your pages/blog posts
* Add Custom code to your web pages (WITHOUT IFRAMES)
* Add Mint site tracking to your webpages

Pricing and Availability
iWeb Buddy is available today for $25.00 USD for a single user license. A family license is also available for $35.00 USD. A demonstration copy is fully functional for the first 30 days and can downloaded from Zarra Studios website.

Congratulations on this release, Marcus and Zarra Studios!

Core Animation Tutorial: Wizard Dialog with Transitions

Wizard TutorialMarcus can really crank these articles out. His latest demonstrates how to create a Wizard interface with next and previous buttons, however, this is not your average every day Wizard interface. This one employs Core Animation. When you click the next or previous buttons, you’ll see that the next view slides in from the right or left depending on which you clicked. Take a look at his latest tutorial post: Core Animation Tutorial: Wizard Dialog with Transitions.

Windows Programmers Shifting to the Mac?

Vista to Core AnimationThe tide is turning or so it seems. Well, maybe it’s not quite so monumentous as a tide shift, but there is certainly a buzz in the air that I think is indicative of at least a gradual shift. I can’t speak for all Windows developers of course, but I’ve talked to enough of my friends to know that the Macintosh is no longer in their eyes some obscure computer made by some obscure computer company that makes computers for artists. It’s got some amazing tech that would appeal to any geek especially programmers who have the initiative to take advantage of it.

I made the switch to the Mac along with a job change back in June of 2007 and I have to say that I am glad I did. It’s not to say that there aren’t good tools for programmers on Windows, but there is most definitely a different approach to everything from the kinds of applications that you build to the way you build them on the Macintosh. These differences in some areas are subtle, but in others they are monumentous.

Core Animation, for instance, a technology that enables programmers to use animation to provide users with visual feedback of many different sorts (e.g. change window frame dimensions and watch the window change in steps rather than having the frame update instantly), is really groundbreaking. It’s not something that you see in Windows programming. There are many other technologies that have a similar appeal. Some frameworks that I’ve used such as WebKit–a framework for building web applications, or QTKit–a framework for building multimedia applications based around QuickTime, are exciting and interesting to work with.

I have made predictions in the past that haven’t come true, so when I make a prediction, you ought to take it lightly. However, on this one I feel pretty confident. Apple just gets doing technology elegantly and they’ve made their tools and libraries available to everyone so that anyone who takes the initiative can build elegant software themselves. This fact, I think, will soon have a snowball effect that will start brining programmers from the Windows world in droves. Considering the disappointment that Vista is and the fact that Microsoft doesn’t even seem to like it, Windows programmers are going to start looking for ways to deepen and broaden their toolsets and even consider new platforms such as the Mac.

So what are the hurdles to entry? I think the following list pretty well summarizes it:

  • The cost of a Mac. I’ll put this one out there even though I don’t agree with it. To get a decent professional laptop, you’ll pay at least as much if not more to get similarly equipped systems to the MacBook Pro line. The MBPs were just recently upgraded and the price is actually excellent for the specs you’re getting
  • The cost of the development environment. Oh wait… XCode is free and comes with every new Macintosh. Scratch that!
  • Objective-C uses square braces. Well, you will need to learn a new language. Just take it by the horns and go for it. It’s not that hard and once you become proficient with it, you’ll prefer it. It’s very powerful and often intuitive.
  • No real hurdles. There really aren’t any major hurdles to entry. Just get a Mac and start coding. It’s that easy. And if you need some pointers, take a look at my XCode 3.0 article to get your started. When you’re done with that, head on over to Cocoa Dev Central for some great tutorials or your can check out Cocoa is My Girlfriend a new site that Marcus and I put together.

If you are planning to switch or have just recently switched to the Mac, you should know that you are far more likely to find work building applications for consumers. Most of the work I’ve done over the years on Windows has been business software. This is one of the main differences. I think this will also shift over time, however, for right now, expect to build applications your mother might use rather than software your company might use. And if you have any questions as to what other benefits there are, talk to any successful independent Macintosh software vendor out there. There are lots of them and they do very well.

Core Animation Tutorial: Window Shake Effect

Marcus and I have started to have NSCoder Nights on Monday’s at Panera Bread on Powers near S. Carefree. We’ve been working to try to learn how to do certain OS X Leopard animation effects using Core Animation. The first challenge we took on was figuring out how to shake a window back and forth to indicate that the user has entered the wrong password in the login window. OS X does this when you try to login and enter your password incorrectly. If you’ve never seen it before, you either don’t have manual login enabled, or you’ve never entered your password incorrectly.

Anyhow, the article is short, but it has an XCode project that you can download and use. Take a look at the post, Core Animation Tutorial: Window Shake Effect and let me know what you think.

NSOperation Example

I just finished writing a new article for Cocoa Is My Girlfriend. I am building on the work Marcus did on his first article called Cocoa Tutorial: NSOperation and NSOperationQueue. In my article I’m taking a slightly more practical approach where I provide a flicker free method for grabbing images from a currently playing QuickTime movie. The NSOperation grabs the image data from the movie and saves it out to a folder in the filesystem specified by the user.

I think it is pretty cool. I hope it will help others see the power and flexibility of NSOperation and NSOperationQueue.

Read the new article, NSOpeartion Example, here.

Cocoa Tutorial: NSOperation and NSOperationQueue

I just finished helping Marcus post an article on our new site Cocoa Is My Girlfriend, a site dedicated to providing blog posts and tutorials on how to develop applications using Objective-C and Cocoa on the Macintosh.

This first post is on how to use the new threading objects found in Mac OS X Leopard (10.5) called NSOperation and NSOperationQueue. They provide a really simple and robust way to do multi-threaded programming. Take a look at the post and let us know what you think.