AVCHD Backup Strategy

Back in December I purchased a new video camera–the High Definition Canon Vixia HF100. Naturally, I did a lot of research before purchasing it as I wasn’t very familiar with the AVCHD format–a high definition H.264 codec that is very efficient and is able to store a lot of video data using a minimal amount of memory. The codec does this by only recording changes in the images rather than full frames. I don’t know the intricate details of the technology, however, it is quite impressive how good the quality is for such an inexpensive camera. You can see the quality this camera can produce by going to Vimeo.com and searching for the tag HF100. So if the video is so great at such a decent price, what’s the catch?

Simply put, the catch is that there are no applications available currently that can edit the AVCHD format natively on the Mac. The Canon camera, and other AVCHD cameras like it, create files that use the .mts file extension. Again, I’m not familiar with all of the technical details, however, I do know the only way to work with the files on the Macintosh is to convert them to some other format. iMovie 08/09 both handle the AVCHD format by importing the .mts files and converting them to the Apple Intermediate Codec (AIC).

iMovie provides several different options for importing the video data, however, if you decide to go with the “Full HD” option when importing, which I recommend if you want to have high quality output, your files will be GINORMOUS! Mine run in the neighborhood of about 1GB per minute of video. That’s huge!

Back in the days of tape, yes probably most people are still using tape, you could simply fill a tape with video and then put a date on it and add it to your physical media archive on a shelf somewhere. This works well as the shelf life for tape, while not infinite, is really quite good. This is exactly how I’ve done archiving in the past (and when I still use my old DV camera for various tasks). The question for me has become, how can I archive such massive amounts of data as efficiently and with some level of confidence that the data won’t get lost. Here are the the various issues I considered in coming up with my backup strategy:

  • Hard Drive: This is a viable option for sure as disks keep getting cheaper, however, I have heard enough stories of failed hard drives that this one makes me a bit nervous. I can see using it as a secondary backup strategy, but I don’t trust hard drives enough. RAID seems like a reasonable solution, but that requires more money as you need redundancy which requires multiple disks.
  • Burning AIC Format to DVD: This strategy would just be too much work. Remember that you can only hold 4.7GB on a single layer DVD-R/DVD+R. This means that you would be able to store less that 5 minutes of video on a single disk. While I worked hard to force myself to record short clips and keep things maintainable at the clip level, manually splitting all of the clips up in order to only get 5 minutes of video per disc seems just unmanageable.
  • Burning AVCHD Format to DVD: The shelf life of optical media is also questionable, however, this is the strategy I am pursuing at the moment. Ideally solid state drives will get much bigger and cheaper at which point I will move everything over. For now, however, burning to DVD is the strategy I’m going to use

So I’ve decided that burning the original files to DVD is the best way to go for now as the shelf life of optical media is reasonable enough that my data should keep in the near term. And I’ve chosen to archive the video in its compressed AVCHD format. Here, however, is where the strategy gets interesting. I want to have a way to get the clips off of the DVD without too much hassle while using iMovie. Your strategy may differ if you use some other editing software. Here is what I mean.

iMovie automatically detects the existence of AVCHD movie clips when you either plug in your camera or mount the SD card with a card reader and loads the import utility. What I have done is taken the data from the SD card; copied it to my hard drive; and then made a disk image (.dmg) file out of it. When this disk image is mounted while iMovie is open, it will automatically detect that the image contains the AVCHD video files and load the import utility just as if it were physical media being mounted. Pretty cool eh? So, yes there is a bit of work on the front end, however, when you burn these disk images to a DVD, you can just mount them directly when you put the DVD in at some point in the future when you are ready to import and start working with the clips. So here are the steps you need to get this done.

  1. Plug in your camera or mount the SD card with a card reader. Then run iMovie
  2. When iMovie runs, it will automatically detect your camera or SD card and will load the import utility. Click the “Archive All…” button at the bottom of the window. Choose your Desktop as the save location (you’ll see why shortly) and give the file a name and iMovie will back up the data to your hard drive.
    Archive All...
    Archive All...
  3. The backup process will take a little while depending upon how much data you are saving. Once it has finished, you need to open Terminal.app, which is located in /Applications/Utilities.
  4. Now we need to create the disk image and will do so using a command line utility called hdiutil. You need to change directories to your Desktop. To do so, at the command prompt in Terminal type cd Desktop and press Enter. You are now in the Desktop folder where we saved the archive.
  5. To save the archive folder as a disk image, type the following command followed by the enter key: hdiutil create -srcdir ArchiveFolder AVCHDImage.dmg. This command will take a little while depending upon how much data your are archiving. You will know the process has finished when the command prompt returns. You should expect it to take a while if you are saving several gigabytes worth of data so be patient as the progress that hdiutil shows isn’t very informative. It is, however, working.

    Note: ArchiveFolder is the name of the folder you saved when you had iMovie create the file on your Desktop. This command will not work if you specified spaces in your folder name. If you do use spaces, you will need to escape them. This means that if you created a folder called ‘Archive Folder’ for example, your command would need to be: hdiutil create -srcdir Archive\ Folder AVCHDImage.dmg. Notice the backslash before the space. Add this backslash before any spaces in your archive folder’s name.

  6. Once hdiutil has finished creating the disk image, you can now burn it to a DVD. Insert a blank DVD and open a new burn folder on the desktop or somewhere handy. Then drag the disk image to the burn folder and click burn.
  7. Once your burn is finished, insert the DVD back into your drive (if it was ejected) and let it mount on the desktop. Open the disk and double click the .dmg file. This will mount the image. You can click Skip during the verification process if you want to have it simply mount without verification. I usually skip it myself as I’m impatient.
  8. Open iMovie again. iMovie should detect the disk image and automatically load the import utility. If it doesn’t then you may have done something wrong. Shoot me specific questions in the comments if you need help here.

There is one caveat I should mention to using this technique. I suggest you keep your data size on your SD card to 4GB or so. In fact I use a 4GB card for precisely this reason–it fits well with my backup-to-DVD strategy since a single layer DVD only holds 4.7GB. If you have a hard drive based camera, you’ll have to keep an eye on the size of the data in order to ensure that your data will fit on a single DVD, but this shouldn’t be too difficult. I have a 16GB SD card for those cases where I need to capture longer shots like when capturing a long presentation, but most of what I do allows me to keep smaller files and therefore reasonable backup sizes.

So that’s it. It’s not a perfect strategy as the shelf life for DVDs is a little disconcerting, however, I have every confidence that solid state memory as it becomes more and more ubiquitous will be the best backup/archival methodology once the drives get bigger and the costs continue to go down. In the mean time, this strategy seems to be the best one for backing AVCHD video. If you have other strategies that have worked for you, share them in the comments. I would love to hear your ideas.

Good luck.

That Sucking Sound Is Java Killing Your Soul

There is nothing fun about Java. Every possible good facet’s goodness is completely predicated on the requirement that you know how to set up and or get around some soul sucking gotcha. Where it’s been said that even a language such as Perl can make “easy things easy and hard things possible”, Java seems to try to make “easy thing hard and hard things infuriatingly impossible”.

Ok, I know. This will get some people steaming. You may not want to continue reading if that’s you. I’m sure that your assumptions are that I’ve never given Java a chance. I’ve never worked with Java long enough to make any sort of real assessment. I’m not a real programmer… blah blah blah. Ok. Sure yeah. I’m not going to try to convince you of anything. I’m sure you’ve found Java just wonderful for your research paper or whatever pet project you’re working on. That’s great. You know so much. Like I said. You may not want to continue reading.

For those of you who can hear it, here’s the point. Why use a language that tries to be everything to everyone when all it does is suck the joy out of being a software engineer. Remember when programming was fun? If you don’t, then you’ve never used anything but Java.

Why do I need to invoke a Factory every time I want just a simple object? Why are there no good GUI design tools–and please, don’t tell me about your favorite GUI design tool. Frankly, they all suck in comparison to pretty much everything else out there. And that’s another thing, why does a GUI based Java application window, when restored from being minimized for several hours, respond so sluggishly?

Java is not fun. It is not exciting. It is not enjoyable. It just makes you grumpy. If you find it fun, I dare you to to tell me how. Go ahead. Put it in the comments now. Let it be known how Java is so fun. Seriously!

I admit that there is no perfect dev environment. They all have their quirks, even my new favorite–xcode on the Macintosh/iPhone, but I just can’t understand the madness. Why do people insist on defending Java–this lousy programming language that is the basis of more once-hot-now-abandoned frameworks than anyone can keep track of. If you don’t believe me, take a quick survey of the latest posts about Java over at Reddit’s Programming Section. Here’s a smattering:

All of these posts are actually not bashing Java. The writers like the language. What they are doing, however, is defending it. Why? Because it needs defending. It’s horrible.

What’s fun about programming is problem solving. Sure enough, you’ll spend a lot of time solving problems when you use Java, but they are not programming problems–they are environment problems. Sure, you can write that web app once you’ve decided which lame web framework is the least bad of them all. But then you’ll find that setting up the least bad framework will take you a full day or longer and then it might not work properly once you deploy it. Oh yeah, and where are you going to deploy it? Not on a shared hosting web server. You’ll have to go co-lo and administer the box yourself if you want it to run Tomcat or some other web app server.

… and I could go on, but why?

If you want to have fun writing code… If you want to get your soul back, start writing for the Mac or get the new iPhone SDK and start writing code for it. For that matter, start writing for 8-bit embedded systems. That’s more fun than Java and probably easier. And for those of you who just at that moment thought about suggesting embedded Java for 8-bit micro-controller programming, you need to put your laptop down right now and seek help. Seriously. Call somebody. You are not well.

If you are wondering what’s fun that I am doing these days, take a look at my other blog Cocoa Is My Girlfriend. Here are a few of my latest posts:

BecomeAnXcoder, And Give, Please!!

This was posted on cocoalab back in October and I’m just now seeing it, but this online book BecomeAnXCoder looks fantastic! I will start pointing people who are interested in Mac OSX programming who have no programming experience there from now on. I hope these guys can continue to produce high-quality content. Which appears debatable–not because they don’t have the ability to create quality content–clearly they do, but because they seem to be having trouble supporting it.

I noticed when I visited the site this note in bold at the top:

Please Donate! Our bandwidth is at record levels: 29 GB in March, and donations do not even come close to covering our costs. If you appreciate our work, please take a minute to send a donation.

I do appreciate their work myself, but it’s mainly because I would send more people, new programmers that is, to them–which in turn is what is giving them the problem of using too much bandwidth in the first place.

To the folks over at Cocoa Lab, this statement really makes you sound desperate. I hope you can get the support you need, but when you give away content for free, a statement like this suggests that maybe your ultimate intention wasn’t to give it away, but rather to give it first so that people may donate. If this is a business model for you, it looks like it’s not working out the way you expected, so, with the deepest sincerity, I thought I might offer a few possible solutions to your predicament.

  • Talk to a publisher. This is really good content that you could probably sell. Show it to a publisher and see what they think. I’ve heard of publishers that *may* be accepting proposals from Mac Programming authors
  • Get cheaper webhosting. There is this company called 1and1 that provides shared hosting (which would work fine for the content you are providing) and give you 2.5TB, yes that was TERABYTES!!, of transfer bandwidth per month for the affordable price of $9.99 per month. You can sign up at 1and1 now. And yes that link is my affiliate link. I will get a kickback if you sign up!! That’s not a sales pitch–just a disclaimer.
  • Convert the site to a blog site. If you make your site into a blog rather than a book and continue to add new content, the benefit to the Cocoa development community continues to grow. It’s great with the content that’s there now, and to those who haven’t checked it out yet there is a *ton* of content, but keep it coming.

I write this not to chastise, but to be helpful. I hate to see good content producers go un-rewarded for their efforts. The solution, however, is either to change your approach, or change your expectations. Best regards to the BecomeAnXCoder writers. Keep up the great work.

Bring On the Comments?

For those who, like me, hear lyrics incorrectly on a regular basis and if you blog, you may be able to appreciate my most recent mistake. I heard the song Bring On the Comets by VHS Or BETA and started to ask my wife when the song got to the chorus, “what are they saying?” “Bring on the comments?”. “Are these guys bloggers?”

Well, my wife laughed (ok, laughed is a little strong. I think she smiled… maybe), but now every time I do a blog post I get this song in my head. So, without further ado, this post is live, bring on the comments… Listen to it on iTunes.
VHS Or BETA - Bring On the Comets - Bring On the Comets

For other misheard songs, check out KissThisGuy – The Archive of Misheard Lyrics

Scott Stevenson on Writing Copy For Product Pages

Scott Stevenson, webmaster and author for his Cocoa development site Cocoa Dev Central and weblog Theocacao, has written a new post on writing copy for product pages. The ‘keep it simple’ meme is the gist, but read what he has to say.

This is quite interesting to me as I have a product release that is imminent. Well, maybe not imminent. The code is almost done. Now I need to write some copy, figure out the PayPal IPN, incorporate Sparkle… etc. etc. Ok. Nevermind. It’s not imminent.

Anyhow, thanks for the post Scott. This is good stuff.

Microsoft Product Launch Less Than Inspiring

Silhouette DudeHeroes?So we drove up to Denver this morning for the Microsoft Product Launch they’ve dubbed Heroes Happen Here (exsqueeze me? What?). We all got signed up late so we had to do the morning sessions which weren’t the developer sessions.

That was really fine because we were all only going for the free software. I’ve gotta say, though, that the sessions were nothing but big snooze-fests. Maybe the developer track was better, but wow. ZZzzzzzzz.

After attending one session, we went to a different track to see if it was any better. Had to leave within a few minutes for fear that the enthusiasm with which these folks presented their topics was enough to make one’s mind seize up from inactivity (doesn’t work that way, but I’m bad at analogies. Work with me here).

Ok. I exaggerate. When we did duck out of the second session early, much to our surprise and delight, we found that they were handing out the goods without requiring the session evaluations. Sweet! We grabbed the software and got outta there. The software included Visual Studio 2008 (Standard), SQL Server 2008, Windows Server 2008, and Windows Vista Ultimate Edition.

I haven’t done any Windows development in a while, but should the time come again that I need to, I will have the latest software. It was worth the trip, but only for that reason. The presentation was, well… lacking.

Don’t think they have that problem on that other platform.

Core Animation Tutorial: Dashboard Effect

I just finished another Core Animation tutorial and posted it at CIMGF over the weekend. This was a pretty fun one. I wanted to duplicate the effect of the Dashboard widgets flying in and out from off screen and this is what I came up with. Take a look and let me know what you think. Core Animation Tutorial: Dashbaord Effect.

Guitar Hero On The Commodore 64?

Shredz64Wow, some people have some serious hacking skills. That, and a lot of time. This guy built his own interface that allows him to connect the Guitar Hero controller for the Playstation up to his Commodore 64 on which he plays a Guitar Hero clone he calls Shredz64.

That’s very geeky… and cool!

Get more information at Toni Westbrook’s website and here is more about his Shredz64.

Zarra Studios releases iWeb Buddy, the post-processor for iWeb

iWeb Buddy Now AvailableColorado Springs, Colorado – Zarra Studios LLC announced today the release of iWeb Buddy. iWeb Buddy is a post processor application for iWeb that is designed to bring additional functionality to iWeb that professionals and “pro-sumers” have come to need on their web sites.

iWeb Buddy allows you to have multiple domain files with iWeb so that you can keep your sites separated into their own files. iWeb Buddy also post processes web pages created in iWeb; allowing the user to add Google Analytics, Social Bookmarking, Haloscan comments and custom HTML code to your website automatically.

In addition, iWeb Buddy remembers your settings, account numbers, etc. making it easy to update your website whenever you make a change in iWeb.

With iWeb Buddy, a user can “monetize” their websites that they have created in iWeb by being able to track the traffic on their sites, add custom html code for shopping carts and more.

Explains Marcus Zarra, owner of Zarra Studios, “iWeb is a great website design application. It is geared for the consumer but it is so close to being truly viable for professionals to use but it is missing a few key features – iWeb Buddy brings those features to iWeb.”

Features:
* Use multiple “domain” files with iWeb
* Add Google Analytics to your website
* Redirect your RSS feeds to sites like Feedburner
* Fix the RSS links on your iWeb blog pages
* Add Social Bookmarking to your blog posts
* Add Haloscan comments to your pages/blog posts
* Add Custom code to your web pages (WITHOUT IFRAMES)
* Add Mint site tracking to your webpages

Pricing and Availability
iWeb Buddy is available today for $25.00 USD for a single user license. A family license is also available for $35.00 USD. A demonstration copy is fully functional for the first 30 days and can downloaded from Zarra Studios website.

Congratulations on this release, Marcus and Zarra Studios!

Core Animation Tutorial: Wizard Dialog with Transitions

Wizard TutorialMarcus can really crank these articles out. His latest demonstrates how to create a Wizard interface with next and previous buttons, however, this is not your average every day Wizard interface. This one employs Core Animation. When you click the next or previous buttons, you’ll see that the next view slides in from the right or left depending on which you clicked. Take a look at his latest tutorial post: Core Animation Tutorial: Wizard Dialog with Transitions.