No more Pink Walls

No updates to this blog in a while I’m afraid, things have just been far too busy to have had anything interesting (read geeky) to write about. That said, I am indulging and making this post purely to show off.

On Monday 25th May at 8:37am, after a rather long day/night, Mel and I welcomed Spencer Bailey Stewart into the world. There were a few little issues throughout the night (Particularly towards the end) and he had some small hurdles to get over in his first hour, but since then he has gone from strength to strength and both he and Mum and now doing amazingly well.
Obligatory Pic:

Spencer Bailey Stewart

He’s a very placid little man and would quite happily sleep through an earthquake, much to our delight. And yes, that is a little penguin he’s holding on to in that pic

So that’s all really. I am very conscious of not becoming a complete baby bore so unless something actually worth writing about happens, this will hopefully be my only baby post for the sake of a baby post.

Amidst a million other things (Today is T-1 days until our little bubs technical due date! No news yet though) I finally managed to get the rallyduino into the car over the weekend. It actually went in a couple of weeks ago, but had to come out again after a few problems were found.

So, the good news, it all works! I wrote an automagic calibration routine that does all the clever number crunching and comes up with the calibration number, so after a quick drive down the road, everything was up and running. My back of the envelope calculations for what the calibration number for our car would be also turned out pretty accurate, only about 4mm per revolution out. The unit, once calibrated, is at least as accurate as the commercial devices and displays considerably more information. Its also more flexible as it can switch between modes dynamically and has full control from the remote handset. All in all I was pretty happy, even the instantaneous speed function worked first time.

To give a little background, the box uses a hall effect sensor mounted next to a brake caliper that pulses each time a wheel stud rotates past. This is a fairly common method for rally computers to use and to make life simpler, the rallyduino is pin compatible with another common commercial product, the VDO Minicockpit. As we already had a Minicockpit in the car, all we did was ‘double adapt’ the existing plug and pop in the rallyduino. This means there’s 2 computers running off the 1 sensor, in turn making it much simpler (assuming 1 of them is known to be accurate) to determine if the other is right.

After taking the car for a bit of a drive though, a few problems became apparent. The explain them, a pic is required:

The 4 devices in the pic are:

1. Wayfinder electronic compass
2. Terratrip (The black box)
3. Rallyduino (Big silver box with the blue screen)
4. VDO Minicockpit (Sitting on top of the rallyduino)

The major problem should be immediately obvious. The screen is completely unsuitable. Its both too small and has poor readability in daylight. I’m currently looking at alternatives and it seems like the simplest thing to do is get a 2×20 screen the same physical size as the existing 4×20. This, however, means that there would only be room for a single distance tracker rather than the 2 currently in place. The changeover is fairly trivial as the code, thankfully, is nice and flexible and the screen dimensions can be configured at compile time (From 2×16 to 4×20). Daylight readable screens are also fairly easily obtainable (http://www.crystalfontz.com is the ultimate small screen resource) so its just a matter of ordering one. In the long term I’d like to look at simply using a larger 4×20 screen but, as you can see, real estate on the dash is fairly tight.

Speaking of screens, I also found the most amazing little LCD controller from web4robot.com. It has both serial and I2C interfaces, a keypad decoder (with inbuilt debounce) and someone has gone to all the hard work of work of writing a common arduino interface library for it and other I2C LCD controllers (http://www.wentztech.com/radio/arduino/projects.html) . If you’re looking for such a LCD controller and you are working with an arduino, I cannot recommend these units enough. They’re available on eBay for about $20 Au delivered too. As much as I loved the unit from Phil Anderson, it simply doesn’t have the same featureset as this one, nor is it as robust.

So that’s where things are at. Apologies for the brain dump nature of this post, I just needed to get everything down while I could remember it all.

Since my last post regarding the conversion of media from Channel 9’s Catch Up service, I have been in discussion with the company behind this technology, Hiro-Media. My concerns were primarily around their use of the open source xvid media codec and whilst I am not a contributor to xvid (and hence do not have any ownership under copyright), I believe it is still my right under the GPL to request a copy of the source code.

First off I want to thank Hiro-Media for their prompt and polite responses. It is clear that they take the issue of license violations very seriously. Granted, it would be somewhat hypocritical for a company specialising in DRM to not take copyright violations within their own company seriously, but it would not be the first time.

I initially asserted that, due to Hiro’s use (and presumed modification) of xvid code, that this software was considered a derivative and therefore bound in its entirety by the GPL. Hiro-Media denied this stating they use xvid in its original, unmodified state and hence Hiro is simply a user of rather than a derivative of xvid. This is a reasonable statement albeit one that is difficult to verify. I want to stress at this point that in my playing with the Hiro software I have NOT in anyway reverse engineered it nor have I attempted to decompile their binaries in any way.

In the end, the following points were revealed:

• The Mac version of Hiro uses a (claimed) unmodified version of the Perian Quicktime component
• The Windows version of Hiro currently on Channel 9’s website IS indeed modified, what Hiro-Media terms an ‘accidental internal QA’ version. They state that they have sent a new version to Channel 9 that corrects this. The xvid code they are using can be found at http://www.koepi.info/xvid.html
• Neither version has included a GPL preamble within their EULA as required. Again, I am assured this is to be corrected ASAP.

I want to reiterate that Hiro-Media have been very cooperative about this and appear to have genuine concern. I am impressed by the Hiro system itself and whilst I am still not a fan of DRM in general, this is by far the best compromise I have seen to date. They just didn’t have a linux version.

This brings me to my final, slightly more negative point. On my last correspondence with Hiro-Media, they concluded with the following:

Finally, please note our deepest concerns as to any attempt to misuse our system, including the content incorporated into it, as seems to be evidenced in your website. Prima facia, such behavior consists a gross and fundamental breach of our license (which you have already reviewed). Any such misuse may cause our company, as well as any of our partners, vast damages.

I do not wish to label this a threat (though I admit to feeling somewhat threatened by it), but I do want to clear up a few things about what I have done. The statement alleges I have violated Hiro’s license (pot? kettle? black?) however this is something I vehemently disagree with. I have read the license very careful (Obviously as I went looking for the GPL) and the only relevant part is:

You agree that you will not modify, adapt or translate, or disassemble, decompile, reverse engineer or otherwise attempt to discover the source code of the Software.

Now I admit to being completely guilty of a single part of this, I have attempted to discover the source code. BUT (and this is a really big BUT), I have attempted this by emailing Hiro-Media and asking them for it, NOT by decompiling (or in any other way inspecting) the software. In my opinion, the inclusion of that specific part in their license also goes against the GPL as such restrictions are strictly forbidden by it.
But back to the point, I have not modified, translated, disassembled, decompiled or reverse engineered the Hiro software. Additionally, I do not believe I have adapted it either. It is still doing exactly the same thing as it was originally, that is taking an incoming video stream, modifying it and decoding it. Importantly, I do not modify any files in any way. What I have altered is how Quicktime uses the data returned by Hiro. All my solution does is (using official OSX/Quicktime APIs) divert the output to a file rather than to the screen. In essence I have not done anything different to the ‘Save As’ option found in Quicktime Pro, however not owning Quicktime Pro, I merely found another way of doing this.

So that’s my conclusion. I will reply to Hiro-Media with a link to this post asking whether they still take issue with what I have done and take things from there.
To the guys from Hiro if you are reading this, I didn’t do any of this to start trouble. All I wanted was a way to play these files on my linux HTPC, with or without ads. Thankyou.

About 6 months ago, Channel 9 launched their ‘Catch Up’ service. Basically this is their way of fighting piracy and allowing people to download Australian made TV shows to watch on their PC. Now, of course, no ‘old media’ service would possibly do this without the wonders of DRM. Channel 9 though, are taking a slightly different approach.

Instead of the normal style of DRM that prevents you copying the file, Channel 9 employs technology from a company called Hiro. Essentially you install the Hiro player, download the file and watch it. The player will insert unskippable ads throughout the video, supposedly even targeted at your demographic. Now this is actually a fairly neat system, Channel 9 actually encourage you to share the video files over bittorrent etc! The problem, as I’m sure you can guess, is that there’s no player for linux.

So, just to skip to the punchline, yes it IS possible to get these files working on free software (completely legally & without the watermark)! If you just want to know how to do it, jump to the end as I’m going to explain a bit of background first.

Hiro

The Hiro technology is interesting in that it isn’t simply some custom player. The files you download from Channel 9 are actually xvid encoded, albeit a bastard in-bred cousin of what xvid should be. If you simply download the file and play it with vlc or mplayer, it will run, however you will get a nasty watermark over the top of the video and it will almost certainly crash about 30s in when it hits the first advertising blob. There is also some trickiness going on with the audio as, even if you can get the video to keep playing, the audio will jump back to the beginning at this point. Of course, the watermark isn’t just something that’s placed over the top in post-processing like a subtitle, its in the video data itself. To remove it you actually need to filter the video to modify the area covered by the watermark to darken/lighten the pixels affected. Sounds crazy and a tremendous amount of work right? Well thankfully its already been done, by Hiro themselves.

When you install Hiro, you don’t actually install a media player, you install either a DirectShow filter or a Quicktime component depending on your platform. This has the advantage that you can use reasonably standard software to play the files. Its still not much help for linux though.

Before I get onto how to create a ‘normal’ xvid file, I just want to mention something I think should be a concern for free software advocates. As you might know, xvid is an open codec, both for encoding and decoding. Due to the limitations of Quicktime and Windows Media Player, Hiro needs to include an xvid decoder as part of their filter. I’m sure its no surprise to anyone though that they have failed to release any code for this filter, despite it being based off a GPL’d work. IA(definitely)NAL, but I suspect there’s probably some dodginess going on here.

Using Catchup with free software

Very basically, the trick to getting the video working is that it needs to be passed through the filter provided by Hiro. I tried a number of methods to get the files converted for use with mplayer or vlc and in the end, unfortunately, I found that I needed to be using either Windows or OSX to get it done. Smarter minds than mine might be able to get the DirectShow filter (HiroTransform.ax) working with mplayer in a similar manner to how CoreAVC on linux works, but I had no luck.

But, if you have access to OSX, here’s how to do it:

1. Download and install the Hiro software for Mac. You don’t need to register or anything, in fact, you can delete the application the moment you finish the install. All you need is the Quicktime component it added.
2. Grab any file from the Catch Up Service (http://video.ninemsn.com.au/catchuptv). I’ve tested this with Underbelly, but all videos should work.
3. Install ffmpegx (http://ffmpegx.com)
4. Grab the following little script: CleanCatch.sh
5. Run:
   chmod +x CleanCatch.sh
   ./CleanCatch.sh <filename>
6. Voila. Output will be a file called ‘<filename>.clean.MP4′ and should be playable in both VLC and mplayer

Distribution

So, I’m the first to admit that the above is a right royal pain to do, particularly the whole requiring OSX part. To save everyone the hassle though, I believe its possible to simply distribute the modified file. Now again, IANAL, but I’ve gone over the Channel 9 website with a fine tooth comb and can see nothing that forbids me from distributing this newly encoded file. I agreed to no EULA when I downloaded the original video and their site even has the following on it:

You can share the episode with your friends and watch it as many times as you like – online or offline – with no limitations

That whole ‘no limitations’ part is the bit I like. Not only have Channel 9 given me permission to distribute the file, they’ve given it to me unrestricted. I’ve not broken any locks and in fact have really only used the software provided by Channel 9 and a standard transcoding package.

This being the case, I am considering releasing modified versions of Channel 9’s files over bittorrent. I’d love to hear people’s opinions about this before doing so though in case they know more than I (not a hard thing) about such matters.

Hallelujah! God [Allah/Budha/FSM] be praised! Something has happened that I’d completely given up hope on. A working, released and effective h.264 hardware decoding solution for linux has been let loose, from the most unlikely of sources, nVidia. OK, so maybe they’re not the most unlikely of sources, that’d be someone like Motorola or something, but given their recent history, it was unexpected.

To put this in perspective, I’ve had a fairly serious linux HTPC setup for about 3 years and it has been our sole source of TV/Music/Video (We don’t own a stereo or DVD player) for the past 18 months or so. Whilst CPU power has incresaed to the point where HW h.264 decoding is less important, when I started, it (or more generally MPEG4 decoding) was the holy grail. High Def h.264 video was virtually unheard of as, until the hardware decode support arrived for Windows, there was no way to play it. Now the two major players, ATI and nVidia, have each had their own implementations of this, UVD and PureVideo(HD) respectively, for quite some time now, but neither released anything for linux. Recently nVidia even went as far as removing XVMC support entirely. Intel have had patchy implementations, even for Windows, but it has only been their newest G45 chipset that has made HighDef decoding possible (Even the G43 and G41 chips don’t do it). Now sure, you might be saying “Who cares? We have fast CPUs!” but the fact of the matter is that to play 1080p content properly, you need at least an E8400 Intel CPU and if anyone tells you different then they haven’t played full resolution/bitrate files. A little while ago my bro in-law gave me a 720p camcorder recording of my nephew and, most embarrasingly, as the whole family sat around we got to watch a slideshow of frames as a 2Ghz Core2 Duo struggled to play it. Clearly, this is an area that linux has severely lagged behind Windows on.

There have been a few false starts along the way for linux (Don’t even get me started on the Via implementation of it!) but now nVidia have released a working, albeit beta, driver and patches for mplayer, ffmpeg etc. Better yet, it actually seems to work! It would be remiss of me not to mention the ATI beta driver that was also released recently with support for something they’re calling XVideo Bitstream acceleration (XvBA), but as no one actually seems to be able to get this to do anything, yet alone produce any meaningful benchmarks, it is a curiousity at best. So, it seems, nVidia take the prize in this pony race.

This puts me in a bit of a bind. A few months ago when my HTPC hardware was giving up at life, I decided I’d stick with Intel as they’re fairly good Open Source players and at the time seemed like the most likely to get out a working h.264 implementation. Now, as Murphy would have it, they’re the last of the major contenders to release something. Now I know they’re still working on it, Keith Packard has spoken about it numerous times and there’s an upcoming talk at linux.conf.au that claims it will provide a working demonstration, but the fact remains that its not here yet. The one thing preventing me from running out and buying a cheap nVidia card right now is the fact that the Intel implementation will be open source.

This brings me around to another issue, standards. Whilst XvMC for MPEG2 was a bit of a mess and never worked the same way twice, at least it was a standard for everyone to converge on. Now it appears as though each of the vendors will have their own way of doing MPEG4, each requiring different patches and doing different things. This is a real shame. After waiting 2+ years, a period of time where any one of the vendors could’ve created a defacto standard simply by getting something out there, we’re now in a position where 3 different, and likely incompatible, solutions get released all around the same time. Sure, people are working on CUDA and Gallium stuff, but given the way nVidia have implemented PureVideo its clear to see that’s where things are heading.

So, in summary, YAY! Finally a sigh of relief for something that has been missing a LONG time. I hope that, if nothing else, its some motivation for Intel to get an open source solution out the door, but until then, a big THANKYOU to nVidia.

[UPDATE] I just noticed that Andy Ritger from nVidia posted a note to the MythTV dev mailing list with some further details: http://www.gossamer-threads.com/lists/mythtv/dev/357332

So it seems that there’s quite the election ‘fever’ (yay sportsfans!) going around at the moment. Obviously the US election has had the bulk of the limelight and I have to admit to having followed it far too closely since late last year (Along with my election buddy, Soporific Frog. Thanks for the memories). Somewhat less grand (yet still of debatable importance) are the Victorian local council elections that are going on at the moment.

I have a colleague who has, for the past few years, worked for both the Victorian and Australian Electoral Commissions and this has lead to some interesting discussions into the way things are run in the background. Here are a few of the more interesting (I think) tidbits:

• There are a worrying number of minor ‘issues’ with the in-house developed election software they use. Certainly nothing that would alter the outcome of an election (I hope), but enough to make me wonder again, why software like this isn’t open source? Surely the best way to debug software like this is to publish the source 6 months before its used and wait for the feedback from a raving pack of civil libertarians.
• As an example of the above, I present the following. You may or may not know that candidates are allowed to provide a 150 word spiel about themselves which is sent out with elections papers so that people get an idea of who all the candidates are. Now whilst there are some guidelines for this (ie no bad language, no personal attacks on other candidates etc) there is basically no editing of this allowed by the electoral commision, Eg if there’s typos or gramatical errors in the text provided, they will get printed verbatim. Now, given the number of candidates in these elections, the process is relatively automated in the software. It does a word count and, assuming all is ok, it goes through. Now a little bird tells me that the algorithm used in this software initially counted the number of spaces and used that to determine the word count (WC=N_Spaces+1). So, combine this algorithm with the fact that gramatical errors will not, as a matter of policy, be corrected, and you might begin to see a problem.
  Example: “Here is my spiel.Do you think its,any good?”
  So the above 10 word sentence, now gets read as being only 7 words long. Whilst this is a small thing, there are a lot of ’small’ people who tend to stand for council elections that take this stuff very serously. It is also something that should’ve been picked up LONG ago. Now, I’m not suggesting this has altered the outcome of anything, but complaint from a candidate would cause great headaches for the electoral commission this late in the game.
• The rules behind preferential voting systems are complex. Whist the premise itself is simple, there are an awful lot of edge case scenarios that have to be dealt with and these make it next to impossible for a human to accurately and quickly determine a result. Computers to the rescue! Specifically, a spreadsheet to the rescue. Yes, all the rules of this system have been applied within an Excel spreadsheet, which has then gone through appropriate certifications, then sold to the VEC, who plugs in votes counts and it spits out the winners. Whilst there’s nothing implicitly wrong with using a spreadsheet for this function, to me it just all seemed a bit amateurish.
• Despite, all of the above, the VEC and AEC do a phenomenal job of running elections.  They enforce processes to cover nearly EVERY scenario and I have nothing but confidence in the elections held within this country. This said, hearing about all these procedures etc only makes me more concerned about the way elections are run in the US. So many things you read about that occur there simply leave me gobsmacked and I can’t get my mind around the fact that a country as developed as the United States, can’t even follow rudimentary (and commonsense) guidelines to ensure a genuine result. Craziness.

Finally, I feel it would be remiss of me not to mention Mr Troy Anthony Platt, a candiadate for the Ballarat North ward. As mentioned above, candidates are allowed to provide a 150 word summary of themselves and, I can only assume, in the tradition of the Official Monster Raving Looney Party, Mr Platt has provided a most interesting blurb:

Vote 1 Troy Anthony Platt, the Starmaster Ranger Wizard of the North ward. North ward Internationale Airport:Romamesque gothic style 5 gates Iternational, Marine Tera Aero tunnel,3 domestic gates, Immigration Hotel 888 levels, 100 security Home Defence Guards, 12 courts rooms, 3 lounges, two customs,motorbike race track,car race track,car workshops,zoo,swampland undersea living trials,Aquanauts harmonic shaft,16 medical centres,swampland viewing displays,horse stables,equestrian centre, Auditorium Entertainment facilities,7 theatres, stock exchange, chicken sale yards,multicultural university, 144 thousands jobs bringing in 101 billion yearly. Internationale Wathaurong Eureka Musical Sounds of Poetry Tour:Canada,USA,Ireland, UK,Spain. Eureka Stockade:The Movie Kristina Bumble Bee. isbn-1-4120-3222-9, A knights Fire Volume 1&2 ISBN 1-4120-6423-6, Fire exits, light fittings (radiation covers)Traffic hazards, customer service behaviour, 10 o clock public House lockout, seven drinks man/four drinks women, legalised canibis, touch one mate, touch waltzing Matilda

The above was indeed published and distributed along with voting papers etc, to all members of the appropriate ward. Now, whilst such randomness maybe common in the UK, apparently here it caused something of a stir. Infact, this went all the way to the VECs legal council to determine whether or not it was allowable. It was eventually decided that it had not broken any of the rules (At least, the rules as they currently stand, ahem) and therefore it had to be published. Good for you Troy Anthony Platt.

“Until a man is twenty-five, he still thinks, every so often, that under the right circumstances he could be the baddest motherfucker in the world. If I moved to a martial-arts monastery in China and studied real hard for ten years. If my family was wiped out by Colombian drug dealers and I swore myself to revenge. If I got a fatal disease, had one year to live, and devoted it to wiping out street crime. If I just dropped out and devoted my life to being bad.

Hiro used to feel this way, too, but then he ran into Raven. In a way, this was liberating. He no longer has to worry about being the baddest motherfucker in the world. The position is taken.”

Snow Crash – Neal Stephenson

I had a moment like this today. Just sat back in awe of someone elses work and realised that no matter what I did, how much money I spent, what mad crazy skills I learned (Even though I would never have the time to learn them)… I could never best what he had done.

Unlike Hiro though, I can’t really say I found it liberating. In a way it was inspiring to see what is actually possible (Some of these things I never would’ve thought could be done previously), but it was also slightly depressing. Made my own efforts seem rather feeble.

Anyway, if ever I needed an excuse to quote some Neal Stephenson, this was it.

As suggested by Stewart Smith:

• Grab the nearest book.
• Open it to page 56.
• Find the fifth sentence.
• Post the text of the sentence in your journal along with these instructions.
• Don’t dig for your favorite book, the cool book, or the intellectual one: pick the CLOSEST.

Result:
“The first step is achieved through effective marketing”

from Systems Analysis & Design (Hawryszkiewycz)

So, yes, the mandatory clean feed has been generating a lot of noise lately and the Federal Member Email/Letter bomb has been dropped. In this spirit, I felt I might as well contribute an entry with the letter I sent of to my local member (Ballarat – Ms Catherine King), last week:

Ms King,

I am writing to express my concern at the proposed mandatory ‘clean feed’ internet concept being put forward by Mr Stephen Conroy. Such a concept poses serious risks to both free speech liberties and Australia’s ability to prosper in a digital world. As each of these is of such importance, I will address my fears about them separately:

• Free speech and censorship: I understand that it is far from the governments intent to create a system capable of wholesale censorship, however intentional or not, this is what a mandatory clean feed will accomplish. Already, proposals have come from other political organisations (See http://www.theage.com.au/articles/2008/10/27/1224955916155.html) that the scope of the filtering should extend to online services such as gambling. This is the thin end of the wedge. Even if we are able to trust the current government not to bow to pressure from such groups, it is not a stretch to believe that future, less trustworthy, governing bodies could extend the blacklist to cover areas that are not in fitting with its ideals (Eg political dissent). Whilst it may not be the Labour party making these changes, they will have certainly enabled the actions. Finally, given the serious opposition this proposal has faced from both individuals and the business sector, I cannot help but question where the motivation for the filters comes from in the first place. If this system is not (and cannot) serve the public interest, who’s interests are it serving? Mr Conroy has so far failed to satisfactorily answer this question.
• Australia in a digital world: Given that I work in the technical sector and have a reasonable understanding of the technology behind a typical Internet Service Provider (ISP), I find it staggering that a proposal such as the mandatory feed can make it past even a cursory investigation into its feasibility. The governments own studies have shown how detrimental the implementation of a system such as this would be to the quality of Australia’s broadband networks. I do not wish to quote statistics from these studies, they are freely available elsewhere, however I do want to note that the government loses much credibility (particularly in the technology industry) if they are going to undertake feasibility studies with no intention of taking any notice of the results. Mr Conroy has listed ’similar’ systems in place overseas that have had success, however this fails to take into account significant differences between the aims of these systems and the one proposed here. Australia is already lagging significantly behind much of the developed world when it comes to high speed broadband availability. On the one hand we have a government willing to hand out $4.1 billion to help improve this situation, yet on the other the same government wishes to cripple the system before it has even left committee.

Given the ever increasing importance of digital infrastructure in an economy, Australia is now at serious risk of becoming a technological backwater and any gains we have made in this area over the past 5-10 years will quickly become irrelevant. I urge you to oppose this mandatory ‘clean feed’ from Mr Conroy and pass on the concerns that I (and others in the Ballarat community) hold regarding it.

I would be happy to discuss any of the points raised above with yourself or a member of your office in case clarification or further information is required.

Thank you,

Josh Stewart

To date, I have not received a reply, though to be honest I do not expect more than a form letter at best.

After my post yesterday about the ‘rallyduino’ a few people emailed me asking for the source, so I threw everything together quickly into a Google Code hosted project. I did want to get it into SVN anyway as having only local copies was starting to get me nervous.

The project page can be found at:
http://code.google.com/p/rallyduino/
Or to view the code directly, checkout:
http://code.google.com/p/rallyduino/source/browse/trunk/rallyduino/rallyduino.pde 

The power regulator I’d ordered arrived last night (Turned out to be well and truly overkill, but it only cost $10) so now just awaiting the Arduino prototyping shields and I can give this a real worlds test.