Richard Russell, 5th May 2008
As Andrew Steer and I have demonstrated, colour recovery has proved to be remarkably straightforward, so long as you don't need to know the signs of U and V! Forcing V to be positive and U to be negative gives quite convincing pictures, particularly if you don't know what the colours are supposed to be. The results are uncannily realistic for the Jimmy Savile frame, which happens to contain virtually no colours outside that quadrant.
However, whilst there may be a few occasions when the +V -U 'false' colours would be acceptable in a final conversion, we really need to extract the full gamut of colours, including blues, purples and bright greens.
The relative ease with which it has been possible to separate U from V stems from the property of the PAL signal whereby the phase of the V component alternates from line to line. This makes it unnecessary to rely on phase information to perform an approximate separation. However there isn't a similar cheat which allows you to distinguish positive U or V from negative U or V, only the phase of the subcarrier carries that information.
Extracting absolute phase information from the film recordings is likely to be very difficult, because of the various geometric distortions occurring in the processes involved. Sources of such distortion include non-linearities in the film-recorder's raster, film stretch and imperfections in the HD telecine. Fundamentally a change in position of the residual PAL chroma signal, caused by such distortions, looks exactly like a change of phase, and teasing one from the other is not going to be straightforward.
It's very early days, but I present here the first primitive results from my attempts to do that. Here is a small part of the TOTP picture which has had the full range of colours recovered (approximately!), shown alongside a reference picture taken from the colour videotape:
The good news is that the recovered image contains some blues, greens and purples, but it has many flaws. In particular the phase detection is very noisy and blocky. I also have to confess that I've chosen to show only this small part of the image for a very good reason!
However, the image was recovered from a single frame, and there was a minimum of manual intervention (just choosing which of the four possible arrangements of quadrants was the correct one). Hopefully there will be ways to improve the performance somewhat, but even if it proves impractical to extract an acceptable image from a single frame there may well be opportunities to tune the process using information from other frames in the sequence.
Notes on the process used[]
I reasoned that, even with the most optimistic expectations, the likelihood is that phase information cannot be recovered with as good a signal-to-noise ratio as achieved by the crude U-V separation process. Thus, extracting full-gamut colour risks significantly degrading the quality of the chroma compared to the early +V-U results, which would be unfortunate.
Therefore what I do is to create a low-resolution 'quadrant map', effectively a 2 bits-per-pixel picture with a width of 96 pixels and a height of 72 pixels. The value of each 'pixel' tells the YUV-to-RGB matrix in which quadrant the chrominance should be, and it selects the appropriate signs for the U and V signals coming from the original 'false colour' process. Because the quadrant map is such a low resolution, the (high frequency) SNR of the chrominance is not seriously degraded.
The downside, of course, is that switches between quadrants can be very obvious and can occur in the wrong place (this can clearly be seen in the above image), giving rise to incorrect colours near the transitions. Even when notionally no chroma is present at the switching point, the colour of the noise will change.
This probably won't be acceptable in the long run, so maybe there will be some way of correlating the 'unsigned' chrominance with the 'quadrant map' so that the quadrant switches are moved closer to where they should occur.
Update 6th May 2008
I found a minor bug in my code, which when fixed slightly improved the results:
You can see that the yellow sleeve is slightly less blocky than before. I've also included a little more of the pink shirt at the left edge. However the problem with V-phase reversal is still apparent, especially at the very top. I'm trying to find a way to fix this whilst continuing to use the 1080-line scans.
Later 6th May 2008
Getting better (original on the right, just in case you can't tell!):
I found a way to fix the earlier problem, by replacing the 2D correlation with two separate 1D correlations (vertical and horizontal). This allowed me to optimise them independently and make them more tolerant of imperfections in the source material. The sample image above now covers the full height of the frame.
There are still difficulties in getting the full frame to come out right (you can see a hue error in the top right-hand corner above) but I understand what the problem is and, even if I can't fix it using strictly intra-frame processing, I am reasonably confident it will be possible to do so by reference to other frames in the sequence.
As a side effect of the process I am getting a feel for the geometric distortions in the source material. The false-colour image below gives a measure of the vertical linearity:
Each coloured band corresponds to a vertical displacement of one line in the 1080-line source material.
7th May 2008
Nearly there. To plagiarise Andrew Steer: who would believe that this came off a piece of black-and-white 16mm film, tele-recorded 30-odd years ago? But it did:
Not quite perfect (there are some blue fingers in there if you look carefully) but a lot better than I've managed before, and at least now I can reveal the entire frame without embarrassment (still all intra-frame processing).
Outline of the process[]
As I stated before, my starting point was the assumption that it would probably be exceedingly difficult, and require complicated spatio-temporal processing, to recover a sufficiently stable and noise-free colour subcarrier to allow 'direct' demodulation of the residual chroma. I wouldn't suggest abandoning that approach altogether, since it would in principle allow the use of 'advanced' PAL decoding techniques (e.g. transform based) to reduce crosstalk between luminance and chrominance, but for the moment it is beyond what I consider practical.
Given the excellent results obtained from the 'single quadrant' colour recovery processes, developed independently by Andrew Steer and myself (see Richard Russell's early experiments), I reasoned that a good way to achieve full-gamut colour recovery would be to use that technique in conjunction with a separate process which would determine the correct quadrant (i.e. the signs of U and V). Determining the quadrant ought to be much less demanding than fully recovering the colour subcarrier.
In essence my quadrant-determining process involves the following steps:
- Divide the picture into a 2D array of blocks. I used blocks of 20 pixels by 15 lines, resulting in 96 blocks across each row and 72 blocks down each column (assuming a 1920x1080 source). Other block sizes could be used, but I found this size to be a good compromise between resolution and ruggedness.
- Use the chroma content of each block (if any) to determine the vertical displacement between its actual position in the picture and where it ought to be (had there been no geometric distortion etc). I do that using a cross-correlation process (pattern matching, if you like).
- Use the chroma content of each block to determine the horizontal displacement between its actual position in the picture and where it ought to be. Again, I use a cross-correlation process.
- Generate U and V 'subcarrier references' for each block, using the measured vertical and horizontal displacements to work out the appropriate phases.
- Demodulate the chroma in each block using the regenerated subcarriers; the quadrant can be determined directly from whether the DC components of the demodulated U and V signals are positive or negative.
This may, with hindsight, sound relatively straightforward, but there are several practical difficulties. Not least of these is that there is insufficient information in each block of chroma to determine its position in isolation; it is necessary to make inferences based on the content of other blocks to obtain the required information. Also, the process has to cope with the situation when a block contains no chroma, or insufficient chroma to make a reliable determination.
My next step is to tidy up my code to a state where I could use it to process a sequence of frames. It remains to be seen whether the low-frequency colour noise which would result from 'quadrant switching' is objectionable.
Later 7th May 2008
Before anybody gets too excited, here's the best I have managed so far with the Jimmy image (which doesn't really benefit from the full-gamut colour anyway):
This far-from-perfect result was obtained using very different 'tuning' parameters from those needed on the other image, so there's a fair way to go before the process could be fully automated (if indeed that's ever possible). The chroma in the top-left corner is so poor that my process struggles to pull out the necessary information.
Progress report, 9th May 2008
I'm trying to make my process less dependent on 'magic numbers', which have to be tuned on each individual frame to get acceptable results. This will be essential if I am to run it on a sequence without excessive degrees of manual intervention.
It's getting better, but I'm still having problems with the horizontal displacement measurements being rather noisy, and thus causing difficulties for the 'disambiguation' algorithm. Disambiguation is necessary because the correlation process is only able to measure the horizontal displacement of a block relative to the nearest half-cycle of subcarrier (about 4 HD pixels); to obtain the needed absolute displacement one has to know which half-cycle of subcarrier it was! Given that the displacements can vary over a frame by several cycles this isn't straightforward.
I can't at the moment decide whether the best approach is to reduce the noise on the displacement measurements (for example that might involve integrating over several frames, if the distortions remain consistent) or to improve the disambiguation algorithm so that it is more tolerant of noise. My preference is for the latter, because it would mean the process remains intra-frame, but I haven't yet come up with a modification that is effective.
I might try a bit of spatial filtering on the measurements (this would have to be mild, because the displacement can vary quite rapidly with position) but to do that I need to get my head around filtering modulo-4 data! For example the mean of 0 and 3 is 3.5! I've done something similar before, when filtering angles in the BBC's Free-d equipment, so I can remind myself of that if necessary.
11th May 2008
Presenting, for your delectation and delight, Full gamut colour recovery - the movie. That's right - all 34 seconds of the Top of the Pops sequence in glorious colour, recovered from the 16mm black-and-white film recording. Click on this link to view an MPEG-4 coded QuickTime movie (just over 43 Mbytes); it should play fine using QuickTime or RealPlayer. If you can't play it, or the download is too slow, you can alternatively view it on YouTube but the quality is far worse (try this link for a better quality version):
There are a few duff frames, where my process hasn't got it right, but if it was perfect you wouldn't believe it! Note the following features of this conversion:
- Entirely intra-frame processing (apart from getting the global quadrant phase right); not dependent on geometric distortions being consistent.
- Completely unattended conversion, with no manual tweaks. At 5 seconds per frame it took around 70 minutes to convert.
- Only one global 'tuning' parameter, to control the tradeoff between the likelihood of making a mistake and failing to decode the quadrant at all.
Known weaknesses:
- The quadrant phase may have to be reset following a cut (although it didn't prove to be necessary on the TOTP sequence).
- If there are two or more non-contiguous areas of colour in a frame, only one will be coloured. To resolve this would require some kind of GUI.
- Doesn't manage to determine the quadrant in the top-left corner of the frame; I have cheated by forcing the +V-U quadrant there, which works well on the TOTP sequence but may not always do so.
I'm pretty pleased with what I've achieved, but comments of all varieties would be welcome.
12th May 2008
I found a way to reduce the number of faulty frames, and have uploaded a slightly improved version of the movie; if you've already viewed the original you may need to flush your internet cache to ensure you pick up the new one.
14th May 2008
Andrew Steer has queried the colour accuracy of the greeny-yellow T-shirt towards the end of the TOTP sequence. Here is a direct comparison between my colour-recovered version and the original:
Apart from the inevitable 'blotchiness' of my version (because of the low-resolution quadrant map) I reckon they're pretty similar.
16th May 2008
One of the potential issues with my colour recovery method is the low resolution of the quadrant map, and how acceptable that will be. As a test, I created an artificial 'film recording' of Test Card J (by PAL coding and scaling to 1920 x 1080) and then ran that through my process. This is the result for the well-known central picture:
Although the low resolution of the chroma is apparent, it doesn't seem too bad to me. It's also interesting to see the cross-colour on the transitions between the frequency gratings, and the effect of the subcarrier notch in the luminance channel. Apologies for the incorrect aspect ratio: I've now adapted my process to output (non-square) Rec.601 pixels.
Here is the quadrant map for the above File:
Colour key: Red = +U +V, Yellow = -U +V, Green = -U -V, Blue = +U -V; I am still forcing the -U +V state to areas where my process hasn't assigned a quadrant. Note that, as an experiment, I am currently using 16 pixel x 15 line blocks, i.e. 120 blocks across the full width rather than 96.
17th May 2008
For those of you interested in the 'state of the art', I've uploaded the latest QuickTime Movie of the Top of the Pops sequence here. The file is just over 43 Mbytes, and will take about 45 seconds to download on a fast 8 Mbps internet connection. It is MPEG-4 coded so you will require a suitable codec.
This represents the best I can do at the moment. Again, it was created fully automatically with no manual intervention. After some tuning, the entire process now runs at about 3 seconds per frame on a 2.8 GHz Pentium 4, so the 34 second sequence took about 45 minutes to render.
19th May 2008
I've started work on the 'JS lights' sequence received from Jonathan today, and it's certainly going to be challenging! As a taster, here's what I've managed so far with just one frame from that sequence:
I haven't any idea whether the background colour is right, since I've not seen the original, but this is what my process reckons it to be. There are obvious problems, such as the green fingers, and this is one of the better frames. However, it's a good example of where a full-gamut approach is essential, because simply forcing the -U+V quadrant makes the background red!
20th May 2008
Seems my process got it right! Jonathan has kindly provided the original for comparison:
I'm particularly pleased that I managed to tease out the bluish regions above Jimmy's shoulders (which are in a different quadrant from the green). My version is considerably darker, but Jonathan comments "It does seem that the FR transfer is darker at this point".
22nd May 2008
Here are some more comparisons between colour-recovered frames and the originals:
These are some of the better frames from the JS lights sequence. It is my opinion that it will never be possible to get good results from the sequence because the rapid changes in background illumination (often taking place between the two fields of a frame), in conjunction with what is suspected to be a small degree of 'EHT breathing' affecting the Film Recording CRT, distorts the subcarrier-dot pattern too much. See this article for an example of what can happen.
If anybody is interested I have a colour-recovered QuickTime movie of the entire sequence, but it's not great so I'm not inclined to upload it for all to see! I've done everything I can think of to improve the performance of my process, but it's fighting against very ropey material. Let's hope this sequence is not typical.
24th May 2008
As Jonathan has pointed out, flashing backgrounds may not be so untypical of Top of the Pops! I have therefore reprocessed the JS lights sequence using the simple error concealment algorithm previously proposed: if the quadrant changes to a different value just for one frame, then changes back to the value it previously had, assume the transient value is an error.
The improvement with this sequence is not that great, since there are many places where two consecutive frames are in error, or the picture is changing too rapidly for the algorithm to have any effect. Nevertheless, James has been kind enough to say "I can't think of a more extreme example ... your process seems to make the most of a 'bad lot' and still works within quite acceptable bounds".
A note on aspect ratios[]
I have assumed, from the point of view of my Colour Recovery process, that the HD film scan corresponds not to a 4:3 raster, but rather that it maps to the full 720x576 'digital' picture (i.e. an aspect ratio of about 4.1:3). This is borne out by a measurement of the pitch of the colour subcarrier 'dots' in the TOTP sequence, which (in the relatively undistorted central part of the image) corresponds to an active line time of about 54 microseconds (a 4:3 raster is 52 us; the 720x576 raster is 53.3 us).
This assumption simplifies the horizontal scaling process, which to convert 1920 pixels down to 720 pixels requires only three filter phases (to convert to 704 pixels would require 11 phases; to convert to 702 pixels would require 117 phases!). Even if this simplification doesn't prove to be accurate for all film recordings, any resulting aspect ratio error is likely to be insignificant, especially considering all the other sources of geometric distortion in the film recording and HD scanning processes.
I have also made no attempt in my process to correct the output picture for the skew (parallelogram) distortion caused by the raster lines not being 'horizontal' (parallel to the HD scan lines). However I do compensate for that in the creation of my reference subcarrier, since that slightly improves the performance of my process.
25th May 2008
The latest algorithm now divides the picture into 120 x 90 blocks (each 16 pixels wide by 12 lines high) compared to the original 96 x 72 blocks. This improves the spatial resolution of the chrominance information somewhat.
29th May 2008
I've now modified my software so that it can output (still in a QuickTime wrapper) uncompressed Rec.601-format YUV video. This has a number of advantages:
- Using a QuickTime wrapper ensures that the file can be imported by most non-linear editing and grading systems, and played by standard PC or Mac media players.
- Storing YUV in the file avoids an unnecessary and undesirable conversion from YUV to RGB (and almost certainly a subsequent conversion back to YUV for grading/editing).
- Storing uncompressed video ensures the best possible picture quality.
Using QuickTime also makes available a wide selection of standard video codecs, giving the opportunity to use mild compression to reduce the file size whilst preserving acceptable quality.
If for any reason the video is needed in another format that should be easily achieved by importing it into an appropriate editing package and exporting in the required format, or even by writing a dedicated conversion utility. The likelihood is that such a conversion operation would be time-consuming, for an entire programme, but it could be carried out overnight as an offline process.
The 34-second Top of the Pops sequence, output as an uncompressed YUV file, is a little under 700 Mbytes and, conveniently, just fits on a CD-ROM. If anybody has a particular wish to see the uncompressed version please contact me privately.
Continued at Full gamut colour recovery (continued)...