Recreating classic performances?

[monkumonku]

Some of you may already be familiar with this technology but it was new to me when I received today's e-mail news from Dr. AIX:

Dr. AIX

As I sat in my university office this morning, I bumped around a few websites after doing a search on high-resolution audio. It's still rather disconcerting that so many writers are hanging on to the HRA company line and writing pieces that are confusing at the best and outright incorrect at worst.

However, I did find an interesting comment on the topic of restoration and re-releasing older masters. The writer stated, "the only way to get a real high-resolution audio recording of an older master is to have mechanically-played instruments recreate an original performance and re-record what they produce". This is brilliant...but of course, impossible. Or is it?

I can tell you that it's already been done (at least with piano)...and with tremendous success.

About Zenph Sound Innovations

Zenph creates revolutionary technology that opens new possibilities for music listening and interaction. The company has developed a fundamentally new method of encoding music as performance data rather than audio. This performance data is easily transformed in a number of ways, enabling users of Zenph products – professionals, hobbyists, students, and listeners – to participate in compelling, interactive musical experiences. The company’s re-performance process was called “a technological miracle” by The Wall Street Journal and one of the “Best Ideas of the Year” by The New York Times Magazine. Zenph’s recordings have received multiple GRAMMY nominations, and its Rachmaninoff Plays Rachmaninoff album on Sony Masterworks was described by the editor of Stereophile Magazine as “perhaps the finest piano recording I have heard.”

Mechanical musicians? How about a player piano...but not just any player piano but one that has practically infinite dynamic resolution and control over pacing and tempo. After all, piano are made up of a bunch of switches (the keys) and three pedals right? How hard could it be to automate them? Actually, not that hard at all. We've had player pianos...even expression pianos for well over a hundred years!

The innovation that Zenph added was to consider a recorded piece of music not as acoustic sounds but as data. For those of you involved in synthesizers or computer music this might sound familiar. It's called MIDI (Musical Instrument Digital Interface) and it records performances as data rather than the acoustic energy coming from a soundboard or vibrating string. When a pianist hits a middle "C" on the keyboard, a set of data emerges from the mechanism and is delivered to a sound generator or a computer that records it. The information includes the note number, the intensity level (how fast the key was depressed) and the timing of the key press and release.

That's really all you need to know. So Zenph came along and analyzed Glenn Gould's debut 1955 recording of Bach's Goldberg Variations. They figured out the precise timing and the dynamic level of each note, the use of the pedals and the balance of each vertical sonority from the original recording. This is their performance data. And then they sent the performance data back to a computerized version of a player piano using an ultra high-resolution form of MIDI. Presto! When they hit play, the sound coming from the piano is "an exact replication" of the original performance buy Glenn Gould. This is the "new" performance that is needed to re-record in high-resolution audio.

It's brilliant, really. Not only can they (in collaboration with Sony Masterworks) capture the "re-performance" with state-of-the-art equipment (although being SONY they released it using DSD 64) and issue it on CD and in 5.1 surround. The Glenn Gould was given all sorts of awards and put Zenph Innovations on the map.

I'm a friend of Dr. John Walker, the guy that came up with this approach. It turns out the copyrights can be refreshed and the recording of the Bach is heard without Glenn's squeaky piano bench, singing and the occasional subway train that is heard during the sessions. I was so impressed that I make available the Zenph productions on the iTrax website. You should check out the "Rachmaninoff plays Rachmaninoff" in 5.1 surround.

But how far can this idea go? The last time I spoke to John, they had raised some serioius funds and were working on building a mechanical acoustic bass and a set of drums. Then they fell off my radar. When I looked at their website today, everything seems a few years out of date. I don't think John is involved anymore. But the original idea was brilliant and as far as making new piano recordings of older performances, it was very successful.

This isn't a way forward for most classic recordings…but it did amaze a lot of people with Gould, Rachmaninoff and Peterson.

I went to the product page at Amazon and read the reviews and listened to samples. There are other recordings done the same way, and I ended up purchasing the above Goldberg Variations SACD and another SACD of Art Tatum's "Piano Starts Here." I'm curious to hear how these re-recordings sound. Along with the DSD and Redbook versions, there is also a binaural recording of each track for both of these albums that you're supposed to listen to through headphones, to hear the way it would sound to the performer as he played.

I found this very interesting. Also, from reading the reviews, a lot of people thought this was a remaster of the original performance. It sort of is, but it really isn't, which brings up the question of just who the artist is for these CD's.

[ÈlTwo]

Wild concept.

But is the data capable of 100% faultless recreation, i.e. can it access the "infinite dynamic resolution and control over pacing and tempo" over not just the instrument, but the effect of the recording studio, or, more often, concert hall?

I'm sure someone will figure out the theoretical limit of the human ear to discern these differences, and then we'll have the same type of discussion about the recreation that we do now about the gamut from low quality mp3s to lossless codecs, as well as having that same conversation about the quality of the new recording.

[garbulky]

Props to the idea. My question echoes Eltwo's. Still quite a unique concept.

[KeithL]

I agree with you, but I disagree that the limiting factor today is the quality of the reproduction OF THE AUDIO SIGNAL at all.

First, "dynamic resolution" and "pacing and tempo" are un-things.

Consider the term "contrast" - we use the word but in reality there's no such "thing" as contrast. I can put two squares, each a different color or brightness, next to each other. We can TALK about the "contrast between them" but, in reality, no such thing exists. We can completely describe the situation by accurately describing the color of each square. "Contrast" is simply a construct we use to describe our perception about how the squares relate to each other. In terms of information, it is extra.

Some audiophiles (and reviewers) insist on making up terms like "pacing" or "tempo". Now, assuming that we accept the dictionary definitions of "pacing" or "tempo", all of which involve time, in that context our digital recording is prefect - far beyond any human being's ability to hear; period; end. The reality is more complicated - and confusing. When most people use terms like "tempo" and "pace" they really aren't talking about timing at all; what they're talking about is other totally unrelated things that happen to psychoacoustically make it SEEM as if the timing is incorrect. (A dip in the upper midrange frequency response can make things like drumbeats seem "slow and dull"; extra second harmonic distortion can make music sound "livelier"; even though neither affects the timing at all.)

Another similar un-term commonly encountered today is "microdynamics"... which, as far as I can discern, is just another (very imprecise) way of talking about how well a certain process or piece of equipment lets you hear minute differences between things. My point is that there is no specification for "microdynamic capability" - it is included in the other (standard) specifications.

Also, incidentally, the idea that analog has "infinite resolution" is quite simply a lie (an exaggeration). It's easy to see the limitations in digital formats - you have "the size of the smallest step" of whatever mechanism you're using (one bit, one pixel etc). - while analog mechanisms SEEM to have infinite resolution. The reality, however, isn't always what it seems. If you start progressively "blowing up" an "analog" picture, you will quickly reach a point where it becomes fuzzy. You don't see pixel edges, but you also don't see more detail once you pass that point. In the case of an analog picture, that point is either the limitation of the lens' sharpness, or of the size of the film grains themselves - which seems an awful lot like digital pixels, doesn't it?

In the case of digital audio, the limitation is related to the size of step recorded by a single bit.  But how about with analog audio; analog is smooth, and so has infinite resolution, right? Well. no. If we're talking tape, we have the size of the magnetic particles on the tape (digital again; hmmmm). On a record, we've got the size of the molecules in the vinyl. But, odds are, in both of those we'll hit the limits of the noise floor for all sorts of reasons. If a sound is quieter than the noise floor (or if a difference is smaller than that), then the noise floor will "obscure it"... which is an nice way of saying that it will disappear and naught be heard. Ad these limits exist everywhere, from the limits on the resolving power of our ears, to the noise floor of the physical world. (A really quiet room, other than a real anechoic chamber, has a noise floor around 30 dB SPL; since the threshold of pain is around 130 dB SPL, that means that you, sitting in a room, are limited to a dynamic range of about 100 dB.)

My point here is that, in terms of all these SIGNAL characteristics, modern digital recordings far exceed our ability to hear, and so could well be "arbitrarily perfect" in that regard. (And made up constructs like "pace" and "microdynamics" all fall under that same basic statement; since they are all simply different ways of describing things we already know how to measure.)

Since we actually CAN reproduce the "two dimensional" characteristics of am audio signal arbitrarily perfectly, our limitations must fall elsewhere. That "elsewhere" is things like the fact that, in a live performance, the sound of each instrument is sent out from that source in different directions, with different dispersion patterns; these then all interact with the room, and your ears, to produce "a Gestalt" of the experience.

Imagine a perfect recording. When you sit in a chair at the perfect spot, you hear EXACTLY what you would have heard if you were at the concert. Now move your head one inch to the left. If you were at the concert, what you hear would change because your head is now in a different position in relation to each of the instruments, and the seat, and the walls. Since you're not really there, and the sound is coming from two (or five, or eleven) speakers instead of a whole bunch of individual instruments, the change YOU experience will be DIFFERENT than if you were at the actual concert. This is the level of "reality" that we are so far unable to capture. (We aren't even close; all we've got are equivalents of "3D" which, while it may be better than 2D, is not at all equal to true holographic presentation.)

At the moment the best we've got is to put two microphones in a dummy head and make a binaural recording.... which could arguably be perfect (as long as you accept the limitation that it will only be perfect if you sit with your head in a clamp to hold it precisely where the dummy's head was located when the recording was made). Actually, we haven't even quite got that much: If you listen through speakers, then you have the effect of the current room you're in added to the original recording, which isn't right either; if you listen to headphones, you'll hear exactly what the dummy "heard", but the sounds won't shift as they should if you move your head, which fact your brain will probably "notice" and find disquieting if not downright odd.

Feb 26, 2014 23:39:18 GMT -5 ÈlTwo said:

Wild concept.

But is the data capable of 100% faultless recreation, i.e. can it access the "infinite dynamic resolution and control over pacing and tempo" over not just the instrument, but the effect of the recording studio, or, more often, concert hall?

I'm sure someone will figure out the theoretical limit of the human ear to discern these differences, and then we'll have the same type of discussion about the recreation that we do now about the gamut from low quality mp3s to lossless codecs, as well as having that same conversation about the quality of the new recording.

[stiehl11]

Back in the early 90's when I was working the (relatively new) electronic frontier of sampling and recorded digital playback I remember recording a piece that I played on a keyboard and imported into (live) a computer. I was able to go back to my performance and correct wrong notes as well as timing, volume and velocity. I then was able to assign any sound I wanted to it; such as a Steinway grand. What resulted was a technically perfect performance of a piece that sounded absolutely wrong.

While I don't doubt that sampling technology and digital music have improved exponentially in the 25 years since I worked with equipment like that, I still doubt that it has improved to the point where it can accurately reproduce the specific model of piano and account for the room acoustics of the venue for the performance. That's not to say that it won't improve or that the result of current technology when applied correctly to a poorly recorded (or damaged) recording wouldn't be an improvement.

[copperpipe]

You're probably referring to midi; midi is just instructions, eg: "play note c3 at this time and for this long, with this instrument sound, velocity etc". Midi is not recording the actual sound that results, it's just the recipe / instructions that tells a sequencer how to play the music. The sequencer is then in charge of taking the instructions and turning those instructions into audible noise.

Sequencers are actually very powerful nowadays; I use "hauptwerk" which is a digital organ. The organ key presses are passed as midi instructions to Hauptwerk which then plays the correct notes. It actually has a library of sounds, one sound per note. These sounds are recorded in original churches / concert halls, note for note, then imported into HW. When hauptwerk is told (via midi) to play note c3, it then looks up the note in it's library and plays back the recorded sound, giving you incredibly realistic sound (it's not computer "made up noises", it's equivalent to playing notes off a cd.

What makes it very cool is how advanced it is; it uses wind modeling (pipe organs basically move air through pipes, just like your mouth passes air through a flute / recorder) so if you play really fast and loud then the amount of air available in the chambers is reduced, so HW affects the pitch of the pipe, the volume etc to emulate this. It also actually has (depending on the sample you upload) multiple recordings per pipe, so if you play note very fast then it will play the "stacato sound", if you hang on to the note for 10 seconds and the recording is only 8 seconds then it will seamlessly loop over or loop onto a second recording of that note etc etc. Very fascinating software...

So yes, room acoustics etc are all accounted for in samplers if the sampler uses pre-recorded sounds from a live concert hall etc.

Anyway, a bit off topic; as an organist, I could on and on about this stuff