A thousand years ago, information
on everything from trade to religion travelled mostly by word of
mouth. One hundred years ago, virtually all knowledge was
disseminated via the printed word. In the last twenty years more
has changed than perhaps in the last thousand. Publishing is a
rapidly changing business. No sooner has desktop publishing
revolutionised printing than a whole host of new media have
arrived to threaten the dominance of the written word. On-line
information from computers all over the world contribute to the
Internet. Children receive their school lessons, and customers
judge rival products, using multimedia CD ROM's rather than
brochures and books. Information is becoming more fun; fine if
you work in education, but what if you work in industry or
commerce? Truth is, it's getting harder to make your mark
against other's more high-profile message delivery. It's
becoming more complicated to engineer and produce too. Remember
how long it took for the office to embrace word-processors?
Think of the learning-curve for creating CD ROM's, creating
virtual realities or writing pages for the World Wide Web.
That's where this book comes in. It covers the skills you'll
need to learn when designing, writing and generating
new-generation electronic media products, whether it's designing
a "brochure" on floppy disc, writing interactive CD ROM's or
pages for the World Wide Web, producing video and audio products
or expanding into the territory of 3D sound and video. A
competent engineer in this new world will need to call on a
backgrounds in both television and sound-recording as well as in
traditional paper publishing and newspapers. She'll need to work
to high standards and to tight deadlines. And to budget and that
means not making too many mistakes.
In many ways it is
convenient to group the disciplines of multimedia and virtual
reality together because they have many common principles and
components. However the ultimate ambitions of multimedia and
virtual reality are quite distinct. Consider the CD-ROM
encyclopaedia where text is combined with sound, film clips and
data navigation software. It demonstrates the four vital
components of multimedia, as it has come to be termed:
Sound/audio media Image/video media Text media
Interaction
This may be an excellent and exciting way of
presenting information, but in no way does it attempt to emulate
a form of reality. One may look up the entry for John F.
Kennedy, read that he was the first Roman Catholic president of
the USA, hear an audio clip, perhaps along the lines of;
"And so my fellow Americans. Ask not what your country can do
for you; ask what you can do for your country."
And
even see a film clip of his last drive through the streets of
Dallas; but your sense of reality will always be that you are
sitting behind a computer, interacting via a keyboard or a
mouse.
Virtual reality (VR) has a very different goal in
mind. The crucial goal is coined in the term immersion. A
virtual reality demonstration or production is successful if
(even to a small or limited degree) during that presentation the
viewer/ listener/ game-player has a sense of being apart from
the world they are truly occupying and experience an immersion
in an alternative environment. But this definition too requires
some qualification. To this must be added that the sense of
immersion must be mediated solely by the senses and not by
mental association and ideas. (I expect you - like me - have
read a gripping novel on an aeroplane only to have glanced up
and experienced the sensation of being dragged away from the
fictional world of the book. If you recognise this, then you
know those seconds of disorientation whilst adapting to the true
reality. But this is an "immersion" which is mediated by the
imagination.) This way, the link with multimedia can be most
clearly seen, since the all embracing task of creating an
empirical alternative sense of reality must, by definition,
involve media which impinge on more than one sense. Moreover,
the true essence of virtual reality is in both creating
convincing sensory inputs and a creating a realistic impression
that one's own actions and behaviours impact and influence a
virtual world in a realistic and convincing way. Simply put, we
must be able to interact with this alternative world.
Many
people - and certainly the popular press - view virtual reality
or VR as the stuff of science fiction. Many worry about the
lacivious uses it will be put; fearing the human race will
evolve into insouciant libertines gorged on a diet of ersatz
sex! For those who dread what the future will bring to a human
race given reign to experience its personal "fantasy realities",
it's worth noting that the stereoscopic viewer - the nearest
anyone could get to a virtual reality system at the time - fell
into ill-repute in Victorian times due to the proliferation of
pornographic material. It seems the engineering progress which
has accompanied the presentation of virtual reality has not been
matched by a commensurate progress in the fantasies which people
hope to have sated by that technology! It is with some sadness
that Gates (1995) predicts that concupiscence will almost
certainly be the driving factor in low-cost virtual reality
systems. So is the future for VR predominantly as a vehicle for
entertainment? Is it going to be good for anyone but salacious
epicures? Certainly the key concept of immersion does perhaps
suggest its predominant applications exist more in the world of
our fantasies than in the world of our duties. However there are
a raft of applications where VR and multimedia concepts and
technologies are beginning to contribute to our abilities to
execute our jobs, perform new jobs or do old jobs differently.
Not strictly pure VR or multimedia, three important categories
are emerging; Virtual design and visualisation, Tele-presence
and Sensory management applications.
Virtual design and
Visualisation
When the Ford Moter Company unveiled the
Taurus in the Autumn of 1995, critics and potential customers
alike were impressed by its design and styling. Yet in the
design of the Taurus, Ford had sidestepped many of the
traditional project phases. Gone were the wind tunnel tests on
life-size clay models of the car; drag performance was predicted
using computer models of physical processes. Visualisation for
mechanical design and styling was performed with advanced 3D
graphics software. Designers and stylists worked across nine
time-zones using interactive graphics and on-line video
communications to keep abreast of each other's progress. These
techniques are transforming the way large companies work.
(Graphics and video techniques are covered in chapters 2, 6, 9
and 10.)
Tele-presence Applications
Tele-presence
applications call to mind the physicist remotely working in
environments contaminated by dangerous high-energy radiation or
the zoologist studying deep-sea creatures via a far-flung
submersible robot. But there are other uses for VR technology
when the problem is much nearer to hand. For instance for the
surgeon operating indeed the body of a patient during endoscopic
procedures. Presently, denied direct sight of the organs and
tissues involved, the surgeon has to make judgements based on
two dimensional television pictures using his experience (rather
than his senses) to judge depth. The application of VR
technology to just this type of problem is covered in Chapter
13.
Sensory Management applications
In many
high-pressure environments, like the flight deck of a modern
high-speed fighter aeroplane, the workload on the pilot is very
high indeed; one fraction of a second's inattention, to one of
the myriad of systems and instruments, can result in a fatal
error. Psychologists, studying these environments, have deduced
that much of the problem lies not in the sheer amount of
information but in the way it is presented. In a rapidly
changing world, it is not only the jet fighter pilot that has to
assimilate ever increasing amounts of data and make reasonable,
yet on-the-spot, decisions on the basis of fleeting megabytes of
text, pictures, video clips and sounds. The link between
multimedia designs for maximum information "uptake" for training
and education is obvious. The manipulation of sensory input in
order that the maximum amount of information can be put across
in the minimum time and bandwidth, and with the minimum strain
of the individual, is a recurring theme throughout the following
chapters. Even if you recoil from the thought of virtual
workplaces, virtual battle or virtual sex, be aware that these
three; virtual design, tele-presence and sensory management,
will touch all our lives from now on.
Multimedia and VR are eclectic
subjects. Underpinning the technology of VR are the "hard"
sciences of audio, video, data manipulation and computing and
also the sciences of perception, the psychology of hearing and
of vision. Part of the study of virtual reality engineering is
realising that understanding the systems inside our bodies is as
important as comprehending the electronic systems outside; that
our own experiences are the proper subjects of scientific
enquiry too. One day, all the mysteries of sight and hearing
will be known but, for now, the "soft" human sciences leave many
'loose ends'. An exquisite irony it is that our brains,
fashioned by millions of years of evolution, have evolved to
apprehend the world outside ourselves, not the world within. In
many ways, knowing our own selves may prove more complex and
difficult than comprehending the rarefied intellectual heights
of solid-state physics!
As well as a whole brace of new
intellectual disciplines, it is likely anyone involved in
multimedia and VR will have to acquire a new range of
operational skills. As the technology emerges, engineers may
find themselves fulfilling many unfamiliar roles - everything
from video camera-person to audio editor or to software
engineer! This book covers some of the ground-work required for
the myriad of duties the multimedia and virtual engineer of
tomorrow will have to face. The pattern of the book falls into
three sections; Media - Sound, Image and Text, Multimedia
Production and Hardware and finally, Virtual Reality.
Media
covers the technical background to the disciplines involved in
multimedia and virtual reality which includes electronics,
perceptual physiology and psychology. Each media "thread" is
examined in a reasonably consistent way. For example, chapter
two, covers sound, then hearing and finally audio electronics.
Chapter 4 follows the same device, covering first light, then
visual perception and finally video and television electronics.
Space is also devoted to waveform generation and synthesis
methodologies, digital audio and digital video technology.
Finally multimedia and VR production and authoring platforms are
surveyed and compared.
Media Production and Hardware
concentrates on the many different operational skills which may
be required of an engineer pursuing, or wishing to pursue, a
career in multimedia and virtual reality. The emphasis here is
less technical - more "hands-on". Sound and vision production
disciplines and techniques are covered including the use of
video and audio production hardware, computer paint and
animation techniques are also described. Finally the process of
multimedia authoring is related and the section concludes with a
look at the preparation and writing of HTML files so that media
elements can be read by Web browser software for inclusion on a
commercial World Wide Web server.
The final section
concentrates on the engineering for the re-creation or synthesis
of artificial realities; aural, visual and interactive. The
emphasis is on practical engineering solutions and applications.
Published here for the first time are my own strategies and
engineering methods for the re-creation of 3D sound-fields by
two loudspeakers as utilised in the Perfect Pitch Music OM 3D
Sound Processor. Also in this section is a description of 3D
film and video techniques and a recountal of the DEV (Depth
Enhanced Video) technique developed for obtaining stereoscopic
depth cues from standard 2D television pictures. Finally the
physics of interaction are examined in relation to the senses -
especially the haptic and kinaesthetic. It is to the senses that
we turn now. After all, without these, we should not be able to
appreciate any reality, real or virtual!
Two philosophical strands
of thought have run through Western civilisation on the matter
of the senses. On the one hand, the Empiricists believe all
human knowledge is mediated via the senses. In other words,
without sensation, the mind is formless. On the other hand, the
Nativists (who can count amongst their number the great
Descartes) think that the mind is not so plastic; that it has
some essential "pre-wiring". Descartes thought the human brain's
"operating-system" had to include certain predispositions - the
notion of God and a predisposition to mathematics for example.
More modern exponents of this line of thought, like Noam
Chomsky, believe the human mind has a predisposition to
language. Like most of the great philosophical dichotomies, the
truth almost certainly lies somewhere in the middle. For the
engineer of the senses (for that is what the virtual reality
engineer ultimately is) this has profound ramifications. We
shall see later on that it is often convenient to regard the eye
as a camera, the ear as a microphone and so on. But this does
the human "machine" a great disservice. The eye is not simply an
open window and physiological studies have demonstrated that
some pre-processing of visual information happens even before
signals pass to the optic nerve. As sentient animals we are
flooded by sensory data every day of our lives but the sense we
make of it is very much our own human sense. Does this make our
job harder rather than easier? The opposite is the case. Humans
want to be fooled by alternate realities. Take, for example, the
act of looking at a monochrome photograph. We can recognise
people - even ourselves in past life - when "snapped" in an
instant in time. We are therefore perfectly happy to accept a
marvellous distortion of our perception of the time dimension;
which normally expects to see people, clouds and trees on the
move, not frozen like statues. Similarly the image is distorted
spatially; the world represented in two dimensions rather than
three. Furthermore the image is mute and colourless. The real
wonder of photography - indeed of all representational art - is
that we recognise the images at all! Look at the optical
illusion below. The mind doesn't give up when presented with a
paradoxical perspective. Instead you can "feel" the brain trying
to make sense out of it, the image swinging this way and that as
it attempts to arrive at a stable spatial interpretation.
Fig. 1.1 Optical Illusion
The Five Senses and Beyond
We are
taught at school that we have five senses; sight, hearing,
smell, taste and touch. But actually we have many more and they
aren't the mysterious senses of ESP either. We feel our bodies
in motion or stationary and we have an impression of our body
(albeit a distorted one) and its component positions which is
independent of the tactual sense. Similarly we "feel" hunger,
fear, agitation and other un-designated internal states.
Fortunately for the multimedia and virtual reality engineer, the
two most important senses are sight and hearing. Most of the
information about the world outside of ourselves is gained via
these and the consideration of these dominates the following.
However, the study and knowledge of haptic, kinaesthetic and
equilibratory senses are sometimes necessary in the design of VR
applications so, for completeness, these are considered in the
last chapter. Until then, sight and hearing dominate the
engineering agenda. But what of the other senses - smell and
taste, will these ever have a role to play in VR applications?
Smell has a more direct route to the brain than any other
sense. The smell receptors high in the nose synapse directly to
the olfactory bulbs of the brain which lie directly below the
frontal lobes. Viewed from an evolutionary perspective, smell is
the most primitive of the senses. It plays a very important role
in lower species, the olfactory cortex of fish occupies the
entire cerebral hemispheres. Perhaps its ancient nature accounts
for the emotionally charged, evocative nature of smell. Given
its power to stir memory, it is surprising that this sense has
played so very minor a role in VR development. Although, the
idea of phials of different odours being dispensed at
appropriate times during a VR presentation is comical! Even more
bizarre is the consideration of taste. Any attempt to administer
a taste sensation is inevitably invasive and unlikely to be
accepted; demonstrating just how far are we from a true
alternative reality system and just how intimate such a system
would need to be. I said at the beginning of the chapter that
more has changed in the last twenty years than in the last
thousand. That pace is not slowing down. If the remaining senses
eventually join the VR fold, I wonder what a VR system will look
like twenty years from now!
