 |
Description  |
|
|
TECHNICAL FIELD
The present invention relates generally to apparatus for obtaining a wide
field of view at a first location without the use of moving parts, and for
selecting a portion or portions of that view under selected viewing
parameters at a second location without the transmission of control
signals from the second location to the first location. Further, the
invention relates to the transformation of the selected view into a
correct perspective for human viewing at the second location.
BACKGROUND ART
The fundamental apparatus, algorithm and method for achieving
perspectively-corrected views of any selected portion of a hemispherical
(or other wide angle) field of view are described in detail in the
above-cited U.S. Pat. No. 5,185,667. This patent is incorporated herein by
reference for its teachings. Through the use of this technology no moving
parts are required for achieving pan, tilt and rotation "motions" as well
as magnification. Briefly, a wide angle field of view image is captured
into an electronic memory buffer. A selected portion of the captured image
containing a region of interest is transformed into a perspective correct
image by an image processing computer. This provides direct mapping of the
wide angle image region of interest into a corrected image using an
orthogonal set of transformation algorithms. The viewing orientation, and
other viewing perimeters, are designated by a command signal generated by
either a human operator or a form of computerized input. The transformed
image is deposited in a second electronic memory buffer where it is then
manipulated to produce the output image as requested by the command
signal.
The invention of that patent was envisioned as being primarily a unitary
system in that all components were located in close proximity. Even in the
subsequent patent applications (Ser. No. 08/014,508, above-cited, and Ser.
No. 08/068,776, filed Jun. 1, 1993) of related technology, the inventions
were envisioned as having all components in close proximity. As such,
there could be ready verification of operation, alignment and any needed
adjustment.
There are applications, however, for the same type of omniviewing of wide
angle images where there is a substantial distance between where the
initial image occurs and the location where the perspectively-corrected
views are to be utilized. For example, in the teleconferencing art some
type of display is exhibited at one location, and persons at a distant
location desire to view all or a selected portion of the display.
According to common practice prior to the development of the basic system
for providing a selected image without the use of moving components,
control signals had to be sent to the site of the display so as to make
necessary adjustments to equipment at that site so as to select a portion
of the display, or enhance a selected portion, for use of the view at the
distant location. Further, it is often desirable to have a plurality of
viewers each individually wishing to observe selected portions of the
image, with those plurality of viewers potentially scattered at separate
viewing locations. The prior art for this situation would require a
plurality of cameras (video sources) and a plurality of control signals
being sent to the site of the images, and each viewer taking a selected
time for their individual viewing.
Accordingly, it is an object of the present invention to utilize variations
on the technology of production of perspective-corrected views, at one or
more locations, of at least portions of an overall image occurring at a
distant location.
It is another object of the present invention to provide for the generation
of a wide angle image at one location and for the transmission of a signal
corresponding to that image to another location, with the received
transmission being processed so as to provide a perspective-corrected view
of any selected portion of that image at the other location.
It is also an object of the present invention is to provide for the
generation of a wide angle image at one location and for the transmission
of a signal corresponding to that image to another location, with the
received transmission being processed so as to provide at a plurality of
stations a perspective-corrected view of any selected portion of that
image, with each station selecting a desired perspective-corrected view.
A further object of the present invention is to provide for the generation
of a wide angle image at one location and for the transmission of a signal
corresponding to that image to a plurality of other locations, with the
received transmission at each location being processed so as to provide a
perspective-corrected view of any selected portion of that image, with the
selected portion being selected at each of the plurality of other
locations.
These and other objects of the present invention will become apparent upon
a consideration of the drawings referred to hereinafter, and the detailed
description thereof.
BRIEF SUMMARY OF THE INVENTION
In accordance with the present invention, there is provided a video camera
at a first location, with that camera having a wide field of view lens,
such as a fish-eye lens, to produce an electrical signal corresponding to
the image as seen through the lens. This electrical signal, which is
distorted because of the curvature of the lens, is inputted to apparatus
for the transmission of the electrical signal to a remote location. The
transmission can be by wire or wireless depending upon the circumstances.
If by telephone wire, the apparatus for transmission includes a
"compression" portion due to the lower band width of these lines. If
transmission is to be wireless, appropriate broadcasting apparatus is
included.
At each location where viewing is desired, there is apparatus for receiving
the transmitted signal. In the case of the telephone line transmission,
"decompression" apparatus is included as a portion of the receiver. The
received signal is then digitized. A selected portion of the digitized
signal, as selected by operator commands, is transformed using the
algorithms of the above-cited U. S. Pat. No. 5,185,667 into a
perspective-corrected view corresponding to that selected portion. This
selection by operator commands includes options of angles of pan, tilt,
and rotation, as well as degrees of magnification.
The system provides for alternate types of receiving command signals. For
example, there can be a plurality of stations for inputting of these
command signals to a single transform unit. Further, there can be the
inputting of command signals at each of several receiving stations, each
of these receiving stations including a transform unit.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a block diagram of one embodiment of the present invention as
applied to the transmission of image signals via telephone lines to a
signal processing station wherein transformation of a selected portion of
a distorted image to a perspective-corrected view is achieved.
FIG. 2 is a block diagram of another embodiment of the present invention as
applied to the transmission of image signals via "broadcast"
(radiobroadcast signal, satellite signal, cable signal, etc) to a signal
processing station wherein transformation of selected portions of a
distorted image perspective-corrected views is achieved, with the possible
input of a plurality of command signals to each select a desired portion
of the image for transformation.
FIG. 3 is a block diagram of a further embodiment of the present invention
wherein the distorted image signal is transmitted to a plurality of
locations, each of these locations having provision for transformation of
selected portions of the image into perspective-corrected views.
BEST MODE FOR CARRYING OUT THE INVENTION
One embodiment of the present invention is illustrated generally at 10 of
FIG. 1, this embodiment being primarily for use with signal transmission
via telephone lines. It will be understood that in this embodiment, as
well as others to be described hereinafter, there are two widely separated
locations, designated as a "remote site" 12 and a "local site" 14.
Situated at the remote site 12 is a wide angle lens 16, such as a
"fisheye" lens, and a video camera 18 for converting any image seen by the
lens 16 into electrical signals corresponding to that image. Typically the
lens is an 8 mm F2.8 lens as manufactured by Nikon, and the camera is a
Videk Digital Camera. These signals are inputted to a compression circuit
20, such as that manufactured as Rembrant VP model, manufactured by
Compression Labs. Inc. Compression is necessary because telephone lines 22
leading from the remote site 12 to the local site 14 have a lower band
width than other methods of signal transfer (see FIGS. 2 and 3). The
compressed signal representing the image is then applied to the phone
lines 22 for transmission to the local site 14.
At the local site 14 the signals on the phone lines 22 are applied to a
decompression circuit 24, such as that manufactured as Rembrant VP model,
manufactured by Compression Labs., Inc., this unit being both the
compression and decompression. Thus, the signal output of the camera 18 is
reconstructed for processing via circuits 26 of the type described in the
above-cited U.S. Pat. No. 5,185,667. For example, the reconstructed signal
is applied to an image capture circuit 28 such as Texas Instrument's TMS
34061 integrated circuits, to be digitized, and then stored in an input
image buffer 30. Typically this buffer (and an output buffer referred to
hereinafter) is constructed using Texas Instrument TMS44C251 video random
access memory chips or their equivalents.
An image processing system consists of an X-MAP and a Y-MAP processor shown
at 32 and 34, respectively. These perform two-dimensional transform
mapping of the signals, and are under control by a microcomputer and
control interface 36. The transformation achieved by these are described
in detail in the above-cited U.S. Pat. No. 5,185,667. The in addition to
determining the desired transformation coefficients based on orientation
angle, magnification, rotation and light sensitivity. Information as to
parameters for these determinations is provided through a user-operated
controller 38 and/or a computer controller 40. Typically, the control
interface 36 can be accomplished with any of a number of microcontrollers
including the Intel 80C196. After the transformation steps, the signal is
passed through an image filter 42 to an output image buffer 44. This
filter 42, as well as the X-MAP and Y-MAP transform processors utilize
application specific integrated circuits (ASICs) or other means as will be
known to persons skilled in the art.
From the output image buffer 44 the transformed signals feed a display
driver 46 for then being displayed on a monitor 48. The driver 46
typically can be Texas Instruments TMS34061 or the equivalent. Its output
is compatible with most commercial television displays.
Another embodiment of the present invention for generating signals
corresponding to a distorted image at one location, and for achieving a
perspectively corrected view at another location, is illustrated at 10' in
FIG. 2. In this embodiment, the same lens 16 and video camera 18 are
utilized as in FIG. 1. However, the electrical signals corresponding to a
distorted image are inputted into a signal transmitter 50. This
transmitter 50 can generate a broadcast signal, a satellite signal, a
cable signal, a sonar signal, etc. at the remote site 12.
As indicated, the transmitted signals 52 are received in a receiver 54
corresponding to the particular type of signal. Thereafter, the signals
representing the distorted image are fed into the processing circuitry 26'
similar to that described with regard to FIG. 1. The only differences are
the illustration of several remote controller inputs 56, 58, and 60 in
addition to the initial controller input 38. While only a total of four
input controllers are illustrated, a larger or smaller number can, of
course, be utilized. The other difference of this circuit from that shown
in FIG. 1 is that the display monitor 48' is adapted to depict four views
as selected by the four input controllers. It will be recognized that a
fewer of a greater number of views can be shown on the display monitor 48'
.
A further embodiment of the present invention is illustrated at 10" in FIG.
3. This embodiment illustrates a combination of elements of FIGS. 1 and 2.
For example, at the remote site there is a wide angle lens 16 and video
camera 18 to produce electrical signals corresponding to a distorted image
to a transmitter 50. This transmitter 50 sends the signals 52 to a number
of receiving stations designated, for example, at six locations 62A
through 62F. Each of these stations has a receiver unit 54A through 54F of
the type shown in FIG. 2. All of the other equipment at each station is
identical with that of FIG. 2, with only a single view monitor 48A through
48F being illustrated; however, a multi-view monitor 48' can be utilized.
The present invention has the capability to achieve pan and tilt of the
image seen by the camera 18 from a local site while the camera is
positioned at a remote site. Further, the image can be rotated any number
of desired degrees up to 360.degree. . This capability provides the
ability to align the vertical image with the gravity vector to maintain a
proper perspective in the image display regardless of the pan or tilt
angle of the image. The processing system at the local site also supports
a change in the magnification (commensurate with a zoom function). These
selectable functions, as well as a selection of a desired portion(s) of
the image are achieved without sending any control signals to the remote
site.
The performance of the transform of a distorted image into a perspectively
corrected image, and the selection of the desired viewing parameters, are
achieved by programming the microcomputer 36, the X-MAP transform
processor 32 and the Y-MAP transform processor 34 based upon the
postulates and equations set forth below as contained in the above-cited
U.S. Pat. No. 5,185,667.
Postulate 1: Azimuth angle invariability
For object points that lie in a content plane that is perpendicular to the
image plane and passes through the image plane origin, all such points are
mapped as image points onto the line of intersection between the image
plane and the content plane, i.e. along a radial line. The azimuth angle
of the image points is therefore invariant to elevation and object
distance changes within the content plane.
Postulate 2: Equidistant Projection Rule
The radial distance, r, from the image plane origin along the azimuth angle
containing the projection of the object point is linearly proportional to
the zenith angle .beta., where .beta. is defined as the angle between a
perpendicular line through the image plane origin and the line from the
image plane origin to the object point. Thus the relationship:
r=k.beta. (1)
Using these properties and postulates as the foundation of the lens system,
the mathematical transformation for obtaining a perspective corrected
image can be determined. Coordinates u,v describe object points within the
object plane. The coordinates x,y,z describe points within the image
coordinate frame of reference.
The object plane is a typical region of interest to determine the mapping
relationship onto the image plane to properly correct the object. The
direction of view vector, DOV[x,y,z], determines the zenith and azimuth
angles for mapping the object plane, UV, onto the image plane, XY. The
object plane is defined to be perpendicular to the vector, DOV[x,y,z].
The location of the origin of the object plane in terms of the image plane
[x,y,z]in spherical coordinates is given by:
x=D sin.beta.cos.differential.
y=D sin.beta.sin.differential. (2)
z=D cos.beta.
where D=scaler length from the image plane origin to the object plane
origin, .beta. is the zenith angle, and .differential. is the azimuth
angle in image plane spherical coordinates. The origin of object plane is
represented as a vector using the components given in Equation 1 as:
DOV[x,y,z]=[Dsin.beta.cos.differential.,Dsin.beta.sin.differential.,Dcos.be
ta.] (3)
DOV[x,y,z]is perpendicular to the object plane and its scaler magnitude D
provides the distance to the object plane. By aligning the YZ plane with
the direction of action of DOV[x,y,z], the azimuth angle .differential.
becomes either 90 or 270 degrees and therefore the x component becomes
zero resulting in the DOV[x,y,z]coordinates:
DOV[x,y,z]=[0, -Dsin.beta.,Dcos.beta.] (4)
The object point relative to the UV plane origin in coordinates relative to
the origin of the image plane is given by the following:
x=u
y=v cos.beta. (5)
z=v sin.beta.
therefore, the coordinates of a point P(u,v) that lies in the object plane
can be represented as a vector P[x,y, z]in image plane coordinates:
P[x,y,z]=[u, vcos.beta.,vsin.beta.] (6)
where P[x,y,z]describes the position of the object point in image
coordinates relative to the origin of the UV plane. The object vector
O[x,y,z]that describes the object point in image coordinates is then given
by:
O[x,y,z]=[DOV[x,y,z]+P[x,y,z] (7)
O[x,y,z]=[u,vcos.beta.-Dsin.beta.,vsin.beta.+Dcos.beta.] 98)
Projection onto a hemisphere of radius R attached to the image plane is
determined by scaling the object vector O[x,y,z]to produce a surface
vector S[x,y,z]:
##EQU1##
By substituting for the components of O[x,y,z]from Equation 8, the vector
S[x,y,z]describing the image point mapping onto the hemisphere becomes:
##EQU2##
The denominator in Equation 10 represents the length or absolute value of
the vector O[x,y,z]and can be simplified through algebraic and
trigonometric manipulation to give:
##EQU3##
From Equation 11, the mapping onto the two-dimensional image plane can be
obtained for both x and y as:
##EQU4##
Additionally, the image plane center to object plane distance D can be
represented in terms of the image circular radius R by the relation:
D=mR (14)
where m represents the scale factor in radial units R from the image plane
origin to the object plane origin. Substituting Equation 14 into Equations
12 and 13 provides a means for obtaining an effective scaling operation or
magnification which can be used to provide zoom operation.
##EQU5##
Using the equations for two-dimensional rotation of axes for both the UV
object plane and the XY image plane the last two equations can be further
manipulated to provide a more general set of equations that provides for
rotation within the image plane and rotation within the object plane.
##EQU6##
where:
A=(cos.phi.cos.differential.-sin.phi.sin.differential.cos.beta.)
B=(sin.phi.cos.differential.+cos.phi.sin.differential.cos.beta.)(19)
C=(cos.phi.sin.differential.+sin.phi.cos.differential.cos.beta.)
D=(sin.phi.sin.differential.-cos.phi.cos.differential.cos.beta.)
and where:
R=radius of the image circle
.beta.=zenith angle
.differential.=Azimuth angle in image plane
.phi.=Object plane rotation angle
m=Magnification
u,v=object plane coordinates
x,y=image plane coordinates
The Equations 17 and 18 provide a direct mapping from the UV space to the
XY image space and are the fundamental mathematical result that supports
the functioning of the present omnidirectional viewing system with no
moving parts. By knowing the desired zenith, azimuth, and object plane
rotation angles and the magnification, the locations of x and y in the
imaging array can be determined. This approach provides a means to
transform an image from the input video buffer 30 to the output video
buffer 44 exactly. Also, the image system is completely symmetrical about
the zenith, therefore, the vector assignments and resulting signs of
various components can be chosen differently depending on the desired
orientation of the object plane with respect to the image plane. In
addition, these postulates and mathematical equations can be modified for
various lens elements as necessary for the desired field-of-view coverage
in a given application.
The input means defines the zenith angle, .beta., the azimuth angle,
.differential., the object rotation, .phi., and the magnification, m.
These values are substituted into Equations 19 to determine values for
substitution into Equations 17 and 18. The image circle radius, R, is a
fixed value that is determined by the camera lens and element
relationship. The variables u and v vary throughout the object plane
determining the values for x and y in the image plane coordinates.
From the foregoing it will be understood by persons skilled in the art that
the art of the omniview motionless camera system has been extended for
applications such as teleconferencing where information displayed at one
location can be transmitted to a second location, with complete control of
the selection of viewing parameters being made at that second site without
any control being transmitted to the first site. This permits a
multi-station receipt of the information and control at each station.
Although certain citations to commercially available equipment are made
herein, there is no intent to limit the invention by these citations.
Rather, any limitation of the present invention is by the appended claims
and their equivalents.
* * * * *
|
|
 |
|
 |
Description |
|
