 |
Description  |
|
|
BACKGROUND OF THE INVENTION
1. Field of the Invention
The present invention relates to computerized film reprocessing techniques
and, more particularly, to techniques for defining masks in frames of
digitized picture stock.
2. Description of Related Art
Film colorization, that is, colorizing black-and-white motion pictures,
turned the film industry on its side in the mid-1980s. With less than
adequate color selection and limited hardware and software capabilities,
early attempts at colorizing notable black-and-white film classics such as
"Casablanca" and "The Big Sleep" produced less than favorable results,
resulting in muddy hues that didn't always stick to the objects they were
meant to color. Indeed, many film purists likened colorization to
vandalism and defacement. However, in the 1990s a demand was created by
the skyrocketing cost of producing new movies and television shows coupled
with the burgeoning demand for movies and television shows to fill up time
slots on the 500 or so cable channels, a demand which has been an
incentive for colorizers to advance their craft to much higher levels of
quality.
Colorizers have also applied their craft to more varied fields, fields
which do not necessarily involve original black-and-white picture stock.
For example, in the past if a director of a picture were unhappy with the
color of a particular shot, the director would have had to reshoot the
shot, which would have incurred high production costs. Further, commercial
artists and advertisers may desire to intensify particular aspects of
television commercials to be more appealing to consumers of target
markets. Other special color effects may also be desired for a particular
film, video, or television show, particularly music videos which are often
intended for the less conservative teenage and young adult audience.
In order to manipulate and modify digitized picture stock, objects such as
actors, cars, fixtures, and so on need to be "masked." A mask defines the
object as a closed region being substantially homogeneous in color. A mask
is typically drawn around an object by a user drawing polygons around the
perimeter of the object, thereby approximating the shape of the object. It
follows that the more complex the object or the shape of the object, the
more lines of the polygon needed to accurately mask the object. This
process requires substantial user time.
Accordingly, it is an object of the present invention to provide edge
detection or region definition technology for masking objects with
increased efficiency by eliminating the need to draw polygons to
approximate the shape of the object.
SUMMARY OF THE INVENTION
Region definition technology of the present invention provides a method for
masking an object based on luminance values of pixels comprising the
object. Generally speaking, region definition technology provides a method
for detecting the curvilinear edge or boundary of a substantially
color-homogeneous region of a frame of digitized picture stock. The
original picture stock may be of any known form and of any length, e.g.,
from a single frame or art still to an entire feature length film. The
digital data from the digitized picture stock is divided or segmented into
individual shots and frames, if necessary. A colorist or other skilled
artist then downloads a digitized image file, i.e., one frame of the
picture, to a computer workstation system.
According to a primary aspect of the region definition technology of the
present invention, a seed point is selected in an object to be mask, which
mask is a region which is substantially homogeneous in color or in
luminance. A test area is generated around the seed point. The luminance
values of the pixels comprising the test area are then determine from
which a range of luminance values is derived. The range of luminance
values has boundary values representative of the highest and lowest
luminance values of the test area.
At this point, the luminance values of the pixels within the region are
determined and compared to the test-area range of luminance values. If a
pixel has a luminance values within the range, then this pixel is
identified. Upon identifying all the pixels in the object or region which
fall within the range, the identified pixels are highlighted. The colorist
may then adjust the boundary values of the range if the highlighted region
does not satisfactorily encompass the object to be masked. If the
highlighted region substantially encompasses the object, then a mask is
defined to comprise the highlighted pixels, which mask may now be modified
as desired.
One of the advantages of the region technology of the present invention is
that it may be more time efficient to mask objects based on their color or
luminance values rather than drawing polygons around the object which can
only approximate the shape of the object. Accordingly, by highlighting the
pixels of an object which have luminance values within a desired range,
the resulting curvilinear edge or boundary substantially accurately traces
the true edge of the object, thereby providing a much more desirable
visual effect.
One of the features of the edge detection technology is that a colorist may
readjust the boundary values of the test-area range in order to more
accurately encompass the desired object. The pixels may then be again
identified, highlighted, and displayed for inspection by the colorist, who
may then again readjust the boundary values until the object is
satisfactorily masked.
Another feature of the present invention is that the test area may be any
desired geometrical shape or may comprise any desired amount of pixels in
order to generate a range of luminance values which is substantially
indicative of the luminance values of all the pixels of the entire region
or object.
Other aspects, advantages, and features of the region definition technology
of the present invention will become apparent to those skilled in the art
from a reading of the following detailed description with reference to the
accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a block diagram of a multiple workstation computer recolorization
network illustrating certain principles of the present invention;
FIG. 2 is a block diagram illustrating a colorization process for picture
stock, particularly showing the creation of a picture database;
FIG. 3 is a block diagram illustrating a frame interpolation process used
in a colorization process according to the present invention;
FIG. 4 is a schematic diagram illustrating a frame interpolation process of
the present invention;
FIG. 5 is a block diagram of a recolorization process illustrating
principles of the present invention; and
FIG. 6 is a block diagram illustrating a region definition method according
to the present invention.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
Referring to the drawings, exemplary embodiments of the present invention
are shown, illustrating the principles of picture recolorization. Upon
reading the following detailed description with reference to the
accompanied drawings, those skilled in the film and colorization arts will
come to realize various alternative and modified embodiments of those
exemplified and described herein. This description provides a foundation
of picture recolorization from which these alternatives and modifications
stem. Accordingly, rather than provide an exhaustive description of all
possible preferred embodiments envisioned by the inventors, the principles
of the present invention are exemplified with only the embodiments
illustrated by the attached drawings and elucidated by the following
description.
FIG. 1 generally shows a multiple workstation computer network 10 including
a central processor unit 12 such as a mainframe computer in communication
with a data storage unit 14 and a plurality of terminals or workstations
16. Each of the workstations 16 may have any combination of the user
interface devices available on the market, but it is preferable for each
workstation 16 to include at least a keyboard with a mouse and a video
display. Digitization pads and the like may also be employed in the
workstations 16. The data storage unit 14 may take on any desired form
available on the market, but as colorization processes require large
amounts of data storage space, the data storage unit 14 should be capable
of storing data on the magnitude of thousands of megabytes (or gigabytes)
or millions of megabytes (or terabytes). The market currently provides
either magnetic tape storage systems, magnetic disk systems, or optical
disc systems which are capable of storing such voluminous capacity. It
follows that it is preferable for the main processor 12 to have data
compression capability to efficiently handle this large amount of data.
Furthermore, each workstation 16 as well as the processor 12 preferably
has dedicated random-access memory (RAM) capability for further efficient
use of the picture recolorization process disclosed herein.
In a commercial implementation of the picture recolorization technology of
the present invention, the color-on-color network 10 may be broken down or
segmented into dedicated function groups or "work bays" in which personnel
performing similar tasks in the recolorization process are located. For
example, colorists, that is, artists or other skilled animators who are
experts on color, may be assigned a certain number of workstations 16;
users who are skilled in the task of masking or drawing polygons around
objects to be recolored may be assigned to a number of workstations 16;
and users who are skilled in the algorithmic function of interpolating,
which is an efficiency function of estimating or fitting color and mask
data to unedited frames of a film, may be assigned to further workstations
16. In any case, any number of defined function workstations 16 may be
manipulated by colorists and users in an efficient orchestration of the
recolorization technology disclosed herein.
At this point a number of definitions of terms in the art will be given in
order to allow those people not specifically skilled in the art to
understand more fully the principles set forth herein. The concept of
color is defined by a combination of the three following qualities: hue,
which indicates the gradation of color or the attribute of colors that
permits them to be classed as red, yellow, green, blue, or an intermediate
color between any contiguous pair of these colors; intensity, which
relates to the density or-brightness qualities of a color; and saturation,
which relates to chromatic purity (i.e., freedom from dilution with white)
or the degree of difference from the gray having the same lightness.
Additional words used in the art of colorization include: luminance, which
relates to the black-and-white aspect of a frame; and chrominance, which
is the hue and saturation component of a color. When speaking of movies or
films and videos, picture indicates a generic term for any motion picture
including movie, film, video, or the like; frame refers to a single
instantaneous exposure of film of the series of exposures on a length of a
picture; and shot refers to an unedited or continuous sequence of frames
between edited breaks or cuts of the picture (i.e., "scenes" in a picture)
or, in other words, an unbroken view of a given scene from a single camera
perspective.
More industry-specific terminology includes: diffision, which relates to
the blending or grading of color at the border of two differently color
objects; precedence, which determines which objects in a frame are more
forward or rearward (i.e., closer or farther from a view's perspective)
than other objects; and baseplane which is the background plane or the
most rearward object to be masked in a frame.
The definition of the concept called colorspace is somewhat more
complicated than those already given. Color television and color computer
monitors (i.e., display units and monitors) normally operate in RGB
colorspace, RGB standing for the additive primary colors red, green, and
blue. These three colors correspond to the three "guns" of color displays
and, in addition, roughly correspond to the three types of color receptors
in the human eye. As colorization processes add color to existing
monochromatic images or modify color of polychromatic images as set forth
herein, the colorspace known as "YCrCb" is preferably chosen for internal
representation and manipulation because YCrCb colorspace separates
luminance information from chrominance information.
In YCrCb colorspace, "Y" represents the monochrome or the luminance portion
of the image, and "Cr" and "Cb" respectively represent the red portion and
the blue portion of the color image, which are read as "red chrominance"
and "blue chrominance." ›The color green is not stored because green can
be algebraically computed from the other three colors, which is known in
the art.! In order to visualize the concept of YCrCb colorspace more
clearly, if the Cr-Cb space were displayed in two-dimensional Cartesian
coordinates, gray would be in the center (0,0) with the Cr and Cb values
both equal to zero. The further from the origin that a point may move
(i.e., the Y or luminance value) the more progressive the intensity of the
color would become, with the hue of the color being defined as the angle
with the origin as its vertex. In addition, a color recipe is a set of
luminance points, which includes at least black and white, for any given
substantially homogeneous color area or mask.
Having provide these basic colorization terms and concepts, picture
recolorization technology according to the present invention generally
entails a method for modifying the color of existing polychromatic or
color picture stock with luminance-to-chrominance mapping techniques.
Generally speaking, picture recolorization provides a method for modifying
colors of a frame of polychromatic picture stock by firstly digitizing the
frame and then identifying at least one area of the frame which is
substantially homogeneous in color. At this point, the luminance values of
each digital unit, e.g., a pixel, of the substantially homogeneous area
are ascertained, with the luminance values mapped to a predetermined set
of color values. The luminance values are then modified as desired by a
colorist to create a particular effect. The modified digitized frame is
then converted back to a desired picture stock. All of these tasks may be
accomplished at the various dedicated workstations 16 of the
recolorization system 10. The substantive description of the present
invention now follows.
Digitizing Picture Stock
With additional reference to FIG. 2, the colorization of black-and-white
pictures or the recolorization of the color pictures, in a general sense,
firstly involves capturing, which is the process of digitizing, frame by
frame, picture stock with a digitizing unit 18 into individual pixels or
digital units as shown in FIG. 1 and represented by block 22 in FIG. 2.
The original stock may be any known monochromatic or polychromatic type of
picture stock, including celluloid motion picture films, television series
or films (for example, 4,096 lines per frame), television commercials,
music videos, and so on. Further examples of source media include
RGB-format videotape, D1 digital videotape, NTSC videotape (i.e.,
videotape with 525 lines per frame), PAL videotape (i.e., 625 lines per
frame), analog and digital videotape in general, and even single art still
and photographic slides as well. As the stock is digitized frame by frame
in the digitizing unit 18, the data are transmitted to the processor 12 of
the computer network 10 and stored in the data storage unit 14. If the
original stock is celluloid or a video master print, then it is preferable
to transfer this valuable stock to Dl videotape first so that the original
celluloid or video master is left untouched by the digitizing unit and
completely intact for archival purposes. At any time in the colorization
or recolorization process, the digital data contained in the data storage
unit 14 may be laid back or converted to a desired form of picture stock
or the original form of the picture stock by a lay back unit 20, which
will be discussed further below.
Creating a Database
Once the digital data from the digitized film stock 22 is stored in the
data storage unit 14, as represented by block 28, a database is created as
shown by block 30. The digital data stored in the storage unit 14
represent the entire film, video, or movie, which is essentially an
summation of identifiable continuous shots. Accordingly, the digital data
are segmented into each original individual shot, as shown by block 32,
with each shot identified. Each shot in turn is essentially a set of
individual continuous frames which have been taken from a given camera
perspective and have a shared set of objects and/or characters, which may
or may not move about the frames during the course of the shot. Each frame
may then also be thought of as a collection of these objects, which may be
actors and actresses, animals, automobiles, trees, fixtures, and so on.
After the shots are segmented 32, a frame from the shot is downloaded to
one of the workstations 16 so that a colorist may work thereon, as
represented by block 33.
A baseplane is an object with a color recipe which covers the entire frame
and is at the lowest precedence, e.g., precedence 1. The baseplane is
typically the "background" (for example, the sky, the ocean, or the wall
of a room) across which all other objects move or of which all other
objects are in front. By defining a baseplane (block 42), the overall
color to be applied to a frame may be quickly applied. The baseplane may
be defined essentially at any point in the process, often beneficially
immediately after the frame is downloaded 33 to the workstation 16 and a
colorist is commencing work thereon.
A process called masking takes each of these objects and delineates a
certain definable color region, which is called a mask and shown by block
34. Objects to be masked are generally selected from each frame based upon
homogeneity of color; that is, the object has substantially the same color
throughout, or the color of the object is substantially constant
throughout. The selected and masked objects may then be assigned names
(block 36) for consistent reference throughout the colorization or
recolorization process. As can be realized, some objects may be a
combination of several masks. For example, if the object in question is a
person, then the various required masks would include the person's shirt,
pants, shoes, face, hair, hands, and so on. This process is called
grouping and is represented by block 38. Therefore, each object may be
thought of as a group of masks. Grouping allows parallel or simultaneous
editing of related masks which increases productivity of the colorist
working at one of the workstations 16. Further, by combining the masked
objects or elements of a principal object into a unified group for
functions such as moving and resealing, the colorist can reduce the amount
of operations required to track objects in motion.
In a preferred embodiment of the present invention, there may be, for
example, 1,024 possible masks available per frame of footage (footage
being defined as a series of frames which typically appear at a rate of 24
frames per second in motion pictures). Accordingly, many of the commands
for editing masks (which will be discussed in detail below) may be
performed on a group of masks and not each individual mask, thereby
greatly reducing the amount of manual labor required to edit frames with a
user operating a computer mouse, for example.
In addition, a colorist may also define hierarchically masks within a given
group, allowing the colorist to manipulate subgroups of a principal
object. However, skilled colorists typically mask objects, after defining
the baseplane, from most rearward to most forward in the frame, thereby
automatically assigning hierarchical precedence values to the masks.
However, if desired, the precedence may be reassigned (block 40) before
the frame data are stored in the database.
Masks are typically generated or defined by drawing polygons, i.e., closed
plane figures bounded by at least three straight lines, around the
substantially homogeneous color region or area of each object. As can be
realized, using more lines for the defining or masking of a polygon
results in a more accurate tracing or definition of the subject object.
Any operation which defines a substantially homogeneous color region of a
frame may be considered a masking operation, including splining techniques
and specifying vertices that correspond to specific pixels or digital
units in the object image.
Each of the object masks and/or each of the masks included in a mask group
of an object is then assigned a color or color recipe, as shown by block
44. Color assignment 44 is a step typically performed by colorists who are
artists or other specially trained animators. For example, in colorizing
an old black-and-white film, the colorists have to research what color a
particular costume would have been at the time the film was made or the
era in which the film takes place. Accordingly, a colorist or art designer
designates the colors in a select number of frames which are
representative of images necessary to establish the artistic look of the
entire film or project. These colorists rely on extensive reference
materials such as color research books, photographs, set and costume
information, and archival files from a movie studio or the Academy of
Motion Pictures Library. All this information plays an integral part in
the decisions regarding color.
In designating or assigning the color of the masked objects in selected
representative frames, the colorist assigns chrominance values or a color
recipe to every mask, each of which may have the same color recipe or a
different color recipe, thereby creating "art stills" from the frames for
each shot or scene. Individual colors are selected from a palette of
approximately 16.8 million colors. These colors comprise a color wheel
defined by luminance and chrominance values in which a colorist may
operate to generate a specific desired color. Once each object is assigned
a color 44, this color information as well as all of the rest of the
information heretofore defined by a user at a workstation 16 is stored in
the database in the data storage unit 14, as shown by block 46.
Accordingly, the recolorization process is able to modify colors of films
that were originally undesirable. However, the recolorization process of
the present invention creates intentionally unrealistic colors for special
effects or to correct flawed colors of specific objects. Furthermore, the
picture recolorization technology of the present invention recolors or
selectively alters existing colors of pictures that were originally in
color and not necessarily in black and white originally.
Masks may be assigned an attribute known as diffusion which relates to the
falloff of values along the border of the mask (i.e., along the lines of
the defining polygon), or the merging of two different colors assigned to
adjacent masks. Diffusion "softens" the borders or edges of the mask,
allowing for smoother blending or grading of the colorization effect of
the object with the surrounding objects. Diffusion further allows a user
to assign a color recipe to a blurry or out-of-focus object such as smoke
which would otherwise be nearly impossible to colorize accurately and
convincingly.
To diffuse a selected mask of an out-of-focus object, each pixel or digital
unit in the mask is replaced with the proportion of the pixels in a box
centered on the given pixel that are inside the outline or polygon of the
mask. This assigns the border of the mask (i.e., polygon) a value of about
0.5 which gradiently increases to 1.0 for a pixel a distance of about
one-half of the defined box inside the mask, and which gradiently
decreases to 0.0 for a pixel an equal distance on the outside of the mask.
The size of the box determines the level or degree of diffusion, which is
controlled by the colorist at the workstation 16. During the actual color
application, the diffused value is used to determine the portion of the
existing color for a given pixel that is replaced by the color from the
colormap of the mask in question. This results in one half of the color of
the polygonal mask at the border thereof comes from the mask itself and
the other half comes from any mask "behind" the mask in question, which
may be either the baseplane or another mask having a lower precedence
value.
Diffusing the color of masks in this manner provides many advantages. First
of all, a user has the ability to color blurry and out-of-focus objects
realistically and convincingly. More importantly, diffusion greatly
reduces the accuracy required by the user or colorist in drawing polygons
around objects, mainly because the receptors in the eye that detect edges
are monochromatic, and the color receptors of the eye have comparatively
poor spatial resolution. It has been found that this diffusing masking
effect allows a colorist to color most human faces with only six points or
vertices of a polygon (i.e., using only a hexagon). A further advantage of
diffusion is the simulation of realistic light reflection and highlights
along the edges of curved objects, resulting in particularly realistic
colorization of faces.
After an initial object in the frame has been masked and assigned a color,
the colorist may continued the masking/assigning color process for as many
objects as desired, as shown by the decision block of FIG. 2.
As mentioned above, masks may be reassigned a precedence value, as
represented by block 40, which allows masks to overlap each other without
interference as the masked objects move relative to each other in prior or
subsequent frames. This allows efficient tracking of an objects's motion
as the object moves spatially from frame to frame within a scene or shot,
possibly occluding other objects in the frame. The ability to overlap
masks increases the colorist's efficiency as objects change position in
each subsequent frame. For example, if a particular shot involves a series
of frames with a ball moving behind a tree, the tree mask should be
assigned a higher precedence than the ball mask so that the ball mask can
move behind the tree in succeeding frames without changing the shape of
the mask or without the ball interfering with the tree. Without this
precedence system, masks would have to fit together like jigsaw pieces
which would require their shape to be changed by the colorist with each
subsequent frame. Such a process would require tremendous user effort and
time.
Blocks or steps 32 to 46 have been described in an exemplary order but may
in reality take place in any order as desired by a user utilizing one of
the workstations 16 of the computer system 10. In addition, once a frame
has been downloaded (block 33) to the workstation 16, any one of the steps
or any combination of the steps may be implemented by a user, with that
particular information created by the user saved or stored in the database
46. Furthermore, this process may be implemented by a group of users at
the workstations 16 with individual users dedicated to a particular task;
for example, one user would only segment or separate shots 32 and then
store this information in the database 46, and another user would then
access this stored shot information to perform the masking of objects 34
and the naming of objects 36, thereafter storing the masking information
in the database 46 in the data storage unit 14. Colorist would then
perform the sophisticated and artistic process of recolorization. This
information may then in turn be accessed by any number of users at the
workstations 16 to perform the various tasks outlined by the block diagram
of FIG. 2.
Upon completion of the database or any portion of the database which has
had color assigned to the object masks 44, the following recolorization
technology may be implemented.
Luminance-to-Chrominance Mapping
The application or assignment of color to masks of the frame images is done
with a proprietary luminance-to-chrominance mapping. In the case of
colorizing black-and-white source material, luminance is derived from the
range of gray values in the black-and-white image. In the case of
modifying the color of a mask in a color frame, luminance is derived from
a weighted average of the color components. The amount of luminance
resolution is dependent on the number of bits used for sampling the source
image. For example, 8-bit sampling yields 256 levels of gray, while 12-bit
sampling yields 4,096 levels of gray.
The basic source of color for the colorization system is the colorspace or
colormap, which is a complete and continuous mapping of luminance to
chrominance. For every luminance value from black to white, the colormap
contains the chrominance values that could be applied. However, because of
the large number of luminance values representable, which may be at least
256, it would be quite tedious for the user to create a colormap from
scratch. Therefore, the colorization system has a higher-level abstraction
called a color recipe, which is a set of luminance points which includes
at least black and white and which the compiler 10 uses to generate a
colormap. In other words, the color recipe allows a colorist to specify
the color for selected points on the luminance range and have these
selected points influence color assignments of nearby luminance values.
The luminance values of every pixel or digital unit a mask which is
substantially homogeneous in color are determined. This may be
accomplished by integrating the area of the mask and generating a
histogram. The resulting histogram displays by percentage the number of
pixels or digital units having a particular luminance or intensity value.
The colorist can | | |
