 |
Description  |
|
|
The present invention relates to systems for determining position in space
and/or orientation in space, that may be used, for example, to instruct an
industrial robot, and the apparatus employing such systems.
There accompanies herewith a paper by the present inventors entitled
"Visual Robot Instruction", Proceedings of the Fifth International
Symposium on Industrial Robots, Chicago, Ill., Sept. 22-24, 1975; said
paper is hereby incorporated herein by reference. Some, but not all, of
the material in the paper is included hereinafter. Most of the description
that follows is centered around instructing industrial-type robots,
sometimes called programmable manipulators, but the concepts herein
disclosed have wider use.
Industry demands a fast and safe method to program industrial robots. Prior
proposals for such programming include the use of potentiometers, voice
control, cathode ray tube (CRT) light pens, master-slave harnesses and
manual controls, as is noted in said paper. Each programming aid offers
advantages, but even when they are used together they cannot specify what
a foreman easily tells a human operator. Each aid has distinct
limitations. Programmers using potentiometers tend to move each
degree-of-freedom one at a time. Joysticks provide some coordination in
controlling two or three degress-of-freedom, but robots typically have
more than three. Voice control is a convenient method for specifying
incremental motion in world or hand coordinates; however, if the increment
is numerically specified, performance is limited in speed by the
programmer's ability to quantify his visual information. Using the voice
control technique of moving the robot in slow motion until a "Stop"
utterance is given may be susceptible to the problem of timing consistency
on the part of the programmer. Additionally, voice control is not well
suited to describing complex trajectories. Currently voice control is also
an expensive programming aid. CRT light pens are convenient for specifying
points on a plane; however specifying points in three dimensional space on
two screens is difficult. Describing a trajectory is very difficult. The
cost of CRTs must also be weighed in a decision to include this
programming aid. The use of a master-slave harness is perhaps the most
expensive robot programming aid. It is also limited in some ways to robots
with an anthropoid configuration. For example, adjustments of harness
elbow elevation to avoid an obstacle may not cause a non-anthropomorphic
arm to adjust appropriately. The programming technique of physically
leading a robot through a sequence of motions is limited by safety
considerations. Uncontrolled behavior is possible if power assist is used.
When using most programming aids, programmers come close to the robot to
inspect the location of the gripper. This proximity endangers the
programmer. For safety, it is desirable if a sequence of motions could be
specified with the programmer in the workspace and the robot inactive.
Accordingly, it is an object of the present invention to provide a system
adapted to facilitate the instruction or programming of
numerically-controlled, industrial robots to enable such robots to perform
new or additional tasks.
Another object is to permit such instruction or programming in a way
similar to that used by one human instructing another.
Still another object is to provide a system that points out obstacles to
the robot.
A further object is to provide a system to aid in programming of other
numerically-controlled machine such as, for example,
numerically-controlled machine tools.
A still further object is to provide such programming in a manner that does
not endanger a human operator.
A still further object is to provide a visual system for industrial robots
and the like that need not be rigidly positioned relative to a work
surface or space.
These and still further objects are discussed hereinafter and are
delineated in the appended claims.
The foregoing objects are achieved in a visual system to determine position
in three-dimensional space and/or orientation in three-dimensional space,
which includes a portable object arbitrarily movable in three-dimensional
space and possessing the discernible properties of position in space
and/or orientation in space. One or more sensors extract visual
information or image data from the portable object and convert that
information to an electric signal or signals. A computer, properly
programmed is connected to receive the signal or signals from the sensor
or sensors and evaluate the same to provide an indication of position in
three-dimensional space and/or orientation in three-dimensional space. The
computer, on the basis of the information received and in the light of its
pre-programs, prepares messages that may be transmitted to the
mechanically-active elements of one or more robots which, in turn, act or
perform some function on the basis of the messages received.
The invention is hereinafter described with reference to the accompanying
drawing in which:
FIG. 1 is a diagrammatic representation of a system employing the present
inventive concepts and shows, among other things, portable means and a
plurality of sensors;
FIG. 2 is a schematic representation, partly block diagram in form, showing
a portion of the system of FIG. 1, the portable means being two lights and
the sensor being a television camera;
FIG. 3 is an isometric view of some of the elements shown in block diagram
form as separate elements in FIG. 1 but combined in FIG. 3;
FIG. 4 is a schematic representation, partly block diagram in form, of a
modification of the system of FIG. 2 in that the later figure shows two
cameras;
FIG. 5A shows a single camera as a sensor and portable means composed of
three lights or light sources;
FIG. 5B shows a cluster of lights which, together, form a single light
source of a type that can be employed as any one of the light sources in
FIG. 5A;
FIG. 5C shows an arbitrarily-shaped distributed light source that can be
used in FIG. 5A;
FIG. 6A is an isometric, diagrammatic representation of a machine that
includes the present teachings;
FIG. 6B is a diagrammatic representation, partly block diagram in form, of
an actual industrial robot system that has been experimented with;
FIG. 7 shows a plan-view layout for a typical industrial operation for the
robot of FIG. 6;
FIG. 8 is a plan view showing typical visual programming parts with respect
to the locations of FIG. 7;
FIG. 9 shows a fiducial array; and
FIG. 10 shows a modification of the portable means of FIG. 1.
As will be evident as the story unfolds, what the present inventors have
done here is present a surrogate for a thing to-be-sensed in
three-dimensional space or for a point, a line, or the like in such space.
From the surrogate, can be obtained visual information or image data that
can be interpreted and then used. The apparatus designated 102 in FIG. 1
includes a visual system 101 to determine position in space and/or
orientation in space; in FIG. 1 the surrogate is the portable means shown
at 1. The portable means 1 is arbitrarily movable in three-dimensional
space and, as later discussed in detail, possesses the discernible
properties of position in space and orientation in space. Sensors 2, 2',
etc., are positioned to extract visual information or data from the
portable means 1 and are operable to convert the visual information or
data to an electric signal or signals. A computer 3 is connected to
receive the electric signal or signals from the sensors 2, 2'... and is
programmed to evaluate the same to provide an indication of position in
space and/or orientation in space on the basis thereof. The apparatus 102
further includes robots 4A, 4B, etc., whose actions and movements are
controlled by the computer on the basis of the visual data, as discussed
in detail in later paragraphs. For convenience of explanation the visual
system 101 in FIG. 1 and later described systems 101A and 101B include
those elements that have to do with locating coordinates in space, the
term "robot" designates the mechanical elements that perform some
mechanical act, and the term "machine" is used to designate the two
combined with some work space (e.g., a machine tool).
The portable means can be one or more lights; in FIG. 2 it is shown at 1A
as two individual sources of light 10A and 10B. Each source acts
effectively as a point source of light in the system 101A of FIG. 2. The
sensor in FIG. 2 is a television camera 2A (two or more cameras can be
employed). The lights 10A and 10B can be modulated ON-OFF, or their
intensity and/or frequency can be modified by a sequencer 11 (the
sequencer need not be a separate unit; sequencing functions can be
performed by the computer); and the orientation of the lights in space and
relative to the camera can be changed as now explained with reference to
FIG. 3 wherein the elements 1A and a keyboard 5A are combined in a single
portable unit or visual programming device (VPD) that is marked 20.
In the device 20, the lights are again designated 10A and 10B and the
distance separating them can be changed by an operator using a control
switch 8 which operates an activator (e.g., small electric motor or
solenoid not shown). The device 20 is moved about by an operator who
grasps the same at a hand grip 9. Electric power to the device 20 is
connected through a cord 6 which serves, as well, to transmit messages
between the device 20 and the computer 3.
The visual programming device 20 can be used to conveniently communicate
spatial information to an industrial robot to cause the robot to perform
some task such as, for example, to grasp an object in its gripper. It is
held in a programmer's hand. It has an array of lights, the lights 10A and
10B, at one end, and a set of switches, the keyboard 5A, at the other end.
The switches communicate information to the computer which, as previously
noted, activates the robot (or robots). The lights may be under computer
control. The VPD 20 shown has the lights 10A and 10B at the tips of a
parallel jaw structure. This structure enables the two lights to be placed
on opposite sides of an object, for example. The center of the two lights
would, in this circumstance, correspond to a position that the programmer
wants the center of the robot's gripper (see fingers 70 and 71 in FIG. 6A)
to assume. A motor activated by the switch 8 on the VPD controls the
distance between the two lights 10A and 10B, as previously discussed.
Distance information may be sensed and communicated to the control
computer 3. This information may be used to verify visual computations.
The distance between light support members 12A and 12B can be made large
to surround large objects without blocking the line of sight from the TV
camera 2A to the lights 10A and 10B. This distance can be made small to
avoid interference with work station structures. Although programming can
be done with the workpiece in position, workpiece presence is not required
by the visual programming technique herein disclosed.
If the vision system has one television camera, as it has in FIG. 2, the
image of a small light can only specify a point in a plane. This plane may
be horizontal and just above the dominant materials handling surface of
the workstation. To specify points in space only a single light is
necessary. A switch on the keyboard 5A can be pressed to record each
point, a numerical keyboard entry may be used to specify how far beyond or
inside the reference plane is the three dimensional command point. This
three dimensional approach suffers either from inaccuracies in estimated
distances or a time consuming measurement process. For applications where
pick and place points are in the same plane, a vision system with one
television camera may suffice. Macros such as "go forward to reference
plane, close gripper, back off from reference plane" might be issued with
a single placement of the VPD 20 and activation of a particular switch on
the VPD keyboard 5A.
If the vision system has two television cameras, as shown at 2A and 2B in
FIG. 4, the three-dimensional location of the lights on the VPD may be
computed directly. The accuracy of this computation depends on the vision
system. For fixed camera mounts, factors affecting accuracy include the
field of view, spatial sampling rate, inter-camera geometry, location of
lights relative to television cameras, and camera geometric distortion. To
quantify the magnitude of the accuracy issue, one can assume a 100 cm line
in a reference plane perpendicular to the optical axis. If this line
covers a horizontal scan line with 250 samples/line and the light
activates a single picture element, location is theoretically limited to
.+-. 2 mm. For cameras with pan/tilt mounts, precision servo-mechanisms
are important.
To describe the proper way to grasp a workpiece, the vector from one robot
fingertip to the other fingertip must be specified. For this purpose, the
design of FIG. 3 is also appropriate. Each light specifies where each
robot fingertip (the robot's fingers are marked 70 and 71 in FIG. 6A and
the fingertips are marked 70A and 71A, respectively) is to be before a
grasping instruction is issued. The centroid between fingertips (points in
space) and orientation between fingertips may be specified at the same
time. However, if a light is obscured, fingertip orientation may be
specified elsewhere in the field of view. Activation of switches on the
keyboard 5A may be used to tell the system how to interpret the visual
instruction. Three types of instruction have been described thus far: (1)
point in space, (2) orientation between fingertips, and (3) point in space
plus orientation between fingertips.
If a VPD with just one light is used, positions and orientation between
fingertips cannot be specified at the same time. However, this can be done
in a two-step procedure. Activation of one control switch specifies a
point in space. Activation of another switch signals the computer to use
the present light position and the most recently specified position to
compute an interfingertips direction vector.
If the two light VPD in FIG. 3 is used to specify a unique orientation
between a pair of robot fingertips which are structurally different, a
distinction between the two lights must be made. This distinction is
possible using colored lights, but then the television camera must have
provisions for color vision. Another method of distinguishing lights is to
turn them on and off sequentially by the sequence 11 under the proper
control of the computer 3. With this scheme, each light is sensed during a
different frame of the television camera. The time required for this
technique is limited by turn-on and turn-off times of the lights and of
the photosensitive element of the television camera. Lights may also be
distinguished by size or intensity. Intensity discrimination requires a
grey scale interface between the television camera and the computer. Light
size does not map directly to image size due to distance attenuation and
directionality of light sources.
A distinction between lights is also useful for the case depicted in FIG. 4
wherein it is assumed that the four points of ray intersection marked 21A,
21B, 21C and 21D are in the same plane. In this case, it is not possible
to distinguish from the light image geometry which of the two combinations
of positions is actually the case without further measures. Even
interlight distance information will not always lead to a distinction
between the two cases since both possibilities may have the same
interlight distance. The region in space where interlight distance is the
same is a function of the angle between the optical axes of the two
television cameras 2A and 2B. This region in space is widened by
limitations on estimating distances using stereo vision. To overcome the
ambiguity presented by the configuration of FIG. 4, it is necessary to
identify the individual light 10A and 10B by, for example, the ON-OFF
procedure above-described.
In addition to the vector between robot fingertips 70A and 71A, a vector
pointing in the direction of the fingers 70 and 71 in FIG. 6A will
frequently have to be specified. For example, to make more rigid contact
with a particular class of workpiece, robot fingers may have contact
surfaces which are not symmetric. Another reason for wanting to specify
finger direction is to prevent the wrist structure shown at 72 in FIG. 6A
from colliding with workstation structures.
Finger direction can be specified by a third light in the array 1B as shown
in FIG. 5 wherein the third light is labeled 10C. Assuming that the three
lights can be located and distinguished, a robot finger direction vector
can be computed which is in the plane defined by lights 10A and 10B and
the third light 10C. The finger direction vector may be specified as being
perpendicular to the line between lights 10A and 10B and passing through
the third light 10C. Finger direction can also be specified by using a VPD
with a single light, as above noted; in this latter case the light would
be located where the third light is located using the three-light
technique. Activation of a switch on the keyboard tells the computer to
interpret this single light location as data to specify finger direction
relative to the most recent fingertips direction vector or to the most
recently specified point in space.
For situations involving a vision system having a single television camera,
as depicted in FIG. 2, it will often be adequate to make the general
specification that robot finger direction is to be perpendicular to the
reference plane. For other situations the general specification of a
vertical finger direction (e.g., horizontal reference plane) will be
appropriate.
Specification of trajectories is useful to avoid obstacles, contouring and
for special applications such as, for example, paint spraying, welding or
applying adhesives. A trajectory is specified by an initial gripper
configuration (location and orientation), an ordered set of intermediate
gripper configurations, a final gripper configuration and the time for
transit between each adjacent pair of gripper configurations. The gripper
passes through the intermediate gripper configurations without halting.
Servomechanisms to achieve this form of motion are available.
As described above, a VPD can conveniently specify gripper configurations.
Thus a VPD is a useful means to specify trajectories. A number of options
are possible to specify the time for each segment of the trajectory. For
many situations, it is appropriate to minimize the time to traverse a
trajectory. A command on the keyboard 5A can specify the minimizing
function. In this case, intermediate gripper configurations may be passed
through with less accuracy. A second approach is to have keyboard entries
specify the time for a trajectory segment immediately after specifying the
concluding gripper configuration for that segment. Rather than use
numerical entries, selection of standard "slow," "medium" and "high"
speeds may be preferable. A third approach is to activate a realtime
sampling algorithm. In this case the servo system would be instructed to
replicate the timing of the robot programmer. Keyboard entries could
specify if a fraction or a multiple of measured realtime performance was
desired for playback.
The VPD 20 specifies gripper position and orientation. Robot arms with
different kinematic and structural configurations will have links
occupying different portions of a workspace for the same gripper position
and orientation. Movement toward a particular gripper configuration may
cause the problem of contact between an arm structure and a workstation
structure. One solution to the problem is to use the VPD to specify a set
of points from which an envelope of workstation structures can be
computed. If such a set of points is specified before initiating standard
VPD programming, a gripper configuration command can be refuted in the
event that a collision is predicted. The robot programmer would be
notified of the result so that he can try another gripper configuration.
For a robot arm with redundant degrees-of-freedom, a search can be
conducted to select an arm configuration which avoids obstacles.
Subsequent refinement of the search for an arm configuration would have as
a goal the stipulation that the configuration be rapidly attained as an
intermediary between the previous gripper configuration command and the
next one. A scheme for modeling the outlines of obstacles and arm links is
discussed in one of the references mentioned in said paper.
The explanation in this and the next several paragraphs is made with
respect to FIGS. 6A and 6B. FIG. 6A illustrates diagrammatically what for
present purposes is called a machine that is assigned the numeral 103; the
machine 103 consists of a worktable 91 having a work surface 90, a visual
system 101B and a robot 4M; other mechanical parts of the robot 4M are
shown in FIG. 6B. In a sense, the whole of the elements in FIGS. 6A and 6B
could be called a robot, but as used herein and as above noted, the term
"robot" in all variations embraces the mechanical and electro-mechanical
parts shown in some detail in FIG. 6B. Also, strictly speaking, the work
surface 90 need not be part of a machine of which the robot 4M is also a
part, but it can be; and it is assumed such for this explanation. Anyway,
the cylindrical object labeled 60 in FIG. 6A is located or found within
the working volume of the machine and, more precisely, its x-y coordinates
are determined (its external dimensions are determinable, as well) by
interaction between the VPD 20, the camera 2A and the computer 3, as now
explained.
Various x-y coordinates on the surface 90 are located with respect to
fiducial or reference means 50 that comprises a plurality of lights 51A
and 51B (there may also be three lights as shown in FIG. 9, or more, and
each elements 51A ... need not be a light, nor a single light). The
important matter here is that the location of the fiducial means 50 be
fixed relative to the workable surface 90 or be determinable relative
thereto. In FIG. 6A the TV camera 2A is attached to a camera arm 74. The
camera arm 74 and the arm 73 of the robot 4M are assumed for present
purposes to bear a known spatial relationship relative to one another and
the computer 3 is pre-programmed to be aware of the relationship. The TV
camera 2A is situated so as to be able to view the lights 51A, 51B, etc.
in the same scene or separate scenes through pan and tilt motion of the
camera. The visual location of the lights 51A, 51B, etc. is noted. These
data are interpreted by the computer 3 to yield x-y coordinates of the
fiducial means 50. In the same way, the location of the object 60 on the
work surface 90 is achieved using the VPD 20 in the manner previously
indicated. The TV camera 2A visually locates the VPD lights 10A, 10B. The
computer 3 interprets these data to yield the x-y position of the object
which is determined relative to the fiducial means 50. The robot 4M has an
end effector or wrist structure 72 comprising the fingers 70 and 71, as
previously mentioned. The computer 3 also extracts orientation information
from the VPD data which permit the wrist structure 72 to be aligned for
purposes of grasping the object 60 with the fingers 70 and 71. The end
purpose here is to have the robot 4M grasp the object 60 in the fingers 70
and 71 and, say, move it to another location in the work space or volume.
The robot 4M can independently move the end effector 72 along each of three
mutually orthogonal axes, one of which is perpendicular to the worktable
surface 90. The robot 4M can also rotate the end effector 72 about an axis
which is perpendicular to the worktable surface 90. The fingers 70 and 71
on the end effector 72 can be closed and opened to permit the robot to
grasp and release the workpiece 60. The fiducial array 50 is r | | |
