In the second half of 1970, after completing his Ph.D. thesis (and getting Marvin Minsky to actually read it), Patrick pulled together a group of researchers to put together a “closed loop” robotics system. That is, one that encompasses sensing, planning, and actuation that affects the objects being sensed.
The group consisted of Patrick Winston, Eugene Freuder, Timothy Finin, and Berthold Horn, and used some code and/or help from Thomas Binford, David Waltz, Jerome Lerman, Robert Woodham, Adolfo Guzman-Arenas, Bill Gosper, Steward Nelson, Richard Greenblatt, and others.
The first task was to image and analyze images of structures built out of simple polyhedral blocks. The resulting line drawings had to be segmented into separate objects. The physical relationships between these objects had to be determined and then used to come up with a disassembly plan for the structure (which would ensure intermediate results were stable). Reversing the disassembly plan led to an assembly plan. The robot arm was then commanded to build this structure (mirror image reversed, just for fun) from parts earlier placed into a “warehouse area” that the arm could reach.
The camera was a random access image dissector from III which had significant radial and tangential distortions which had to be corrected. The transformation between “eye” coordinates and “arm” coordinates had to be calibrated by moving the AMF Versatran hydraulic arm in the work space while the camera tracked a “surveyor's mark” held by the hand. This was a monocular vision system that relied on knowledge of the working surface relative to the camera and the arm.
Succesful completion of the task was filmed while Dr. Donald Michie happened to be visiting from Edinburgh University. Fragments of that footage made it into the final cut of “Eye of the Robot” (script by Patrick Winston, narrated by David Waltz), some of which is devoted to Marvin Minsky explaining Patrick's thesis (he had finally read it). An actual sequence of the robot building a copy of a structure may be found about 10 minutes into the 15 minute film.
Sadly, the copy demo project was not documented well because the people working on it dispersed (post Ph.D.) and Patrick was soon roped into managing the Artificial Intelligence Laboratory, with little time for writing papers about past projects. As a result it is perhaps not as widely known as say Stanford's Shakey project, which also demonstrated visual sensing, planning and actuation. When I asked him about this in 2018, he responded:
I think at the time there were no obvious places to publish and Marvin wasn't one to push publication. I can't remember if we even did an AI Memo. Possibly too stupid to realize what a milestone it was. I do remember we sent Marvin a telegram, probably showing off.
