Researchers teach computers how to name images by 'thinking'

Penn State researchers have "taught" computers how to interpret images using a vocabulary of up to 330 English words, so that a computer can describe a photograph of two polo players, for instance, as "sport," "people," "horse," "polo."

The new system, which can automatically annotate entire online collections of photographs as they are uploaded, means significant time-savings for the millions of Internet users who now manually tag or identify their images. It also facilitates retrieval of images through the use of search terms, said James Wang, associate professor in the Penn State College of Information Sciences and Technology, and one of the technology's two inventors.

The system is described in a paper, "Real-Time Computerized Annotation of Pictures," given at the recent ACM Multimedia 2006 conference in Santa Barbara, Calif., and authored by Jia Li, associate professor, Department of Statistics, and Wang. Penn State has filed a provisional patent application on the invention. Major search engines currently rely upon uploaded tags of text to describe images. While many collections are annotated, many are not. The result: Images without text tags are not accessible to Web searchers. Because it provides text tags, the ALIPR system-Automatic Linguistic Indexing of Pictures-Real Time-makes those images visible to Web users.

ALIPR does this by analyzing the pixel content of images and comparing that against a stored knowledge base of the pixel content of tens of thousands of image examples. The computer then suggests a list of 15 possible annotations or words for the image.

"By inputting tens of thousands of images, we have trained computers to recognize certain objects and concepts and automatically annotate those new or unseen images," Wang said. "More than half the time, the computer's first tag out of the top 15 tags is correct."

In addition, for 98 percent of images tested, the system has provided at least one correct annotation in the top 15 selected words. The system, which completes the annotation in about 1.4 seconds, also can be applied to other domains such as art collections, satellite imaging and pathology slides, Wang said. The new system builds on the authors' previous invention, ALIP, which also analyzes image content. But unlike ALIP which characterized images by incorporating computational-intensive spatial modeling, ALIPR characterizes images by modeling distributions of color and texture.

The researchers acknowledge computers trained with their algorithms have difficulties when photos are fuzzy or have low contrast or resolution; when objects are shown only partially; and when the angle used by the photographer presents an image in a way that is different than how the computer was trained on the object. Adding more training images as well as improving the training process may reduce these limitations-future areas of research.

Source: Penn State

Related stories:

Scientists create touch-based illusion
Anyone who has seen an optical illusion can recall the quirky moment when you realize that the image being perceived is different from objective reality. Now, a team of scientists from MIT, Harvard and McGill has designed a new illusion involving the sense of touch, which is helping to glean new insights into perception and how different senses—such as touch and sight—work together.
First images of solar system's invisible frontier
NASA's sun-focused STEREO spacecraft unexpectedly detected particles from the edge of the solar system last year, allowing University of California, Berkeley, scientists to map for the first time the energized particles in the region where the hot solar wind slams into the cold interstellar medium.
Astrotechnology Brings Nanoparticle Probes Into Sharper Focus
While pondering the challenges of distinguishing one nanosize probe image from another in a mass of hundreds or thousands of nanoprobes, two investigators at Emory University and the Georgia Institute of Technology made an interesting observation. The tiny, clustered dots of light looked a lot like a starry sky on a clear night.
Astronomy technology brings nanoparticle probes into sharper focus
While pondering the challenges of distinguishing one nano-sized probe image from another in a mass of hundreds or thousands of nanoprobes, researchers at Georgia Tech and Emory University made an interesting observation. The tiny, clustered dots of light looked a lot like a starry sky on a clear night.
Microscope Sees with Nanoscale Resolution
Researchers have recently built an x-ray microscope that has a pixel resolution of just 15 nanometers, allowing scientists to study the properties of materials at the molecular scale and beyond.
New software advances photo search and management in online systems
Searching for digital photographs could become easier with a Penn State-developed software system that not only automatically tags images as they are uploaded, but also improves those tags by "learning" from users' interactions with the system.
High-resolution images herald new era in Earth sciences
High-resolution images that reveal unexpected details of the Earth's internal structure are among the results reported by MIT and Purdue scientists in the March 30 issue of Science.
Cheaper Color Printing by Harnessing Ben Franklin's Electrostatic Forces
Recent advances in the basic science of electrostatics could soon lead to color laser printers that are cheaper and up to 70 percent smaller than current models, a physicist reports at this week's AVS International Symposium and Exhibition in San Francisco.

News discussion:

Technology news