Montage:
Managing Images by Colour, Texture, Sketch, and Shape
Figure 1: Philosopher in Meditation by Rembrandt (1632); Louvre Museum, Paris
When Words Fail

A picture is worth a thousand words. This adage is especially true when it comes to describing visually rich images. Sometimes it is just impossible to summarize the content or the message or the characteristics of a picture in words, not to mention in a few words.

To complicate matters, the same picture may elicit different responses from different viewers, who may use very different words to describe the picture.

In a database when all data are actual images, therefore, the use of words or keywords may fail to be an effective way of retrieving the data desired, or classifying the many images stored.

Consider the picture in Figure 1. What keywords should be used to describe it so that people can easily access the image in the database? What keywords should be used so that people can easily retrieve all other images similar to Figure 1 from the database?

A group of researchers from the Department of Computer Science and Engineering at The Chinese University have developed an innovative system of managing and retrieving digital information that is stored in the form of images. Their project, commissioned by the Industry Technology Development Council from mid-1995 to mid-1997, has a specific objective of providing the local textile and fashion industries with an effective tool to manage and search fabric and design patterns.
Figure 2: Montage snapshots
Colour Sketch: It allows users to draw a pattern with specific colours. Montage will then search similar matches from the database.
Colour Similarity: Users can search similar images from a database based on the overall impression of colours.
Classification Tree: A tree-like organization tool which allows users to easily manage and organize their database. Users can search an image or build a complicated query by simple clicks. This reduces search time and avoids the typing mistakes made by traditional keyword search.

Montage Comes in to Help

The new system designed by the principal researchers¾Profs. I. King, A. Fu, L.W. Chan, and L. Xu¾is called 'Montage'. It is a content-based image database system supporting content-based retrieval of digital images by colour, texture, sketch, and shape which runs on the Windows NT/95 platform. It is also an image management system which allows users to catalogue, query, and manage their image assets efficiently and effectively. Its usefulness lies in its capability to search the database for images with approximately similar user-specified features.

Figure 2 illustrates some of these functions: (a) retrieving the painting in Figure 1 using the colour sketch pad plug-in, (b) retrieving fashion images using the colour similarity plug-in, and (c) retrieving fabric patterns using the classification tree tool.

The design of Montage is based on an open system architecture, which makes the system flexible, modular, and extensible. It includes a plug-in framework that makes the customization of a plug-in module for a specific application easy. This allows users to extend the function of the Montage system to suit their own needs.

How Does Montage Work?


Figure 3: Three major components of Montage

Figure 3 shows the three major components of Montage's content-based search engine.

The first component deals with image processing and pattern recognition. This includes image analysis, image manipu-lation, and feature extraction.

Feature extraction of an image comes in different forms. Currently, Montage contains four general purpose plug-in modules which process colour similarity, textile texture, colour sketch, and polygon shape information. Research is being done to develop better optimized basic plug-ins as well as more specialized plug-ins to be used in different applications, for example, for face recognition.

The second component deals with advanced database technology which tries to optimize the storage and organization of processed image features so that effective retrieval can be performed. The research involves finding the type of indexing structure that is most suitable for high-dimensional image feature data so that retrieval is more efficient.

The last major component concerns software/hardware system issues. Here, researchers are faced with the system integration part of Montage: designing user-friendly interface, optimizing and debugging programming code, defining specifications for plug-ins and communication protocols.

Beneficiaries of the New Technology

Montage is now used by the researchers to maintain and update an image database of textile and fabric patterns collected from various local manufacturers in the textile and fashion industries. Fashion designers can use its user-friendly interface to look up previous designs for inspiration. Corporations can create electronic catalogues with its search engine built in for product browsing and electronic commerce.

The usefulness of Montage can be extended to other industries. With the right set of plug-ins, it can be used in biomedical applications, image authentication, image archival, graphics design, and the publishing industry.

Further Developments

Upon the completion of the first version of Montage, the research team has started integrating the search engine into a website which will host textile and fabric sourcing and trading for Hong Kong. Montage's search engine will be used to locate fabrics with similar colour and texture features and fabric-related information for buyers.

The researchers also plan to add server and internet capabilities to Montage, making it easy for people to use its search engines to locate images over the Internet. With sufficient support and development, they hope that Montage will become an essential image retrieval tool in the computerization of the textile and other industries.


From left:
Prof. Irwin King, Prof. Ada Wai-chee Fu, Prof. Lai-wan Chan and Prof. Lei Xu

Prof. Irwin King received his B.Sc. degree in engineering and applied science from the California Institute of Technology in 1984, and his M.Sc. and Ph.D. degrees in computer science from the University of Southern California in 1988 and 1993 respectively. He joined The Chinese University in 1993. His research interests include multimedia systems¾content-based retrieval methods for image databases; image processing¾face analysis and computing; and neural networks¾unsupervised learning theory for visual processing.

Prof. Ada Wai-chee Fu received her B.Sc. degree in computer science from The Chinese University in 1983, and her M.Sc. and Ph.D. degrees in computer science from Simon Fraser University of Canada in 1986 and 1990 respectively. She worked at Bell Northern Research from 1989 to 1993 on a wide-area distributed database project and joined The Chinese University in 1993. Her research interests include issues in distributed databases, replicated data, data mining, content-based retrieval in multimedia databases, and parallel and distributed systems.

Prof. Lai-wan Chan received her BA and MA degrees in electrical science and her Ph.D. degree in information engineering from Cambridge University, England.
Her research interests are the learning and modelling of artificial neural networks, and the application of neural networks in time series prediction, image recognition, Cantonese speech recognition, image database systems, and data mining.

Prof. Lei Xu joined the Department of Computer Science and Engineering at The Chinese University in September 1993. Prior to that he worked as a postdoctoral/senior research associate for four years in several universities in Finland, Canada, and the USA, including Harvard and MIT. He is a past president of Asian-Pacific Neural Networks Assembly, an IEEE senior member, and an associate editor for six international journals on neural networks. He has published over 180 papers on neural networks, computer vision and pattern recognition, signal processing, and artificial intelligence. He has received several international and Chinese national academic awards.