MediAssist: Managing Personal Digital Photo Archives

by Noel Murphy, Cathal Gurrin and Gareth J. F. Jones

MediAssist is creating a range of applications to help people to manage their digital photo archives by employing context to automatically annotate photo collections.

Recent years have seen a revolution in photography with a move away from analog film towards digital technologies. Many users of digital cameras are now accumulating very large numbers of personal digital photos. While digital storage offers ample capacity to store these collections, technology for managing digital photos has not kept pace with advances in capture and storage technologies. The MediAssist project at the Centre for Digital Video Processing (CDVP) at Dublin City University is developing applications to enable users to efficiently search their image archives.Users often remember where and when a photo was taken, and indeed may recall this more clearly than the actual contents of a photo, so clearly context information of this type should be effective when searching digital photo archives. MediAssist applications use the capture and exploitation of context data as the basis for organising and searching personal photo collections. The time and date of photo capture are easily accessible from the camera, and this can be augmented with location information using coincident GPS data.

A typical operational scenario for a user using MediAssist tools would require a GPS enabled digital camera and although available, these are currently expensive. While awaiting the arrival to market of consumer-grade digital cameras with integrated GPS, we use a separate GPS device and we automatically match photos with their location from a GPS tracklog. Photos are uploaded from the camera to a PC which then automatically annotates each photo. This annotation stage extends the time labels to include weekday, weekend, month, season and year of capture, and the GPS location labels the town, city/state and country. Data from the camera is used to determine whether the photo was taken indoors or outdoors, The MediAssist tools use external information sources to further annotate images based on the time and location of capture to include whether the environment was light or dark, and even the prevailing weather conditions. In addition, various automatic content analysis tools are employed to add further content to a digital photo archive. Based on other research activities within the CDVP, we have integrated technologies which annotate each photo to identify (re-occurring) faces and buildings.

Once a user has their digital photos organized in the archive, MediAssist applications provide support for searching and browsing. We have two streams of research, producing a desktop and a mobile device interface.

Figure 1 shows the MediAssist desktop search application. Using this the user can search for photos based on location and/or time of capture, and to filter results based on light status, weather conditions, indoor or outdoor, and whether the photos contain people or buildings. Location based searching allows the user to select combinations of desired country, city, and town, while time based searching enables selection of a specific time period, eg summer, or time ranges, eg between February and April.

When examining the results of a search, a user is presented either with an exhaustive list of images or a list of events, where each event is a combination of photos taken at the same time and place, eg a birthday party, family outing, or trip to the zoo. Each event is represented in the desktop interface with up to five key photos automatically extracted from the event, to best represent the varied themes within the event. Selecting an event automatically displays all photos from that event.

In addition to the desktop interface, we are developing a mobile photo management application. This is based on the same underlying architecture as the desktop application, with an interface tailored to suit mobile devices.

It is known that the reduced screen space and limited options for interaction with mobiles devices means that user interaction with mobile devices should differ greatly from those of the standard desktop applications. Figure 2 shows the interface to the MediAssist mobile application, this addresses interactivity issues by primarily presenting a user with a personalised list of recommended ‘favourite’ photos. As a secondary access methodology it also supports location and feature based searching of the user’s archive. The search window in the mobile device is hidden from view and only appears when the user wishes to search the photo archive. The search options are more limited than in the desktop environment, and are primarily based on location and easily selected useful features, such as season and weather. The result of a search is a list of events, with a single most representative photo chosen for each event. If a user wishes, they can further browse the event by selecting the orange arrow.

The current MediAssist applications will be enhanced to incorporate advances in the annotation technologies as these become available within the CDVP. These improvements will include advances in recognition of content features within the photos, and also extensions to the annotation process to include further external knowledge.

The support of the Informatics Directorate of Enterprise Ireland is gratefully acknowledged.

Link:
Centre for Digital Video Processing: http://www.cdvp.dcu.ie/

Please contact:
Noel Murphy, Centre for Digital Video Processing, Dublin City University /Irish Universities Consortium
E-mail: murphyneeng.dcu.ie