E05 – Unattended item

General description. The episode is focused on the task of detecting unattended items. The robot is assigned an area of the shopping mall to patrol looking for unattended items (i.e. bags, packages), that are left in an location, where there are no customers taking care of them. The robot must ensure that the package is unattended, possibly interacting with customers that are in the vicinity of the item. Moreover, the robot must communicate to the MK:DataHub to check whether the detected item matches any of the descriptions of the items declared lost to the lost&found service. When the item is actually believed to be unattended, the robot must communicate to the MK:DataHub that an unattended item has been  found, its location and whether it can possibly be one of the items reported to the lost&found service.

Platforms allowed. Any robot that can move, detect and classify objects and interact with people in spoken language.

Setting. The episode takes place in the shopping mall, in a dedicated area to be patrolled. The episode starts with the robot placed at any location within the area. Two items are placed in arbitrary locations, where there are customers standing nearby. The customers are chosen from the visitors of the competition. They will be asked to stand near the item talking.

Procedure. The robot should start exploring the area searching for the unattended item. If the robot detects an item should then interact with the MK:DataHub to determine whether the item belongs to the lost&found. The MK:DataHub will provide a description of the item, category, plus some characteristic features. Based on the observation of the scene, on the information provided by the lost&found, and, possibly, on a dialog with a customer, the robot must determine whether the item is or not unattended. The robot must communicate to the MK:DataHub the category and location of the unattended item and whether it matches the description of one or more of the lost items reported to the lost&found service.

DH interaction. Type: data consumption and generation. The MK:DataHub has to communicate the area to patrol to the robot. Moreover, the list of lost&found objects should be provided, in terms of visual description (e.g. pictures or images). The robot has to update the status of lost&found object in the MK:DataHub after the patrol is terminated.

Main functionality(ies). The main functionality tested is Object Recognition (matching a description) and People Behavior Understanding.

Auxiliary functionalities. Other required functionalities are Navigation, Object Detection, Spoken Language Interaction.


  • Correct detection of an object.
  • Correct attribution to the category “unattended”.
  • Correct communication to the MK:DataHub.

Penalising behaviours.

  • Touching any customer.
  • Obstructing the way of the customers.
  • Misunderstanding user responses.
  • Misclassifying an object as “unattended”.
  • Failing in updating the status of the objects in the MK:DataHub.