Integrated Intelligence in Robotics: Vision Language Planning

Catalog description

Recent progress in language models and vision-language models are impressive but similar breakthroughs have not been observed in the field of robotics. Some experts believe it is only a matter of time until enough training data is available for robots to achieve a similar kind of breakthrough observed in language models, while others argue there are critical challenges that data alone would not be able to address, e.g., dexterity, safety, and social intelligence. In this course, we study how we can address those challenges related to robot intelligence. This course covers interdisciplinary topics on cognitive intelligence in robotic systems. Cognitive abilities constitute high-level, humanlike intelligence that exhibits reasoning or problem-solving skills. Such abilities as semantic perception, use of language, and task planning can be built on top of low-level robot autonomy. The topics covered may bridge multiple technical areas, for example, vision-language intersection and language-action/plan grounding. Project Themes Filmmaking presents varied robotics and machine learning challenges ranging from content generation, e.g. scenario generation or scene/video synthesis/editing to robotics automation, e.g. autonomous camera control or autonomous stop-motion control. Arts in Robotics is a fast growing field that connects robotics with humanity such as creativity and compassion. Sample projects include robot painting, robot sculpting, pottery making, etc. Following natural language instructions in navigation or manipulation tasks remain difficult despite the powerful foundation models. This theme encourages the students to design an innovative intelligent architecture that can be a game changer in this fast moving field. The projects on social robots will explore how robots can recognize and understand human emotions or social norms to choose suitable…