Foundations of Foundation Models

    13:30 - 14:30, Dec. 20, 2024, UTC+8
    Shuicheng Yan
    About the Speaker
    Prof. Yan Shuicheng is currently the Honorary Advisor of Kunlun Tech and was formerly the Chief Scientist at Sea. He is a member of the Singapore Academy of Engineering and a Fellow of AAAI, ACM, IEEE, and IAPR, and his research interests include computer vision, machine learning, and multimedia analytics. To date, Prof. Yan has published more than 800 papers in top international journals and conferences with an H-index of 140+. Prof. Yan Shuicheng's team has won more than ten awards in two core competitions, Pascal VOC and ImageNet (ILSVRC). In addition, his team has won more than ten best paper and best student paper awards, especially a grand slam at ACM Multimedia, the top conference in multimedia, including three best paper awards, two best student paper awards, and a best demo award.
    Abstract
    In this presentation, I divide the research on foundation models into three aspects, corresponding to three types of errors: approximation error, estimation error, and optimization error. For approximation error, I will share two recent advancements in model architecture, MoE++ and MoH. For optimization error, I will present our new optimizer called Adan and discuss how we optimize individual layers based on samples with varying levels of difficulty. Additionally, I will introduce two of our new products, SkyMusic and SkyReels.