2020 in Review With Johan A.K. Suykens

In 2020, Synced has covered a lot of memorable moments in the AI community. Such as the current situation of women in AI, the born of GPT-3, AI fight against covid-19, hot debates around AI bias, MT-DNN surpasses human baselines on GLUE, AlphaFold Cracked a 50-Year-Old Biology Challenge and so on. To close the chapter of 2020 and look forward to 2021, we are introducing a year-end special issue following Synced’s tradition to look back at current AI achievements and explore the possible trend of future AI with leading AI experts. Here, we invite Prof. Johan A.K. Suykens to share his insights about the current development and future trends of artificial intelligence.

Meet Johan A.K. Suykens

Johan A.K. Suykens is a full professor with KU Leuven, ESAT-Stadius and the Leuven.AI Institute. He is an IEEE and ELLIS Fellow. He has been awarded the International Neural Networks Society INNS 2000 Young Investigator Award, and European Research Council (ERC) Advanced Grant 2011 and 2017. He is currently serving as associate editor of the IEEE Transactions on Neural Networks and Learning Systems and the IEEE Transactions on Artificial Intelligence, and as program director of Master AI at KU Leuven.

The Best AI Technology Developed in the Past 3 to 5 Years: “Deep Learning in Connection to Kernel Methods With a Double-Descent Curve”

One of the fundamental questions in AI is about learning and generalization. After training a model on given training data, under which conditions will it generalize well on unseen data? This question is especially relevant for deep learning architectures, which contain a lot of unknown parameters. All classical theories about generalization are characterized by a U-shaped curve, related to the bias-variance trade-off, where one selects a best compromise solution avoiding over-fitting and under-fitting.

However, in recent years in studies on deep learning in connection to kernel methods, new insights have been obtained, with a double-descent curve instead of the U-shaped curve. In the new additional part of the curve one has the so-called interpolation regime with an over-parameterized model, which can still yield good generalization. Previously, one already had great insights on the use of weight decay regularization and early stopping to implicitly reduce to an effective amount of parameters. But this new shape instead of the U-shape can have important consequences on how one selects models in AI. Further refined analysis and deeper understanding of this phenomenon is definitely needed in the future.

The Most Promising AI Technology in the Next 1 to 3 Years: “Existing Models Enhanced With Additional Interpretability and Context Information”

With deep learning several powerful models have been proposed. Increasingly in AI there is the quest for having explainable models. Existing models enhanced with additional interpretability and context information should therefore be promising in the near future.

The Biggest Challenge in the Field of AI: “Get the Big Picture”

The biggest challenge in AI in my opinion is to get the big picture, i.e. understanding and guiding the full AI spectrum on theory, algorithms, applications, philosophical, ethical and legal dimensions. Currently, I see this mainly threefold:

Firstly, to get a deep and broad interdisciplinary understanding on the theory, models, methods and learning paradigms, across all related research fields. It is important to establish solid foundations with new unifying frameworks, applicable to a wide range of different application domains.

Secondly, to map such unifying theoretical frameworks to the many application domains, engineering the mathematics and conceiving new AI systems visions for it. Understanding AI model representations and optimally matching and tailoring them to given application characteristics is also a key aspect in this.

Thirdly, as AI is becoming increasingly powerful, the Ethics & AI dimension is of growing importance, at the individual persons-, societal- and planet-level. The safe contours should be outlined within which AI can flourish and be safely developed, paving the way for a bright AI future. Since the Asilomar AI principles and the Montreal Declaration Responsible AI, initiatives were taken on this by the European Commission, G20, Council of Europe, OECD, UNESCO, and others. Reaching a full worldwide consensus on Ethics & AI principles will be a milestone result for humankind.

The Latest Noteworthy Development: “New Synergies Between Deep Learning, Neural Networks and Kernel Machines, Working Towards a Unifying Framework”

We obtained new synergies between deep learning, neural networks and kernel machines, working towards a unifying framework. Through duality principles we were able to establish new connections between restricted Boltzmann machines (RBM) and deep Boltzmann machines on the one hand and kernel-based methods such as kernel principal component analysis (KPCA) and least squares support vector machines (LS-SVM) on the other hand. The resulting restricted kernel machines (RKM) are promising for deep learning, generative models, multi-view and tensor based models, latent space exploration, robustness and explainability. They can be used in primal form with neural network parametrizations or convolutional feature maps, as well as in dual form for deep kernel machines. Further challenges in this context include for example deep clustering models, semi-supervised learning, symmetry and invariance properties, optimal transport principles, efficient optimization schemes and large scale algorithms.

One of the future trends along this AI direction might be to have further synergies between deep learning, neural networks and kernel machines. With neural networks and deep learning several powerful and flexible architectures have been proposed, while kernel machines have solid foundations in optimization and learning theory. Deep kernel machines for which one can understand the role of each neuron in each layer in its neural network interpretation, might emerge.

---

Synced Report | A Survey of China’s Artificial Intelligence Solutions in Response to the COVID-19 Pandemic — 87 Case Studies from 700+ AI Vendors

This report offers a look at how China has leveraged artificial intelligence technologies in the battle against COVID-19. It is also available on Amazon Kindle. Along with this report, we also introduced a database covering additional 1428 artificial intelligence solutions from 12 pandemic scenarios.

Click here to find more reports from us.

---

We know you don’t want to miss any news or research breakthroughs. Subscribe to our popular newsletter Synced Global AI Weekly to get weekly AI updates.