AI through the Ages

Exploring the legacy and prospects of Artificial Intelligence

Welcome back to our immersive exploration of the AI domain. In my inaugural post, we glimpsed the vast expanse of Artificial Intelligence, sketching the outlines of high level questions such as the what, how and why of AI. Today, we broaden our lens, tracing AI's lineage back to the days of symbol-powered systems, delving into some details of contemporary machine learning, and casting our gaze toward the future challenges where the future of AI stretches out before us.

Good Old Fashioned AI (GOFAI)

The early phase of AI can conveniently be said to have begun with the Dartmouth summer workshop in 1956. The term Good Old Fashioned Artificial Intelligence(GOFAI) was coined by John Haugeland. GOFAI refers to the early phase of artificial intelligence that was heavily rooted in symbolic processing. This phase imagined intelligence in terms of discrete, logical manipulation of symbols. The following are some points regarding the features and limitations of GOFAI:

• Features of GOFAI:

  • Physical Symbol System Hypothesis: It was presumed that any system capable of manipulating symbols could produce intelligent action. Allen Newell and Herbert Simon made the following claim:

    “A physical symbol system has the necessary and sufficient means for general intelligent action.”

  • Knowledge Representation: It was believed that knowledge could be represented in a form of logical rules and semantic networks. A well known and ambitious project which aimed to codify human knowledge was Cyc which was initiated in the 1980s by Doug Lenat.

• Limitations of GOFAI:

  • Complexity and Scalability: Symbolic systems struggled to scale with the complexity of real-world scenarios.

  • Context Sensitivity: GOFAI systems often failed to account for the nuanced context in which information is embedded.

  • Ontological Rigidity: GOFAI systems were inflexible to changes in the knowledge base or environment, due to their rigid ontological commitments.

By far, the greatest limitation of GOFAI has to be the ontological rigidity. By viewing the world as discrete, well-defined objects which can be mapped arbitrarily to symbols, it failed to capture the structural complexity and uncertainty inherent in real world signals(audio, video, images, language). Arguably, the success of current Deep Learning approaches is their representational richness-they are better able to encode the complexity of real world signals.

Present Day AI: Centered Around Machine Learning

We can mark the beginnings of this current AI era dominated by Deep Learning to the 2012 AlexNet paper. However, much progress began in the 1980s with Geoffrey Hinton’s paper Learning Representations by Back-propagating Errors. In addition, Yann Lecun proposed Convolutional Neural Networks in 1989. These methods did not gain ground at the time due to limited data and compute resources.

Contemporary AI has largely moved away from handcrafted knowledge representation to data-driven machine learning(ML) approaches. Of the data-driven ML approaches, Deep Learning occupies the prime spot. Deep Learning is characterized by the use of deep neural networks the combined use of gradient descent algorithms and neural networks enabling high powered statistical pattern discovery. Here are some features and limitations of present-day AI:

• Features of Present-Day AI:

  • Statistical Learning: Modern AI focuses on methods for detection and extraction of patterns in large datasets.

  • Data Availability and Computational Power: There is the assumption that sufficient and relevant data are available to train models. Availability of enormous computational resources is also taken as a given. In a widely discussed article titled The Bitter Lesson, Rich Sutton points in the direction of methods which scale well.

    “One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The two methods that seem to scale arbitrarily in this way are search and learning.”

    Rich Sutton is a giant in the AI field and his thoughts have had great impact on current generation of AI/ML practitioners.

  • Performance Metrics: AI systems are often gauged on specific performance metrics, assuming that improvement in these metrics translates to better intelligence.

• Limitations of Present-Day AI:

  • Dependency on Data and Data Bias: The reliance on large datasets can limit the ability to learn from few examples(few-shot learning), unlike human learning. In addition, AI systems can inherit and amplify biases present in training data.

  • Explainability: Many machine learning models, especially deep neural networks, are often described as ‘black boxes’-this is a reference to the difficulty in determining why a neural network

  • Generalization: Present AI systems can struggle to generalize knowledge across different domains.

I continue to be astounded and grateful for Deep Learning’s continued progress. I think that their greatest success has been in “encoding” real world signals in a form which we can perhaps utilize for reasoning and other cognitive tasks. Murray Shanahan in his paper Talking about Large Language Models has this to say:

“Indeed, it can reasonably be claimed that one emergent property of an LLM is that it encodes kinds of knowledge of the everyday world and the way it works that no encyclopedia captures…“

I would go as far as extending the above statement to Deep Learning as a whole.

Current and Future Challenges in AI

As we stand on the threshold of a new era in artificial intelligence, several challenges beckon our collective intellectual and innovative prowess:

• Continual Learning: Modern AI must evolve to learn continually, accumulating knowledge over time without forgetting previous learning—a challenge not adequately addressed by current deep neural net models.

• One-shot Learning: The ability to learn from from very few examples. What is desirable here is sample efficiency. Currently deep learning based methods need a huge amounts of data.

• Meta-Learning: AI systems need to learn how to learn, optimizing their learning process across various tasks to become more efficient learners. Here is an excerpt from Stuart Russell’s book(Human Compatible) which captures the goal of Meta Learning:
  ”Training a robot to stand up requires that the human already knows what standing up means so that the reward can be defined. What we want is for the robot to discover for itself that standing up is a thing-a useful abstract action, one that achieves the precondition(being upright) for walking or running or shaking hands or seeing over a wall and so forms part of many abstract plans for all kinds of goals.”

• Common Sense: Infusing AI with commonsense reasoning, enabling systems to reason about the world as humans do, remains a significant hurdle. Gary Marcus is the most prominent voice in articulating the need for common sense in AI systems. How to achieve this is not too clear at the moment. In any case, we’ll need to develop clarity on what constitutes common sense. I have strong feelings about the central importance of common sense knowledge and reasoning and will revisit this topic in a future article.

• Quest for Generalization: Beyond narrow specialization, the future of AI lies in generalization, the ability to apply learned knowledge to new, unseen problems and domains.

• Embodiment: Most AI systems today lack physical presence, ignoring the role of an embodied existence in cognitive processing. Over the past 3 decades, there is increasing recognition that embodiment is crucial for a robust, useful AI system. Advances in robotics and availability of robotics datasets could bring rapid progress to this area.

I am convinced that the evolution of AI will exceed the bounds of larger datasets and more compute. Creation of intelligent systems that mirror the learning, adaptation, and reasoning capacities of biological minds will involve more inspiration from biology. Take, for instance, Convolutional Neural Networks, which drew inspiration from the local receptive fields of the human visual cortex. However, this biological inspiration should not be seen as mere mimicry; rather, it serves as a stepping stone towards a deeper comprehension and amplification of the nature of intelligence.