Welcome to this Monday's edition of Fresh from the arXiv: bringing you a hand-picked selection of the recent AI preprints that I found interesting. This time I kept the number of preprints short and instead chose to dive deeper into what has been happening to one of my favorite artificial neural network architectures - the Transformers (my other favorite being GANs).
With the advent of the Transformer neural network architecture, self-attention is popping up all over the deep learning field. In Temporal Self-Attention Network for Medical Concept Embedding, self-attention is used to embed medical concepts associated, for instance, with the course of a person's treatement (diagnoses, procedures, medications, etc), into a fixed-size vector that can then be used for further modeling. Just like in natural language, various medical concepts comprise a vocabularly of sorts, and the order of elements in the sequence matters, so it is only natural that the recent breakthroughs in NLP carry over to this domain.
Speaking of NLP, what about the not-so-natural languages - in particular, the ones used for programming? Transformers and self-attention are making their way into those as well. One of the useful features of Integrated Development Environments (IDEs) is code completion: IDE suggesting libraries, code bits etc based on the code that you have so far. The team behind A Self-Attentional Neural Architecture for Code Completion with Multi-Task Learning trains a Transformer network to perform this task, claiming state-of-the-art results in the process.
If you think about it, representing series of events or, to put it more generally, elements, as ordered sequences is an approach that applies in a very wide range of real-world scenarios. Since the number of possible elements (i.e., the size of the problem's vocabularly) is often extremely large, so is the dimensionality of the one hot encoded vectors used to represent such inputs in the machine. It is therefore advantageous to embed each element into a lower-dimensional space, hopefully in a way that makes sense - and contextual embeddings, that take into account the other elements of the sequence (e.g. those based on self-attention), tend to help with the making-sense part. Additionally, sequences usually have variable lengths, which complicates their further processing. One solution, used in the first preprint I mentioned above, is to embed the entire sequence into a fixed-sized vector, regardless of the sequence's original length. Since Transformer networks and self-attention have proven useful in both element and sequence embedding, I expect to keep seeing them applied to many more domains outside of natural language processing in the near future.
If you have been at all following what has been going on in AI lately, I hardly need to convince you that the field has seen some pretty incredible successes. Machine learning models have outperformed humans even at tasks that were previously thought to be out of AI's reach, yet this in itself does not necesserily get us any closer to the so-called Artificial General Intelligence (AGI). AGI can be defined as the ability of a machine to perform any task that a human can, and it is this generalization property (*any* task) that makes it tricky. An image classification model may be an expert at figuring out what is on the image, but don't bother asking it to translate "Hello" from English to French.
Trying to get a machine to match the human mind sounds like a difficult task, to put it lightly. What about a non-human one? The study of animal cognition is a field centered around probing the mental capacities of non-human animals. For instance, can the husky pictured above manipulate its environment to get a treat? (As a dog parent, I can tell you that it can surely manipulate its owner!) The idea behind the Animal-AI Olympics competition is to subject AI models to the same kind of tests that animals have been evaluated on. What I find cool about this is not that it makes me think of artificially intelligent (or artificial and intelligent?) kittens, and not even that aiming for an AGI with a limited scope seems like a reasonable thing to do, but mainly that it addresses the fact that the very notion of intelligence should not be limited to the way that our human minds work.
The authors of The Animal-AI Environment: Training and Testing Animal-Like Artificial Cognition present an environment designed - you guessed it! - to train and test animal-like artificial cognition. The environment can be "interacted with" via a Python API and consists of different arenas containing various movable and immobile objects, as well as rewards. Indeed, it is similar to a game environment, but with the goal of testing the agent for some sort of general understanding, rather than merely completing well-defined objectives. Looking foward to seeing what comes out of this!
Till next week :)