Having spent ages in academia, I still measure time in terms of academic years. Among other things, this means that my professional/educational New Year resolutions tend to be scheduled for the 1st of September, rather than January. This time, I made the decision to get back into checking out everything (well, everything here, here, and here) that is new on the arXiv on the daily basis. If you are interested in what pre-prints drew my attention in the previous week, read on!

The first two are review papers. I love a good review paper: the kind that starts out with an overview of the field, outlines the different research directions, and finishes up with the currently open questions.


We all heard of the embarassing examples of AI technology perpetuating, or rather, amplifying, the biases that are already present in our daily lives. AI models used to make decisions on who gets approved for a mortgage and whose resume will stand a chance to appear before an actual human being, learn from past data. If a group of people was historically underrepresented, or put at a disadvantage, tough luck - a naive ML model will very likely pick this up and make it official. What is worse, few models (hardly any that come from the deep learning side) can be cast into a transparent enough form to explain their decisions. Fairness in Deep Learning: A Computational Perspective is a recent overview of the ML research meant to both detect and mitigate such undesired biases. At 8 pages, it makes for a good morning read. While we are at it, let me mention a second paper dealing with this subject that came out last week: On Measuring and Mitigating Biased Inferences of Word Embeddings, discussing the use of the natural language inference (NLI) task to measure the bias present in word embeddings.

Example from Fairness in Deep Learning: if being male significantly increases the doctor prediction confidence of a classifier, this indicates the model’s bias against women.

Another active research area that will only continue to grow is the medical applications of AI. Reinforcement Learning in Healthcare: A Survey is a new pre-print providing a rather comprehensive review of both reinforcement machine learning and its applications in the healthcare field. In fact, the authors make a good point that reinforcement learning and the healthcare domain are well suited to work in tandem:

Unlike traditional supervised learning methods that usually rely on one-shot, exhaustive and supervised reward signals, RL tackles with sequential decision making problems with sampled, evaluative and delayed feedback simultaneously. Such distinctive features make RL technique a suitable candidate for developing powerful solutions in a variety of healthcare domains, where diagnosing decisions or treatment regimes are usually characterized by a prolonged and sequential procedure.

Are you concerned about the evil AI technologies used to track your every move? Fear not, according to AdvHat: Real-world adversarial attack on ArcFace Face ID system you can trick a state-of-the-art face recognition model by simply putting a color rectangle on your forehead hat (anyone senses a retail business opportunity here?)

At the other end of the spectrum, if you are concerned with people trying to fool your deep neural network by sticking colored rectangles on their head via adversarial examples, check out A Statistical Defense Approach for Detecting Adversarial Examples.


Since I am currently working on a machine reading comprehension project, I cannot possibly pass on anything that contains BERT in the title. The tabloid-worthy (at least, in name) Revealing the Dark Secrets of BERT finds that the multiple attention heads used in BERT's architecture are, in fact, too multiple, and some are better off being removed. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks looks at a problem that, in my opinion, does not get nearly enough attention from the BERT enthusiasts. Many of the state-of-the-art results accomplished with BERT & Co (including various sentence-pair tasks with wide practical applciations) come from architectures that are very much unsuitable for production. In the (very slightly modified) words of the authors:

BERT  has set a new state-of-the-art performance on sentence-pair regression tasks like semantic textual similarity (STS). However, it requires that both sentences are fed into the network, which causes a massive com- putational overhead: Finding the most similar pair in a collection of 10,000 sentences requires about 50 million inference computations (~65 hours) with BERT. In this publication, we present Sentence-BERT (SBERT) ... [which]... reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT.

Hope you enjoyed this by-no-means-exhaustive list of recent arXiv pre-prints, and I'll see you next week!