Welcome to the selection of the arXiv preprints that caught my eye this past week!

I am thinking Fresh from the arXiv may not be sensational enough. It is a little too straight to the point, isn't it. Speaking of which, if you feel like your newsfeed has been short on clickbaits recently, you are in luck: they can now be generated automatically thanks to folks behind Clickbait? Sensational Headline Generation with Auto-tuned Reinforcement Learning among others. In the past headline generation has been mainly linked to the text summarization NLP task, but who are we kidding, the only place where the goal of the headline is to summarize the contents is probably the arXiv itself.

Preprints on adversarial attacks on neural networks (both on how to orchestrate and how to avoid them) never cease coming, and the one that caught my eye this past week was  Localized Adversarial Training for Increased Accuracy and Robustness in Image Classification. Most of the adversarial samples I have seen up until now looked like hallucinogenic dreams at best, and as such were easy to identify by any reasonably sober human being. Not the ones from this paper, however:

Left column: original images; middle column shows the bounding boxes around the objects being classified; in the right column the area outside the bounding boxes has been replaced with adversarial pixels. See how subtle the difference between the left and the right sides is? Not to the Inception v3 network! While it correctly identifies both original and bounding boxed pairs of images as a "bassoon" and a "great white" respectively, it ends up being at a loss for what is on the upper right image (its best guess, with 12% certainty, is an "electric fan"). On the other hand, Inception is 99% sure that the lower right photo is that of a "parachute"!

Naturally, the Related Works section of the preprint indicates that I have been living under a rock, and, I quote, "the observation that imperceptible changes could reliably fool image classifiers was first made in 2014" (with a list of appropriate references to follow). In any case, the authors propose a simple solution to an apparently common problem in the so-called adversarial training (adding adversarial examples to the training set, in the hopes that it will make the classifier more robust to attacks. While the approach may very well succeed on that front, the resulting models tend to loose their original accuracy on unaltered images). The authors' idea is to include images with adversarial backgrounds, like the ones in the right column in the Figure above, in the training set, in order to train the model to focus on the object rather than the background.

In a yet another spin on the BERT model that has taken the field of Natural Language Processing by storm, the authors of Knowledge Enhanced Contextual Word Representations  have developed KnowBert: BERT-based embeddings enhanced with facts coming from existing knowledge bases. Personally, I would guess that the extent of the advantage of KnowBert over BERT will vary greatly depending on the NLP task at hand, but contextualized word embeddings combined with knowledge bases/graphs sure sound like a promising research direction. Another recent preprint on this subject, Commonsense Knowledge + BERT for Level 2 Reading Comprehension Ability Test, attemps to take BERT reading comprehension abilities to [quite literally] the next level. It is also worth noting that Baidu's ERNIE (Enhanced Language RepresentatioN with Informative Entities), which does make use of knowledge graphs during its pre-training stage, has been shown to outperform Google's BERT on multiple NLP tasks.

See you next week!