Deep learning\u00c2\u00a0(also known as\u00c2\u00a0deep structured learning\u00c2\u00a0or\u00c2\u00a0hierarchical learning) is part of a broader family of\u00c2\u00a0machine learning\u00c2\u00a0methods based on artificial neural networks. Learning can be\u00c2\u00a0supervised,\u00c2\u00a0semi-supervised\u00c2\u00a0or\u00c2\u00a0unsupervised.<\/p>\n
Deep learning architectures such as\u00c2\u00a0deep neural networks,\u00c2\u00a0deep belief networks,\u00c2\u00a0recurrent neural networks\u00c2\u00a0and\u00c2\u00a0convolutional neural networks\u00c2\u00a0have been applied to fields including\u00c2\u00a0computer vision,\u00c2\u00a0speech recognition,\u00c2\u00a0natural language processing, audio recognition, social network filtering,\u00c2\u00a0machine translation,\u00c2\u00a0bioinformatics,\u00c2\u00a0drug design, medical image analysis, material inspection and\u00c2\u00a0board game\u00c2\u00a0programs, where they have produced results comparable to and in some cases superior to human experts.<\/p>\n
Artificial Neural Networks\u00c2\u00a0(ANNs) were inspired by information processing and distributed communication nodes in biological systems. ANNs have various differences from biological\u00c2\u00a0brains. Specifically, neural networks tend to be static and symbolic, while the biological brain of most living organisms is dynamic (plastic) and analog.<\/p>\n
Deep neural networks are generally interpreted in terms of the\u00c2\u00a0universal approximation theorem\u00c2\u00a0or\u00c2\u00a0probabilistic inference.<\/sup><\/p>\n The classic universal approximation theorem concerns the capacity of\u00c2\u00a0feedforward neural networks\u00c2\u00a0with a single hidden layer of finite size to approximate\u00c2\u00a0continuous functions.\u00c2\u00a0In 1989, the first proof was published by\u00c2\u00a0George Cybenko\u00c2\u00a0for\u00c2\u00a0sigmoid\u00c2\u00a0activation functions\u00c2\u00a0and was generalised to feed-forward multi-layer architectures in 1991 by Kurt Hornik.\u00c2\u00a0Recent work also showed that universal approximation also holds for non-bounded activation functions such as the rectified linear unit.<\/p>\n The universal approximation theorem for\u00c2\u00a0deep neural networks\u00c2\u00a0concerns the capacity of networks with bounded width but the depth is allowed to grow. Lu et al.\u00c2\u00a0proved that if the width of a\u00c2\u00a0deep neural network\u00c2\u00a0with\u00c2\u00a0ReLU\u00c2\u00a0activation is strictly larger than the input dimension, then the network can approximate any\u00c2\u00a0Lebesgue integrable function; If the width is smaller or equal to the input dimension, then\u00c2\u00a0deep neural network\u00c2\u00a0is not a universal approximator.<\/p>\n The\u00c2\u00a0probabilistic\u00c2\u00a0interpretation\u00c2\u00a0derives from the field of\u00c2\u00a0machine learning. It features inference, as well as the\u00c2\u00a0optimization\u00c2\u00a0concepts of\u00c2\u00a0training\u00c2\u00a0and\u00c2\u00a0testing, related to fitting and\u00c2\u00a0generalization, respectively. More specifically, the probabilistic interpretation considers the activation nonlinearity as a\u00c2\u00a0cumulative distribution function.\u00c2\u00a0The probabilistic interpretation led to the introduction of\u00c2\u00a0dropout\u00c2\u00a0as\u00c2\u00a0regularizer\u00c2\u00a0in neural networks.\u00c2\u00a0The probabilistic interpretation was introduced by researchers including\u00c2\u00a0Hopfield,\u00c2\u00a0Widrow\u00c2\u00a0and\u00c2\u00a0Narendra\u00c2\u00a0and popularized in surveys such as the one by\u00c2\u00a0Bishop.<\/p>\n Deep learning revolution:<\/span><\/p>\n In 2012, a team led by George E. Dahl won the “Merck Molecular Activity Challenge” using multi-task deep neural networks to predict the\u00c2\u00a0biomolecular target\u00c2\u00a0of one drug.\u00c2\u00a0In 2014, Hochreiter’s group used deep learning to detect off-target and toxic effects of environmental chemicals in nutrients, household products and drugs and won the “Tox21 Data Challenge” of\u00c2\u00a0NIH,\u00c2\u00a0FDA\u00c2\u00a0and\u00c2\u00a0NCATS.<\/p>\n Significant additional impacts in image or object recognition were felt from 2011 to 2012. Although CNNs trained by backpropagation had been around for decades, and GPU implementations of NNs for years, including CNNs, fast implementations of CNNs with max-pooling on GPUs in the style of Ciresan and colleagues were needed to progress on computer vision. In 2011, this approach achieved for the first time superhuman performance in a visual pattern recognition contest. Also in 2011, it won the ICDAR Chinese handwriting contest, and in May 2012, it won the ISBI image segmentation contest.\u00c2\u00a0Until 2011, CNNs did not play a major role at computer vision conferences, but in June 2012, a paper by Ciresan et al. at the leading conference CVPR\u00c2\u00a0showed how max-pooling CNNs on GPU can dramatically improve many vision benchmark records. In October 2012, a similar system by Krizhevsky et al.\u00c2\u00a0won the large-scale\u00c2\u00a0ImageNet competition\u00c2\u00a0by a significant margin over shallow machine learning methods. In November 2012, Ciresan et al.’s system also won the ICPR contest on analysis of large medical images for cancer detection, and in the following year also the MICCAI Grand Challenge on the same topic.\u00c2\u00a0In 2013 and 2014, the error rate on the ImageNet task using deep learning was further reduced, following a similar trend in large-scale speech recognition. The\u00c2\u00a0Wolfram\u00c2\u00a0Image Identification project publicized these improvements.<\/p>\n Image classification was then extended to the more challenging task of generating descriptions (captions) for images, often as a combination of CNNs and LSTMs.<\/sup><\/p>\n Some researchers assess that the October 2012 ImageNet victory anchored the start of a “deep learning revolution” that has transformed the AI industry.<\/p>\n In March 2019,\u00c2\u00a0Yoshua Bengio,\u00c2\u00a0Geoffrey Hinton\u00c2\u00a0and\u00c2\u00a0Yann LeCun\u00c2\u00a0were awarded the\u00c2\u00a0Turing Award\u00c2\u00a0for conceptual and engineering breakthroughs that have made deep neural networks a critical component of computing.<\/p>\n The above is a brief about Deep Learning. Watch this space for more updates on the latest trends in Technology.<\/p>\n","protected":false},"excerpt":{"rendered":" Deep learning\u00c2\u00a0(also known as\u00c2\u00a0deep structured<\/p>\n","protected":false},"author":1,"featured_media":1522,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[492,491,7],"tags":[494,493,18],"class_list":["post-1521","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-artificial-neural-networks","category-deep-learning","category-techtrends","tag-artificial-neural-networks","tag-deep-learning","tag-technology"],"_links":{"self":[{"href":"https:\/\/blog.amt.in\/index.php\/wp-json\/wp\/v2\/posts\/1521","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.amt.in\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.amt.in\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.amt.in\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.amt.in\/index.php\/wp-json\/wp\/v2\/comments?post=1521"}],"version-history":[{"count":1,"href":"https:\/\/blog.amt.in\/index.php\/wp-json\/wp\/v2\/posts\/1521\/revisions"}],"predecessor-version":[{"id":1523,"href":"https:\/\/blog.amt.in\/index.php\/wp-json\/wp\/v2\/posts\/1521\/revisions\/1523"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.amt.in\/index.php\/wp-json\/wp\/v2\/media\/1522"}],"wp:attachment":[{"href":"https:\/\/blog.amt.in\/index.php\/wp-json\/wp\/v2\/media?parent=1521"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.amt.in\/index.php\/wp-json\/wp\/v2\/categories?post=1521"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.amt.in\/index.php\/wp-json\/wp\/v2\/tags?post=1521"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}