home / Firmware/ Neural networks in simple words. In simple words about the complex: what are neural networks? History of neural networks

Neural networks in simple words. In simple words about the complex: what are neural networks? History of neural networks

James Loy, Georgia Tech. A beginner's guide, after which you can create your own neural network in Python.

Motivation: focusing on personal experience in learning deep learning, I decided to create a neural network from scratch without a complex learning library like. I believe it is important for a beginner Data Scientist to understand internal structure.

This article contains what I have learned and hopefully it will be useful for you too! Other helpful related articles:

What is a neural network?

Most articles on neural networks draw parallels with the brain when describing them. It's easier for me to describe neural networks as a mathematical function that maps a given input to a desired result without going into details.

Neural networks are made up of the following components:

input layer, x
arbitrary amount hidden layers
output layer, ŷ
kit scales and displacements between each layer W and b
selection for each hidden layer σ ; in this work we will use the activation function of Sigmoid

The diagram below shows the architecture of a two-layer neural network (note that the input layer is usually excluded when counting the number of layers in the neural network).

Creating a Neural Network class in Python looks simple:

Neural network training

Exit ŷ a simple two-layer neural network:

In the above equation, the weights W and the biases b are the only variables that affect the output ŷ.

Naturally, the correct values for the weights and biases determine the accuracy of the predictions. Process fine tuning the weights and biases from the input are known as.

Each iteration of the training process consists of the following steps

calculating the predicted output ŷ called forward propagation
updating weights and biases called

The sequential graph below illustrates the process:

Direct distribution

As we saw in the graph above, forward propagation is just an easy computation, and for a basic 2-layer neural network, the output of the neural network is given by:

Let's add feedforward to our Python code to do this. Note that for simplicity, we have assumed that the offsets are 0.

However, we need a way to assess the "goodness" of our forecasts, that is, how far our forecasts are). Loss function just allows us to do it.

Loss function

There are many loss functions available, and the nature of our problem should dictate our choice of loss function. In this work, we will use sum of squares of errors as a loss function.

The sum of squared errors is the average of the difference between each predicted value and the actual value.

The goal of training is to find a set of weights and biases that minimizes the loss function.

Back propagation

Now that we have measured our forecast error (loss), we need to find a way propagating the error back and update our weights and biases.

To find out the appropriate amount to correct for the weights and biases, we need to know the derivative of the loss function with respect to the weights and biases.

Recall from the analysis that the derivative of the function is the slope of the function.

If we have a derivative, then we can simply update the weights and biases by increasing / decreasing them (see diagram above). It is called .

However, we cannot directly calculate the derivative of the loss function with respect to the weights and biases, since the equation of the loss function does not contain weights and biases. Therefore, we need a chain rule to aid in the calculation.

Fuh! It was cumbersome, but it allowed us to get what we need - the derivative (slope) of the loss function with respect to the weights. We can now adjust the weights accordingly.

Let's add the backpropagation function to our Python code:

Checking the operation of the neural network

Now that we have our complete Python code for performing forward and backpropagation, let's take a look at our neural network by example and see how it works.

The perfect set of weights

Our neural network needs to learn the ideal set of weights to represent this function.

Let's train a neural network for 1500 iterations and see what happens. Looking at the iteration loss graph below, we can clearly see that the loss monotonically decreases to a minimum. This is consistent with the gradient descent algorithm we talked about earlier.

Let's look at the final prediction (output) from the neural network after 1500 iterations.

We did it! Our forward and backward propagation algorithm showed the neural network to work successfully, and the predictions converge on true values.

Note that there is a slight difference between predictions and actual values. This is desirable because it prevents overfitting and allows the neural network to better generalize invisible data.

Final reflections

I learned a lot in the process of writing my own neural network from scratch. While deep learning libraries like TensorFlow and Keras allow deep networks to be built without fully understanding inner work neural network, I find it helpful for novice Data Scientists to gain a deeper understanding of them.

I have invested a lot of my personal time in this work and I hope it is useful to you!

Many of the terms in neural networks are biologically related, so let's start at the beginning:

The brain is a complex thing, but it can also be divided into several main parts and operations:

The causative agent may be internal(like an image or an idea):

Now let's take a look at the basic and simplified parts brain:

The brain is generally like a wired network.

Neuron- the basic unit of calculus in the brain, it receives and processes chemical signals from other neurons, and, depending on a number of factors, either does nothing, or generates an electrical impulse, or an Action Potential, which then sends signals to neighboring ones through synapses bound neurons:

Dreams, memories, self-regulating movements, reflexes and in general everything that you think or do - everything happens thanks to this process: millions, or even billions of neurons work on different levels and create connections that create various parallel subsystems and represent a biological neural network.

Of course, these are all simplifications and generalizations, but thanks to them we can describe a simple
neural network:

And to describe it formalized using a graph:

Some clarification is required here. The circles are neurons, and the lines are the connections between them,
and, in order not to complicate at this stage, interconnection represent the direct movement of information from left to right... First neuron in this moment active and grayed out. We also assigned a number to it (1 - if it works, 0 - if not). The numbers between neurons show weight communication.

The graphs above show the moment in time of the network, for a more accurate display, you need to divide it into time intervals:

To create your own neural network, you need to understand how weights affect neurons and how neurons are trained. Let's take a rabbit (test rabbit) as an example and put it under the conditions of a classic experiment.

When a safe stream of air is directed at them, rabbits, like humans, blink:

This behavior can be drawn with graphs:

As in the previous diagram, these graphs show only the moment when the rabbit feels the breath, and thus we coding whiff as boolean. In addition, we calculate whether the second neuron is firing based on the weight value. If it is equal to 1, then the sensory neuron is triggered, we blink; if the weight is less than 1, we do not blink: the second neuron has limit- 1.

Let's introduce one more element - safe sound signal:

We can model rabbit interest like this:

The main difference is that the weight is now zero so we didn't get a blinking rabbit, well, at least not yet. Now we will teach the rabbit to blink on command, mixing
irritants (beep and breath):

It is important that these events take place at different times epochs, in graphs it will look like this:

The sound itself does nothing, but the airflow still makes the rabbit blink, and we show this through the weights times the stimuli (in red).

Education complex behavior can be simplified as a gradual change in weight between linked neurons over time.

To train the rabbit, repeat the steps:

For the first three attempts, the circuits will look like this:

Please note that the weight for the sound stimulus grows after each repetition (highlighted in red), this value is now arbitrary - we chose 0.30, but the number can be anything, even negative. After the third repetition, you will not notice a change in the rabbit's behavior, but after the fourth repetition, something amazing will happen - the behavior will change.

We removed the exposure with air, but the bunny still blinks when he hears the beep! Our last diagram can explain this behavior:

We trained the rabbit to respond to sound by blinking.

In a real experiment of this kind, it may take more than 60 repetitions to achieve the result.

Now we will leave the biological world of the brain and rabbits and try to adapt everything that
learned to create an artificial neural network. First, let's try to do a simple task.

Let's say we have a four-button machine that dispenses food when the correct one is pressed.
buttons (well, or energy if you are a robot). The task is to find out which button gives out the reward:

We can depict (schematically) what the button does when pressed like this:

It's better to solve such a problem in its entirety, so let's look at all the possible results, including the correct one:

Click on the 3rd button to get your dinner.

To reproduce a neural network in code, we first need to make a model or graph with which we can map the network. Here is one graph that is suitable for the task, besides, it displays its biological counterpart well:

This neural network simply receives input - in this case, it will be the perception of which button was pressed. Next, the network replaces the input information with weights and draws an inference based on the addition of the layer. It sounds a bit confusing, but let's see how the button is represented in our model:

Note that all weights are 0, so the neural network, like a baby, is completely empty but completely interconnected.

Thus, we associate the external event with the input layer of the neural network and calculate the value at its output. It may or may not coincide with reality, but we will ignore this for now and begin to describe the task in a computer-understandable way. Let's start by entering the weights (we'll use JavaScript):

Var inputs =; var weights =; // For convenience, these vectors can be called
The next step is to create a function that collects the input values and weights and calculates the output value:

Function evaluateNeuralNetwork (inputVector, weightVector) (var result = 0; inputVector.forEach (function (inputValue, weightIndex) (layerValue = inputValue * weightVector; result + = layerValue;)); return (result.toFixed (2));) / / It may seem complex, but all it does is match the weight / input pairs and add the result
As expected, if we run this code, we get the same result as in our model or graph ...

EvaluateNeuralNetwork (inputs, weights); // 0.00
Live example: Neural Net 001.

The next step in improving our neural network will be a way to check its own output or resulting values comparable to the real situation,
let's first encode this specific reality into a variable:

To detect inconsistencies (and how many), we'll add an error function:

Error = Reality - Neural Net Output
With it, we can evaluate the performance of our neural network:

But more importantly, what about situations where reality is beneficial?

Now we know that our neural network model isn't working (and we know how much), great! This is great because we can now use the error function to control our learning. But all this will make sense if we redefine the error function as follows:

Error = Desired Output- Neural Net Output
An imperceptible but such an important discrepancy, tacitly showing that we will
use previously obtained results for comparison with future actions
(and for training, as we will see later). This also exists in real life, complete
repeating patterns, so it can become an evolutionary strategy (well, in
in most cases).

Var input =; var weights =; var desiredResult = 1;
And a new function:

Function evaluateNeuralNetError (desired, actual) (return (desired - actual);) // After evaluating both the Network and the Error we would get: // "Neural Net output: 0.00 Error: 1"
Live example: Neural Net 002.

Let's summarize the intermediate result... We started with a task, made it simple model in the form of a biological neural network and got a way to measure its performance in comparison with reality or the desired result. Now we need to find a way to correct the inconsistency - a process that can be viewed as learning for both computers and humans.

How to train a neural network?

The basis for learning both biological and artificial neural networks is repetition
and learning algorithms so we will work with them separately. Let's start with
training algorithms.

In nature, learning algorithms are understood as changes in physical or chemical
characteristics of neurons after experiments:

A dramatic illustration of how two neurons change over time in the code and our "learning algorithm" model means that we will simply change something over time to make our life easier. Therefore, let's add a variable to indicate the degree of ease of life:

Var learningRate = 0.20; // The higher the value, the faster the learning process will be :)
And what difference does that make?

This will change the weights (just like a rabbit!), Especially the weight of the output we want to get:

How to code such an algorithm is your choice, for the sake of simplicity I add the learning factor to the weight, here it is as a function:

Function learn (inputVector, weightVector) (weightVector.forEach (function (weight, index, weights) (if (inputVector> 0) (weights = weight + learningRate;)));)
When used, this learning function will simply add our learning factor to the weight vector active neuron, before and after the round of learning (or repetition), the results will be as follows:

// Original weight vector: // Neural Net output: 0.00 Error: 1 learn (input, weights); // New Weight vector: // Neural Net output: 0.20 Error: 0.8 // If it's not obvious, the output of the neural network is close to 1 (output of a chicken) - that's what we wanted, so we can conclude that we are moving in the right direction
Live example: Neural Net 003.

Okay, now that we're heading in the right direction, the final piece of this puzzle will be implementation. repetitions.

It's not that difficult, in nature we just do the same thing over and over again, and in the code we just specify the number of repetitions:

Var trials = 6;
And the implementation of the function of the number of repetitions into our training neural network will look like this:

Function train (trials) (for (i = 0; i< trials; i++) { neuralNetResult = evaluateNeuralNetwork(input, weights); learn(input, weights); } }
Well, our final report:

Neural Net output: 0.00 Error: 1.00 Weight Vector: Neural Net output: 0.20 Error: 0.80 Weight Vector: Neural Net output: 0.40 Error: 0.60 Weight Vector: Neural Net output: 0.60 Error: 0.40 Weight Vector: Neural Net output: 0.80 Error : 0.20 Weight Vector: Neural Net output: 1.00 Error: 0.00 Weight Vector: // Chicken Dinner!
Live example: Neural Net 004.

Now we have a weight vector that will give only one result (chicken for dinner) if the input vector matches reality (pressing the third button).

So what is so cool we just did?

In that specific case our neural network (after training) can recognize the input data and say, which will lead to the desired result (we still need to program specific situations):

In addition, it is a scalable model, toy and tool for our training with you. We were able to learn something new about machine learning, neural networks and artificial intelligence.

Caution to users:

The mechanism for storing the studied weights is not provided, so this neural network will forget everything it knows. When updating or rerunning the code, you need at least six successful retries for the network to fully learn if you think that a person or a machine will press buttons in a random order ... It will take some time.
Biological networks for learning important things have a learning rate of 1, so only one successful repetition will be needed.
There is a learning algorithm that closely resembles biological neurons, with a catchy name: widroff-hoff rule, or widroff-hoff training.
Neuron thresholds (1 in our example) and overfitting effects (with a large number of repetitions, the result will be greater than 1) are not taken into account, but they are very important in nature and are responsible for large and complex blocks of behavioral responses. So are negative weights.

Notes and bibliography for further reading

I tried to avoid the math and hard terms, but if you're interested, we've built a perceptron, which is defined as a supervised learning (supervised learning) algorithm for double classifiers - a tough thing.

The biological structure of the brain is not an easy topic, partly because of imprecision, partly because of its complexity. Better to start with Neuroscience (Purves) and Cognitive Neuroscience (Gazzaniga). I've modified and adapted the bunny example from Gateway to Memory (Gluck), which is also a great guide to the world of graphs.

Another great resource, An Introduction to Neural Networks (Gurney), will suit all your AI needs.

Now in Python! Thanks to Ilya Andshmidt for providing the Python version:

Inputs = weights = desired_result = 1 learning_rate = 0.2 trials = 6 def evaluate_neural_network (input_array, weight_array): result = 0 for i in range (len (input_array)): layer_value = input_array [i] * weight_array [i] result + = layer_value print ("evaluate_neural_network:" + str (result)) print ("weights:" + str (weights)) return result def evaluate_error (desired, actual): error = desired - actual print ("evaluate_error:" + str (error) ) return error def learn (input_array, weight_array): print ("learning ...") for i in range (len (input_array)): if input_array [i]> 0: weight_array [i] + = learning_rate def train (trials ): for i in range (trials): neural_net_result = evaluate_neural_network (inputs, weights) learn (inputs, weights) train (trials)
Now on GO! Thanks to Kieran Macher for this version.

Package main import ("fmt" "math") func main () (fmt.Println ("Creating inputs and weights ...") inputs: = float64 (0.00, 0.00, 1.00, 0.00) weights: = float64 (0.00, 0.00, 0.00, 0.00) desired: = 1.00 learningRate: = 0.20 trials: = 6 train (trials, inputs, weights, desired, learningRate)) func train (trials int, inputs float64, weights float64, desired float64, learningRate float64) ( for i: = 1; i< trials; i++ { weights = learn(inputs, weights, learningRate) output:= evaluate(inputs, weights) errorResult:= evaluateError(desired, output) fmt.Print("Output: ") fmt.Print(math.Round(output*100) / 100) fmt.Print("\nError: ") fmt.Print(math.Round(errorResult*100) / 100) fmt.Print("\n\n") } } func learn(inputVector float64, weightVector float64, learningRate float64) float64 { for index, inputValue:= range inputVector { if inputValue >0.00 (weightVector = weightVector + learningRate)) return weightVector) func evaluate (inputVector float64, weightVector float64) float64 (result: = 0.00 for index, inputValue: = range inputVector (layerValue: = inputValue * weightVector result = result + layerValue) return result ) func evaluateError (desired float64, actual float64) float64 (return desired - actual)

You can help and transfer some funds for the development of the site

Neural networks are in vogue now, and for good reason. With their help, you can, for example, recognize objects in pictures or, conversely, draw the nightmares of Salvador Dali. Thanks to convenient libraries, the simplest neural networks are created in just a couple of lines of code, and it will not take more to appeal to the artificial intelligence of IBM.

Theory

Biologists still don't know exactly how the brain works, but how it works individual elements the nervous system is well studied. It consists of neurons - specialized cells that exchange electrochemical signals with each other. Each neuron has many dendrites and one axon. Dendrites can be compared to the inputs through which data enters the neuron, while the axon serves as its output. The connections between dendrites and axons are called synapses. They not only transmit signals, but can also change their amplitude and frequency.

The transformations that occur at the level of individual neurons are very simple, but even very small neural networks are capable of a lot. All the variety of behavior of the worm Caenorhabditis elegans - movement, search for food, various reactions to external stimuli, and much more - are encoded in just three hundred neurons. And okay worms! Even ants have enough 250,000 neurons, and what they do is definitely beyond the power of machines.

Almost sixty years ago, American researcher Frank Rosenblatt tried to create computer system, arranged in the image and likeness of the brain, however, the possibilities of its creation were extremely limited. Since then, interest in neural networks has flared up repeatedly, but time after time it turned out that computing power not enough for any advanced neural networks. Much has changed in this regard over the past decade.

Electromechanical brain with a motor

Rosenblatt's car was called the Mark I Perceptron. It was designed for image recognition, a task that computers still do so poorly. The Mark I was fitted with a resemblance to the retina of the eye: a square array of 400 photocells, twenty vertically and twenty horizontally. Photocells were randomly connected to electronic models of neurons, and they, in turn, to eight outputs. Rosenblatt used potentiometers as synapses connecting electronic neurons, photocells and outputs. When training the perceptron, 512 stepper motors automatically rotated the potentiometer knobs, adjusting the voltage on the neurons depending on the accuracy of the result at the output.

Here's how a neural network works in a nutshell. An artificial neuron, like a real one, has several inputs and one output. Each input has a weighting factor. By changing these coefficients, we can train the neural network. The dependence of the output signal on the input signals is determined by the so-called activation function.

In the Rosenblatt perceptron, the activation function adds the weights of all inputs to which a logical one arrived, and then compares the result with a threshold value. Its disadvantage was that a slight change in one of the weighting factors with this approach can have a disproportionately large impact on the result. This makes learning difficult.

Modern neural networks usually use non-linear activation functions such as sigmoid. In addition, the old neural networks had too few layers. Nowadays, one or more hidden layers of neurons are usually located between the input and output. This is where all the fun happens.

To make it easier to understand what is at stake, take a look at this diagram. It is a feedforward neural network with one hidden layer. Each circle corresponds to a neuron. On the left are the neurons of the input layer. On the right is the neuron of the output layer. In the middle is a hidden layer with four neurons. The outputs of all neurons in the input layer are connected to each neuron in the first hidden layer. In turn, the inputs of the output layer neuron are connected to all outputs of the hidden layer neurons.

Not all neural networks are designed this way. For example, there are (although less common) networks in which the signal from neurons is fed not only to the next layer, like the feedforward network from our circuit, but also in the opposite direction. Such networks are called recurrent. Fully connected layers are also just one of the options, and we'll even touch on one of the alternatives.

Practice

So, let's try to build the simplest neural network with our own hands and figure out how it works along the way. We will be using Python with the Numpy library (we could have done without Numpy, but with Numpy, linear algebra will take less effort). This example is based on Andrew Trask's code.

We need functions to calculate the sigmoid and its derivative:

Continuation is available only to participants

Option 1. Join the "site" community to read all the materials on the site

Membership in the community within the specified period will open you access to ALL Hacker's materials, increase your personal cumulative discount and allow you to accumulate a professional Xakep Score!

In the first half of 2016, the world heard about a lot of developments in the field of neural networks - their algorithms were demonstrated by Google (network player in the go AlphaGo), Microsoft (a number of services for image identification), startups MSQRD, Prisma and others.

To bookmarks

The editors of the site tell what neural networks are for, what they are for, why they captured the planet right now, and not years earlier or later, how much money can be made on them, and who are the main market players. Experts from MIPT, Yandex, Mail.Ru Group and Microsoft also shared their views.

What are neural networks and what tasks they can solve

Neural networks are one of the directions in the development of artificial intelligence systems. The idea is to simulate as closely as possible the functioning of the human nervous system - namely, its ability to learn and correct errors. This is main feature any neural network - it is able to independently learn and act on the basis of previous experience, each time making fewer and fewer mistakes.

The neural network imitates not only activity, but also the structure of the human nervous system. Such a network consists of a large number of individual computational elements ("neurons"). In most cases, each "neuron" refers to a specific layer of the network. The input data is sequentially processed on all layers of the network. The parameters of each "neuron" can change depending on the results obtained on the previous sets of input data, thus changing the order of the entire system.

Andrey Kalinin, Head of Mail.ru Search at Mail.Ru Group, notes that neural networks are capable of solving the same problems as other machine learning algorithms, the only difference is in the approach to learning.

All tasks that neural networks can solve are somehow related to learning. Among the main areas of application of neural networks are forecasting, decision making, pattern recognition, optimization, data analysis.

Vlad Shershulsky, director of Microsoft technology cooperation programs in Russia, notes that neural networks are now widely used: “For example, many large Internet sites use them to make reactions to user behavior more natural and useful to their audience. Neural networks are at the core of most modern systems speech recognition and synthesis, as well as image recognition and processing. They are used in some navigation systems, be they industrial robots or self-driving cars. Algorithms based on neural networks protect Information Systems from malicious attacks and help to identify illegal content on the network. "

In the short term (5-10 years), Shershulsky believes, neural networks will be used even more widely:

Imagine an agricultural harvester whose actuators are equipped with multiple video cameras. It takes five thousand pictures per minute of each plant in its trajectory strip and, using a neural network, analyzes whether it is a weed, whether it is infected with a disease or pests. And he treats each plant individually. Fantastic? Not quite anymore. And in five years it may become the norm. - Vlad Shershulsky, Microsoft

Head of the Laboratory of Neural Systems and Deep Learning at the Center for Living Systems of the Moscow Institute of Physics and Technology Mikhail Burtsev gives a tentative map of the development of neural networks for 2016-2018:

systems for recognition and classification of objects in images;
voice interaction interfaces for the Internet of Things;
service quality monitoring systems in call centers;
troubleshooting systems (including time predictors Maintenance), anomalies, cyber-physical threats;
intelligent security and monitoring systems;
bots replacing some of the functions of call center operators;
video analytics systems;
self-learning systems that optimize material flow management or the location of objects (in warehouses, transport);
intelligent, self-learning control systems for production processes and devices (including robotic ones);
the emergence of universal translation systems "on the fly" for conferences and personal use;
the emergence of bots-consultants technical support or personal assistants, in functions close to a person.

Grigory Bakunov, Director of Technology Dissemination at Yandex, believes that the basis for the dissemination of neural networks in the next five years will be the ability of such systems to make various decisions: “The main thing that neural networks are doing for a person is to save him from unnecessary decision-making. So they can be used almost everywhere where not too intellectual decisions are made by a living person. It is this skill that will be exploited over the next five years, replacing human decision-making with a simple automaton. ”

Why neural networks have become so popular right now

Scientists have been developing artificial neural networks for over 70 years. The first attempt to formalize a neural network dates back to 1943, when two American scientists (Warren McCulloch and Walter Pitts) presented an article on the logical calculus of human ideas and neural activity.

However, until recently, says Andrey Kalinin from Mail.Ru Group, the speed of neural networks was too low to be widely used, and therefore such systems were mainly used in developments related to computer vision, and other algorithms were used in other areas. machine learning.

A laborious and time consuming part of the neural network development process is its training. In order for the neural network to correctly solve the assigned tasks, it is required to "run" its operation on tens of millions of input data sets. It is with the emergence of various accelerated learning technologies that Andrei Kalinin and Grigory Bakunov associate the spread of neural networks.

The main thing that has happened now is that various tricks have appeared that make it possible to create neural networks that are much less susceptible to retraining. - Grigory Bakunov, Yandex

“First, there is a large and publicly accessible array of labeled pictures (ImageNet) on which to learn. Secondly, modern video cards make it possible to train and use neural networks hundreds of times faster. Thirdly, ready-made, pre-trained neural networks have appeared that recognize images, on the basis of which you can create your applications without taking a long time to prepare the neural network for work. All this provides a very powerful development of neural networks in the area of pattern recognition, ”says Kalinin.

What are the volumes of the neural networks market

“It's very easy to count. You can take any area that uses low-skilled labor — for example, call center operators — and simply subtract all human resources. I would say that we are talking about a multi-billion dollar market, even within a single country. How many people in the world are engaged in low-skilled jobs is easy to understand. So even in a very abstract way, I think we are talking about the one-hundred-billion-dollar market all over the world, ”says Grigory Bakunov, Director of Technology Dissemination at Yandex.

According to some estimates, more than half of the professions will be automated - this is the maximum volume by which the market for machine learning algorithms (and neural networks in particular) can be increased. - Andrey Kalinin, Mail.Ru Group

“Machine learning algorithms are the next step in the automation of any process, in the development of any software... Therefore, the market at least coincides with the entire software market, but rather surpasses it, because it becomes possible to make new intelligent solutions that are inaccessible to old software, ”continues Andrey Kalinin, head of Mail.ru Search at Mail.Ru Group.

Why Neural Network Developers Build Mass Market Mobile Apps

In the past few months, several high-profile entertainment projects using neural networks have appeared on the market at once - this is also a popular video service that social network Facebook, and Russian applications for image processing (investments from Mail.Ru Group in June) and others.

The abilities of their own neural networks were demonstrated both by Google (AlphaGo technology beat the champion in Go; in March 2016, the corporation auctioned 29 paintings drawn by neural networks, and so on), and Microsoft (the CaptionBot project, which recognizes images in images and automatically generates captions to them ; the WhatDog project, which determines the dog's breed from a photograph; HowOld service, which determines the age of a person in the picture, and so on), and Yandex (in June, the team built a service for recognizing cars in pictures into the Auto.ru application; presented a musical album; in May she created the LikeMo.net project to draw in the style of famous artists).

Such entertainment services are created rather not to solve global problems that neural networks are aimed at, but to demonstrate the capabilities of a neural network and conduct its training.

“Games are a characteristic feature of our behavior as a species. On the one hand, almost all typical scenarios of human behavior can be simulated in game situations, and on the other hand, game creators and, especially, players can get a lot of pleasure from the process. There is also a purely utilitarian aspect. A well-designed game not only brings satisfaction to the players: as they play, they train a neural network algorithm. After all, neural networks are based on learning by example, ”says Vlad Shershulsky from Microsoft.

“First of all, this is done in order to show the capabilities of the technology. There really is no other reason. If we are talking about Prisma, then it is clear why they did it. The guys have built some kind of pipeline that allows them to work with pictures. To demonstrate this, they chose a fairly simple way of styling for themselves. Why not? This is just a demonstration of the algorithms, ”says Grigory Bakunov from Yandex.

Andrey Kalinin from Mail.Ru Group has a different opinion: “Of course, it is impressive from the point of view of the public. On the other hand, I would not say that entertainment products cannot be applied to more useful areas. For example, the task of stylizing images is extremely relevant for a number of industries (design, computer games, animation are just a few examples), and the full use of neural networks can significantly optimize the cost and methods of creating content for them. "

Major players in the neural network market

As Andrei Kalinin notes, by and large, the majority of neural networks present on the market differ little from each other. “The technologies are roughly the same for everyone. But the use of neural networks is a pleasure that not everyone can afford. To independently train a neural network and carry out many experiments on it, you need large training sets and a fleet of machines with expensive video cards. Obviously, such opportunities are available for large companies", He says.

Among the main market players, Kalinin mentions Google and its division Google DeepMind, which created the AlphaGo network, and Google Brain. Microsoft has its own developments in this area - they are being handled by the Microsoft Research laboratory. Neural networks are being created by IBM, Facebook (a division of Facebook AI Research), Baidu (Baidu Institute of Deep Learning) and others. A lot of development is taking place at technical universities around the world.

Grigory Bakunov, Director of Technology Dissemination at Yandex, notes that interesting developments in the field of neural networks are also found among startups. “I would remember, for example, the ClarifAI company. This is a small start-up, once made by people from Google. Now they are perhaps the best in the world at identifying the contents of a picture. " These startups include MSQRD, Prisma, and others.

In Russia, not only startups, but also large technology companies are engaged in developments in the field of neural networks - for example, the Mail.Ru Group holding uses neural networks to process and classify texts in Search, and analyze images. The company is also conducting experimental development related to bots and dialogue systems.

Yandex is also engaged in the creation of its own neural networks: “Basically, such networks are already used in working with images, with sound, but we are exploring their capabilities in other areas as well. Now we are doing a lot of experiments in using neural networks in working with text. " Developments are carried out at universities: Skoltech, MIPT, Moscow State University, Higher School of Economics and others.

To bookmarks

What are neural networks and what tasks they can solve

Neural networks are one of the directions in the development of artificial intelligence systems. The idea is to simulate as closely as possible the functioning of the human nervous system - namely, its ability to learn and correct errors. This is the main feature of any neural network - it is able to independently learn and act on the basis of previous experience, each time making fewer and fewer mistakes.

Vlad Shershulsky, director of Microsoft technology cooperation programs in Russia, notes that neural networks are now widely used: “For example, many large Internet sites use them to make reactions to user behavior more natural and useful to their audience. Neural networks underlie most modern speech recognition and synthesis systems, as well as image recognition and processing. They are used in some navigation systems, be they industrial robots or self-driving cars. Algorithms based on neural networks protect information systems from malicious attacks and help identify illegal content on the network. "

In the short term (5-10 years), Shershulsky believes, neural networks will be used even more widely:

Imagine an agricultural harvester whose actuators are equipped with multiple video cameras. It takes five thousand pictures per minute of each plant in its trajectory strip and, using a neural network, analyzes whether it is a weed, whether it is infected with a disease or pests. And he treats each plant individually. Fantastic? Not quite anymore. And in five years it may become the norm. - Vlad Shershulsky, Microsoft

systems for recognition and classification of objects in images;
voice interaction interfaces for the Internet of Things;
service quality monitoring systems in call centers;
systems for identifying problems (including predicting the time of maintenance), anomalies, cyber-physical threats;
intelligent security and monitoring systems;
bots replacing some of the functions of call center operators;
video analytics systems;
self-learning systems that optimize material flow management or the location of objects (in warehouses, transport);
intelligent, self-learning control systems for production processes and devices (including robotic ones);
the emergence of universal translation systems "on the fly" for conferences and personal use;
the emergence of bots-consultants of technical support or personal assistants, in functions close to a person.

Why neural networks have become so popular right now

The main thing that has happened now is that various tricks have appeared that make it possible to create neural networks that are much less susceptible to retraining. - Grigory Bakunov, Yandex

What are the volumes of the neural networks market

According to some estimates, more than half of the professions will be automated - this is the maximum volume by which the market for machine learning algorithms (and neural networks in particular) can be increased. - Andrey Kalinin, Mail.Ru Group

“Machine learning algorithms are the next step in the automation of any process, in the development of any software. Therefore, the market at least coincides with the entire software market, but rather surpasses it, because it becomes possible to make new intelligent solutions that are inaccessible to old software, ”continues Andrey Kalinin, head of Mail.ru Search at Mail.Ru Group.

Why Neural Network Developers Build Mass Market Mobile Apps

In the past few months, several high-profile entertainment projects using neural networks have appeared on the market at once - this is also a popular video service that is a social Facebook network, and Russian applications for image processing (investments from Mail.Ru Group in June) and others.

Such entertainment services are created rather not to solve global problems that neural networks are aimed at, but to demonstrate the capabilities of a neural network and conduct its training.

Major players in the neural network market

As Andrei Kalinin notes, by and large, the majority of neural networks present on the market differ little from each other. “The technologies are roughly the same for everyone. But the use of neural networks is a pleasure that not everyone can afford. To independently train a neural network and carry out many experiments on it, you need large training sets and a fleet of machines with expensive video cards. Obviously, large companies have such opportunities, ”he says.

Neural networks in simple words. In simple words about the complex: what are neural networks? History of neural networks

What is a neural network?

Neural network training

Direct distribution

Loss function

Back propagation

Checking the operation of the neural network

Final reflections

Notes and bibliography for further reading

Theory

Electromechanical brain with a motor

Practice

Continuation is available only to participants

Option 1. Join the "site" community to read all the materials on the site

What are neural networks and what tasks they can solve

Why neural networks have become so popular right now

What are the volumes of the neural networks market

Why Neural Network Developers Build Mass Market Mobile Apps

Major players in the neural network market

What are neural networks and what tasks they can solve

Why neural networks have become so popular right now

What are the volumes of the neural networks market

Why Neural Network Developers Build Mass Market Mobile Apps

Major players in the neural network market

Password recovery