TLDR: This writeup is for enthusiasts of AI, automated gaming, and application’s of machine (deep) learning in general.

Self-driving vehicles, Industrial scale automations, Operations in unsafe zones like oil rigs, Increased successes in surgical operations … these are part of the many promises of artificial intelligence.

Do you know that the highest grandmaster of the hardest board game in the world GO is an AI? 9 March 2016 – 15 March 2016 was the week AlphaGo broke the internet by beating Lee Sedol an 18-time world champion by 4 – 1. It was a feat considered impossible as the number of possible moves in the game of GO is more than the number of atoms in the universe. And that was the week I was seduced from the medical career I was pursuing and everything became history.

During this past few weeks I have been able to explore another part of the AI promise, automated gaming.
Games are part entertainment and part logical and strategical so training an AI agent through reinforcement learning could help us develop other important agents like the case of an agent trained by google that was able to cut their electricity bill of a server building by more than 70%.

In this post, I will be showing the result of my experiment with an AI agent trained with Pytorch that plays the game snake. Here is the final result.

[img]https:///media/10-24-2020/Jax6Kp.gif[/img]

The training took about 26 hours on an hyper threaded Intel Xeon CPU (good reward aware policy was learned around 5 hours). Although I tried training with a free tesla K80 GPU from google but I was getting almost the same result from my benchmark because most of the work the agent is doing is on the CPU.

And here is the agent in it’s infancy stage about 0 – 2 hours of training. It can be seen performing random actions.

[img]https:///media/10-24-2020/--Uk7D.gif[/img]

The agent is trained using the principles of Q Learning, Bellman Equation and the Markov’s decision process. The above is used to exploit an environment with a number of states by an agent performing an action on the environment. In this experiment, the environment is the snake game.

The environment gives our agent a reward for each action performed. The reward is positive when the agent eats the apple, and negative when the agent crashes the snake on the wall or when the agent crashes the snake on itself. The purpose of this reward is to learn a good policy i.e. eating apples is good while crashing is bad.

For the training loop, the environment provides the agent with a state containing vital information the agent needs to navigate the board. The state includes the direction of the location of the apple (i.e. north, west, south, east), the current direction the snake is travelling, and details of the surrounding position of the snake’s head whether they are occupied or not.

The reward function for the training is the same as the one described above, but I added an extra negative -0.05 reward for each action the agent takes that doesn’t result to the snake eating the apple. This will tell the agent to eat the apple as soon as possible and not to roam around the board.

The loss function is the hubber loss. A hybrid of the Mean square error loss and Mean absolute error loss function. Another viable loss function is the trusted Mean square error loss itself.

The agent itself is nothing but three layers of feed forward neural networks with 64 hidden layers, nonlinear activation functions, and an output of three choices LEFT, RIGHT or keep moving STRAIGHT.

Conclusion

The experiment was a bit of success although the agent didn’t maximize its potential mainly because it can only sense the board only through the region around it’s head and a better option is the snake seeing the whole board as an image through a convolutional neural network. The agent was able to eat about 50 apples at max.

For the next experiment, I will be training an agent to play the old shooting game CONTRA and I am really excited about the training. I will also build upon the lesson learnt by this experiment by giving the agent more awareness of the environment by providing the agent the direct image render of the screen. IF the next experiment is successful, other prospective games for experimenting include Asphalt, Mortal Kombat etc. and also maybe soccer (but definitely the old versions like winning eleven 2008 and will probably need the help of a C++ dev to tune the game to a usable environment)
Adios guys.

Interesting. I am also an AI enthusiast and my interest is on the Microcontroller programming (TinyML). Interestingly I don’t like Pytorch, I use Tensorflow for deep leaning and neural network. I’m currently working a traffic light(Computer Vision) model to be deployed to PyBoard and Arduino.

Predstan:
Interesting. I am also an AI enthusiast and my interest is on the Microcontroller programming (TinyML). Interestingly I don’t like Pytorch, I use Tensorflow for deep leaning and neural network.

I started with Tensorflow actually, but it doesn't feel pythonic to me so I switched to pytorch but with the Keras api now merged with the new Tensorflow, I think I might start to use it again.

I’m currently working a traffic light(Computer Vision) model to be deployed to PyBoard and Arduino.


Sounds interesting, could you explain more about it. And why PyBoard and not raspberry pi.

So how did you start learning this?

Everything here is good beautiful...

Marvel1206:
So how did you start learning this?

Learning Python was the first step. Microsoft did an online lecture about python around that time and I followed on Youtube. Once I was comfortable with Python I got the book Hands on machine learning with Scikit learn and Tensorflow which gave me good introduction to machine learning with each chapter about a different kind of ML algorithm/Model (I read many other books but none is as influential as this book). Subsequently i took a few courses on Udemy about Reinforcement learning that introduce me to the field.

Interesting work you have there I'm also an AI enthusiast. With Tensorflow and keras I have been able to build deep learning models on breast cancer prediction, diabetes prediction and heart disease prediction among other works.

Abcruz:
Interesting work you have there I'm also an AI enthusiast. With Tensorflow and keras I have been able to build deep learning models on breast cancer prediction, diabetes prediction and heart disease prediction among other works.

Sounds cool. These are very important areas that can benefit humans greatly.

Enceladus:


Sounds cool. These are very important areas that can benefit humans greatly.

Thanks for the compliment.
AI is such an interesting field of practical knowledge because it can be applied to so many areas of our lives in automating our repetitive task with a measurable level of accuracy.

Enceladus:


I started with Tensorflow actually, but it doesn't feel pythonic to me so I switched to pytorch but with the Keras api now merged with the new Tensorflow, I think I might start to use it again.

I’m currently working a traffic light(Computer Vision) model to be deployed to PyBoard and Arduino.


Sounds interesting, could you explain more about it. And why PyBoard and not raspberry pi.

It’s a neural Net model for object and signage detection. The agent basically will be able to detect the traffic signs and make decisions from there. Like self-driving car. The signs are just the usual American traffic signs; the stop signs, yield signs, traffic lights. The agent will control the speed of a motor to either stop, slow down or run based on its decision through the model from the camera and sensor.
The choice of Arduino and PyBoard is just because that’s what I have and I have been using for some electronic projects.

Amazing!

I am a Data Science enthusiast and I really like everything you wrote. Before commencing any online course I usually check Quora and Reddit for reviews. I understand that there is no particular online course that will teach you everything about DS.

Could you please enumerate the courses you took while climbing the ladder of Data Science?

I'd really appreciate. Thanks in anticipation.

Predstan:


It’s a neural Net model for object and signage detection. The agent basically will be able to detect the traffic signs and make decisions from there. Like self-driving car. The signs are just the usual American traffic signs; the stop signs, yield signs, traffic lights. The agent will control the speed of a motor to either stop, slow down or run based on its decision through the model from the camera and sensor.
The choice of Arduino and PyBoard is just because that’s what I have and I have been using for some electronic projects.

I get now. Sounds very interesting. I think reinforcement learning could also be applied here.

BelieverDE:
Amazing!

I am a Data Science enthusiast and I really like everything you wrote. Before commencing any online course I usually check Quora and Reddit for reviews. I understand that there is no particular online course that will teach you everything about DS.

Could you please enumerate the courses you took while climbing the ladder of Data Science?

I'd really appreciate. Thanks in anticipation.

Sure. Data science is too broad for a course.

Courses from SuperDataScience on reinforcement learning (2) and one on machine learning helped a lot. The book hands on machine learning too was influential.

Medium.com and towardsdatascience.com blogpost also keeps you updated with new trends.

Enceladus:


Sure. Data science is too broad for a course.

Courses from SuperDataScience on reinforcement learning (2) and one on machine learning helped a lot. The book hands on machine learning too was influential.

Medium.com and towardsdatascience.com blogpost also keeps you updated with new trends.

How did you create those animated photos?

Abcruz:


How did you create those animated photos?

I used obs studio to record the screen, premier pro to edit, then I used an online service to convert the final video to .gif animated image.

Enceladus:


Sure. Data science is too broad for a course.

Courses from SuperDataScience on reinforcement learning (2) and one on machine learning helped a lot. The book hands on machine learning too was influential.

Medium.com and towardsdatascience.com blogpost also keeps you updated with new trends.

The head of SuperDataScience is Kirill Eremenko. @Ejiod suggested his Machine Learning course to me on udemy. I suppose I'll have to follow his learning path, if any.

I never liked the idea of using books to learn Machine Learning, but now that you've recommended this book, I'll definitely download it!

Of late I had interest in machine learning and artificial intelligence ...
I am just learning the basics now .....
If possible I wanna code and build a personal ai assistant like that of the Jarvis and I greet everyone in this thread

Enceladus:


I used obs studio to record the screen, premier pro to edit, then I used an online service to convert the final video to .gif animated image.

Nice! thanks for sharing.

BelieverDE:


The head of SuperDataScience is Kirill Eremenko. @Ejiod suggested his Machine Learning course to me on udemy. I suppose I'll have to follow his learning path, if any.

I never liked the idea of using books to learn Machine Learning, but now that you've recommended this book, I'll definitely download it!

Kirill and his team are quite good. Same here too but that book is quite different.

Kinglyte56:
Of late I had interest in machine learning and artificial intelligence ...
I am just learning the basics now .....
If possible I wanna code and build a personal ai assistant like that of the Jarvis and I greet everyone in this thread

Quite possible. Mark Z built a Jarvis himself. His source code should be somewhere on github, that will give you some idea about ow to start. GL

It would better if we can apply this reinforcement knowledge to games like Fifa and Pes.

Nice work op!!!