Many bloggers finds this difficult, optimising your blog's robot.txt isn't as much as fun as it may seem.

Robot.txt is very important when it comes to the segment of SEO, it helps the search engines to locate your blog and understand what it's all about.

What is robot.txt ?

Robot.txt is a text file that a blog or website owners use or create to help tell the search engine bots, how to operate (crawl, index and submit) pages on their blog/website.

Robot.txt are usually stored in the root directory (main folder) of a blog/website.

Robot.txt file are always in the format below:

1. User-agent : [user-agent name]
Disallow:[URL string not to be crawled]
2. User-agent:[user-agent name]
Disallow:[URL string to be crawled]
Sitemap:url of your xml sitemap

You are free to have multiple lines of instructions to allow or disallow specific URLs and also add multiple sitemap.

I will advice you to always define your instruction because if you don't disallow a URL then search engine bots will then assume you allowed them to crawl it.

Read more>>> https://www.techlypost.com/2019/07/03/how-to-create-a-fully-optimized-wordpress-robot-txt-file