concept previous chapter in category reinforcement learning

This is an excerpt from Manning's book Grokking Deep Reinforcement Learning.
Tweetable Tweet
Do some work on your own and share your findings
Every chapter, I’ll give some ideas on how to take what you have learned to the next level. If you’d like, share your results with the rest of the world and make sure to check out what others have done, too. It’s a win-win situation, and hopefully, you take advantage of it.
For this chapter, here are some ideas to explore:
· #gdrl-ch03-tf01: Many of the grid-world environments out there have MDPs available that can be solved with the Policy Iteration, and Value Iteration functions presented in this chapter. Surprised? Simply use `env.unwrapped.P` and pass that variable to the functions in this chapter. More explicitly, do that for a few environments that we did not used in this chapter, environments created by others, or perhaps the environment you created yourself in the last chapter.
· #gdrl-ch03-tf02: The discount factor, gamma, was introduced in the previous chapter as part of the MDP definition. However, we didn’t go into the details of this, very crucial variable. How about you run Policy Iteration and Value Iteration with several different values of gamma, capture the sum of rewards agents obtain with each, and the optimal policies. How do these compare? Can you find anything interesting that can help others better understand the role of the discount factor?
· #gdrl-ch03-tf03: Policy Iteration and Value Iteration both do the same thing, they take an MDP definition and solve for the optimal value function and optimal policies. However, an interesting question is how do these compare? Can you think of an MDP that is challenging for Policy Iteration and easy for Value Iteration to solve or the other way around? Create such an environment as a Python package, and share it with the world. What did you find that others may want to know? How do VI and PI compare?
· #gdrl-ch03-tf04: In every chapter, I’m using the final hashtag as a ‘catch-all’ hashtag. Feel free to use this one to discuss anything else that you worked on relevant to this chapter. There is no more exciting homework that the one you create for yourself. Make sure to share what you set yourself to investigate and your results.
· Write a tweet with your findings, tag me @mimoralea (I’ll retweet,) and use the particular hashtag from the list above to help folks interested find your results. There are no right or wrong results; you share your findings and check others’ findings. Take advantage of this to socialize, contribute, and get yourself out there! We’re waiting for you!
Here is a tweet example:
“Hey, @mimoralea. I created a blog post with a list of resources to study deep reinforcement learning. Check it out at <link>. #gdrl-ch01-tf01”
I’ll make sure to retweet and help others find your work
After gathering data, there are multiple things an agent can do with this data. Some agents, for instance, learn to predict expected returns or value functions. In the previous chapter, you learned about many different ways of doing so, from using Monte-Carlo to TD targets, from every-visit to first-visit MC targets, from n-step to λ-return targets. There are many different ways of calculating targets that can be used for estimating value functions.