Lab: Passive Reinforcement Learning

CSC261 - Artificial Intelligence - Weinman

Summary:

We prepare to cut to the chase by learning policies as well as how to learn world models when they are unknown.

Preparation

1. Move to your directory for materials from the prior lab.

   $ cd somewhere/mdp
2. Copy starter materials for this lab (note the dot at the end of the command!). The -i flag asks to overwrite any files that may exist. If you made any changes to your Makefile you may want to save them, otherwise it is fine to overwrite the prior version. Object (.o) files will be copied as well. If you are unsure whether yours are functional, you should overwrite your own.

   $ cp -i ~weinman/courses/CSC261/code/adp/* .

Exercises

A: Policy Iteration

1. Open policy_iteration.c and locate the main function. Read through it and make sure you understand what it does.
2. Verify that you can build and link the default (do-nothing) policy iteration program

   $ make mdp policy
3. Verify that the policy iteration program runs.

   $ ./policy_iteration 0 0 4x3.mdp

   It should print a random policy for 12 states consisting of the four movement actions 0-3.

B: Adaptive Dynamic Programming

1. Open adp.c and locate the main function. Read through it and make sure you understand what it does.
2. Verify that you can build and link the default (random action) passive reinforcement learning agent.

   $ make adp

   You can ignore the warning about line 14; gamma(3) is a deprecated function and we're using gamma as a variable name. gcc wants to make sure we know about that.
3. Verify you can run the ADP agent using a policy file you copied (each entry corresponds to an action for the state).

   $ ./adp 0 0 4x3.mdp 10 < 4x3.policy

   It should print zero-valued utilities for all 12 states.

Lab Assignment