Skip to content

Commit 2c0bf70

Browse files
avaaminiavaamini
committed
student 2021 update submission instructions
1 parent 4a79e10 commit 2c0bf70

1 file changed

Lines changed: 32 additions & 42 deletions

File tree

‎lab3/RL.ipynb‎

Lines changed: 32 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
"nbformat_minor": 0,
44
"metadata": {
55
"colab": {
6-
"name": "RL_Solution.ipynb",
6+
"name": "RL.ipynb",
77
"provenance": [],
88
"collapsed_sections": [
99
"jrI6q7RmWQam"
@@ -28,7 +28,7 @@
2828
" Visit MIT Deep Learning</a></td>\n",
2929
" <td align=\"center\"><a target=\"_blank\" href=\"https://colab.research.google.com/github/aamini/introtodeeplearning/blob/master/lab3/RL.ipynb\">\n",
3030
" <img src=\"https://i.ibb.co/2P3SLwK/colab.png\" style=\"padding-bottom:5px;\" />Run in Google Colab</a></td>\n",
31-
" <td align=\"center\"><a target=\"_blank\" href=\"https://github.com/aamini/introtodeeplearning/blob/master/lab3/solutions/RL_Solution.ipynb\">\n",
31+
" <td align=\"center\"><a target=\"_blank\" href=\"https://github.com/aamini/introtodeeplearning/blob/master/lab3/RL.ipynb\">\n",
3232
" <img src=\"https://i.ibb.co/xfJbPmL/github.png\" height=\"70px\" style=\"padding-bottom:5px;\" />View Source on GitHub</a></td>\n",
3333
"</table>\n",
3434
"\n",
@@ -232,10 +232,9 @@
232232
" # First Dense layer\n",
233233
" tf.keras.layers.Dense(units=32, activation='relu'),\n",
234234
"\n",
235-
" # TODO: Define the last Dense layer, which will provide the network's output.\n",
236-
" # Think about the space the agent needs to act in!\n",
237-
" tf.keras.layers.Dense(units=n_actions, activation=None) # TODO\n",
238-
" # [TODO Dense layer to output action probabilities]\n",
235+
" '''TODO: Define the last Dense layer, which will provide the network's output.\n",
236+
" # Think about the space the agent needs to act in!'''\n",
237+
" # [TODO: Dense layer]\n",
239238
" ])\n",
240239
" return model\n",
241240
"\n",
@@ -276,14 +275,13 @@
276275
" # add batch dimension to the observation if only a single example was provided\n",
277276
" observation = np.expand_dims(observation, axis=0) if single else observation\n",
278277
"\n",
279-
" '''TODO: feed the observations through the model to predict the log probabilities of each possible action.'''\n",
280-
" logits = model.predict(observation) # TODO\n",
281-
" # logits = model.predict('''TODO''')\n",
278+
" '''TODO: feed the observations through the model to predict the log \n",
279+
" probabilities of each possible action.'''\n",
280+
" logits = model.predict('''TODO''')\n",
282281
" \n",
283282
" '''TODO: Choose an action from the categorical distribution defined by the log \n",
284283
" probabilities of each possible action.'''\n",
285-
" action = tf.random.categorical(logits, num_samples=1)\n",
286-
" # action = ['''TODO''']\n",
284+
" action = ['''TODO''']\n",
287285
"\n",
288286
" action = action.numpy().flatten()\n",
289287
"\n",
@@ -331,12 +329,12 @@
331329
" # Add observations, actions, rewards to memory\n",
332330
" def add_to_memory(self, new_observation, new_action, new_reward): \n",
333331
" self.observations.append(new_observation)\n",
332+
"\n",
334333
" '''TODO: update the list of actions with new action'''\n",
335-
" self.actions.append(new_action) # TODO\n",
336-
" # ['''TODO''']\n",
334+
" # TODO: your update code here\n",
335+
"\n",
337336
" '''TODO: update the list of rewards with new reward'''\n",
338-
" self.rewards.append(new_reward) # TODO\n",
339-
" # ['''TODO''']\n",
337+
" # TODO: your update code here\n",
340338
"\n",
341339
"# Helper function to combine a list of Memory objects into a single Memory.\n",
342340
"# This will be very useful for batching.\n",
@@ -363,7 +361,7 @@
363361
"source": [
364362
"## 3.3 Reward function\n",
365363
"\n",
366-
"We're almost ready to begin the learning algorithm for our agent! The next step is to compute the rewards of our agent as it acts in the environment. Since we (and the agent) is uncertain about if and when the game or task will end (i.e., when the pole will fall), it is useful to emphasize getting rewards **now** rather than later in the future -- this is the idea of discounting. This is a similar concept to discounting money in the case of interest. Recall from lecture, we use reward discount to give more preference at getting rewards now rather than later in the future. The idea of discounting rewards is similar to discounting money in the case of interest.\n",
364+
"We're almost ready to begin the learning algorithm for our agent! The next step is to compute the rewards of our agent as it acts in the environment. Since we (and the agent) is uncertain about if and when the game or task will end (i.e., when the pole will fall), it is useful to emphasize getting rewards **now** rather than later in the future -- this is the idea of discounting. This is a similar concept to discounting money in the case of interest. ecall from lecture, we use reward discount to give more preference at getting rewards now rather than later in the future. The idea of discounting rewards is similar to discounting money in the case of interest.\n",
367365
"\n",
368366
"To compute the expected cumulative reward, known as the **return**, at a given timestep in a learning episode, we sum the discounted rewards expected at that time step $t$, within a learning episode, and projecting into the future. We define the return (cumulative reward) at a time step $t$, $R_{t}$ as:\n",
369367
"\n",
@@ -439,13 +437,11 @@
439437
"def compute_loss(logits, actions, rewards): \n",
440438
" '''TODO: complete the function call to compute the negative log probabilities'''\n",
441439
" neg_logprob = tf.nn.sparse_softmax_cross_entropy_with_logits(\n",
442-
" logits=logits, labels=actions) # TODO\n",
443-
" # neg_logprob = tf.nn.sparse_softmax_cross_entropy_with_logits(\n",
444-
" # logits='''TODO''', labels='''TODO''')\n",
440+
" logits='''TODO''', labels='''TODO''')\n",
445441
" \n",
446442
" '''TODO: scale the negative log probability by the rewards'''\n",
447-
" loss = tf.reduce_mean( neg_logprob * rewards ) # TODO\n",
448-
" # loss = tf.reduce_mean('''TODO''')\n",
443+
" loss = tf.reduce_mean('''TODO''')\n",
444+
" \n",
449445
" return loss"
450446
],
451447
"execution_count": null,
@@ -474,12 +470,10 @@
474470
" logits = model(observations)\n",
475471
"\n",
476472
" '''TODO: call the compute_loss function to compute the loss'''\n",
477-
" loss = compute_loss(logits, actions, discounted_rewards) # TODO\n",
478-
" # loss = compute_loss('''TODO''', '''TODO''', '''TODO''')\n",
473+
" loss = compute_loss('''TODO''', '''TODO''', '''TODO''')\n",
479474
"\n",
480475
" '''TODO: run backpropagation to minimize the loss using the tape.gradient method'''\n",
481-
" grads = tape.gradient(loss, model.trainable_variables) # TODO\n",
482-
" # grads = tape.gradient('''TODO''', model.trainable_variables)\n",
476+
" grads = tape.gradient('''TODO''', model.trainable_variables)\n",
483477
" optimizer.apply_gradients(zip(grads, model.trainable_variables))\n"
484478
],
485479
"execution_count": null,
@@ -696,23 +690,20 @@
696690
" # First, 32 5x5 filters and 2x2 stride\n",
697691
" Conv2D(filters=32, kernel_size=5, strides=2),\n",
698692
"\n",
699-
" # TODO: define convolutional layers with 48 5x5 filters and 2x2 stride\n",
700-
" Conv2D(filters=48, kernel_size=5, strides=2), # TODO\n",
701-
" # Conv2D('''TODO'''),\n",
693+
" '''TODO: define convolutional layers with 48 5x5 filters and 2x2 stride'''\n",
694+
" # [your Conv layer here]\n",
702695
"\n",
703-
" # TODO: define two convolutional layers with 64 3x3 filters and 2x2 stride\n",
704-
" Conv2D(filters=64, kernel_size=3, strides=2), # TODO\n",
705-
" Conv2D(filters=64, kernel_size=3, strides=2),\n",
706-
" # Conv2D('''TODO'''),\n",
696+
" '''TODO: define two convolutional layers with 64 3x3 filters and 2x2 stride'''\n",
697+
" # [your Conv layers here]\n",
707698
"\n",
708699
" Flatten(),\n",
709700
" \n",
710701
" # Fully connected layer and output\n",
711702
" Dense(units=128, activation='relu'),\n",
712-
" # TODO: define the output dimension of the last Dense layer. \n",
713-
" # Pay attention to the space the agent needs to act in\n",
714-
" Dense(units=n_actions, activation=None) # TODO\n",
715-
" # Dense('''TODO''')\n",
703+
"\n",
704+
" '''TODO: define the output dimension of the last Dense layer. \n",
705+
" Pay attention to the space the agent needs to act in'''\n",
706+
" # [TODO: your Dense layer here]\n",
716707
" \n",
717708
" ])\n",
718709
" return model\n",
@@ -904,17 +895,16 @@
904895
"\n",
905896
" '''TODO: determine the observation change.\n",
906897
" Hint: this is the difference between the past two frames'''\n",
907-
" frame_diff = mdl.lab3.pong_change(previous_frame, current_frame) # TODO\n",
908-
" # frame_diff = # TODO\n",
898+
" frame_diff = # TODO\n",
909899
"\n",
910900
" '''TODO: choose an action for the pong model, using the frame difference, and evaluate'''\n",
911-
" action = choose_action(model, frame_diff) # TODO \n",
912-
" # action = # TODO\n",
901+
" action = # TODO\n",
902+
"\n",
913903
" # Take the chosen action\n",
914904
" next_observation, reward, done, info = env.step(action)\n",
915905
"\n",
916906
" '''TODO: save the observed frame difference, the action that was taken, and the resulting reward!'''\n",
917-
" memory.add_to_memory(frame_diff, action, reward) # TODO\n",
907+
" memory.add_to_memory('''TODO''', '''TODO''', '''TODO''')\n",
918908
"\n",
919909
" previous_frame = current_frame\n",
920910
" \n",
@@ -1104,7 +1094,7 @@
11041094
"* How does the complexity of Pong relative to CartPole alter the rate at which the agent learns and its performance? \n",
11051095
"* What are some things you could change about the agent or the learning process to potentially improve performance?\n",
11061096
"\n",
1107-
"Try to optimize your **Pong** model and algorithm to achieve improved performance. **MIT students and affiliates will be eligible for prizes during the IAP offering.** To enter the competition, please [email us](mailto:introtodeeplearning-staff@mit.edu) with your name and the following:\n",
1097+
"Try to optimize your **Pong** model and algorithm to achieve improved performance. **MIT students and affiliates will be eligible for prizes during the IAP offering.** To enter the competition, MIT students and affiliates should upload the following to the course Canvas:\n",
11081098
"\n",
11091099
"* Jupyter notebook with the code you used to generate your results, **with the Pong training executed**;\n",
11101100
"* saved video of your Pong agent competing;\n",

0 commit comments

Comments
 (0)