This number is decreased to roughly 400 sensible plans (e.g. looking 1 second into the future), this leads to roughly 3.5 billion possible chains of actions. pressing W+A followed by two times W converts game state vector X into Y).įor 9 possible actions (W, W+A, W+D, S, S+A, S+D, A, D, none) and 10 timesteps (i.e. frame/screenshot at 10fps) is embedded via convolutions and fully connected layers to a vector.įrom that vector, future embeddings (the successors) are predicted.Įach such prediction (for each timestep) is dependent on a chosen action (e.g. However, that component is only trained for some batches, so it is skipped here.ĭuring application, each game state (i.e. That way the LSTMs only have to predict the changes (of the embeddings) that were caused by the actions.Īside from these, there is also an autoencoder component applied to the embeddings of Embedder 2. This module uses an addition to the previously generated embedding (i.e. These future embeddings can then be used to predict future direct and indirect rewards (using the two previous models).
for (s, a, r), (s', a', r') it predicts r when being in s'. Direct Reward: A model that predicts the direct reward, i.e.The steering wheel position is approximated using a separate CNN.) Not merging this component with Embedder 1 allows to theoretically keep the weights from pretraining fixed. The current gear state and the speed is read out from the route advisor. (These are: (1) Previous actions, (2) whether the gear is in reverse mode, (3) steering wheel position, (4) previous and current speeds. Embedder 2: Takes the results of Embedder 1 and converts them into a vector.the mirrors are always at roughly the same positions, so it is logical to detect them partially by their position.) Instance Normalization was used, because Batch Normalization regularly broke, resulting in zero-only vectors during test/eval (seemed like a bug in the framework, would usually go away when using batch sizes >= 2 or staying in training mode). The two gradient inputs (see image) are just gradients from 1 to 0 which are supposed to give positional information. Embedder 1: A CNN that is pretrained in semi-supervised fashion.positions of cars and lanes in example images)Īs well as some automatically generated ones (e.g. That training uses some manually created annotations (e.g. To make training faster, a semi-supervised pretraining is applied to the first stage of the whole model (similar to Loss is its own Reward: Self-Supervision for Reinforcement Learning, though here only applied once at the start). (While that paper uses only predictions for the next timestep, here predictions for the next T timesteps are generated via an LSTM.) The basic training method follows the standard reinforcement learning approach from the original Atari paper.Īdditionally, a separation of Q-values in V (value) and A (advantage) - as described in Dueling Network Architectures for Deep Reinforcement Learning - is used.įurther, the model tries to predict future states and rewards, similar to the description in Deep Successor Reinforcement Learning. it can not directly set the steering wheel angle). It is trained (mostly) via reinforcement learning and only has access to the buttons W, A, S and D The resulting AI will automatically steer, accelerate and brake.
Download euro truck simulator 2 ppsspp code#
This repository contains code to train and run a self-driving truck in Euro Truck Simulator 2.
0 kommentar(er)
