A Bayesian Viewpoint on Q-Studying

Unique work by Dabney et al. means that the mind represents reward predictions as likelihood distributions

Experiments were completed on mice using single-unit recordings from the ventral tegmental condominium.
.
This contrasts towards the broadly adopted capacity in reinforcement discovering out (RL) of modelling single scalar
portions (expected values).
In level of truth, by utilizing distributions we’re ready to quantify uncertainty within the decision-making task.
Uncertainty is terribly crucial in domains the build making a mistake would possibly per chance per chance per chance discontinuance up within the shortcoming of skill to win better

Examples of such domains consist of self sustaining automobiles, healthcare, and the monetary markets.
. Research in risk-unsleeping reinforcement discovering out has emerged to contend with such issues
.
Nevertheless, one other crucial utility of uncertainty, which we focal level on listed right here, is efficient exploration
of the mutter-action condominium.

Introduction

The cause of this text is to obviously uncover Q-Studying from the perspective of a Bayesian.
As such, we employ a tiny grid world and a straightforward extension of tabular Q-Studying for instance the fundamentals.
Namely, we uncover methods to prolong the deterministic Q-Studying algorithm to model
the variance of Q-values with Bayes’ rule. We focal level on a sub-class of issues the build it is sensible to make a selection that Q-values
tend to be distributed
and safe insights when this assumption holds genuine. Lastly, we uncover that making employ of Bayes’ rule to change
Q-values comes with a anguish: it is at risk of early exploitation of suboptimal insurance policies.

This article is basically per the seminal work from Dearden et al. .
Namely, we broaden on the perception that Q-values tend to be distributed and overview totally different Bayesian exploration
insurance policies. One key distinction is that we model $$mu$$ and $$sigma^2$$, whereas the
authors of the distinctive Bayesian Q-Studying paper model a distribution over these parameters. This allows them to quantify
uncertainty of their parameters to boot to the expected return – we easiest focal level on the latter.

Since we’re modelling $$mu$$ and $$sigma^2$$, we initiate by evaluating the prerequisites below which it is acceptable
to make a selection Q-values tend to be distributed.

When Are Q-Values In overall Dispensed?

The readers who’re unsleeping of Q-Studying can skip over the collapsible field below.

We can employ the Central Limit Theorem (CLT) as the root to word when Q-values tend to be
distributed. Since Q-values are sample sums, then they should always stare increasingly more customarily distributed as the sample dimension
increases .
Nevertheless, the first nuance that we are able to level to is that rewards should always be sampled from distributions with finite variance.
Thus, if rewards are sampled distributions such as Cauchy or L&eacutevy, then we just isn’t going to make a selection Q-values tend to be distributed.

Otherwise, Q-values are approximately customarily distributed when the model of efficient timesteps
$$widetilde{N}$$ is enormous

We are able to mediate efficient timesteps as the model of elephantine samples.
.
This metric is created from three components:

• $$N$$ – Preference of timesteps: As $$N$$ increases, so does $$widetilde{N}$$.
• $$xi$$ – Sparsity: We clarify sparsity as the model of timesteps,
  on moderate, a reward of zero is deterministically received in between receiving non-zero rewards
  
  In the Google Colab pocket book, we ran simulations to uncover that $$xi$$ reduces the efficient model of timesteps by $$frac{1}{xi + 1}$$:
  
  Experiment in a Notebook
  
  .
  When sparsity is uncover, we lose samples (since they’re continually zero).

  As a result of this truth, as $$xi$$ increases, $$widetilde{N}$$ decreases.

• $$gamma$$ – Discount Element:
  As $$gamma$$ will get smaller, the agent areas more weight on immediate rewards relative to a ways away ones, which blueprint
  that we just isn’t going to condominium a ways away rewards as elephantine samples. As a result of this truth, as $$gamma$$ increases, so does $$widetilde{N}$$.

Discount Element and Combination Distributions

We can clarify the total return as the sum of discounted future
rewards, the build the nick worth factor $$gamma$$ can rob on any tag between $$0$$ (myopic) and $$1$$ (a ways-sighted).
It helps to mediate the ensuing distribution $$G_t$$ as a weighted mixture distribution.


G_t=r_{t+1} + gamma r_{t+2} + gamma^2 r_{t+3} + … + gamma^{N-1} r_{t+N}


After we station $$gamma lt 1$$, the mixture weights for the underlying distributions commerce from equal weight
to time-weighted, the build immediate timesteps get a elevated weight. When $$gamma=0$$, then right here is
such as sampling from easiest one timestep and CLT would not contend with. Use the slider
to stare the attain $$gamma$$ has on the mixture weights, and within the damage the mixture distribution.

$$$$
$$$$

$$$$
$$$$

$$$$
$$$$

$$gamma$$=

We mix the components above to formally clarify the model of efficient timesteps:

widetilde{N}=frac{1}{xi + 1}sum_{i=0}^{N-1}gamma^{i}

Below we visually uncover how every factor impacts the normality of Q-values

We scale the Q-values by $$widetilde{N}$$ on account of in any other case the distribution of Q-values
strikes farther and farther to the factual as the model of efficient timesteps increases, which distorts the visible.
:

Skew-Customary

Bernoulli

There is a caveat within the visible analysis above for environments which get a terminal mutter. Because the agent strikes closer
to the terminal mutter, then $$N$$ will progressively win smaller and Q-values will stare less customarily distributed.
On the opposite hand, it is sensible to make a selection that Q-values are approximately customarily distributed for many
states in dense reward environments if we employ a massive $$gamma$$.

Bayesian Interpretation

We preface this section by noting that the following interpretations are
easiest theoretically justified when we pick Q-values tend to be distributed. We initiate by defining the final
change rule using Bayes’ Theorem:

text{posterior} propto text{likelihood} occasions text{prior}

When using Gaussians, we get an analytical solution for the posterior

A Gaussian is conjugate to itself, which simplifies the Bayesian updating
task vastly; as an different of computing integrals for the posterior, we get closed-carry out expressions
.
:

mu =frac{sigma^2_1}{sigma^2_1 + sigma^2_2}mu_2 + frac{sigma^2_2}{sigma^2_1 + sigma^2_2}mu_1


sigma^2=frac{sigma^2_1sigma^2_2}{sigma^2_1 + sigma^2_2}

By attempting at a color-coded comparison, we can leer that deterministic Q-Studying is such as updating the mean
using Bayes’ rule:

initiate{aligned}
&color{green}mu&
&color{unlit}=&
&color{orange}frac{sigma^2_1}{sigma^2_1 + sigma^2_2}&
&color{red}mu_2&
&color{unlit}+&
&color{red}frac{sigma^2_2}{sigma^2_1 + sigma^2_2}&
&color{blue}mu_1&

\ \

&color{green}q(s,a)&
&color{unlit}=&
&color{orange}alpha&
&color{red}(r_{t+1} + gamma q(s^prime,a^prime))&
&color{unlit}+&
&color{red}(1 – alpha)&
&color{blue}q(s,a)&
discontinuance{aligned}

What does this uncover us about the deterministic implementation of Q-Studying, the build $$alpha$$ is a hyperparameter?
Since we do not model the variance of Q-values in deterministic Q-Studying, $$alpha$$ does not explicitly depend
on the easy job in Q-values. As an different, we can clarify $$alpha$$ as being the ratio of how implicitly certain
the agent is in its prior, $$q(s,a)$$, relative to the likelihood, $$r + gamma q(s^prime,a^prime)$$

Our dimension is $$r + gamma q(s^prime,a^prime)$$ since $$r$$ is recordsdata given to us straight from the
atmosphere. We insist our likelihood as the distribution over this dimension:
$$mathcal{N}left(mu_{r + gamma q(s^prime,a^prime)}, sigma^2_{r + gamma q(s^prime,a^prime)}factual)$$.
.
For deterministic Q-Studying, this ratio is in overall constant and the uncertainty in $$q(s,a)$$ does not commerce
as we win more recordsdata.

What happens “below the hood” if we contend with $$alpha$$ constant?
Fair sooner than the posterior from the outdated
timestep turns into the prior for the latest timestep, we prolong the variance
by $$sigma^2_{text{prior}_{(t-1)}} alpha$$

When $$alpha$$ is held constant, the variance of the prior implicitly undergoes the following transformation:
$$sigma^2_{text{prior}_{(t)}}=sigma^2_{text{posterior}_{(t-1)}} + sigma^2_{text{prior}_{(t-1)}} alpha$$.

Derivation

Allow us to first mutter that $$alpha=frac{sigma^2_text{prior}}{sigma^2_text{prior} + sigma^2_text{likelihood}}$$, which would possibly per chance per chance per chance even be deduced
from the color-coded comparison within the main text.

Given the change rule

$$
sigma^2_{text{posterior}_{(t)}}=frac{sigma^2_{text{prior}_{(t)}} occasions sigma^2_{text{likelihood}_{(t)}}}{sigma^2_{text{prior}_{(t)}} + sigma^2_{text{likelihood}_{(t)}}}
$$, all americans is conscious of that $$sigma^2_{text{posterior}_{(t)}} lt sigma^2_{text{prior}_{(t)}}$$

We also know that the change rule works in such a approach that $$sigma^2_{text{prior}_{(t)}}=sigma^2_{text{posterior}_{(t-1)}}$$

As a result of this truth, we can mutter that $$sigma^2_{text{prior}_{(t)}} lt sigma^2_{text{prior}_{(t-1)}}$$ if we pick
$$sigma^2_text{likelihood}$$ does not commerce over time. This implies that $$alpha_{(t)} neq alpha_{(t-1)}$$

In declare to plan $$alpha_{(t)}=alpha_{(t-1)}$$, we get to prolong $$sigma^2_{text{posterior}_{(t-1)}}$$
sooner than it turns into $$sigma^2_{text{prior}_{(t)}}$$. We solve for this quantity below:

$$
initiate{aligned}
sigma^2_{text{posterior}_{(t-1)}} + X &=sigma^2_{text{prior}_{(t-1)}} \
frac{sigma^2_{text{prior}_{(t-1)}} occasions sigma^2_text{likelihood}}{sigma^2_{text{prior}_{(t-1)}} + sigma^2_{likelihood}} + X &=sigma^2_{text{prior}_{(t-1)}} \
X &=sigma^2_{text{prior}_{(t-1)}} left(1 – frac{sigma^2_text{likelihood}}{sigma^2_{text{prior}_{(t-1)}} + sigma^2_text{likelihood}} factual) \
X &=sigma^2_{text{prior}_{(t-1)}} alpha
discontinuance{aligned}
$$

.

This retains the uncertainty ratio between the likelihood and the prior constant

An different interpretation is that the variance for the prior and likelihood are both lowering in such a approach
that retains the ratio between them constant. Nevertheless, we attain not own it is sensible to make a selection
that the variance of the sampled reward would repeatedly decrease as the agent turns into more certain in its prior.
.

Below we visualize this interpretation by comparing the “recurring” Bayesian change to the constant $$alpha$$ change:

Starting $$alpha$$=

Now that all americans is conscious of what happens below the hood when we contend with $$alpha$$ constant, it is worth noting that not all americans
holds it constant.
In word, researchers also decay $$alpha$$ for the agent to count less on novel recordsdata (implicitly turning into more
certain) for every subsequent timestep .
Even supposing deterministic Q-Studying largely depends on heuristics to manufacture a decay agenda, Bayesian Q-Studying has
it built in:

alpha=frac{sigma^2_{q(s,a)}}{sigma^2_{q(s,a)} + sigma^2_{r + gamma q(s^prime,a^prime)}}

As our agent updates its perception about the sector this can naturally fabricate
a decay agenda that corresponds to how certain it is in its prior. As uncertainty decreases, so does the discovering out rate.
Imprint that the discovering out rate is bespoke for every mutter-action pair on account of it is possible to
was more confident specifically mutter-action pairs sooner than others

Some causes consist of visiting these mutter-action pairs more ceaselessly than others, or merely on account of they’re inherently less noisy.
.

Exploration

Exploration Insurance policies

There are lots of methods we can employ a distribution over Q-values to detect as a change to the $$varepsilon$$-greedy
capacity. Below we outline a number of, and overview every within the last section of this text.

• Epsilon-Greedy: We station $$varepsilon$$ as a hyperparameter. It represents the likelihood of deciding on a
  random action (i.e. deviating from deciding on the action with the most sensible possible Q-tag).
• Bayes-UCB:
  We contend with the actions with the largest factual tails, using some
  self belief interval (we employ 95% in our analysis)
  
  Since we model Q-tag distributions as Gaussians, to calculate the 95% self belief interval we employ
  $$mu_{q(s,a)} + sigma_{q(s,a)} occasions 2$$.
  
  . Truly, we’re deciding on the action that has
  the most sensible possible Q-tag
  
  There will be a deterministic implementation of Higher Self belief Lag, the build the bonus is a function of the
  model of timesteps which get passed to boot to the model of occasions the agent has visited a particular mutter-action
  pair
  .
  .
• Q-Payment Sampling: We sample from the Q-tag distributions and resolve the action
  with the largest sampled Q-tag. This carry out of exploration is identified as Q-tag sampling within the case of Q-Studying
  
  and Thompson sampling within the final case .
• Myopic-VPI: We quantify a myopic leer of coverage enchancment with tag of most sensible possible recordsdata (VPI)
  
  $$text{VPI}(s,a)=int^infty_{-infty}text{Ticket}_{s,a}(x)Pr(mu_{s,a}=x)dx$$, which would possibly per chance per chance per chance even be intuitively
  described as the expected enchancment over the latest most productive action.
  . It is “myopic” on account of it easiest considers the construction for the latest timestep.
  We contend with the action that maximizes $$mu_{s,a} + text{VPI}(s,a)$$.

Below we visualize the lots of exploration insurance policies in action:

By interacting with the visible above, one would possibly per chance per chance per chance wonder if we can infer what the “exploration parameter” is for the
other stochastic coverage, Q-tag sampling, which does not explicitly clarify $$varepsilon$$.
We detect this inquire within the following section.

Implicit $$varepsilon$$

In distinction to deterministic Q-Studying, the build we explicitly clarify $$varepsilon$$ as the exploration hyperparameter,
when we employ Q-tag sampling there is an implicit epsilon $$hat{varepsilon}$$.
Before defining $$hat{varepsilon}$$, we are able to win some
notation out of the approach. Let’s clarify two likelihood distributions, $$x_1 sim mathcal{N}(mu_1, sigma^2_1)$$ and
$$x_2 sim mathcal{N}(mu_2, sigma^2_2)$$. To calculate the likelihood that we sample a tag $$x_1 gt x_2$$, we
can employ the following equation, the build $$Phi$$ represents the cumulative distribution function
:

initiate{aligned}
&mu =mu_1 – mu_2 \
&sigma=sqrt{sigma^2_1 + sigma^2_2} \
&Pr(x_1 gt x_2)=1 – Phileft(frac{-mu}{sigma}factual)
discontinuance{aligned}

With this equation, we can now calculate the likelihood of sampling
the next Q-tag for a reference action $$hat{a}$$ relative to 1 other action.
If we attain this for every action that an agent can plan (with the exception of the reference action)
and calculate the joint likelihood, then
we win the likelihood that the sampled Q-tag for $$hat{a}$$ is higher than all other actions

In a given mutter, the Q-tag for one action should always be self sustaining of the opposite Q-values in that mutter.
It is a ways on account of you potentially would possibly per chance per chance per chance easiest rob one action at a time, and we customarily word
Q-discovering out to MDPs, the build the Markov property holds (i.e. history does not matter).
Thus, to calculate the joint likelihood, it is merely a multiplication of the marginal potentialities.
:

bar{P}_{hat{a}}=prod_{a}^{mathcal{A}}Pr(x_{hat{a}} gt x_a), quad text{for} ,, a neq hat{a}

We then procure the action with the largest $$bar{P}_{a}$$ on account of that is the action that we’d contend with if we weren’t
exploring

Since we’re using neatly-liked distributions, $$text{arg}max{bar{P}_{a}}$$ happens to correspond to the Q-tag with the largest mean.
.

a_{max}=text{arg}max{bar{P}_{a}}, quad forall ,, a in mathcal{A}

Then, if we sum up the potentialities of sampling the largest Q-tag, for all actions instead of the exploitation action,
then we win the likelihood that we are able to detect:

hat{varepsilon}=frac{1}{C}sum_{a}^{mathcal{A}}bar{P}_{a}, quad text{for} ,, a neq a_{max}

Where $$C$$ is the normalizing constant (sum of all $$bar{P}_{a}$$)

Making employ of Bayes’ Rule

We can now set aside the speculation into word! By inspecting the discovering out task, we can leer that there is
a key anguish in making employ of Bayes’ rule to Q-Studying.
Namely, we focal level on diverging Q-tag distributions, which can cause agents to was confident in suboptimal insurance policies.

Game Setup

As researchers within the monetary markets, we designed the atmosphere after a sub-class of issues that share identical
characteristics. These issues are characterized by
environments that give a reward at every timestep, the build the mean and variance of the rewards depends on the mutter
that the agent is in

That is such as the return received for any change/investment, the build the expected return and volatility
depends on the market regime.
. To achieve this, we employ a modified model of the Cliff World atmosphere
:

Examining the Realized Distributions

Below we uncover the Q-tag distributions realized by our agent for every mutter-action pair.
We employ an arrow to spotlight the realized coverage.

By hovering our mouse over the path, we word that the agent does not learn the “genuine” Q-tag distribution
for all mutter-action pairs. Only the pairs that recordsdata it by the path appear like genuine.
This happens for the reason that agent stops exploring once it thinks it has found the optimal coverage

Despite the indisputable truth that agents attain not learn the real Q-values, they are able to restful learn the optimal coverage if
they learn the relative tag of actions in a mutter.
The relative tag of actions is ceaselessly known as the advantage .
.
Below we leer that discovering out plateaus once exploration stops:

One factor that continually happens when using Bayes’ rule (after enough episodes) is that the agent finds its solution to the arrangement with out falling
off the cliff. Nevertheless, it does not continually procure the optimal path.
Below we color states per how ceaselessly they’re visited right by coaching – darker shades insist elevated visitation rates.
We leer that mutter visitations outdoors of the arrangement trajectory are virtually non-existent for the reason that agent turns into anchored
to the path that leads it to the arrangement.

Let’s dig into the staunch mutter that is accountable for the agent either discovering the optimal coverage or not. We can name this
the “severe mutter” and highlight it with a valuable particular person within the figure above.
When examining what happens right by coaching, we leer that the cause on the aid of
the anguish is that the Q-tag distributions diverge. We can employ Q-Payment sampling for the following analysis.
For the reason that agent explores by technique of Q-Payment sampling, once the
density of the joint distribution approaches 0, the agent will continually sample a elevated
Q-tag from one distribution relative to the opposite. Thus, it can per chance not ever rob the action from the Q-tag distribution
with a decrease mean.
Let’s stare at a visual illustration of this idea:

$$mu$$=

We can insist the distribution that we toggle as $$x_1$$ and the static distribution as $$x_2$$.
The main bar represents $$Pr(x_1 gt x_2)$$ and the 2d bar represents $$hat{varepsilon}$$. When visualized,
it is glaring that $$hat{varepsilon}$$ is genuine the overlapping condominium below the two distributions

The agent easiest explores when there is a likelihood of sampling a elevated tag from either distribution, which is easiest the
case when there is a tight quantity of overlap between the distributions.
.
Allow us to now peep the discovering out progress on the severe mutter:

Optimum
Suboptimal

Episode=

Whether the agent finds the optimal coverage or the suboptimal coverage, we own about that exploration stops as soon as the
Q-values diverge a ways enough. This will be considered as the coaching progress
flat traces for the action with a decrease mean.
As a result of this truth, a risk in making employ of Bayes’ rule to Q-discovering out is that the agent does not
detect the optimal path sooner than the distributions diverge.

Impact of Protection on Perception

We can employ the agent that realized the suboptimal coverage for a quickly experiment. At the severe mutter, all americans is conscious of that
the Q-tag distributions diverge in such a approach that the agent would possibly per chance per chance per chance not ever sample a Q-tag for $$text{Down}$$ that is
elevated than $$text{Fair}$$, and
thus it can per chance not ever switch down. Nevertheless, what if we force the agent to switch down and leer what it does from that level on?
Strive it out below:

By forcing the agent to switch down, we word that there are occasions when it goes all over the hazard zone to the arrangement.
We can uncover what’s happening with an analogy:

Imagine entering into a car accident at intersection X whenever you are discovering out to force.
That you just would possibly per chance per chance per chance partner that intersection with a unfriendly result (low Q-tag) and rob a detour going ahead.
Time past regulation that chances are you’ll per chance win better at driving (coverage enchancment) and whenever you accidentally discontinuance up at intersection X,
that chances are you’ll per chance attain genuine beautiful. The anguish is that you by no blueprint revisit intersection X on account of it is laborious to decouple the unfriendly
memory from the truth that you were a unfriendly driver on the time.

This anguish is highlighted in one of David Silver’s lectures, the build he states that even supposing Thompson
sampling (Q-tag sampling in our case) is titanic for bandit issues, it does not contend with sequential recordsdata successfully in
the elephantine MDP case . It
easiest evaluates the Q-tag distribution using the latest coverage and does not rob into tale the truth that the coverage
can toughen. We can leer the final consequence of this within the following section.

Dialogue

To deem the exploration insurance policies beforehand talked about, we overview the cumulative remorse for every capacity
in our recreation atmosphere.
Feel sorry about is the distinction between the return received from following the optimal coverage in comparison to the staunch coverage
that the agent followed

If the agent follows the optimal coverage, then this can get a remorse of $$0$$.
.

Median
Median with Vary

Even supposing experiments in our recreation atmosphere suggest that Bayesian exploration insurance policies detect more efficiently
on moderate, there appears to be like to be a principal broader fluctuate of outcomes.
Moreover, given our analysis on diverging Q-tag distributions, all americans is conscious of that there are occasions when Bayesian agents can
was anchored to suboptimal insurance policies.
When this happens, the cumulative remorse appears to be like like a diagonal line $$nearrow$$,
which would possibly per chance per chance per chance even be considered protruding from the fluctuate of outcomes.

In conclusion, whereas Bayesian Q-Studying sounds titanic theoretically, it can per chance even be nerve-racking to have a study in staunch
environments. This anguish easiest will get harder as we switch to more sensible environments with higher
mutter-action areas. On the opposite hand, we mediate modelling distributions over tag functions is a thrilling condominium of
overview and has the skill to attain mutter of the art (SOTA) results, as demonstrated in some associated works on distributional
RL.

Related Work

Even supposing we focal level on modelling Q-tag distributions in a tabular setting,
lots of intriguing overview has long gone into using function approximations to model these distributions
. More recently, a sequence of
distributional RL papers using deep neural networks get emerged reaching SOTA leads to Atari-57.
The main of such papers presented the divulge DQN (C51) structure as a solution to discretize Q-values into bins and
then place a likelihood to every bin .

Considered some of the weaknesses in C51 is the discretization of Q-values to boot to the truth that you favor to specify
a minimal and maximum tag. To beat these weaknesses, work has been carried out to “transpose” the anguish with
quantile regression .
With C51 they regulate the likelihood for every Q-tag fluctuate, but with quantile regression they regulate the Q-values for every
likelihood fluctuate

An opportunity fluctuate is more formally identified as a quantile – on account of this truth the title “quantile regression”.
.
Following this overview, the implicit quantile community (IQN) used to be presented to learn the elephantine quantile function
as towards discovering out a discrete station of quantiles .
One of the recent SOTA improves on IQN by totally parameterizing the quantile function; both the quantile fractions
and the quantile values are parameterized .

Others specifically focal level on modelling tag distributions for efficient exploration
.
Osband et al. also focal level on efficient exploration, but in distinction to other distributional RL approaches,
they employ randomized tag functions to approximately sample from the posterior
.
One other attention-grabbing capacity for exploration makes employ of the uncertainty Bellman equation to propagate uncertainty
all over more than one timesteps .