Search results for "Markov process"
showing 10 items of 147 documents
Sequence Q-learning: A memory-based method towards solving POMDP
2015
Partially observable Markov decision process (POMDP) models a control problem, where states are only partially observable by an agent. The two main approaches to solve such tasks are these of value function and direct search in policy space. This paper introduces the Sequence Q-learning method which extends the well known Q-learning algorithm towards the ability to solve POMDPs through adding a special sequence management framework by advancing from action values to “sequence” values and including the “sequence continuity principle”.
Is Admission-Controlled Traffic Self-Similar?
2002
It is widely recognized that the maximum number of heavy-tailed flows that can be admitted to a network link, while meeting QoS targets, can be much lower than in the case of markovian flows. In fact, the superposition of heavy-tailed flows shows long range dependence (self-similarity), which has a detrimental impact on network performance. In this paper, we show that long range dependence is significantly reduced when traffic is controlled by a Measurement-Based Admission Control (MBAC) algorithm. Our results appear to suggest that MBAC is a value added tool to improve performance in the presence of self-similar traffic, rather than a mere approximation for traditional (parameter-based) ad…
Protecting entanglement by adjusting the velocities of moving qubits inside non-Markovian environments
2017
Efficient entanglement preservation in open quantum systems is a crucial scope towards a reliable exploitation of quantum resources. We address this issue by studying how two-qubit entanglement dynamically behaves when two atom qubits move inside two separated identical cavities. The moving qubits independently interact with their respective cavity. As a main general result, we find that under resonant qubit-cavity interaction the initial entanglement between two moving qubits remains closer to its initial value as time passes compared to the case of stationary qubits. In particular, we show that the initial entanglement can be strongly protected from decay by suitably adjusting the velocit…
Random walks in dynamic random environments and ancestry under local population regulation
2015
We consider random walks in dynamic random environments, with an environment generated by the time-reversal of a Markov process from the oriented percolation universality class. If the influence of the random medium on the walk is small in space-time regions where the medium is typical, we obtain a law of large numbers and an averaged central limit theorem for the walk via a regeneration construction under suitable coarse-graining. Such random walks occur naturally as spatial embeddings of ancestral lineages in spatial population models with local regulation. We verify that our assumptions hold for logistic branching random walks when the population density is sufficiently high.
Modeling the coupled return-spread high frequency dynamics of large tick assets
2015
Large tick assets, i.e. assets where one tick movement is a significant fraction of the price and bid-ask spread is almost always equal to one tick, display a dynamics in which price changes and spread are strongly coupled. We introduce a Markov-switching modeling approach for price change, where the latent Markov process is the transition between spreads. We then use a finite Markov mixture of logit regressions on past squared returns to describe the dependence of the probability of price changes. The model can thus be seen as a Double Chain Markov Model. We show that the model describes the shape of return distribution at different time aggregations, volatility clustering, and the anomalo…
Bayesian Markov switching models for the early detection of influenza epidemics
2008
The early detection of outbreaks of diseases is one of the most challenging objectives of epidemiological surveillance systems. In this paper, a Markov switching model is introduced to determine the epidemic and non-epidemic periods from influenza surveillance data: the process of differenced incidence rates is modelled either with a first-order autoregressive process or with a Gaussian white-noise process depending on whether the system is in an epidemic or in a non-epidemic phase. The transition between phases of the disease is modelled as a Markovian process. Bayesian inference is carried out on the former model to detect influenza epidemics at the very moment of their onset. Moreover, t…
On the derivation of a linear Boltzmann equation from a periodic lattice gas
2004
We consider the problem of deriving the linear Boltzmann equation from the Lorentz process with hard spheres obstacles. In a suitable limit (the Boltzmann-Grad limit), it has been proved that the linear Boltzmann equation can be obtained when the position of obstacles are Poisson distributed, while the validation fails, also for the "correct" ratio between obstacle size and lattice parameter, when they are distributed on a purely periodic lattice, because of the existence of very long free trajectories. Here we validate the linear Boltzmann equation, in the limit when the scatterer's radius epsilon vanishes, for a family of Lorentz processes such that the obstacles have a random distributio…
Statistics of return times for weighted maps of the interval
2000
For non markovian, piecewise monotonic maps of the interval associated to a potential, we prove that the law of the entrance time in a cylinder, when renormalized by the measure of the cylinder, converges to an exponential law for almost all cylinders. Thanks to this result, we prove that the fluctuations of Rn, first return time in a cylinder, are lognormal.
Componentwise adaptation for high dimensional MCMC
2005
We introduce a new adaptive MCMC algorithm, based on the traditional single component Metropolis-Hastings algorithm and on our earlier adaptive Metropolis algorithm (AM). In the new algorithm the adaption is performed component by component. The chain is no more Markovian, but it remains ergodic. The algorithm is demonstrated to work well in varying test cases up to 1000 dimensions.
Bayesian analysis of a Gibbs hard-core point pattern model with varying repulsion range
2014
A Bayesian solution is suggested for the modelling of spatial point patterns with inhomogeneous hard-core radius using Gaussian processes in the regularization. The key observation is that a straightforward use of the finite Gibbs hard-core process likelihood together with a log-Gaussian random field prior does not work without penalisation towards high local packing density. Instead, a nearest neighbour Gibbs process likelihood is used. This approach to hard-core inhomogeneity is an alternative to the transformation inhomogeneous hard-core modelling. The computations are based on recent Markovian approximation results for Gaussian fields. As an application, data on the nest locations of Sa…