online learning result not matching batch (using natural parameters)

I’m trying to access and use the natural parameter vector of the Gaussian distribution in an online learning application. I’m using a simple example I found in the documentation to estimate the mean of a Gaussian with a sequence of data points. When I run inference on the entire batch of data points I get an expected mean of: Gaussian(16.03, 1.235). I would think that summing all the data likelihoods (via the natural parameter vectors) and adding the initial prior would produce the same result, but I get an expected mean of: Gaussian(16.49, var= 0.0308). Is there something conceptually wrong with my thinking or am I not using the natural parameter vector correctly (I notice there is a simplified form of the natural parameter vector used in Infer.NET)? Or is it inference related?

Here is my code:

double[] dataArray = { 14.32,  14.96,  14.11,  18.63,  14.74,  21.97,  20.11,  15.59,
                                   19.67,  19.54,  14.52,  14.56,  19.69,  18.93,  19.02,  14.87,
                                   14.17,  20.14,  13.98,  18.65,  13.09,  17.93,  15.66,  18.78,
                                   18.31,  18.89,  19.53,  17.8 ,  13.35,  16.72,  12.21,  13.58,
                                   14.54,  18.37,  19.48,  14.84,  13.22,  17.63,  12.65,  12.45,
                                   18.97,  18.44,  12.92,  19.09,  18.00,  12.93,  17.07,  17.75,
                                   19.38,  17.8 ,  13.02,  14.16,  13.36,  16.78,  15.26,  12.47,
                                   21.34,  12.97,  14.22,  19.54,  12.33,  12.56,  14.92,  17.07,
                                   17.02,  13.33,  17.78,  19.96,  18.16,  12.5 ,  12.56,  12.13,
                                   12.76,  17.76,  14.74,  15.65,  18.11,  18.53,  16.53,  17.75 };

            /********** inference **********/
            var engine = new InferenceEngine();
            engine.Algorithm = new VariationalMessagePassing();
            engine.ShowProgress = false;

            /********* model *********/
            Variable<double> mean = Variable.GaussianFromMeanAndVariance(0.0, 100).Named("mean");

            Variable<double> x = Variable.New<double>().Named("x");
            x = Variable.GaussianFromMeanAndVariance(mean, 100);

            //Range n = new Range(dataArray.Length);
            //VariableArray<double> x = Variable.Array<double>(n).Named("x");
            //x[n] = Variable.GaussianFromMeanAndVariance(mean, 100).ForEach(n);

            //x.ObservedValue = dataArray;
            //Gaussian meanExpected = engine.Infer<Gaussian>(mean);
            //Console.WriteLine(meanExpected);
            ////result: Gaussian(16.03, 1.235)

            Variable<Gaussian> meanMessage = Variable.Observed<Gaussian>(Gaussian.Uniform()).Named("meanMessage");

            Variable.ConstrainEqualRandom(mean, meanMessage);

            mean.AddAttribute(QueryTypes.Marginal);
            mean.AddAttribute(QueryTypes.MarginalDividedByPrior);

            Gaussian meanMarginal = Gaussian.Uniform();

            double[] GaussNatural = { 0, 0 };

            for (int t = 0; t < dataArray.Length; t++)
            {
                x.ObservedValue = dataArray[t];

                // this is the posterior distribution over the mean
                meanMarginal = engine.Infer<Gaussian>(mean);

                double postMean, postVar;
                meanMarginal.GetMeanAndVariance(out postMean, out postVar);

                // when we divide by the prior (note this is the original prior) we are left with only the data likelihood
                Gaussian dataLikelihood = engine.Infer<Gaussian>(mean, QueryTypes.MarginalDividedByPrior);

                double dataLikeMean, dataLikeVar;
                dataLikelihood.GetMeanAndVariance(out dataLikeMean, out dataLikeVar);

                meanMessage.ObservedValue = new Gaussian(dataLikeMean, dataLikeVar);

                double meantimesprecision, precision;
                dataLikelihood.GetNatural(out meantimesprecision, out precision);

                GaussNatural[0] = GaussNatural[0] + meantimesprecision;
                GaussNatural[1] = GaussNatural[1] + precision;
            }

            double mPrior = 0;
            double bPrior = 1 / 100.0;

            GaussNatural[0] = GaussNatural[0] + mPrior;
            GaussNatural[1] = GaussNatural[1] + bPrior;

            Console.WriteLine("mean= {0}, var= {1}", GaussNatural[0] / GaussNatural[1], 1 / GaussNatural[1]);

[tminka]

There should not be any difference between EP and VMP on this example.

[jacowp357]

Hi Tom, will the way I am adding and re-scaling the natural parameters, to create a new meanMessage in the code above, work with EP as well?