[Question] Impact of x scale on results

Hello, I am trying to fit some observations with the simplest model from the tutorials:

class ExactGPModel(gpytorch.models.ExactGP):
    def __init__(self, train_x, train_y, likelihood):
        super(ExactGPModel, self).__init__(train_x, train_y, likelihood)
        self.mean_module = gpytorch.means.ConstantMean()
        self.covar_module = gpytorch.kernels.ScaleKernel(gpytorch.kernels.RBFKernel())

    def forward(self, x):
        mean_x = self.mean_module(x)
        covar_x = self.covar_module(x)
        return gpytorch.distributions.MultivariateNormal(mean_x, covar_x)

# Training data (omitted for brevity).
#train_y = ...
#train_x = ...

likelihood = gpytorch.likelihoods.GaussianLikelihood()
model = ExactGPModel(train_x, train_y, likelihood)

# Training code (omitted for brevity).
# Optimizer: Adam, lr=0.1
# Loss: ExactMarginalLogLikelihood

After training, I am evaluating the model by feeding a linspace with 100 points with this pattern:

with torch.no_grad(), gpytorch.settings.fast_pred_var():
    test_x = torch.linspace(0, train_x[-1], 100, dtype=torch.double)
    observed_pred = likelihood(model(test_x))

However, I am experiencing very different results depending on how I choose to scale the x dimension.

X in [0…1]

X in [0…100]

X in [0…1000]

I am confused: I did expect a result invariant w.r.t. the scale chosen for X dimension. Is this behaviour consistent with the theory of Gaussian Processes?

Thanks in advance for any help!

[Balandat]

@LMolr are you observations indeed noiseless? As Wesley said you can crank down the likelihood noise sigma and not estimate it as part of the fitting. In the limit sigma -> 0 you will get the model mean pass through the observations, but a zero value may lead to numerical issues b/c of ill conditioned linear solves.

[wjmaddox]

To try to induce the model to more closely fit the data, one suggestion would be to set the likelihood noise to be small on initialization (or even to fix it). For example, model.likelihood.noise = 1e-3 * torch.ones(1)