Webui not correctly linking to local gateway when running over LAN at .local domain

Checklist

• This is a bug report, not a question. Ask questions on discuss.ipfs.io.
• I have searched on the issue tracker for my bug.
• I am running the latest kubo version or have an issue updating.

Installation method

built from source

Version

# ipfs version --all
Kubo version: 0.15.0
Repo version: 12
System version: arm64/linux
Golang version: go1.18.5

Config

# ipfs config show

{
  "API": {
    "HTTPHeaders": {
      "Access-Control-Allow-Methods": [
        "PUT",
        "POST"
      ],
      "Access-Control-Allow-Origin": [
        "https://djnofp2u5mvjsewk7anxsj4iep3n5uvez362lek4ixjysgeefk5esxyd.local", #api/webui domain
        "http://localhost:3000",
        "http://127.0.0.1:5001",
        "https://webui.ipfs.io",
        "http://ipfs.embassy:5001",
        "https://rfu6pj36capximdisyvwxyjnhlwyeovf4gcmz4hvuptaetwduslav3id.local", #gateway domain
        "https://0.0.0.0:8080"
      ]
    }
  },
  "Addresses": {
    "API": "/ip4/0.0.0.0/tcp/5001",
    "Announce": [],
    "AppendAnnounce": [],
    "Gateway": "/ip4/0.0.0.0/tcp/8080",
    "NoAnnounce": [],
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/::/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/quic",
      "/ip6/::/udp/4001/quic"
    ]
  },
  "AutoNAT": {},
  "Bootstrap": [
    "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt"
  ],
  "DNS": {
    "Resolvers": {}
  },
  "Datastore": {
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
            "sync": true,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "10GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": true,
      "Interval": 10
    }
  },
  "Experimental": {
    "AcceleratedDHTClient": false,
    "FilestoreEnabled": false,
    "GraphsyncEnabled": false,
    "Libp2pStreamMounting": true,
    "P2pHttpProxy": false,
    "StrategicProviding": false,
    "UrlstoreEnabled": false
  },
  "Gateway": {
    "APICommands": [],
    "HTTPHeaders": {
      "Access-Control-Allow-Headers": [
        "X-Requested-With",
        "Range",
        "User-Agent"
      ],
      "Access-Control-Allow-Methods": [
        "GET"
      ],
      "Access-Control-Allow-Origin": [
        "*"
      ]
    },
    "NoDNSLink": false,
    "NoFetch": false,
    "PathPrefixes": [],
    "PublicGateways": {
      "localhost": null,
      "rfu6pj36capximdisyvwxyjnhlwyeovf4gcmz4hvuptaetwduslav3id.local": {
        "Paths": [
          "/ipfs",
          "/ipns"
        ]
      }
    },
    "RootRedirect": "",
    "Writable": false
  },
  "Identity": {
    "PeerID": "12D3KooWPCNH8gBqGuuCxerv3wLEjBEEWHYJXxa8eUcsNSjLS6cm"
  },
  "Internal": {},
  "Ipns": {
    "RecordLifetime": "",
    "RepublishPeriod": "",
    "ResolveCacheSize": 128
  },
  "Migration": {
    "DownloadSources": [],
    "Keep": ""
  },
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns"
  },
  "Peering": {
    "Peers": null
  },
  "Pinning": {
    "RemoteServices": {}
  },
  "Plugins": {
    "Plugins": null
  },
  "Provider": {
    "Strategy": ""
  },
  "Pubsub": {
    "DisableSigning": false,
    "Router": ""
  },
  "Reprovider": {
    "Interval": "12h",
    "Strategy": "all"
  },
  "Routing": {
    "Type": "dht"
  },
  "Swarm": {
    "AddrFilters": null,
    "ConnMgr": {
      "GracePeriod": "20s",
      "HighWater": 900,
      "LowWater": 600,
      "Type": "basic"
    },
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": false,
    "RelayClient": {
      "Enabled": true
    },
    "RelayService": {},
    "Transports": {
      "Multiplexers": {},
      "Network": {},
      "Security": {}
    }
  },
  "Transports": {
    "Network": {
      "Relay": true
    }
  }
}

Description

I am trying to package IPFS for embassy-os. See this discussion forum post for some background on embassy-os.

1. Run kubo v0.15.0 in a docker container on a raspberry pi on my home network using https://github.com/chrisguida/ipfs-wrapper. The Dockerfile here is pretty much the same as the Dockerfile included in the kubo repo, with some slight modifications to adapt it to .
2. configure kubo as in this entrypoint.
3. In the webui, go to the “Files” tab and select any file to view it. The ipfs.io public gateway sometimes displays the file after a long wait. Other times, if I open the browser console I can see this error:

(As you can see, the browser does not like that the request to http://0.0.0.0:8080 is cleartext HTTP, while the domain I’m viewing the webui through is HTTPS (“https://djnofp2u5mvjsewk7anxsj4iep3n5uvez362lek4ixjysgeefk5esxyd.local” in the above example config). Actually, I’m not even sure what http://0.0.0.0:8080 is supposed to me. I do know that if I run kubo locally on my laptop, this request succeeds. How firefox is able to resolve this URL, ever, is beyond me.)

But since the point of embassy-os is to self-host, we want to avoid “public” gateways and use the gateway on our own device, which responds much faster anyway. If I go to the webui “Settings” tab and enter in my gateway address (“https://rfu6pj36capximdisyvwxyjnhlwyeovf4gcmz4hvuptaetwduslav3id.local” in the example config) into the “public gateway” field, clicking on files now works, and sends the files through my local gateway properly.

Issue ipfs/kubo#1: I would like to be able to configure this setting programmatically, ie in docker_entrypoint.sh. Requiring the user to manually find their gateway domain and input it into this settings field adds a lot of friction to the UX. However, that can be worked around for now if it’s the only way. The real bug is below:

Issue ipfs/kubo#2: When I click “Download” on any file, even after manually setting the gateway in the webui, the browser still requests http://0.0.0.0:8080, rather than the local gateway I’ve configured. This of course results in an error. This makes it so that the user cannot download a file through the UI. The bizarre thing is that if I manually alter the download URL to replace http://0.0.0.0:8080 with the local gateway URL (“https://rfu6pj36capximdisyvwxyjnhlwyeovf4gcmz4hvuptaetwduslav3id.local” in the example config), the file can be viewed through the browser just fine. For some reason, the webui is unable to simply link the user directly to the local gateway, even when it is specifically configured in the webui.

It’s possible I’m missing a config value somewhere, as I’m fairly new to kubo, but this definitely seems like at least a documentation issue, and probably a bug in the webui.

Please let me know how I can help debug/fix this, as our community is asking for access to IPFS on embassy-os!

Thanks in advance,

–Chris

[chrisguida]

Hey @lidel thanks again for getting back to me. Apologies for the delayed response.

ipfs-webui is designed to talk to local Kubo node via /api/v0 RPC, within the same origin and use gateway on the same box

Does this mean that using two different domains for the API and Gateway will not work? I can put them on the same domain if so.

/api/v0 endpoint grants ADMIN level permissions to your IPFS daemon, and the only officially upported Origin in browser context is localhost

Yes, I understand, this is not a problem since .local is the TLD designed for mDNS, which only works over LAN. Obviously since embassy-os is designed to run on a personal server, and not on the user’s local machine, we cannot use localhost here. But surely there are workarounds for people who wish to run their IPFS node from a server in their house rather than from a client machine. If not, that would seem to [partially] defeat the purpose of running one’s own IPFS node. But perhaps I’m misunderstanding IPFS’s goal. Further, I can already access the gateway by manually copying and pasting the gateway domain in place of 0.0.0.0:8080 when opening a gateway URL from the webui. And again, this functionality is already supported for clicking on a file to view it, but not to download it. So this functionality is already partially supported. All that is needed is a tweak to the frontend to have it respect the user’s gateway settings in the Settings tab for downloads in addition to file viewing.

(this is to say, you have very custom setup that maintainers have no bandwidth to support, but read ideas below)

That’s ok; as I said, I’m willing to contribute the work necessary to add this functionality to IPFS. Again, I think allowing users to run IPFS nodes from an always-on server in their homes is a uniquely beneficial use case of IPFS. It allows the user to fully contribute to the IPFS network by hosting files for others; it also allows the user to access IPFS in a fully self-sovereign way, as they don’t need to rely on public gateways. But perhaps there are other ways to achieve this.

is running on *.local a hard requirement?

It’s more of a soft requirement that will save dozens of hours of dev work and customer support. Embassy-os only supports access to services via .local when connecting via LAN, and over .onion when connecting remotely. We are planning to add support for direct IP access over LAN, but for now .local/mDNS is the only officially supported method of accessing services over LAN. But again, this is a soft requirement; if I can find a way to connect to the gateway using an IP instead of a domain, that is much better than no gateway access from the webui at all. I can simply hack in a workaround in embassy-os for now if IP addresses will work.

I see you have TLS cert, so i guess you don’t care or don’t have mixed-content issues

Correct, we need to access the .local domain over HTTPS, and the webui is using http; that is one of the problems that we need to fix or work around.

but the usual advice for people who want to access API and webui remotely is SSH tunneling of API and gateway ports both looks like localhost – it solves all issues in browser context, and may be more future-proof than using custom LAN hostnames.

Ok, I can try this and see whether it will be too cumbersome. The goal of embassy-os is to provide a large marketplace of services that are plug-and-play for people who are new to self-hosting. Asking them to create an SSH tunnel in order to use the gateway will most likely result in a lot of support load, as the majority of our users don’t have SSH access, but we’ll do it if there’s no other way.

For a browser, the “listen on all interfaces” 0.0.0.0 is turned into 127.0.0.1 (loopback IP, same as localhost) – that is why it works locally in your Firefox 😃

Excellent, I suspected this, but couldn’t find any online resources explaining this behavior. Thanks for confirming 😃

any reason why you disabled subdomains on locahost by setting it to null in PublicGateways? you may want to remove that line, just so you are not suprised if you ever need it.

This was from a suggestion somewhere (I’ll try to find it) which recommended disabling localhost subdomains when using IPFS non-locally. Thanks for the reminder, but again, I’m trying to come up with a solution that doesn’t require an SSH tunnel. The goal is to provide a zero-configuration experience for the end user, so I need all configuration done automatically inside the docker container if at all possible. Is there any reason to have localhost enabled here if I’m not using an SSH tunnel between the raspberry pi and the user’s client device?

ipfs-webui will try to use local gateway before resorting to public one

It reads the first item from Addresses.Gateway in the config (this field can be either a String, or an array of strings) and tries to use it instead of public gateway (which is used only as last resort).

Ok, thanks for the info. I haven’t tried using an array of addresses yet; I’ve only tried setting it to single strings, which have all failed in one way or another.

👉 Your Addresses.Gateway is “/ip4/0.0.0.0/tcp/8080”, and ipfs-webui parses that as localhost (which is what 99,9% use cases want), but that is not the behavior your custom setup requires.

Potential fix here is to change Addresses.Gateway to array and put your an explicit address there first, so ipfs-webui picks it up and uses it instead of 0.0.0.0

If your gateway is at https://ipfsgw.local or LAN IP A.B.C.D then try seting Addresses.Gateway to one of the arrays below and see if any of them works (you need to reboot daemon for the config to be applied):

[“/dns4/ipfsgw.local/tcp/443”, “/ip4/0.0.0.0/tcp/8080”] [“/ip4/A.B.C.D/tcp/443”, “/ip4/0.0.0.0/tcp/8080”]

I did try your top example (as a string, not as an array with 0.0.0.0 as the second value), and kubo exits because it cannot see the .local address. This makes sense, because the .local address is not visible from within the docker container. This is because we are using a reverse proxy on the host machine to direct traffic coming into the .local domain to the docker container’s IP address. I have two questions about this:

1. Most servers allow a “listening” address that is different from the “dialing” address that the client uses to connect. It appears as though the webui does not allow dialing an address that kubo itself cannot dial. Is this correct? If so, I think this is the feature that needs to be added, and as I mentioned previously, I’m willing to put in the work to add this feature if you guys would accept a PR that does this.
2. Does it matter if I use only a single string rather than an array of strings, if I’m not intending to use localhost? Does kubo no longer complain that it cannot dial the first address, as long as the second address is dialable? I will test this to confirm.

I doubt there is an IP address which is both listenable and dialable by kubo, without port forwarding in embassy-os (again, this feature is forthcoming, currently it will be a hack). The docker container has its own unique VLAN address (ie 172.18.0.x), but this address is not visible outside the host from another computer on the LAN. The embassy’s own LAN IP address is visible outside the embassy, but is not visible to kubo. Perhaps there’s some docker network hacking I can do to work around this.

It sounds like there are 5 potential fixes to this issue:

1. use an SSH tunnel (not desirable, but potentially workable for our advanced users)
2. hack in port forwarding using socat inside embassy-os, and somehow get kubo to recognize this as a valid listening address (perhaps using network=host in docker?)
3. use an array of strings instead of a string for the Addresses.Gateway setting, perhaps this will magically work?
4. fix the webui so that it will respect what the user puts in their Settings tab, which it already does for “click to view”, but does not work for downloads (this seems easiest? not optimal, since the user needs to set this in their UI for every client device)
5. allow kubo to listen on an address that kubo itself cannot dial (optimal, since this would allow zero-configuration for the end user).

I will attempt 1-3 and get back to you. It sounds like you guys are reluctant to look into 4-5, which is sad since I think this would be not too much work for a lot of gain. Again, I am willing to take responsibility for these enhancements myself, as long as the IPFS community can help answer my questions.

Thanks for helping me out on this one, I really appreciate it 😃

[chrisguida]

Ok never mind, enabling IPFS Companion fixes the “download” behavior, but breaks the “click to view” behavior xD

Will see if there’s some combination of webui and companion settings that fixes this.