mt-daapd over HTTPS

[Adar]

I’ve got an odd request that I’d like some feedback on.

At home I’ve got a huge collection of music served up via mt-daapd. The server running mt-daapd sits behind a NAT/firewall router on my ADSL which is somewhat bandwidth constrained.

At work I’d like access to all of this music. However, my work firewall is also quite restrictive. No incoming connections are allowed, and all outgoing connections must pass through an HTTP proxy. HTTP and FTP are generally allowed, as long as they go through this proxy, and while HTTPS must also travel through the proxy, once the proxy identifies the traffic as HTTPS it is ignored.

However, it’s not enough that I simply run apache-ssl and forward the right ports. I want my music at home to be shared to the rest of my work network. As such, I need to run an mt-daapd server at work that communicates to apache-ssl at home to retrieve files. Two major problems:

1) Retrieving tag information over a slow (768kbps) ADSL connection would be a nightmare for the quantity of music that I have. I’d like to share the mt-daapd database at home over HTTPS and run a cron job at work that updates the work mt-daapd db nightly with the most recent DB. This would require either changing all of the local pathnames to URL’s, or doing on-the-fly changes when opening the file itself.

2) In order to actually get the file, it would seem that only the last step (opening a file) should change. That is, at that stage, instead of opening a local file and filling up a client buffer, the server should open a remote file passing through the proxy and using HTTPS. Would mt-daapd be conducive to such a change?

Lastly, at home I use 0.2.1.1, so my DB is the old-style. Would hacking at mt-daapd require that I get the latest CVS, which uses the new style? I suppose I can update mt-daapd at home to use the latest CVS and change the DB to sqlite, then export that one. Does the CVS code even work?

I’m willing to do the development as long as I’m not underestimating the amount of work necessary to pull this off. Thoughts?

[rpedde]

There are at least two ways to do this (off the top of my head)

1. The hackish way:

a. I’d probably put up a cvs version (since the sqlite db is easier to modify and manipulate) and get a database of your “at home” file layout.

b. Once you have that, set up the same version at work. Copy in the database from home. In dispatch.c, on about line 695 it does an open from the file info from the path. Rather than a fopen on the path from the pmp3 struct, you’ll be constructing a url (http://your.server/databases/1/items/%5Bpmp3->id].mp3) and feeding that back to the client.

It’s a single-purpose hack, but not to difficult. Just make sure the db is in sync with the home database, and start up mt-daapd with a -s (skip initial mp3 scan) and scan_interval set to 0.

That should work.

2. tcp/ip tunneling over http?

http://www.nocrew.org/software/httptunnel.html

[Adar]

rpedde wrote:

There are at least two ways to do this (off the top of my head)

1. The hackish way:

a. I’d probably put up a cvs version (since the sqlite db is easier to modify and manipulate) and get a database of your “at home” file layout.

b. Once you have that, set up the same version at work. Copy in the database from home. In dispatch.c, on about line 695 it does an open from the file info from the path. Rather than a fopen on the path from the pmp3 struct, you’ll be constructing a url (http://your.server/databases/1/items/%5Bpmp3->id].mp3) and feeding that back to the client.

It’s a single-purpose hack, but not to difficult. Just make sure the db is in sync with the home database, and start up mt-daapd with a -s (skip initial mp3 scan) and scan_interval set to 0.

That should work.

2. tcp/ip tunneling over http?

http://www.nocrew.org/software/httptunnel.html

Thanks for the reply. I can see how #1 can work (though making mt-daapd use an https connection sounds like a headache). But how will tunneling over HTTP get the job done? Sure I can establish a connection to port 3689, but what about mDNS on 5353? I don’t see how httptunnel can work for UDP, so I’m at a bit of a loss as to how this can work.

[rpedde]

Aaah… locally, I’d use a local mdns proxy responder, like the ones described in the FAQ under “How can I play mt-daapd through a ssh tunnel”. Yours would be similar, but through a HTTP tunnel.

I bring that one up just because I always really wondered how well the http tunnel worked. 🙂

— Ron

[velociped]

I was prepared to make that same suggestion, Ron. That is until I visited the referenced site for GNU httptunnel. I can find no explicit statement that the HTTP tunnel will support UDP packets. In fact, looking through the archives of their mailing list, there is an unanswered question from over two years ago in which someone asks about UDP support.

In looking at other, similar solutions, I noted that one such product, HTTP-Tunnel, states that their particular implementation does not support UDP tunneling.

Adar, I would certainly encourage you to give it a try. In order to get the remote library to be visible, it will be necessary to tunnel the DAAP beacon, by proxy, to the local, client subnet. Should httptunnel prove to be a robust and versatile alternative for situations wherein more restrictive firewalls are utilized (i.e. where ssh tunnels are blocked), it would be a great piece of information for the FAQ and the project in general.

Please report back with your findings should you decide to try it.

Herman

[velociped]

Upon further consideration, I have thought of an indirect way one might go about implementing the UDP tunnel over HTTP. The success rate and reliability are unknown, but it is worth a try if a direct UDP tunnel cannot be established.

SSH is a TCP protocol. In theory, one could establish an SSH connection through the HTTP tunnel. A subsequent UDP connection could be routed through an SSH tunnel and the DAAP beacon relayed through that pipe.

Such a convoluted solution would represent a great deal of stacking and, given that you have indicated your DSL connection to be somewhat flaky as it is, may or may not be reliable. However, it may be worth a try.

Herman

[Adar]

Okay, so I’ve tried various tunneling mechanisms since those appeared to be the easiest:

1) I set up an http tunnel between my work machine and my server at home. This was done by running “hts -F localhost:3689 50100” at home, so that the home tunnel end listens to port 50100 and forwards packets to port 3689. On the work machine, I ran “htc -P myproxy -F 4000 my.ip:50100” which set up the work tunnel end to listen on port 4000 and forward packets to my.ip:50100 through my work proxy.

Through the tunnel I can telnet, type a few things, and watch mt-daapd respond with “Bad request” in HTML. In addition, I installed howl’s mdns beacon and published “local 4000” as a daap server, and in a different machine at work I could get the DB through the HTTP tunnel.

However, it seems that httptunnel only allows one connection through at a time. As such, I could telnet or get the DB just fine, but when I tried playing a song, nothing happened. iTunes hung on “Connecting…” and telnet never responded with mt-daapd stuff. Through netstat it looked as if another connection to the home machine never occurred. Netstat on my work machine shows an established connection between telnet’s ephemeral port and the htc process, but after that I can’t tell what’s going on. My conclusion is that httptunnel only supports tunneling one connection at a time, and since iTunes tries to connect to the same published port for both DB requests and streams, I don’t know if it’ll work.

2) I briefly looked at corkscrew, but it only works with stdin/stdout, so I wasn’t sure how to leverage it in this situation, nor does it have proxy support.

3) I tried httptunnel with ssh layered over it. That is, on the home machine “hts -F localhost:22 50100”. On the work machine “htc -P myproxy -F 12345 my.ip:50100”. Then on the work machine: “ssh -p 12345 localhost -N -f -L 4000:localhost:3689”. As such, ssh was connecting to port 12345 on the work machine (one end of the tunnel) and forwarding port 4000 to port 3689 at the home machine end.

With this done, I could again telnet into the mt-daapd through the ssh tunnel. However, iTunes could not connect even to get the DB. It complained about firewall settings and not being able to establish a connection. I made sure that I was publishing a daap server of 4000 locally through howl, and I was, so I don’t know what went wrong.

Anyway, that’s the extent of my trials, and as you can see, none of them fully worked. Any ideas?

[velociped]

Working from attempt #3, a question followed by a comment…

Let me see if I am following you on this. You are attempting to reroute SSH through HTTP on your remote/home/server end on port 50100. You are then initiating an HTTP tunnel from the client/work end from 12345, locally, to 50100 on the remote/home/server end. Finally, you are creating a tunnel from port 4000, locally, to port 3689 on the remote/home/server machine. Am I understanding this correctly?

This seems unduly complicated to me. Forgive my ignorance of the HTTP proxy, but why not simply create the HTTP tunnel directly between the client/work end and the ssh port on your remote/home/server? (e.g. htc -P myproxy -F 12345 my.ip:22. Then it simply becomes a matter of routing all of the other requests through the SSH tunnel.

Next, it is not necessary to create a bogus DAAP port locally. In fact, it may be counter productive. Unless you have other DAAP shares on the local subnet, simply reroute that port to the remote/home/server. (e.g. ssh -l [email protected] -N -f -L 3689:my.ip:3689 — assuming the machine at the remote/home/server is the same as that running mt-daapd. If using NAT, you will want to substitute the internal address for the server in port forwarding parameter (e.g. -L 3689:internal.address:3689).

Your recent post also neglected to mention your handling of the mDNS beacon. You will need to republish the remote/home/server beacon to the client/work subnet. This is accomplished by issuing a command such as mdns -R 127.0.0.1 squeal “shareName” _daap._tcp. 3689 & on the client/work local end (or using one of the GUI utilities mentioned in the document wiki. In fact, you should basically be able follow the directions in that document verbatim once the SSH connection is initiated through the HTTP tunnel.

Herman

[Adar]

velociped wrote:

Working from attempt #3, a question followed by a comment…

Let me see if I am following you on this. You are attempting to reroute SSH through HTTP on your remote/home/server end on port 50100. You are then initiating an HTTP tunnel from the client/work end from 12345, locally, to 50100 on the remote/home/server end. Finally, you are creating a tunnel from port 4000, locally, to port 3689 on the remote/home/server machine. Am I understanding this correctly?

Yes. An HTTP tunnel through which I shove an SSH tunnel.

This seems unduly complicated to me. Forgive my ignorance of the HTTP proxy, but why not simply create the HTTP tunnel directly between the client/work end and the ssh port on your remote/home/server? (e.g. htc -P myproxy -F 12345 my.ip:22. Then it simply becomes a matter of routing all of the other requests through the SSH tunnel.

Unfortunately it’s not quite so simple. From what I understand, httptunnel requires both a client (htc) and server (hts) process to function. This is because it adds an additional HTTP header to every packet, and as such, needs a server process to strip the headers before pushing them out of the tunnel. Because of this, the tunneling cannot simply forward to the SSH server.

I want to clarify why I even bothered building an additional SSH tunnel. From my first attempt it seemed as if httptunnel could not multiplex multiple connections through the tunnel. I thought perhaps using SSH as the sole connection and relying on the SSH tunnel to multiplex the connections would work.

Next, it is not necessary to create a bogus DAAP port locally. In fact, it may be counter productive. Unless you have other DAAP shares on the local subnet, simply reroute that port to the remote/home/server. (e.g. ssh -l [email protected] -N -f -L 3689:my.ip:3689 — assuming the machine at the remote/home/server is the same as that running mt-daapd. If using NAT, you will want to substitute the internal address for the server in port forwarding parameter (e.g. -L 3689:internal.address:3689).

I failed to mention that I did try with port 3689 too. When it first failed, I tried 4000, thinking that ssh may short circuit port forwards of “3689:localhost:3689” since there are situations where that would be meaningless.

Your recent post also neglected to mention your handling of the mDNS beacon. You will need to republish the remote/home/server beacon to the client/work subnet. This is accomplished by issuing a command such as mdns -R 127.0.0.1 squeal “shareName” _daap._tcp. 3689 & on the client/work local end (or using one of the GUI utilities mentioned in the document wiki. In fact, you should basically be able follow the directions in that document verbatim once the SSH connection is initiated through the HTTP tunnel.

I appreciate your diligence in ensuring that I followed the correct steps, but I’m almost positive that I carried this out correctly. I’m using the mdnsresponder (based on Howl) available through the Debian repository, which has a config file where all of the published services live. I followed the examples listed there and, in attempt one, this was good enough to get my iTunes computer to see the forwarded mt-daapd server AND get the DB. So I’m nearly positive that this part of the procedure is working. I’ll give it another shot tomorrow at work and report back with my success.

Herman

[velociped]

Thank you for explaining the philosophy of the HTTP tunnel. The requirement of a client and server scheme makes some sense, but really only adds one layer to the attempt.

That said, I think we are in agreement that the SSH tunnel is the way to go. It will be possible to plumb all of the services through that conduit. My only concerns are your previous mention that your home ADSL line is somewhat bandwidth constrained and the potential for latency issues due to routing the DAAP and mDNS packets through an SSH pipe, which is, in turn, itself crammed through an HTTP tunnel. However, it should be feasible with the only issue being degree of reliability.

I failed to mention that I did try with port 3689 too. When it first failed, I tried 4000, thinking that ssh may short circuit port forwards of “3689:localhost:3689” since there are situations where that would be meaningless.

I can provide some assurance that no short circuit will result from forwarding a standard local port to a standard remote port will occur. This is the technique of which I make use on an almost daily basis.

I do have a concern regarding the formula you are utilizing. Perhaps I am misreading your intent, but where is the “localhost” machine? Is it truly local to your work subnet or is it local to the network on the remote end? The machine name/number in that parameter should be the address of the remote server. IOW, the -L switch is telling SSH to reroute the local port 3689 to port 3689 on the remote machine with the address provided in the parameter.

Finally, your original SSH commnd contained the -p switch. I am not sure that is correct as implemented. The -p switch is for connecting to non-standard ports on the remote end. I think you are wanting to bind (-b switch) SSH to port 12345 on the local machine. Once the HTTP tunnel is established between the networks, you will want to use the standard SSH port (22) on the remote end. The trick is rerouting local SSH traffic through the tunnel. This, I think, requires the bind switch.

Herman

[Author]

Posts