26 January 2010

Revisiting rtmpdump [updated!]

The original Comedy Central logo used from 199...Image via Wikipedia

EDIT 2012-05-15: Updated guide so it uses the "new" git code repository! Thanks BecomingJulie!

Time for an updated guide - I told you previously that things change quickly! :)
The new major release 2.x of rtmpdump brings new tools which make things much easier. How much? Well downloading shows of The Colbert Report, The Daily Show and episodes from South Park Studios is now almost a breeze!
What is this gamechanger you say? It's the new rtmpsrv program which comes with rtmpdump 2. It does the guessing game for us and gives a copy-pastable command. In short you have to position it between your browser and the internet and allow it to perform essentially a man-in-the-middle attack on the handshake of the proprietary protocol.
Compilation on GNU/Linux (for other system please read the README file):
sudo apt-get install build-essential git zlib1g-dev libssl-dev
git clone git://git.ffmpeg.org/rtmpdump
cd rtmpdump
make SHARED= SYS=posix
The program is now compiled for use on your system! Now the harder part - you need root access to configure the iptables firewall which should be a part of any modern Linux distro. This is the part which will reroute the RTMP communication of the flash plugin in your browser through a local port:
sudo iptables -t nat -A OUTPUT -p tcp --dport 1935 -j REDIRECT
Excellent, now start the program rtmpsrv which will listen on the local port:
./rtmpsrv
Browse to a media site and wait for rtmpsrv to write out a command in the console. If you see multiple chapters on the media site, try skipping to another one. I'll describe why in a minute.
OK! You should see at least one long string starting with rtmpdump. rtmpsrv did its job so close the browser tab and stop rtmpsrv with a Control+C signal. Ending rtmpsrv seems to be a problem. The cleaning up process seems to go nowhere at times but sending several Ctrl+C events usually produces a segfault. :/
Also delete the firewall redirect so that you can access the media server directly:
sudo iptables -t nat -D OUTPUT -p tcp --dport 1935 -j REDIRECT
If you're in luck taht is it and you only need to run the output of rtmpsrv, like this:
./rtmpdump -r "rtmpe://cp9950.edgefcs.net:1935/ondemand?ovpfv=1.1" -a "ondemand?ovpfv=1.1" -f "LNX 10,0,42,34" -W "http://media.mtvnservices.com/player/release/?v=4.2.2-8" -t "rtmpe://cp9950.edgefcs.net:1935/ondemand?ovpfv=1.1" -p "http://www.colbertnation.com/full-episodes/thu-january-14-2010-kathleen-sebelius" -C Z: -y "mp4:mtvnorigin/gsp.comedystor/com/colbert/season_06/episode_008/cr_06008_act1_768x432_1720.mp4" -o output.flv
The problem with Comedy Central shows are commercial breaks which cut the show into different separate chapters called acts (see the end of command above). This gives us a bit more work but it's so worth it avoiding the horrible full screen flash experience! Smooth full-screen playback has arrived, Randall! :)
This particular episode of The Colbert Report has 4 acts - all you need to do is change the actX in the command and the output filename must be different for each act too. Ending up with 4 differnet videos is kind of lame but manageable. If you know of a FLOSS stitching solution, please comment! (Avidemux doesn't wanna, PiTiVi wants to transcode, haven't tried LiVES yet).
SouthParStudios.com has a different gotcha - the first clip is the intro which is the same for all shows. Only the following acts are show specific. I managed to get a second command from rtmpsrv by seeking in the browser as mentioned above. Behold:
./rtmpsrv
RTMP Server v2.1c
(c) 2010 Andrej Stepanchuk, Howard Chu; license: GPL

Streaming on rtmp://0.0.0.0:1935
WARNING: Trying different position for client digest!


rtmpdump -r "rtmpe://cp40493.edgefcs.net:1935/ondemand?ovpfv=1.1" -a "ondemand?ovpfv=1.1" -f "LNX 10,0,42,34" -W "http://media.mtvnservices.com/player/release/?v=4.2.1" -t "rtmpe://cp40493.edgefcs.net:1935/ondemand?ovpfv=1.1" -p "http://www.southparkstudios.com/episodes/251890" -C Z: -y "mp4:mtvnorigin/gsp.comedystor/com/sp/extras/intros/PromoIntros/HDintroPromoXX_768x432_750.mp4" -o output.flv


rtmpdump -r "rtmpe://cp40493.edgefcs.net:1935/ondemand?ovpfv=1.1" -a "ondemand?ovpfv=1.1" -f "LNX 10,0,42,34" -W "http://media.mtvnservices.com/player/release/?v=4.2.1" -t "rtmpe://cp40493.edgefcs.net:1935/ondemand?ovpfv=1.1" -p "http://www.southparkstudios.com/episodes/251890" -y "mp4:mtvnorigin/gsp.comedystor/com/sp/season_13/1313/acts/sp_1313_act1_768x432_750.mp4" -o output.flv

^CCaught signal: 2, cleaning up, just a second...
And we have lift-off! See the act1 above? Incrementing this value will give you all the acts.
I am the first to admit this should be a total overkill for a good viewing experience but the current situation isn't really bearable. While fast modern multi-core desktop processors are capable of much smoother playback my aging Pentium M and a masses of netbooks struggle with this task. And of course (legally) downloading the shows and watching them offline is now possible! I think modern smartphones and media devices should have no problem with playback of these local files but as always YMMV.
Creators of rtmpdump, on behalf of the Linux masses suffering from stuttering full-screen playback with flash, I salute you!

Enhanced by Zemanta

23 comments:

  1. Before anyone asks: the source code of rtmpdump comes with a README file. In there are more instructions, e.g. using the pf firewall from OpenBSD.

    How to do this on Windows? There's a compiled version right on the rtmpdump homepage but I don't know how to easily redirect traffic.

    ReplyDelete
  2. hello !

    such a pretty tuto ! thanks a lot :)

    But I have a question : i'm with mac os x snow leopard ... so I don't have iptables on my system ... what can I do ?

    thanks !

    carmelo

    ReplyDelete
  3. @Carmelo
    MacOS X fortunately uses the ipfw firewall that is common in *BSD operating systems.

    Fortunately the README of rtmpdump conatins instructions for ipfw as well:

    ipfw add 40 fwd 127.0.0.1 1935 tcp from any to any 1935
    ipfw add 40 fwd 127.0.0.1 1935 tcp from any to any 1935 not uid proxy

    You'll have to find a way to delete these rules after using rtmpsrv yourself!

    For more see the ipfw manpage and a graphical interface called NoobProof.

    ReplyDelete
  4. hey !
    thanks for your answer !

    so .. with this command, there is a problem :

    MacPro:~ Carmelo$ ipfw add 40 fwd 127.0.0.1 1935 tcp from any to any 1935
    ipfw: missing ``from''
    MacPro:~ Carmelo$


    the ipfw tools doesn't like 'any' parameters ... I will look at man page and NoobProof

    thanks again :)

    ReplyDelete
  5. @Carmelo
    Try this one:
    # ipfw add 41 fwd 127.0.0.1,1935 tcp from me to any dst-port 1935
    Deleting the rule might work with:
    # ipfw delete 41
    To make sure the rule is applied or deleted, see if it is in the ruleset with:
    # ipfw show

    ReplyDelete
  6. thanks !

    but i tried your command.
    i can see with ipfw show the rule #41 ...

    but still nothing with rtmpsuck ...

    (i have tried under ubuntu with same video site : no problem, rtmpsuck shows good informations)

    ReplyDelete
  7. rtmpsrv 2.1d automatically downloads videos using rtmpdump now.

    no more copy/pasting.

    ReplyDelete
  8. Hello everyone!

    I don't know if rtmpdump can be installed into a IP Cameras? The OS of IP Camera is embedded Linux. Because we try to publish the live streaming of IP Camera via RTMP protocol. Could you kindly give me more information about this?
    Thank you very much!

    ReplyDelete
  9. Hello, Anonymous poster!
    There certainly are server implementations. take a look on lkcl.net (search for crtmpserver on that page).

    ReplyDelete
  10. Thanks to the help here and a couple of other places, I was finally able to get this to work.

    I kept getting segmentation faults when running rtmpdump and it turned out that I have librtmp installed on my machine (Fedora Core 14).

    In order to get around that - as stated at the very end here - you must use the LD_PRELOAD variable to specify the librtmp that comes with (or is compiled with) the rtmpdump package/source.

    My basic routine for doing this now consists of this:

    sudo iptables -t nat -A OUTPUT -p tcp --dport 1935 -j REDIRECT

    rtmpsrv

    sudo iptables -t nat -D OUTPUT -p tcp --dport 1935 -j REDIRECT

    export LD_PRELOAD=/home/lavid/lib/librtmp.so.0

    rtmpdump -r [...]

    Hope this helps someone...

    ReplyDelete
  11. Awesome tutorial! Haha, now to go back through the man pages with this in hand and get closer to actually learning how to fish.

    ReplyDelete
  12. Thank you very much for this. You are a star!

    Missed the first episode of My Transsexual Summer and tried for ages to get the stream to download. Obviously needed rtmpdump, and it needed to be 2.4 out of git to work with "version 9". Grr. Oh well. I'd already built the one from the tarball, so I knew I had the dependencies. Segmentation fault! Turned out to be cos some tart forgot to replace /usr/local/lib/librtmp.so.0 with the new one. Used tcpflow to get the xml file with the StreamUri in it, and it almost worked. But only almost.

    Even tried running a shareware program in Wine. Didn't work, quelle surprise, but the fact that others had had success under real Windows suggested it was at least possible to pull the streams as I wanted. What I couldn't work out was how to get rtmpsrv to listen to the address it was supposed to. Fortunately, before I went as far as installing and configuring BIND9 just for the purpose of a bit of DNS redirection, I guessed there must be a trick with iptables.

    Now, I probably would have figured it out for myself sooner or later, me being such a kewl h4xx0r chick and all that :) (Well, *eventually* the chick bit, but that's another story.) But thanks anyway for taking the time to write this up.

    BTW, since I believe in giving back, here's a tip from me. If you ever want to do a bunch of commands in a row that all need sudo, try

    $ sudo /bin/bash

    The prompt will change to a comment mark, and you're now effectively root (just like on any old-fashioned Unix). Press Ctrl+D to get back to normal.

    ReplyDelete
  13. Hi everyone,

    I have a problem with Rtmpdump . I hope someone can help me...

    I did everthing up here but error has occured. I want to download stream from this adress "http://www.ttnetmuzik.com.tr/radio". I wrote this command on console window.

    ercan@ubuntu:~$ sudo iptables -t nat -A OUTPUT -p tcp --dport 1935 -j REDIRECT
    ercan@ubuntu:~$ rtmpsrv
    RTMP Server v2.3
    (c) 2010 Andrej Stepanchuk, Howard Chu; license: GPL

    Streaming on rtmp://0.0.0.0:1935

    rtmpdump -r "rtmpe://streaming.ttnetmuzik.com.tr/ttnet?QzlHbWxPZFJWNndCUnBKSDZNM1VmNEpZYUVvN0pXQTM=" -a "ttnet?QzlHbWxPZFJWNndCUnBKSDZNM1VmNEpZYUVvN0pXQTM=" -f "LNX 11,2,202,228" -W "http://www.ttnetmuzik.com.tr/TTNETMuzik/miniplayer/swf/flowplayer.rtmp-3.1.3.swf?v1" -p "http://www.ttnetmuzik.com.tr" -y "mp4:20835b420fda63a6bd7c9c944939df31ffc446a059001fc7.m4a" -o mp4_20835b420fda63a6bd7c9c944939df31ffc446a059001fc7.flv

    Closing connection... done!

    RTMPDump v2.3
    (c) 2010 Andrej Stepanchuk, Howard Chu, The Flvstreamer Team; license: GPL
    Connecting ...
    WARNING: HandShake: Type mismatch: client sent 6, server answered 8
    INFO: Connected...
    Duplicate request, skipping.
    Closing connection... ERROR: RTMP_ReadPacket, failed to read RTMP packet header
    done!

    Can anyone help me this error?

    ReplyDelete
  14. Hi, ercan!

    The website will only stream to Turkish IPs: "We are sorry to say that due to licensing constraints, we can not allow access to visitors located outside of Turkey." but from your error it might be that the server at ttnetmuzik.com.tr only allows clients that know how to talk RTMP version 8. Your program only seems to offer version 6 but the server does not downgrade the protocol version. This seems to be a misconfiguration on the server.

    On the homepage of rtmpdump the authors say that it should support up to protocol version 9. I suggest you (re)do the steps in the blog post above so you have the absolute newest program. Then try dumping the stream again.

    ReplyDelete
  15. This comment has been removed by the author.

    ReplyDelete
  16. I'm very glad to interest my problem. Thank you very much for helping me.

    I installed newest version of rtmpdump. I's new version is V2.4. It's suported protocol 9. But this time another error ocured. Error is shown like this:

    ercan@ubuntu:~/rtmpdump$ rtmpsrv
    RTMP Server v2.3
    (c) 2010 Andrej Stepanchuk, Howard Chu; license: GPL

    Streaming on rtmp://0.0.0.0:1935

    rtmpdump -r "rtmpe://streaming.ttnetmuzik.com.tr/ttnet?VDg4TGJEV3VxQ0grc2RLdUE1QjlqOEs1UEI5OXVVOTk=" -a "ttnet?VDg4TGJEV3VxQ0grc2RLdUE1QjlqOEs1UEI5OXVVOTk=" -f "LNX 11,2,202,233" -W "http://www.ttnetmuzik.com.tr/TTNETMuzik/miniplayer/swf/flowplayer.rtmp-3.1.3.swf?v1" -p "http://www.ttnetmuzik.com.tr" -y "mp4:c03ac34907a9946890aa7c16b39653494fe44fd9d26a9419.m4a" -o mp4_c03ac34907a9946890aa7c16b39653494fe44fd9d26a9419.flv

    Closing connection... done!

    RTMPDump v2.3
    (c) 2010 Andrej Stepanchuk, Howard Chu, The Flvstreamer Team; license: GPL
    Failed to open file! mp4_c03ac34907a9946890aa7c16b39653494fe44fd9d26a9419.flv


    If i write sudo begining of the code line (ercan@ubuntu:~/rtmpdump$ sudo rtmpsrv
    ) program give same error which i tried earlier.

    What should i do to overcome this error ?

    ReplyDelete
  17. sudo apt-get install git
    git clone git://git.ffmpeg.org/rtmpdump
    sudo apt-get install build-essential subversion zlib1g-dev libssl-dev
    cd /home/ercan/rtmpdump rtmpdump
    make SHARED= SYS=posix

    I wrote this code respectively. But after that i run the "rtmpsrv" code. The error is given like this.

    "rtmpsrv" is not installed. You can install by typing the following lines
    "sudo apt-get install rtmpdump"

    When i wrote this line, v2.3 is installed. Then error is the same as previously error. I couldn't solve this problem.

    ReplyDelete
    Replies
    1. ercan, did you forget the ./ before the rtmpsrv command? Try running $rm -rf ~/rtmpdump and then the commands from the above post again.

      Also thanks for your comments, they belatedly made me realize my guide was a bit obsolete. I should have read BecomingJulie's comment more thoroughly.

      I updated the post above with the new repository location.

      Delete
  18. Thank you very much for this great tutorial!
    It helped me a lot.
    :-)

    ReplyDelete
  19. There's a further refinement you can do that saves you the bother of cutting and pasting.

    The latest rtmpsrv actually tries to do the rtmpdump command itself, but it fails because the command it was trying to send to the server gets redirected to itself ad nauseam. So try this:

    First make a new folder to save your downloaded files in, and navigate to there:

    $ mkdir downloaded_video
    $ cd downloaded_video

    Now start iptables -- note the slight modification:

    $ sudo iptables -t nat -A OUTPUT -p tcp --dport 1935 -m owner --uid-owner jules -j REDIRECT
    (you will need to replace "jules" above with your own login name.) This will only redirect packets if the owner of the connection is "jules".
    $ sudo /path/to/rtmpsrv
    (you may not need to give the full path, if rtmpsrv is copied somewhere in your PATH.)

    Now, since we used sudo, rtmpsrv is running as "root"; so the connection set up by rtmpsrv will belong to "root", not "jules". Therefore, the packets coming from our person in the middle will *not* match the rule, and so will not be redirected, but sent straight to the real server.

    When you're done, press ctrl+C to close down rtmpdump -- but you are not quite ready to close the terminal yet. The files were downloaded by, and still belong to, "root". So now you need to use the "chown" command to CHange the OWNer of the files to yourself.

    Type

    $ sudo chown jules:jules *
    (replace "jules" with your own login name, of course)

    Now the files you saved belong properly to you. Before you close down the terminal, run this one final command:
    $ sudo iptables -t nat -D OUTPUT -p tcp --dport 1935 -m owner --uid-owner jules -j REDIRECT

    This will remove the iptables redirection we applied earlier.

    ReplyDelete
  20. Nice tutorial...

    As for a "FLOSS stitching solution" this works for me most of the time. .

    Run this on each of the parts:

    ffmpeg -i INCOMING.(FLV/MP4) -vcodec copy -acodec copy VIDEO_(01,02).mkv


    then:

    mkvmerge -o FULL_VIDEO.mkv VIDEO_01.mkv +VIDEO_02.mkv +VIDEO_03.mkv


    Results can vary depending on your source files!

    Thanks the tutorial. . .

    ReplyDelete
  21. Hi all,

    First of all Thanks for the great tutorial.

    Now for the shitty part... I'm running this on osx using ipfw and when I set the rule for port 1935, the flash player will load the stream from https instead, like it's understanding that I'm trying route it to localhost. When I add a second rule, this time IP specific I do see traffic going through it, but the player does nothing on the browser and I see no output on rtmpsrv :(

    Do you think it might have anything to do with me being on osx? Has anyone had any luck getting Colbert report lately?

    Thanks,
    Pedro

    ReplyDelete
    Replies
    1. Hello Pedro!

      Yes, apparently MTV changed something on their Adobe media servers and rtmpdump developers have not yet come up with a solution. I tried some older versions of rtmpdump and they all don't work for TDS & TCR.

      I can't really comment on the ipfw firewall issue though, I don't have access to any Macs. To see if the firewall really has anything to do with your issues it would be interesting to try dumping the video on Linux. Can you share the website address which is giving you grief?

      Delete

Yin & Yang!