Announcing Flazr – download RTMP (Flash video) streams

A quick post to announce my new open-source project: http://flazr.com

I had done most of the coding early this year and was really feeling bad about sitting on the code and not having time to package and release as open source. Judging by the number of hits I keep getting on this blog entry I made previously on “how to download RTMP streams with Red5” – there appears to be a lot of demand for this.

Flazr is a Java implementation of the RTMP protocol using the excellent Apache Mina project. I think I was able to come up with a far more concise and readable implementation than what the Red5 project uses – which is not that surprising – because the scope of Red5 is much bigger and Flazr focuses only on the client side. I do feel that Flazr will be useful as an additional reference for those interested in understanding the details of the RTMP protocol.

One of the highlights of Flazr (especially from a Java perspective) is the usage of Groovy for scripting. Groovy allows the end-user to script things such as scraping the HTML from a given URL, parsing it and then invoking the RTMP client routine with the right parameters. All this in a platform-independent manner, without the need to compile anything and using a normal text-editor. I expect Flazr’s Groovy approach to be much easier to use (and arguably more powerful) than the PERL-driven approach that projects like “get_iplayer” and “rtmpdump” use.

Flazr has been designed so that end-users can extend the capabilities far beyond what the core supports – using just some plain-text Groovy scripts. This means that end-users won’t need to depend on the project team (*ahem* just me for now :) to pitch in and make changes. I’m quite interested to see how this turns out in practice.

All in all I’m quite positive about Groovy’s useful-ness, the example scripts that come with the Flazr distribution demonstrate usage of the nifty XmlSlurper and implementing a Java interface on-the-fly. End users should be able to even plug in third-party libraries if they so desire and make use of all that the JVM offers.

How to record RTMP flash video streams using Red5

Update 2009-04-04: I’ve released an open source project that you can use to download RTMP video streams in a much simpler way. You can find it here: http://flazr.com

Red5 is an open source Flash server written in Java. It does not include a standalone client yet but I was able to write a Java program that uses Red5 to connect to an RTMP video stream and record / save it to a file. Code is provided below and also some tips on how to get the details required to download flash videos that you come across on the internet.

Here’s the code of “MyRtmpClient.java”, you are free to use and modify this in any way you want.

import java.io.File;

import org.apache.mina.common.ByteBuffer;
import org.red5.io.IStreamableFile;
import org.red5.io.ITag;
import org.red5.io.ITagWriter;
import org.red5.io.flv.impl.FLVService;
import org.red5.io.flv.impl.Tag;
import org.red5.io.utils.ObjectMap;
import org.red5.server.api.event.IEvent;
import org.red5.server.api.event.IEventDispatcher;
import org.red5.server.api.service.IPendingServiceCall;
import org.red5.server.api.service.IPendingServiceCallback;
import org.red5.server.net.rtmp.Channel;
import org.red5.server.net.rtmp.RTMPClient;
import org.red5.server.net.rtmp.RTMPConnection;
import org.red5.server.net.rtmp.codec.RTMP;
import org.red5.server.net.rtmp.event.AudioData;
import org.red5.server.net.rtmp.event.IRTMPEvent;
import org.red5.server.net.rtmp.event.Notify;
import org.red5.server.net.rtmp.event.VideoData;
import org.red5.server.net.rtmp.message.Header;
import org.red5.server.net.rtmp.status.StatusCodes;
import org.red5.server.stream.AbstractClientStream;
import org.red5.server.stream.IStreamData;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class MyRtmpClient extends RTMPClient {

	private static final Logger logger = LoggerFactory.getLogger(MyRtmpClient.class);

	private String saveAsFileName = "test.flv";

	public static void main(String[] args) {

		String host = "localhost";
		String app = "oflaDemo";
		final String name = "IronMan.flv";
		int port = 1935;
		final int duration = 10000; // milliseconds, -2 means until end of stream

		final MyRtmpClient client = new MyRtmpClient();
		logger.debug("connecting, host: " + host + ", app: " + app + ", port: " + port);
		
		IPendingServiceCallback callback = new IPendingServiceCallback() {							
			public void resultReceived(IPendingServiceCall call) {
				logger.debug("service call result: " + call);
				if ("connect".equals(call.getServiceMethodName())) {					
					client.createStream(this);
				} else if ("createStream".equals(call.getServiceMethodName())) {													
					Integer streamId = (Integer) call.getResult();
					logger.debug("createStream result stream id: " + streamId);
					logger.debug("playing video by name: " + name);						
					client.play(streamId, name, 0, duration);
				}								
			}						
		};
		
		client.connect(host, port, app, callback);

	}

	private RTMPConnection conn;	
	private ITagWriter writer;

	private int videoTs;
	private int audioTs;

	@Override
	public void connectionOpened(RTMPConnection conn, RTMP state) {
		logger.debug("connection opened");
		super.connectionOpened(conn, state);
		this.conn = conn;
		init();
	}

	@Override
	public void connectionClosed(RTMPConnection conn, RTMP state) {
		logger.debug("connection closed");
		super.connectionClosed(conn, state);
		if (writer != null) {
			writer.close();
			writer = null;
		}
		System.exit(0);
	}

	@Override
	public void createStream(IPendingServiceCallback callback) {
		logger.debug("create stream");
		IPendingServiceCallback wrapper = new CreateStreamCallBack(callback);
		invoke("createStream", null, wrapper);
	}

	@Override
	protected void onInvoke(RTMPConnection conn, Channel channel, Header header, Notify notify, RTMP rtmp) {
		super.onInvoke(conn, channel, header, notify, rtmp);		
		ObjectMap<String, String> map = (ObjectMap) notify.getCall().getArguments()[0];
		String code = map.get("code");
		if (StatusCodes.NS_PLAY_STOP.equals(code)) {
			logger.debug("onInvoke, code == NetStream.Play.Stop, disconnecting");
			disconnect();
		}
	}	

	private void init() {
		File file = new File(saveAsFileName);
		FLVService flvService = new FLVService();
		flvService.setGenerateMetadata(true);
		try {
			IStreamableFile flv = flvService.getStreamableFile(file);
			writer = flv.getWriter();
		} catch (Exception e) {
			throw new RuntimeException(e);
		}
	}

	private class CreateStreamCallBack implements IPendingServiceCallback {

		private IPendingServiceCallback wrapped;

		public CreateStreamCallBack(IPendingServiceCallback wrapped) {
			this.wrapped = wrapped;
		}

		public void resultReceived(IPendingServiceCall call) {
			Integer streamIdInt = (Integer) call.getResult();
			if (conn != null && streamIdInt != null) {
				MyNetStream stream = new MyNetStream();
				stream.setConnection(conn);
				stream.setStreamId(streamIdInt.intValue());
				conn.addClientStream(stream);
			}
			wrapped.resultReceived(call);
		}

	}

	private class MyNetStream extends AbstractClientStream implements IEventDispatcher {

		public void close() { }

		public void start() { }

		public void stop() { }

		public void dispatchEvent(IEvent event) {
			if (!(event instanceof IRTMPEvent)) {
				logger.debug("skipping non rtmp event: " + event);
				return;
			}
			IRTMPEvent rtmpEvent = (IRTMPEvent) event;
			if (logger.isDebugEnabled()) {
				logger.debug("rtmp event: " + rtmpEvent.getHeader() + ", "
						+ rtmpEvent.getClass().getSimpleName());
			}
			if (!(rtmpEvent instanceof IStreamData)) {
				logger.debug("skipping non stream data");
				return;
			}
			if (rtmpEvent.getHeader().getSize() == 0) {
				logger.debug("skipping event where size == 0");
				return;
			}
			ITag tag = new Tag();
			tag.setDataType(rtmpEvent.getDataType());
			if (rtmpEvent instanceof VideoData) {				
				videoTs += rtmpEvent.getTimestamp();
				tag.setTimestamp(videoTs);
			} else if (rtmpEvent instanceof AudioData) {
				audioTs += rtmpEvent.getTimestamp();
				tag.setTimestamp(audioTs);
			}
			ByteBuffer data = ((IStreamData) rtmpEvent).getData().asReadOnlyBuffer();
			tag.setBodySize(data.limit());
			tag.setBody(data);
			try {
				writer.writeTag(tag);
			} catch (Exception e) {
				throw new RuntimeException(e);
			}
		}
	}

}

To compile and run the code you need the following libraries:

commons-collections-3.2.jar
jcl104-over-slf4j-1.4.3.jar
logback-classic-0.9.8.jar
logback-core-0.9.8.jar
mina-core-1.1.6.jar
red5.jar
slf4j-api-1.4.3.jar
spring-beans-2.0.8.jar
spring-context-2.0.8.jar
spring-core-2.0.8.jar

We need the latest Red5 JAR built from version control (not the latest official release) and I have uploaded these files here: red5-rtmp-client-libszip.pdf – just rename the file to end with “.zip” after downloading.

The code shown is hard-coded to connect to “localhost”, app name “oflaDemo” and stream name “IronMan.flv” which can be tested on the official Red5 server distribution which you can download and run. When it comes to downloading flash streams from the internet, WireShark can be used to sniff out the values you need.

Let’s take this site for example: http://videolectures.net/ Start a WireShark capture session before clicking on a video on the site to play it. Let WireShark grab all the information exchanged between your PC and the remote flash server and you can stop the capture once the video begins to play, we are only interested in what goes on during the connection handshake. I will use this video as an example: http://videolectures.net/ff06_chomsky_szmp/ [update Jun-2008: looks like they changed this particular video to Windows media instead of Flash, so try other videos or other sites]

In WireShark you can filter for protocol “rtmpt” and the first few entries would be handshake or “invoke” operations. Examining the “Handshake part 3” we can easily get the value of the required “app” property. Below we can see it is “video/2006/other/ff06/chomsky_noam”:

For the host name, the IP address should do fine for most sites, but we can easily figure out the host name of the stream server from what appears after “rtmp://”. Note that WireShark allows you to search the text contents of captured packets. Here below we can see that the host name is “velblod.videolectures.net”:

And finally when the “play” command is issued – we need the value of the stream name. Below we see it is “chomsky_noam_01”:

So with the right values of hostname, app and stream name set – you can run the program and download the stream to your local drive for offline viewing. To download the whole stream – just change the duration to ‘-2’ as hinted in the source code comment. There are many free Flash players available you can use to play downloaded content such as FLV Player.

Do let me know if this works for you and if you find any additional parameters that need to be passed for other sites.