Export a timeline

Send a descriptor of your edit to ffmpeg.timeline_export.v1: clips on a canvas, a sound mix, and captions. PennyCompute returns one finished MP4.

Describe the picture

Set the canvas: aspect_ratio 16:9, 9:16, 1:1 or 4:3, resolution 720p, 1080p, 4k or proxy_480p, and fps from 1 to 120. List the clips in order, each a public link with optional in_seconds and out_seconds, a volume and a fit. For stills, speed changes and plain colour, list picture_segments instead: a video, an image with an optional crop and camera move, or a placeholder.

Mix the sound

Add tracks to mix: narration, music, sound_effects, user_audio or an overlay_video, each with where it starts, how long it plays, its volume and fades. music_duck_db lowers music under narration, and primary_volume_under_narration lowers the clips’ own sound.

Request body
{
  "service_id": "ffmpeg.timeline_export.v1",
  "input": {
    "canvas": {
      "aspect_ratio": "16:9",
      "resolution": "1080p",
      "fps": 30
    },
    "clips": [
      {
        "url": "https://example.com/shot-1.mp4",
        "in_seconds": 0,
        "out_seconds": 6.2
      },
      {
        "url": "https://example.com/shot-2.mp4",
        "fit": "cover"
      }
    ],
    "mix": {
      "music_duck_db": -10,
      "tracks": [
        {
          "kind": "narration",
          "url": "https://example.com/voice.wav",
          "duration_seconds": 12,
          "fade_out_seconds": 0.5
        },
        {
          "kind": "music",
          "url": "https://example.com/bed.m4a",
          "duration_seconds": 20,
          "volume": 0.6
        }
      ]
    },
    "captions": {
      "lines": [
        {
          "start_seconds": 0,
          "end_seconds": 2.4,
          "text": "Point Reyes, at dusk."
        }
      ]
    }
  }
}

Burn captions and finish the sound

captions takes up to 300 lines with a start, an end and text, drawn in the style you set. style.font names the face, Arial when it is left out. Renderers have system faces such as Arial, Helvetica, Times New Roman, Georgia, Verdana and Courier New, and DejaVu Sans in regular and bold. They have no other DejaVu face, such as DejaVu Sans Oblique, DejaVu Serif or DejaVu Sans Mono, and no Liberation or Noto faces; a face a renderer lacks is drawn in a substitute the renderer chooses, Helvetica when tested, and the job does not fail or say so. sound sets the AAC bitrate, sample rate and channels, and peak_ceiling_dbfs, the level the mix is limited at.

Finish the sound of a video you already have

Send picture: copy with source_url to keep a finished video’s picture untouched. Add source_audio_url to replace its sound, or tracks to mix over it. source_url is an MP4, MOV or Matroska file whose picture is H.264 or H.265; a picture an MP4 cannot carry, such as ProRes, fails the job with runner_failed, so send that video as one clip with picture: encode. With source_audio_url the source’s own sound is dropped: the replacement plays at mix.primary_volume and tracks mix over it. A source with no sound is accepted: tracks play over silence, and with no tracks the output has a silent sound track. duration_seconds is optional here: omitted, the output is as long as the picture; given, it has to be within 0.2 seconds of the picture’s length.

Submit and retrieve

Send the descriptor as input to POST /v1/jobs with an Idempotency-Key. Unknown fields are refused, and a refused request lists each field and why in issues. When the job succeeds, GET /v1/jobs/{id}/output returns the link to the MP4, and the job’s result carries its duration, width and height and a report on the sound.

Price and limits

An export is billed like other FFmpeg work, for the length of its output by size: $0.00375 per minute for SD, $0.0075 for HD and $0.015 for 4K, by the second with a ten-second minimum. A descriptor can have up to 240 clips or picture segments, 512 sound tracks and 3600 seconds of output. Every source is a link of up to 1 GB.

Picture: clips, segments and length

When any picture_segments are given the picture is made from them and clips are not used; otherwise the picture is the clips joined in order. duration_seconds is the total length: the picture is padded with black and the sound with silence up to it, then both are cut at it. With picture: copy, source_url names one finished video whose picture is kept untouched and whose size is the output’s; canvas, clips and picture_segments are not sent, and the job fails when the picture’s length differs from duration_seconds by more than 0.2 seconds.

Picture: which source frame is shown

The output has a constant frame rate and is as long as asked, to the frame. Output frame n of a clip shows the source frame on display at the clip’s in-point plus n output frames: the last source frame that begins at or before that instant, or up to one millisecond after it, on any time base. That millisecond is there for files whose frame times are rounded, such as a 30 fps file with times written to the millisecond or a WebM file: every frame of such a file is shown once. An in-point that sits within a millisecond before a frame’s start begins on that frame. A clip whose in-point falls between two source frames starts on the earlier one, so an in-point of 0.2 seconds in a 24 fps clip, frame 4.8, starts on frame 4; an in-point on a frame starts with that frame. A change of rate repeats or drops frames by the same rule: a 24 fps clip on a 30 fps canvas shows source frames 0, 0, 1, 2, 3, 4, 4, 5, 6, 7, 8, 8 and so on. A look or an opening fade does not move the frames. A video picture segment follows the same rule with its speed in it: output frame n shows the source frame on display at source_in_seconds plus n times playback_rate divided by the frame rate, with the same millisecond counted in source time, so a speed above 1 skips source frames and a speed below 1 repeats them. A look on a segment counts in source time from the in-point, and each output frame takes the look in effect at its own source instant: a look that starts at 0.5 seconds, at 1.5 times the speed and 24 fps, first shows on output frame 8. A segment’s sound starts at the in-point and plays at the speed. in_seconds and source_in_seconds count from the file’s start, the earliest start of its picture or its sound, and picture and sound are cut at the same file time. A picture frame is shown from the first output instant at or after its own start, so in a file whose picture begins a few milliseconds after its sound, as in a WebM with Opus sound, the picture runs up to one output frame behind the sound and never ahead of it. Where a file’s picture begins later than its sound, its first frame is shown until then. One limit: in a source whose frames are more than a second apart, a clip or a segment with an in-point between two of them starts on the later frame.

Picture: sources whose frame times wander

A camera, a phone or a browser recording often writes each frame a few milliseconds early or late, around a steady rate. Such a source is first put on that rate’s grid, and the rule above then applies: every frame gets its own place, so a 30 fps recording on a 30 fps canvas shows each frame once, at any in-point. The rate is read from the frame starts themselves, not from the rate the file states, and is taken to a usual rate such as 24, 25, 29.97, 30, 50, 59.94 or 60 when it is within 0.4 percent of one. Each frame start moves to the nearest place on the grid, two frames that share a place show the later one, and real gaps stay gaps. An in-point keeps its distance from the start of the frame on display at it, so an in-point given as a frame’s own start begins on that frame. A source whose frames are within 0.75 milliseconds of a steady grid, exact or rounded to the millisecond, is left exactly as it is. A source with no steady rate, or one whose frame starts sit about a third of a step or more from any grid, as a 60 fps browser recording can, is also left as it is and follows the rule on its own times. The reference file below is the definition: the rate, the grid, each frame’s place and the in-point, as code.

Picture: encode, fit and camera

An encoded picture is H.264 in yuv420p with the index at the front of the file, at CRF 23 for 720p, 18 for 1080p, 20 for 4k and 30 for proxy_480p. The encoder is libx264, at preset fast for proxy_480p and 720p and veryfast for 1080p and 4k. With captions or an overlay_video the picture is encoded a second time at the same CRF; with only a sound mix it is encoded once. canvas.crf, from 0 to 51, and canvas.preset, a libx264 preset from ultrafast to veryslow, replace those defaults for every picture encode in the job and do not change the price. A copied picture is not encoded and takes neither. A clip is letterboxed on black unless its fit is cover. A video segment is always letterboxed. An image with a crop is stretched to the canvas; without one it is scaled to cover and cropped to the centre. The camera on an image eases out along a cubic curve, with scale between 1 and 1.2 and pan within 3 percent.

Colour

A clip or a video picture segment may carry luts, each a colour table by link with the stretch it covers. Times count from the piece’s in-point, in its source time before any change of speed, start included and end excluded. Tables are read trilinear and applied in RGB after frames are fixed to the output rate; the picture is then scaled to the canvas and taken to 4:2:0 in one step. Input with no colour matrix tag is taken as BT.709 and its samples are not converted. A full-range input is brought to limited range. Input tagged with another matrix, such as BT.601, or coded as RGB is converted to BT.709. HDR input is refused, and the output is tagged BT.709, limited range. A clip’s luts are refused when picture_segments are given: put the looks on the segments.

Sound: tracks and ducking

An overlay_video is drawn, and its sound plays when its volume is above 0. A track’s sound is required, where a source with no sound fails the job, or optional, where it is skipped. A clip with no sound gets silence, and mono becomes stereo. Narration lowers music by music_duck_db and the clips’ own sound to primary_volume_under_narration. The duck is a side-chain compressor, not a step in level: threshold 0.01, attack 120 ms, release 350 ms, and a ratio of the decibels divided by 1.5, kept between 2 and 20, so how deep it goes follows how loud the narration is. After the last narration ends the ducked sound returns to full level over the release. user_audio and sound_effects are never ducked, and music is ducked only when music_duck_db is below zero.

Sound: finishing

The mix is limited at peak_ceiling_dbfs, with 4x oversampling, a 5 ms attack, a 50 ms release and no make-up gain, then encoded, decoded and measured. When the decoded peak passes 0 dBFS the mix is encoded again, lower by the excess plus 1 dB, up to three times and 24 dB; a fourth overshoot fails the job. peak_ceiling_dbfs: null skips the limiter and the correction. sample_rate and channels may be source, with picture: copy, to keep the soundtrack’s own. edit_list: false writes the file with no edit list.

What fails an export

After a renderer has started, an export fails with invalid_input when a required source has no sound, when the finished sound is not in the codec, rate and channels asked for, when the sound’s length is off by more than a quarter of a second, or when the picture is shorter than 85 percent of the expected length. A link that cannot be fetched fails it with download_failed.

The result report

A finished export’s result has duration_seconds, width and height as measured, and sound: codec, sample_rate, channels, bitrate_kbps as measured on the file, duration_seconds, true_peak_dbtp, integrated_lufs, peak_correction_db, how much lower the final encode is than the first, and correction_passes, how many times it was encoded again. A silent or nearly silent mix measures a very low bitrate_kbps, a few kbps, whatever bitrate was asked for.

Every limit

Up to 240 clips, 240 picture segments, 512 sound tracks and 32 looks on a piece. Clip volume 0 to 2 and other volumes 0 to 8. A track plays at 0.25 to 4 times speed and a video segment at 0.5 to 4. Fades up to 30 seconds. Captions up to 300 lines of 240 characters. Output up to 3600 seconds, at 1 to 120 frames a second, on a 16:9, 9:16, 1:1 or 4:3 canvas.