How auto-highlights work

One of StreamHero's most-requested features is automatic highlight detection. Here's a look under the hood at how we find the best moments in your streams.

The pipeline

When you submit a VOD URL, StreamHero runs the following steps:

1. Audio transcription — We transcribe the full broadcast using AssemblyAI, which gives us the complete spoken word timeline.
2. Chat event detection — We fetch the VOD's chat replay and look for statistically significant spikes (hype trains, mass reactions, subscriber surges).
3. Scene analysis — A vision model samples frames at regular intervals and classifies the type of content happening (gameplay, facecam reaction, cutscene, etc.).
4. Moment scoring — Each candidate moment is scored across several dimensions: chat activity, audio energy, transcription sentiment, and scene type.
5. Clip extraction — The top-scoring moments are padded with a few seconds of context and queued for video processing on our Cloud Run workers.

Rarity tiers

Not all highlights are equal. We classify each clip into one of three rarity tiers:

• Common — A good moment; solid content worth keeping.
• Rare — An above-average moment with strong chat engagement or dramatic audio spike.
• Epic — A once-per-stream event. These are the clips that go viral.

Potential moments

Beyond confirmed highlights, StreamHero also surfaces potential moments — segments that scored above average but didn't quite make the highlight cut. These are useful when you want to personally review borderline clips before publishing.

Pro and Premium subscribers see all highlights and potential moments. Free accounts see up to 3 highlights and 4 potential moments.

What's next

We're actively working on improving the scoring model with more signal sources, including viewer count data and real-time engagement metrics. Follow this blog for updates.