My PhD (before 10+ years) was in video search and one of the proposed methods for video comparison was using the shot durations of the video. Notice that with shots I refer to cuts in the camera flow, IIRC hollywood movies have a such a shot cut every 4 seconds as an average (for example when two people talk the camera will move from one person to the other).
I remember that I used two techniques for extracting scene cuts:
* Difference in the brightness (Y) histogram of the YUV video between frames; when that difference is more than a threshold there's a scene cut
* Counting the number of Intra macroblocks per frame on an H.264 encoded video; if that number was more than a threshold then there's a scene cut
Author of PySceneDetect here. The current implementation does exactly what you hint at, except instead of YUV, it considers deltas in the HSV domain (specifically differences in hue and color).
Other techniques being considered for future work include use of optical flow, background subtraction, and analyzing histograms.
From what I remember the Y (luma) component in a YUV video has more information than the other two components and it could also be extracted without the need to fully decompress the video (in mpeg compressed videos). Of course this info is more than 10 years old (I don't really do any video research any more) so I guess there should have been progress in that area.
This is indeed correct, I'm just using HSV instead of YUV, but the primary source of information is the luma/brightness component (although currently all 3 of the HSV components are averaged, so perhaps a better weighting may improve precision).
The image for an MPEG compressed video is splitted in 16x16 blocks. For each frame (excluding the 1st of course), the compressing algorithm tries to to matchthat particular block with a block in the previous frame (it searches the previous frame to see where there are the fewest differences). If it can do it it only encodes the differences and the position in the previous frame; this is called an inter(predicted) block. If however it can't match that block with the previous frame then it needs to re-encode from scrach; that's the intra macroblock. As you can understand after a shot cut there will be much more intra macroblocks.
Even if that is fixed, a link to the repo on the left column[1] would be nice. Maybe you don't want to edit that page, just encountered the docs first, e.g. from HN :D.
This project sounds pretty interesting, I'll take a look!
Funny you should mention that - initially home videos were what I was doing this with, as I was unable to find any existing programs that could handle them reliably (or had various shortcomings, or weren't hackable/extensible, etc...). Then I realized that what I came up with actually competed pretty well with the existing solutions (benchmarks below [1]), so decided to open source it and share it with the world.
Of course, the project is still very basic in it's current form, but does offer a good platform for testing more advanced scene detection methodologies (and has also been used as a baseline in various research/academic contexts). There's been plenty of suggestions for other detection methods (e.g. using histograms) which I'm very interested in looking into adding for future releases.
Lastly, one thing I do want to address going forwards is the performance, to make the algorithms more suitable for real-time systems (have decided to rewrite the core algorithms in C++ when I have some time, and interface that with the Python library/CLI). That being said, I'm always open to feedback or different ways of approaching things. If you or anyone else has any feedback or suggestions, feel free to create a new issue on the Github issue tracker [2] and I would be happy to discuss the possibility of including it in a future release of PySceneDetect.
Author of PySceneDetect here: Thank you all for the thought-provoking discussions, and the attention you've given my side project. There are some specific cases where PySceneDetect achieves great accuracy (like fast cuts or fades), and some where it's currently not so good at (like sudden flashes or large obstructions). That being said I do want to track these things and come up with solutions to improve the robustness of the content detection algorithm over time.
I'm most open to any feedback or feature requests/ideas/suggestions; feel free to checkout the issue tracker on Github, or create a new entry:
So this is likely beyond the scope of your project, but I've always thought a really good project would be a website to host scene indexes for movies and TV.
Eg. Let's say that you wanted to watch a prerecorded football game or baseball game without all the commercials, timeouts, commentators talking about the fans, etc.
Or... Let's say that you wanted to re-cut a movie in a certain way, by re-ordering the scenes, you could just generate a new scene data file and let the encoder/player use that.
This is still relevant I think :) What you mentioned is very similar to an edit decision list (EDL [1]), of which I only learned recently. I had a feature request [2] to support EDL as an ouptut format, and upon further investigation, it seems like the format is very similar to what you're talking about. The Wikipedia page also indicates that VLC supports xspf files, but I haven't done much research into that yet ("XML Shareable Playlist Format").
So I can look forward to a "Next scene" shortcut in my video player of choice soon? (Without needing embedded "DVD" chapters) While also getting nice thumbnails for them?
Pornhub et al often show scene markers, but I have assumed they're manual or extracted in the same way as DVD chapters.
Tested it out on an anime episode to see if it could detect accurately when the opening and closing credits play. It seemed to be able to detect the ending of both even with a high threshold, but it didn't really detect the start. And with a low threshold, it gave me like 350 chapters for a 22 minute video, which seemed a bit excessive.
Would you be able to share a small sub-set of the episode, in particular the area where you're unable to detect the starting segment? (If not, no worries!)
There's a few issues with PySceneDetect currently that may lead to false or missed detections, but these are things that I would like to solve in the long run:
- threshold is heuristic/fixed right now, but I would like to change it to an adaptive/statistical method which can dynamically change
- single-frame events can trigger false scene changes
Thanks for your feedback, and feel free to share any other suggestions you might have.
I'm trying to help a friend in media industry. His requirement is ability to identify different voice in a movie and have the output time stamped - Ex. Voice A: 00.00.00sec - 00.05.30sec, Voice B: 00.05.31 - 00.06.30, etc.
It would be very helpful if anyone can point to any tools that exists that can do that (open source or otherwise).
If AWS gets you going definitely go for it but if you're looking for existing tools type 'python speaker diarization' in github.'
Most of what you'll find would require downsampling your audio to 16khz and you'll find a combination of NN based diarizer and hmm based models pre-ML.
One thing to note a lot of the systems will work well for interviews, broadcast media and footage taken from a camera because the audio will tend to be clean.
Film and movies will be a challenge because of the background music being identified as a separate voice. It tends to to confuse it.
Haven't used the cloud based systems with audio the background sounds you tend to find in film movies.
Transcribe seems to be more for speech-to-text irrespective to who is making the speech.
Here the requirement is to identify the unique voices. Ex. if "Mary had a little lamb" is voiced by two different voices then the engine should identify Voice A said "Mary" at 00.00.00sec-00.00.01sec and then Voice B said "had a little" at 0.00.02sec-00.00.03sec, then Voice A again said "lamb".. etc.
Amazon Transcribe is able to recognize when the speaker changes and attribute the transcribed text appropriately. This can significantly reduce the amount of work needed to transcribe audio with multiple speakers like telephone calls, meetings, and television shows.
So they claim to be able to do it. However, having to do speaker diarization doesn't exactly make speech recognition easier, so you should adjust your expectations regarding the error rate accordingly.
And then came across this comparison between various cloud transcription service, which should be helpful for evaluating on diarization aspect as well.
Transcribe does “Recognize Multiple Speakers” — it’s on the page that you linked to, in the list of features.
I would also recommend comparing it to Google’s and maybe MS/Azure’s services.
Aside from the fidelity of the transcription itself, and the accuracy of disambiguating the different speakers, I’m also not convinced that all of these services will give you an end timestamp (start timestamps are sometimes there, but not necessarily for every word or sentence).
You could try a multi prong approach by doing the transcript to get text and speakers only (using the services mentioned above), and then using an aligner such as “gentle” [0] to find the start / end times.
You will have some gaps and wrongly transcribed words, but it may be a start..!
Just curious, what didn't you like about the output quality? What version of PySceneDetect were you using?
The latest version (v0.5.x) uses ffmpeg instead of mkvmerge for output by default now, which produces significantly more accurate and higher quality output than before.
That being said, you are correct in that optical flow and keyframe information is currently not being used during the detection phase. There are several proposals to incorporate this into a future release, however, along with several other techniques:
- histogram analysis
- edge detection
- background subtraction
- automatic or dynamic threshold detection
This was my assignment in the "artificial intelligence" course I took at my University in Naples (Italy) in 2003.
My NN could recognise scenes cuts with about 50% probability :)
a Film is a collection of shots, scenes and sequences. Shots are fairly easy to detect. Most editing software comes with built in shot detection.
This approach promises to detect scenes. I’m not enough of a coder to know if it is valid, but I can certainly imagine that it is a solvable problem. Most scenes are defined by location, and movies makers like each locatation to be distinct from the preceding one.
To my mind, sequences are where the action is at. Sequences are defined by narrative development... These can usually be defined by a trained human, but are probably tough to computationly define.
There is also a huge difference between film genres. The scenes and sequences of Finding Nemo are pretty easy to define. But you try the same approach on an art house film like The Scent of Green Papaya, and see how far you get.
Does this program really detect scenes by the film-making definition? In video encoding terminology, shots are commonly called "scenes". E.g. x264's "scenecut" option really sets the sensitivity for detecting new shots, so that I-frames can be inserted there to avoid the inefficiency of encoding differences from a different shot.
Author of the program here. You are correct, the program detects shots rather than scenes, but I didn't want to give the impression that this project was related to the existing ShotDetect program. I felt that the documentation explained this well enough, but I'm open to considering an alternative project name if anyone has a suggestion.
I had fun some years ago working on some sentiment analysis, and found if you took subtitles you could often see where general changes happened. Splitting that over where the histogram jumped you could more reasonably split things over camera cuts. Plot an exponentially smoothed sentiment and you could see patterns for types of content (sadness then uplift at the end of current affairs shows for example).
I built it for auto-summarisation of video and categorisation as a few day hack and it was OK (better than I expected, not good enough to use). Totally failed at sports though, as the commentary usually has more varied speech when there's little happening.
Detecting scenes seems incredibly difficult to me. Even in a basic edit you could have back to back shots looking at very different parts of a single set. Take the "Jack Rabbit Slims" scene from Pulp Fiction - Vincent Vega walking through the diner, the car table head to head scene about the milkshake with linking cuts around the diner when they discuss the two Marilyns, and the beginning of the Twist competition are three scenes that use the same location, actors, background, etc. Finding the boundaries would be very hard.
One approach might be to estimate the intrinsic parameters of the camera based on keypoint tracking. Assuming the camera was adjusted between shots, you might be able to get that to work.
Or you could make some kind of generic feature vector from each frame that includes contrast, etc. The interiors are the same, but the camera focus/fov will likely be different.
Nowadays just take an annotated movie and throw it into an encoder/decoder network to assess if a frame is from the same scene?
Surprisingly, this uses a fairly simple heuristic to accomplish just that (HSV deltas). I'm the author of the program, and it actually works pretty well with Pulp Fiction as a whole.
The one area where PySceneDetect does struggle with currently, however, is dealing with sudden/rapid flashes, or momentary obstructions to the viewing scene.
It would be kind of awesome if we could “hash tag” scenes such that at some point we could search fot “remember that #action scene in movie X where the [something happens]” and it can pull up that clip of the scene...
You could get pretty close to that by allowing people to bookmark and tag timestamps in films. In fact I'm pretty sure we are close to being able to do it, but implementations lag because it would be a feature for services that do not generally compete on features but only on content.
I've seen a proof of concept which combined the output of PySceneDetect with subtitle information and computer vision to allow you to do something like "go to the scene with the big castle" or something similar. I can't remember what it's called off the top of my head, but it seemed like a pretty cool concept.
Author of the program here. This is definitely something that is worth while considering, although I'm not too sure if a correlation always exists between the audio and picture patterns (e.g. in a music video, scene transitions may not be exactly synchronized with the audio).
I've also seen people combine the cuts detected by PySceneDetect with subtitle (.srt) information to generate a more comprehensive list of scenes, which are sets of shots joined together by some context (to allow for jumping forward to a particular scene by it's description).
Beyond all that, good filmmakers often use edits to manipulate or confound expectations about time and space.
However, there are scenes in the narrative sense, but also scene main the way an AD making a schedule will see them, that is, scene headings (sluglines), which would be easier to differentiate and detect.
Author of the program here. Yes, the program detects shots rather than scenes, but I didn't want to give the impression that this project was related to the existing ShotDetect program. I felt that the documentation explained this well enough, but I'm open to considering an alternative project name if anyone has a suggestion.
Odd, scene differentiations often seemed like arbitrary slices just meant for trying to get to a certain point in the movie, long ago when I'd look at them on DVD's. Not something where you could objectively say "this is where one scene ends and another begins" and others would come to the same conclusion on their own.
Can it detect subliminal frames? Would an adequate threshold also lead to many false positives (triggering on any brutal massive change, for example a blast-inducing explosion)?
You could differentiate single-frame inserted "subliminal" (like the ones in Fight Club) images from other flashy things like explosions, by the fact that the frames right before and after the flash should be nearly identical or very similar. While after an explosion, things usually changed a bit.
This is definitely a good idea, and something that I'm most open to considering for a future release of PySceneDetect. Admittedly the current version does not handle single-frame "upsets" like this, but this does seem like a logical and reasonable approach to a first attempt at filtering them out.
I would do an exponential smoothing of pixel values over some timescale, say, 0.2 seconds, before further detection of scene changes. That should do the trick.
I remember that I used two techniques for extracting scene cuts:
* Difference in the brightness (Y) histogram of the YUV video between frames; when that difference is more than a threshold there's a scene cut
* Counting the number of Intra macroblocks per frame on an H.264 encoded video; if that number was more than a threshold then there's a scene cut