#phonecamera #linuxphone #librem5
Librem 5 camera/kernel can do three possible resolutions, ~1024x768 @ ~24fps, ~2048x.. @ ~31 fps and ~4096x.. @ ~15fps. Debayering is actually easier and better quality if we downscale at the same time, and that allows best framerate, so we do that (2048x.. resolution).
ARM has problems with cache coherency w.r.t. DMA, and kernel solution is to simply disable cache on DMAbufs for userspace, which means accessing video data is 10x slower than it should be on the CPU. Which means debayering on GPU is attractive, and that's what we do. (gold.frag). GPU can do more image signal processing functions easily, too, so we do some of that.
Unfortunately, we hit the same uncached memory problem at the GPU output. So we use separate thread to copy. All this unfortunately does not fit on one core, so we need two threads, one controlling GPU debayer on frame n+1, while the other one copies video data from frame n. (heart.c). We save resulting RGBA data to ramdisk. This all costs maybe 80% of one core.
From there, Python scripts can pick them up: ucam.py displaying the viewfinder and mpegize.py handling the video encoding via gstreamer. There's basically 0% cpu left, but I can encode ~1024x.. video. Unfortunately that's without audio and with viewfinder at 1fps. Plus, combination of C + Python is great for prototyping, but may not be that great for performance.
Code is here: gitlab.com/tui/tui/-/tree/mast… .
At this point I'd like viewfinder functionality merged into the rest of GPU processing. Ideally, I'd like to have a bitmap with GUI elements, combine it with scaled RGBA data, and rendering it to screen. I know SDL and Gtk, SDL looked like better match, but I could not get SDL and GPU debayering to work in single process (template SDL code is here gitlab.com/tui/debayer-gpu/-/b… ).
If you can integrate main.c and heart.c, that would be welcome. If you have example code that combines SDL with processing on GPU, that would be nice, too. If you know someone who can do GPU/SDL, boost would not be bad, I guess.
Sebastian Krzyszkowiak
in reply to Pavel Machek • • •I played with it over the last days and I already have 526x390 30 FPS encoding with live viewfinder, perfectly synced audio, color correction, lens shading correction, tone mapping, AWB and AE - consuming about 40% CPU. Still needs chromatic shading correction and AF, and I started experimenting with enabling PDAF.
I can also make the sensor output 60 FPS and RAW10. Patches incoming ;) Still only 2 MIPI lanes though, so no 13MP 30FPS yet.
Sebastian Krzyszkowiak
in reply to Sebastian Krzyszkowiak • • •Sebastian Krzyszkowiak
in reply to Sebastian Krzyszkowiak • • •source.puri.sm/Librem5/linux/-…
s5k3l6xx: Read OTP AWB/AF data (#411) · Issues · Librem5 / linux · GitLab
GitLabSebastian Krzyszkowiak
in reply to Sebastian Krzyszkowiak • • •Pavel Machek
in reply to Sebastian Krzyszkowiak • • •@dos I hope you played with it in your day job for about 8 hours a day, because otherwise I'll feel kind of bad 😀. Thanks!
Do you also get constant clicking from the AF system? I thought it was gone on Mobian, but then it came back.
526x390 is easy, can you do 1052x..? Is it possible to peek at sources somewhere?
Thanks again.
Sebastian Krzyszkowiak
in reply to Pavel Machek • • •It's 526x390, but properly binned (each channel's sample consists of 4 raw pixels averaged), which reduces noise. The shader got heavy though, does only ~35 FPS at this res - but there should be room for optimization. I've been more concerned with its correctness than performance so far.
Stats counting on the CPU with NEON is fast enough for full frame with some subsampling.
I'm giving it some finishing touches and will then publish it of course 😁
Sebastian Krzyszkowiak
in reply to Sebastian Krzyszkowiak • • •Also, my stats are center-weighted. Millipixels annoyed me with its reluctance to overexpose the sky 😄
Pavel Machek
in reply to Sebastian Krzyszkowiak • • •Sebastian Krzyszkowiak
in reply to Pavel Machek • • •Pavel Machek
in reply to Sebastian Krzyszkowiak • • •Sebastian Krzyszkowiak
in reply to Pavel Machek • • •Pavel Machek
in reply to Sebastian Krzyszkowiak • • •Sebastian Krzyszkowiak
in reply to Pavel Machek • • •Sebastian Krzyszkowiak
in reply to Sebastian Krzyszkowiak • • •Pavel Machek
in reply to Sebastian Krzyszkowiak • • •@dos I have feeling that you'll post few more advancements and then you'll disappear, with that code being gone forever. Probably abducted by aliens as that's clearly alien code 😀.
Please publish it somewhere.... we don't want another "Fermat's last theorem" situation.
Also, yes, at 512x.. resolution, things are likely easier. I had a lot of fun trying to get gtk/gstreamer working, and while I got it to work _somehow_, I was not really able to understand/debug performance issues.
Sebastian Krzyszkowiak
in reply to Pavel Machek • • •Displaying the viewfinder is free with (gl)waylandsink, as the dmabuf gets passed directly to the compositor in a subsurface, so the resolution is not important - it goes straight to the GPU for compositing. It's important for encoding and that's where uncached buffers can bite you, but since my shader is currently too heavy to handle higher res anyway it's not a thing I concern myself with right now.
And the code is basic, it just takes time to get familiar with the problem space 😜
Pavel Machek
in reply to Sebastian Krzyszkowiak • • •Sebastian Krzyszkowiak
in reply to Pavel Machek • • •Outputting AYUV is trivial and makes encoding slightly faster, but then you lose the viewfinder for free as that's not a format that can be passed to the compositor.
Ideally we would output planar YUV (perhaps with multiple render targets?) as that can be sampled from with recent etnaviv: gitlab.freedesktop.org/mesa/me…
For now I'm limiting myself to what's possible on bookworm/crimson though 😀
etnaviv: add multi-planar YUV support (!3418) · Merge requests · Mesa / mesa · GitLab
GitLabPavel Machek
in reply to Sebastian Krzyszkowiak • • •@dos So, this is what I was targetting with my "gstreamer will downscale UV components" comment: GPU is doing heavy computation to make sure you have great color values for each pixel, only for gstreamer to discard it in the next step.
Anyway, I'd love to see the code 😀.
Sebastian Krzyszkowiak
in reply to Pavel Machek • • •Pavel Machek
in reply to Sebastian Krzyszkowiak • • •@dos Lens corrections -- are we speaking lens shading or geometry corrections?
I am still eager to see your code. I'd like to try to tweak it for 1024x.. resolution, to get good quality videos.
Sebastian Krzyszkowiak
in reply to Pavel Machek • • •Both, including chromatic shading. They're not that bad in still photos, but I find them very distracting when in movement.
I've got a song for you, but you'll have to translate it yourself 😁 youtube.com/watch?v=FdIid5IJEd…
I'm relying on kernel changes, so I need to put everything in place first. It also took some time for me to get reasonably confident that what I'm doing is correct. I'm pretty much done though, just some cleanups left to do.
Nie poganiaj mnie bo tracę oddech - Maanam
YouTubeSebastian Krzyszkowiak
Unknown parent • • •@lord Specialized hardware. Phones usually don't have to do image processing nor video encoding on CPU or GPU at all, they have hardware ISP and encoders. L5 does not.
On other hardware there's also a matter of whether there's driver and software support for it, so Android may be able to use that, but pmOS - not necessarily.
Sebastian Krzyszkowiak
in reply to Pavel Machek • • •By the way, the actual framerate limits (verified to work) are 120 FPS @ 1052x780, 60 FPS at 2104x1560 and 20 FPS (8-bit) or 15 FPS (10-bit) at 4208x3120. Full res could go up to 30 FPS, but only once we get 4 lane configuration to work.
I can even produce 120 FPS video, although it doesn't skip frames only once downscaled to 127x93 right now 😂
Pavel Machek
in reply to Sebastian Krzyszkowiak • • •@dos I guess that for 120 fps video, you'd want to capture a burst to available memory, and do the conversion after the capture is done.
Could we get some intermediate (maybe cropped) modes, too? For digital zoom, or maybe for cases where performance is not available to deal with full image.
Sebastian Krzyszkowiak
in reply to Pavel Machek • • •