RKMPP not using hardware decoding when no OpenCL even when tone-mapping is not needed

[zhujunsan]

The bug

I'm using OrangePi 3B w/ 8G memory, which equipped with a Rk3566 CPU. mali driver is not installed

When turning on Hardware Acceleration by choosing API as RKMPP, and enabled Hardware decoding, the encoding part works fine, but ffmpeg is not using rkrga to resize the video, which result to high cpu usage and low speed.

here's the ffmpeg command immich is using (running command by hand in container, args from ps aux|grep ffmpeg):

/usr/bin/ffmpeg \
-i upload/library/some.mp4 \
-y -c:v h264_rkmpp -c:a copy -movflags faststart -fps_mode passthrough \
-map 0:0 -map 0:1 -strict unofficial -g 256 -v verbose \
-vf scale=-2:720,zscale=t=linear:npl=100,tonemap=hable:desat=0,zscale=p=709:t=709:m=709:range=pc,format=yuv420p \
-level 51 -rc_mode CQP -qp_init 23 /path/to/encoded-video/some.mp4

frame= 716 fps=3.7 q=-0.0 Lsize= 25130KiB time=00:00:23.85 bitrate=8629.0kbits/s speed=0.122x

here's what can be used: (running command by hand in container, args comes from codebase)

/usr/bin/ffmpeg \
> -hwaccel rkmpp -hwaccel_output_format drm_prime -afbc rga -noautorotate \
> -i upload/library/some.mp4 \
> -y -c:v h264_rkmpp -c:a copy -movflags faststart -fps_mode passthrough \
> -map 0:0 -map 0:1 -strict unofficial -g 256 -v verbose \
> -vf "scale_rkrga=-2:720:format=nv12:afbc=1" \
> -level 51 -rc_mode CQP -qp_init 23 /path/to/encoded-video/some.mp4

frame= 716 fps=109 q=-0.0 Lsize= 19464KiB time=00:00:23.85 bitrate=6683.6kbits/s speed=3.63x

a warping 30 times faster.

I know mali driver is not installed, and RK3566 doesn't support HW tone-mapping, but it still can handle hw decoding and rescaling non hdr videos quite well.

I think there might be some imperfect logic in determine which args ffmpeg shoud use in server/src/services/media.service.spec.ts and maybe server/src/utils/media.ts

The OS that Immich Server is running on

Armbian Community v24.11 bookworm

Version of Immich Server

v1.118.2

Version of Immich Mobile App

not related

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    group_add:
      - video
    security_opt: # enables full access to /sys and /proc, still far better than privileged: true
      - apparmor=unconfined
      - systempaths=unconfined
    devices:
      - /dev/dri
      - /dev/dma_heap
      - /dev/rga
      - /dev/mpp_service
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - stack.env
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: unless-stopped
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - stack.env
    restart: unless-stopped
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:2ba50e1ac3a0ea17b736ce9db2b0a9f6f8b85d4c27d5f5accc6a416d8f42c6d5
    healthcheck:
      test: redis-cli ping || exit 1
    restart: unless-stopped

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 1m
      #start_interval: 30s
      #start_period: 5m
    command:
      [
        'postgres',
        '-c',
        'shared_preload_libraries=vectors.so',
        '-c',
        'search_path="$$user", public, vectors',
        '-c',
        'logging_collector=on',
        '-c',
        'max_wal_size=2GB',
        '-c',
        'shared_buffers=512MB',
        '-c',
        'wal_compression=on',
      ]
    restart: unless-stopped

volumes:
  model-cache:

Your .env content

UPLOAD_LOCATION=/docker/immich/library
DB_DATA_LOCATION=/docker/immich/postgres
TZ=Asia/Shanghai
IMMICH_VERSION=release
DB_PASSWORD=postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

1. Turn on hw accl and hw decoding on RK3566 cpu
   

2. That's it

...

Relevant log output

No response

Additional information

No response

[mertalev]

Are you sure it can do hardware decoding without a Mali driver?

Edit: Ah, I see it's only used for tone-mapping. I think we can relax this to let it do hardware decoding when not tone-mapping.