eskater/proctoring repository overview

Based on webdevops/php-nginx:8.1

🤡🤡🤡🤡🤡🤡🤡

1. Redis - for queue (laravel horizon)
2. librosa - floating point time series from file wav
3. Mediapipe (facemesh) - detect & pose head
4. face_recognition - compare_faces
5. janus-pp-rec - decode mjr to webm and opus
6. ffmpeg - merge audio+video and convert to mp4; convert mp4 to wav for librosa .

⁠Env

1. REDIS_PASSWORD=1234
2. MEDIA_PATH=/app/media - mjr files
3. RESULT_PATH=/app/result - mp4 files
4. API_ACCESS_TOKEN=1234 - Bearer token
5. HORIZON_ACCESS_TOKEN=1234 - /horizon/dashboard?token=1234
6. RESULT_ENDPOINT="" - jobs send result to external url (https://example.com/give_me_result⁠)
7. RESULT_ENDPOINT_ACCESS_TOKEN=""
8. PROCTORING_FPS=24
9. PROCTORING_RECOGNITION_FPS=120
10. PROCTORING_VIOLATION_GAP=5 - seconds

⁠Jobs

1. Proctoring - run script proctoring.py and send result to $RESULT_ENDPOINT with $feedback
2. Transcoder\Mjr - run bin janus-pp-rec & ffmpeg convert to mp4 by format videoroom-rid-uid-(camera|screen)-stamp-(audio|video).mjr and send result to $RESULT_ENDPOINT {"48": {"22": "***.mp4"}} with $feedback

"result": {
    "24": {
        "48": {
            "camera": {
                "1644862277": "d9766ff2-3eeb-4513-b5b7-9dd7cd8ff955.mp4"
            },
            "screen": {
                "1644862277": "b92af415-09ae-4131-b467-17ca82186b47.mp4"
            }
        }
    }
},

⁠Endpoint

{'headers': {'Authorization': 'Bearer $API_ACCESS_TOKEN'}}

1. POST /api/transcoder/mjr {path *1: 'videoroom-88-48-', delete: false, feedback: []} - wildcard glob(%s*.mjr); run job Transcoder\Mjr convert mjr to mp4; response {result: true}
2. GET /api/transcoder/files?path=videoroom-100 - mjr files
3. POST /api/proctoring/file {file *2: 'room-82-user-48.mp4', recognize: File<Image>, feedback: []} - run job Proctoring analyze video; response {result: true}
4. POST /api/recognition/check {images: File<Image>[], reference: File<Image>} - recognizing faces by reference face image; response {result: [true, false, true]}
5. POST /api/recognition/encodings {images: File<Image>[]} - get encodings from faces; response {result: [encodings...]}
6. POST /api/recognition/check_encodings - recognizing faces by reference face encodings; response {result: [true, false, true]}

*1 - directory $MEDIA_PATH (mjr files); *2 - directory $RESULT_PATH (mp4 files)

⁠proctoring.py

Video analysis by script proctoring.py

{'sound': {'min': 0.0015672889, 'avg': 0.2298492444626225, 'max': 0.7851088, 'noises': [[2, 0.37563697], [18, 0.19933486], [30, 0.13773125], [42, 0.20286202]]}, 'video': [[3.0, 'face:pose', 'right'], [5.0, 'face:many', 2]]}

1. video: [second, type, value]: types: face:pose, face:none, face:many, face:recognition
2. audio noises: [second, value]: value - sound.min > 0.1