Blind and Visually Impaired Community
↑
3
↓
Educational App for VIP (with demo video) (self.Blind)
submitted by DNNenthusiast
I wrote a post here a couple of months ago. I received valuable feedback from the community members and I am grateful to those who spend their time and completed the survey.
Now I am attaching a short demonstration $1 of the app (This is not a final version of the product yet).
Let me describe what's happening there. The user holds a phone in one hand (phone holder can be used as well) and aims its camera towards a tactile graphics (TG). The UK map is printed on the image together with markers which are located in the corners. These markers are required to calibrate the image location at every frame instance and compensate for the camera fluctuations. At least three markers have to be visible for the app to operate properly (This might be the main challenge for the user but if he/she has an experience with using a phone camera, it should be OK ). In the meantime, the user starts exploring the TG with the other hand. The app detects the index finger position and corresponding audio feedback is triggered notifying the user what he/she is pointing upon at the moment. Here it is important to note that the app is capable of detecting each finger class independently, so the user can explore the image with the whole hand (or both). Then the user starts exploring a new TG with the image of a human skeleton. First, a QR code located on the image's backside is scanned and corresponding information is downloaded from the server. The internet connection is required only for this step. Finally, the image is explored in the same manner.
With the help of deep learning algorithms, the app is capable of working in real-time and even under poor lighting conditions (as in the video). Also, the app might be useful during the current situation when most of the students are studying from home. Since no guidance from the instructor is required.
If you want this app to be polished and then released or have any ideas for the additional features, please follow this survey $1 $1 Do not hesitate to pm me, if you have any questions.
Thanks!
Now I am attaching a short demonstration $1 of the app (This is not a final version of the product yet).
Let me describe what's happening there. The user holds a phone in one hand (phone holder can be used as well) and aims its camera towards a tactile graphics (TG). The UK map is printed on the image together with markers which are located in the corners. These markers are required to calibrate the image location at every frame instance and compensate for the camera fluctuations. At least three markers have to be visible for the app to operate properly (This might be the main challenge for the user but if he/she has an experience with using a phone camera, it should be OK ). In the meantime, the user starts exploring the TG with the other hand. The app detects the index finger position and corresponding audio feedback is triggered notifying the user what he/she is pointing upon at the moment. Here it is important to note that the app is capable of detecting each finger class independently, so the user can explore the image with the whole hand (or both). Then the user starts exploring a new TG with the image of a human skeleton. First, a QR code located on the image's backside is scanned and corresponding information is downloaded from the server. The internet connection is required only for this step. Finally, the image is explored in the same manner.
With the help of deep learning algorithms, the app is capable of working in real-time and even under poor lighting conditions (as in the video). Also, the app might be useful during the current situation when most of the students are studying from home. Since no guidance from the instructor is required.
If you want this app to be polished and then released or have any ideas for the additional features, please follow this survey $1 $1 Do not hesitate to pm me, if you have any questions.
Thanks!
siriuslylupin6 1 points 2y ago
Sounds interesting for sure.
KillerLag 1 points 2y ago
You mention it detects the index finger position... Does the hand have to be splayed open for it to detect? What would happen if I only had two fingers out?
What would it say if you were tracing along the border between the two countries?
Does the device need to be a set distance away? It seems to be on a stand, but most people wouldn't have a stand like that around. But if they hold it in one hand while feeling with the other, the distance may change/alter the focus.
If you wanted to change "head" to "skull", can it be done remotely (a teacher over the app)?
Interesting app you have there.
What would it say if you were tracing along the border between the two countries?
Does the device need to be a set distance away? It seems to be on a stand, but most people wouldn't have a stand like that around. But if they hold it in one hand while feeling with the other, the distance may change/alter the focus.
If you wanted to change "head" to "skull", can it be done remotely (a teacher over the app)?
Interesting app you have there.
DNNenthusiast [OP] 1 points 2y ago
Yes, since I trained my algorithm on the splayed open hand images, it detects the fingers when all of them are visible. But I can retrain my model for different cases. That's why I need some feedback from the potential end users.
I developed an interactive webpage for creating annotations. So, it depends how instructor annotates the graphics. He/she may leave some space between the objects and the app will stay silent on that region.
The app is capable of detecting images in about 10 fps (frames per second). So, it can easily handle the phone camera movements. I tried to shake my hand while I was holding the phone and the app still performed well. If 3 markers are visible, the app detects the images from \~ 20cm - 1meter range. But again I need to conduct the experiments before. For now everything works smoothly.
The object names can be easily changed via the interactive tool.
Thanks!
I developed an interactive webpage for creating annotations. So, it depends how instructor annotates the graphics. He/she may leave some space between the objects and the app will stay silent on that region.
The app is capable of detecting images in about 10 fps (frames per second). So, it can easily handle the phone camera movements. I tried to shake my hand while I was holding the phone and the app still performed well. If 3 markers are visible, the app detects the images from \~ 20cm - 1meter range. But again I need to conduct the experiments before. For now everything works smoothly.
The object names can be easily changed via the interactive tool.
Thanks!
This nonprofit website is run by volunteers.
Please contribute if you can. Thank you!
Our mission is to provide everyone with access to large-
scale community websites for the good of humanity.
Without ads, without tracking, without greed.
scale community websites for the good of humanity.
Without ads, without tracking, without greed.
©2023 HumbleCat Inc • HumbleCat is a 501(c)3 nonprofit based in Michigan, USA.