Hand tools and fabrication machines
Imagine using a book, a table, or a wall as a touchpad. Combined with projector technology, you can replicate touch screens on any surface. This project aims to bring this utility and ease of use by through simple socket protocols and Walabot, a consumer-level 3D imaging device.
As I wanted this solution to be both practical in everyday use and universally accessible, so I opted for the least hardware-intensive setup. To balance ease of setup with imaging accuracy, I implemented a basic configuration loop that runs each time the Walabot starts.
This tutorial covers the setup procedure for the Raspberry Pi and the cursor control on a host computer.How Does It Work?
The project relies on a Raspberry Pi 3 client that takes readings from a Walabot Creator sensor (thanks Walabot!), and a host machine which in this case is my laptop.
The general user flow is as follows:
- Raspberry Pi sends trigger signal to the Walabot
- Walabot detects a target and locates target
- Raspberry Pi retrieves this data, determines the user action, and sends it to the socket host through the local WiFi connection
- Host machine receives the command and conducts cursor control through PyAutoGUI
In ProgressSetting Up the Pi
First, we need to set up the pi so that it's usable and has all the necessary libraries installed.
- If your Pi isn't preloaded with an OS, you can follow this tutorial to install Raspbian with NOOBS. Raspbian is the official OS of the Raspberry Pi and is by far the most popular.
- If you don't have a monitor and want to access your pi headless, I'd recommend you setting up VNC or SSH and adding a shutdown button to the Pi.
- We won't be needing IO so you can leave the header pins unsoldered.
- Insert the SD card into the Pi, and connect it to display and power. The Pi should boot up.
- Connect to the internet, and open Chromium.
- Navigate to the Walabot download page
- Download the Raspberrypi Installer and click keep to confirm the download
- Go back to the desktop, open the terminal and type
sudo apt-get update
sudo dpkg -i walabot_maker_1.0.34_raspberry_arm32.deb
In summary, the commands updates the system's package lists, and installs the WalabotSDK
- After going through a few agreement prompts, you should have the Walabot SDK installed. Then type
sudo apt-get dist-upgrade
This updates the installed packages and reboots the RPi.
For greater speed, I'd recommend disabling the desktop GUI as shown here.Walabot Interfacing
Now this is where the Walabot comes in. It's a 3D microwave imaging sensor that can do a whack load of things, from breathing sensing, seeing through walls, and detecting multiple targets. You can check em out on their youtube channel!
The functionality that's really useful for this project is the RF tridimensional sensing, which can detect the x, y, and x coordinates of a target. Also, its dynamic filtering capability which removes static signals can be combined with it's filtering of fixed reflectors, enabling you to track an object accurately even through other materials.
However one caveat of using such a powerful device is it's high power consumption. The Walabot uses 400-900 mA, which is too high for the raspberry's default USB output. This can be solved by navigating to /boot/config.txt and adding this line: max_usb_current=1.That'll increase the max current to 1200 mA.Target Detection
Making sure the Walabot detects only my finger when I want it to is key, or else my computer may randomly move the cursor, possibly even out of bounds. For my project, I interfaced with the Walabot API using the GetSensorTargets() function and turned MTI(Moving Target Identification) mode off to prevent the filtering of the finger signal over longer periods of time. I set the radius of detection (r) in my project to a maximum of 10 cm to set the arena to the area immediately above the Walabot. This changes based on the thickness of the material.
I tried setting the arena dimensions in x, y, and z values, but that only worked best on the short range profile which was only available on the Walabot developer edition. When I tried this on the default scanner mode, I got nearly zero target detection, probably as the scanner mode reads radially instead of through Cartesian coordinates. Thus I have set a standard scanner profile, set the arena radially, and convert the radial dimensions to Cartesian coordinates.
The target objects have x, y, z, and amplitude parameters, of which I only used the x and y values. As the y axis is along the length of the Walabot, I translated this to the x axis on the screen when the Walabot is placed horizontally like a touchpad. When I compare the coordinates to the arena bounds, I can find the relative location of the target to the arena and translate that to the relative location of the cursor on the host screen.
However, since my arena was initialized based on r, theta and phi values, I had to manually find the usable arena bounds through measurement of the lowest and highest values of the x and y coordinates. I then automated this at setup so that the program can find the bounds when the user slides their finger along the x axis and y axis. Even though there is formulae to convert polar coordinates to Cartesian, the set bounds differed from the actual measurable arena as it can't take into account the interference of another surface on the registered area.
To determine if a user wants to move the cursor, right click or left click, I made use of the multiple target detection features of the Walabot. Movement would be 1 point of contact, a right click would be 2 points of contact and a left click would be 3. It seems solid, time to test it out.
Argh! This turned out to be much more difficult to set up. I initially planned for the code to click the respective buttons on the mouse when the Walabot detectes two or more targets, but at the very moment in which the fingers enter the field of view, the Walabot detects one target, thus moving the mouse before clicking.
The obvious solution is to decrease the arena range itself, which naturally decreases the z-range. This has the side effect of producing a touchpad that's too small to use. The other route I took was to manually filter out targets with a z-attribute that's too high, but that doesn't work as a target along the z axis is still registered higher at the edge than along the center. Amplitude doesn't work as well, as it's centered around the most sensitive part of the module. I considered manually changing the z-parameter value for all targets by adding abs(phi angle - 90 degrees) * multiplier to all targets, but unfortunately the phi angle parameter is not an accessible sensor target parameter. At the end, I could only solve this by either using a curved touchpad to match the curvature of the range of detection, or by getting a Walabot Developer edition.Bluetooth Serial (Optional)
Since the goal was to create an intuitive and effective touchpad sensor that one can stick under any surface, I initially opted to set up data transfer from the rpi to the laptop through Bluetooth. I had to abandon this step as my Rpi could not scan for Bluetooth signals. I'll still cover this means of sending serial data as it should run on working raspberry pis.
First, pip install pybluez and pyautogui to python 2.7 on the raspberry pi.
If you're using a windows 10 pc as the host, pip install the PyBluez-0.22-cp27-none-win_amd64.whl file for your pc from here, and pip install pyautogui. I had difficulties in getting a connection, which was resolved by changing my Bluetooth adapter to the Generic Bluetooth Adapter.
Skip the next step as you'll be using the Bluetooth version of the client and server code. Copy the client and server files to the rpi and pc respectively and follow the instructions.TCP Socket Communication
As my initial attempt at getting Bluetooth serial failed on my rpi, I implemented serial communication over the socket. The downside to this is that both the server and the client have to be on the same wifi network.
On both your rpi and your pc, type the following in the console:
pip install pyautogui
Pyautogui is the library that we'll be using for mouse control.
Now verify that both devices are connected to the same network, and copy the server.py and client.py code to your rpi and pc respectively. Follow the instructions on the files.Optimization
When there are power disconnects I want to rpi to run the client.py program at startup. I can do this by typing sudo nano /etc/profile in the console and then typing sudo python with the path to my script. For example:
sudo python /home/pi/Documents/Github/Walabot-Touchpad/client.py
Type “Ctrl+X” to exit, then “Y” to save followed by “Enter” twice.
If you're using bluetoothClient.py, then simply replace the file path with it's location and name instead.
To improve the scanning speed on my rpi, I also disabled the GUI. This allows for a much smoother touchpad experience. To do this,
- open the terminal and type sudo raspi-config followed by the enter key
- navigate to Boot Options > Desktop/CLI
- select Console Autologin
Now exit the config menu, and select yes for the restart prompt. Now the client should start in console mode, which automatically runs client.py at startup.
To view the client output, you can connect to the pi through SSH. Here's a really handy tutorial for most operating systems.Enclosure Construction
Now, the only thing left to do it to put everything together. Plug the pi to the Walabot, and the pi to a power bank. Now build any enclosure you like to keep everything in one box. I used cardboard and some suction cups to build a nice enclosure for everything.
The final product proved to be quite useful and responsive! The low profile and discreetness of the design is a big win, as it means I can slap this project under any surface without anyone noticing. For the fun of it, I might replace my PC mouse on my computer and see how long it will take for others to discover my hidden touchpad.
The Walabot was also highly accurate and efficient. From my experience working with ultrasonic sensors I assumed there would be many glitches and hidden restrictions, but the SDK proved to be extremely intuitive and reliable. The only downside was trying to make do with the standard sensor profile as the short range capability was restricted to the Walabot Developer edition.
For future improvements, I'm planning on either using another rpi for bluetooth serial communication or the nmap library to automatically scan for my pc's IP address. There's also the untapped potential of using the Z- axis for more interfacing, potentially for zooming in and out in games and such.
If I do get a projector, I'll be sure to update this project with a setup where the screen is casted on a wall and the touchpad is calibrated under the projection area in a 1:1 ratio.
Thanks for reading and I hope you found this interesting!