How to send my sensor data to my web app?

Hi all!

First time deploying any sort of app so forgive me if the answer is super obvious.

I’ve got a raspberry pi that’s continuously ingesting sensor information at a rate of 100hz+

I’d like to somehow send the sensor data directly to a dashboard hosted on Qovery without the use of a database.

I’ve never attempted anything like this before and am unsure where to begin. What are the standard, best practices? Through limited research I’ve come across websockets but am unsure if this is the best path to pursue?

Any suggestions, links, or reading material is super appreciated! Preferred language is Python if it matters.

2 Likes

Hi @Teejay , welcome to the Qovery Community. I am going to try to give you a concise answer to this design question :slight_smile:

Can you describe the type of your application and how you are going to keep the data? Do you need to keep the data?

The first questions that come to my mind are:

  1. What is the volume of data that you want to send?
  2. What is the size of the data?
  3. Is it a hobby or a professional project? → this is important to know if you need reliability.

Websocket is a solution but there are tons of other solutions like HTTP, GRPC, any binary protocol…

Happy to discuss about it :slight_smile:

Hi @rophilogene, thank you for the warm welcome!

I’ve got a prototype dashboard built using Streamlit. The raspberry pi is monitoring a CAN bus. For each unique id on the bus, the dashboard creates a live plot with the payload data of the last 100 messages. There can be about 15-30 id’s in total at any given time. I’ve got this working successfully locally over TCP at the moment.

  1. Hopefully I interpret this correctly, but the volume is only a single bus connected to the pi.
  2. Each message is a max of 8 bytes.
  3. Definitely a hobby project – just a learning experience and an opportunity for me to get my feet wet!

I’ve done a bit of reading about gRPC per your suggestion and it definitely seems like a great solution. I also came across an alternative RPC, Pyro5, which seems a bit higher level and easier to prototype with.

I look forward to any suggestions! Thank you again for the help.

I took a look and Streamlit looks very cool product. Thanks for sharing.

From what I see, the question is, do you need real-time analytics or not? Because real-time means keeping the connection open with your server app to display the data. So GRPC / Websocket or even your own TCP protocol will fit this purpose. But all of that at the cost of more complexity to stream the data in real-time.

On the other hand, if you don’t need a real-time info display, you can batch the data from your raspberry and send it every X time or X batch size. This is better if your raspberry internet connection is not so good.

It’s just a matter of choice, I don’t see any difficulty here.

gRPC for real-time is a good choice. Approved by several big companies :slight_smile: I don’t know about Pyro5. But gRPC is not so difficult but definitely required some time to dig into. Maybe this tutorial fits your need?

Let me know.

Batching the data should be totally fine!

I implemented a very basic gRPC example locally on my laptop and got the server successfully working in a container. I’m able to make calls from the client via localhost:50051with no issue.

Dockerfile

FROM python:3.8
EXPOSE 50051
WORKDIR /app
RUN pip3 install grpcio grpcio-tools
COPY . .
CMD ["python", "-u", "server.py"]

I then tried deploying the server on Qovery which was successful. I made the app and port publicly accessible which to my understanding essentially binds that port to the router url(?). But when I try connecting to the url from my client I always get a failure. Is this not possible?

Again, I apologize if this is obvious. My knowledge regarding cloud infrastructure/networking is extremely limited!

Hi @Teejay,

Unfortunately at the moment is impossible to route gRPC traffic outside of the private network with Qovery (can you confirm @Pierre_Mavro / @Erebe ?). Better support of other protocols than HTTP will be available in our next release planned for September 2021.


@Pierre_Mavro @Erebe do you have a workaround to propose to Teejay?

Yes, I confirm and there is no workaround at the moment, unfortunately.

@rophilogene @Pierre_Mavro no worries! thank ya’ll for the confirmation.

So assuming I have a connected device outside the network, is the recommended approach to utilize a simple rest api for data transfer?

1 Like

Yes, with authentication for sure :wink:

Do not hesitate if you need help here :slight_smile: