Transfering large dataset

V R

unread,

Jan 31, 2022, 10:44:15 AM 1/31/22

to Protocol Buffers

Hello,

I am working with large images and large meshes in medical field and I need to transfer and serialize those objects directly.

I read here https://developers.google.com/protocol-buffers/docs/techniques that "Protocol Buffers are not designed to handle large messages. As a general rule of thumb, if you are dealing in messages larger than a megabyte each, it may be time to consider an alternate strategy." Does that mean that I can't use gRPC to transfer a mesh with e.g., 2MBytes ?

Thank you

David Raleigh

unread,

Jan 31, 2022, 1:46:04 PM 1/31/22

to V R, Protocol Buffers

HTTP2 Multiplexing won't work so hot with one very large message. To gain performance you'd want to break the file into pieces. Or you might design your system differently so that the large images / meshes are in blob storage someplace and your gRPC message contains the path to the item.

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com .
To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/8478b682-ee3c-4b37-8742-a3ce7160fc00n%40googlegroups.com .

V R

unread,

Jan 31, 2022, 6:13:00 PM 1/31/22

to David Raleigh, Protocol Buffers

Is there a limit ? By large I mean 4MBytes. And how about streaming with gRPC: is it not possible to use that ?

David Raleigh

unread,

Jan 31, 2022, 6:57:20 PM 1/31/22

to V R, Protocol Buffers

I think the default is 4MB. and you can raise it to something monstrous like 1gig or 2 gigs. You'll need to increase the max message size on both client and server.

Adam Cozzette

unread,

Jan 31, 2022, 7:08:58 PM 1/31/22

to David Raleigh, V R, Protocol Buffers

I think this part of the documentation is a bit out of date because we now routinely use messages much larger than 1 MB in size. The only enforced requirement is that serialized messages have to be strictly less than 2 GiB in size. It's probably still not a great idea to use huge messages hundreds of megabytes in size, because they end up stored in memory all at once. But if the message is just a few MB then that's generally no problem at all.

To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/CAA9YwXoTT94jiX8PaA%2BM6%2BbBnMurArma-TSsUgSy256nf91WjQ%40mail.gmail.com .

V R

unread,

Jan 31, 2022, 10:22:18 PM 1/31/22

to Adam Cozzette, David Raleigh, Protocol Buffers

Thank you very much for pointing that the doc is outdated and that you don't have issues.

Best regards.

Reply all

Reply to author

Forward