Rust Types and Required Messages

119 views
Skip to first unread message

Yordis Prieto

unread,
Jun 2, 2025, 3:46:15 PM Jun 2
to Protocol Buffers
As requested by [ @esrauchg ]( https://github.com/esrauchg ), moving the conversation here:

Although this originated in a `prost` discussion, the underlying issue concerns
how the official protobuf toolchain treats field presence. The intent here is to
clarify the expected stance and behavior of the generator.

**What language does this apply to?**

Primarily **Rust** , but also interested in **Go** and **TypeScript** as they
relate to protobuf code generation.

**Describe the problem you are trying to solve.**

The generated types are structurally imprecise: **all fields are wrapped in `Option<T>` ** ,
even those marked as `required` (explicitly or implicitly) in the proto
definition.

This leads to:

- Verbose boilerplate ( `Some(...)` wrappers)
- Increased cognitive overhead
- Risk of runtime errors when unwrapping values that should be guaranteed

```rust
fn calculate_minimum_bid_increment (cmd : & StartAuctionRun ) -> Result < MoneyAmount , Error > {
match & cmd . minimum_bid_increment_policy {
Some (policy) => match & policy . policy {
Some ( minimum_bid_increment_policy :: Policy :: Fixed (fixed_policy)) => {
//...
}
// ...
// NOTE: this should never happen, it is required
None => Err ( Error :: MinimumBidIncrementPolicyRequired ),
},
// NOTE: this should never happen, it is required
None => Err ( Error :: MinimumBidIncrementPolicyRequired ),
}
}
```

Despite the domain clearly requiring this field, the type system does not
enforce it. Structurally speaking.

We're using protobuf as the canonical schema for all:

- Commands
- Events
- Aggregate Snapshot

That applies across multiple language runtimes (via WASM modules) and is
critical for:

- Schema evolution and change detection
- Consistent interop across services
- Serialization correctness
- Avoiding runtime reflection or manual encoders/decoders

We treat protobuf-generated types as the source of truth and only validate
`commands` post-deserialization (or via protovalidate extension).

Here is the existing callbacks (thus far):

```rust
pub type InitialState < State > = fn () -> State ;
pub type IsTerminal < State > = fn (state : State ) -> bool ;
pub type Decide < State , Command , Event , Error > =
fn (state : & State , command : Command ) -> Result < Decision < State , Command , Event , Error >, Error >;
pub type Evolve < State , Event > = fn (state : & State , event : Event ) -> State ;

pub type GetStreamId < Command > = fn ( Command ) -> String ;
pub type IsOrigin < Event > = fn ( Event ) -> bool ;
pub type GetEventID < Event > = fn ( Event ) -> String ;
```

**Describe the solution you'd like**

The code generator should respect the `required` field modifier in `.proto`
definitions, emitting non-optional Rust fields where appropriate.

That would:

- Better align with schema intent
- Eliminate unnecessary `Option<T>` wrappers
- Improve safety and ergonomics

**Describe alternatives you've considered**

Creating application-level copies of protobuf types, but such types
have very little value in our context. The distintion between application
and serialization matters better little to us, especially when protobuffer
files can not break change.

**Additional context**


Em Rauch

unread,
Jun 2, 2025, 4:00:07 PM Jun 2
to Yordis Prieto, Protocol Buffers
Can you clarify a bit more what it is you are trying to achieve, and if there are any official implementations where Option/nullability is a pain point?

So in general the official Google Protobuf implementations do not expose Option<SomeMessage> or nullable-SomeMessage* in almost any of our implementations (as per the page I mentioned on the github issue: https://protobuf.dev/design-decisions/nullable-getters-setters/ )

If I'm following collectly, on all official implementations you should be able to get the behavior you're describing by just declaring the child messages and just don't check their has-bits. As far as the observable behavior is concerned all of the submessages being eagerly allocated or lazily allocated should be an implementation detail.

proto2 `required` is considered strongly regretted and we discourage its use, but just considering the semantics of required with Prost shape of API, I think it could not omit the Option<> on required message fields because the semantic of required is inherently only on wire and not in-memory: its the normal design that you would first construct a message without the required fields set, set them, and then eventually serialize it. Even in terms of data that was from parsed data, most Google official Protobuf implementations support a "parsePartial" which means "parse these bytes but don't enforce required, and then me write the logic against hassers directly".


--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com .
To view this discussion visit https://groups.google.com/d/msgid/protobuf/b5386744-e770-430e-bccb-959409a05641n%40googlegroups.com .

Yordis Prieto

unread,
Jun 2, 2025, 5:05:46 PM Jun 2
to Protocol Buffers
> Can you clarify a bit more what it is you are trying to achieve, and if there are any official implementations where Option/nullability is a pain point?

To be clear, I haven't use the official Rust package. I would love to, the conversation started in Prost, and the short comings there. I am trying to figure out where to find the official v4 rust version to figure out the situation there.


I think it could not omit the Option<> on required message fields because the semantic of required is inherently only on wire and not in-memory: 

I figured, that is where I wish I couldn't even construct a "broken" type, if a given field is required, then it must be set in order to construct the message. Otherwise, the type system isn't helping much and we must figure out the issues at runtime. I am unsure of the details about parsePartial and memory vs. wire; but I would expect two types based on my needs (PartialMessage<Message> and Message).

I guess I am going against what the philosophy is, which it is OK by me, just feel suboptimal, I want to get the most I can out of the type system without requiring that much discipline from developers.

Em Rauch

unread,
Jun 3, 2025, 8:29:06 AM Jun 3
to Yordis Prieto, Protocol Buffers
>  but I would expect two types based on my needs (PartialMessage<Message> and Message).
> I guess I am going against what the philosophy is, which it is OK by me, just feel suboptimal, I want to get the most I can out of the type system without requiring that much discipline from developers.

This does make perfect sense in general: if required was newly created in 2025 it maybe could try to have semantics of always having Builders, being able to parsePartial into Builders. Note that not many of the official implementations of Protobuf do not use builder pattern though, so anything like that would be very difficult to retrofit.

But the context is more that the idea of "blanket enforcement of presence of a field at parse time" is considered a misfeature (as discussed in the "Required is Strongly Deprecated" section  here ). Under the expected usecase patterns of Protobuf, its considered a big footgun since many people believe they have a field that is obviously always strictly required, but actually a few years later they will want to replace it with something different, and hard enforcement at parse time makes that evolution almost impossible.

Zooming out a bit: for the usecase you're describing, for the official Google supported implementations, I think just doing the normal / natural thing should be able to achieve the effective behavior you're thinking about. Make a normal message, make it have normal message-typed fields, treat it as normal child structs which are just "always there". Treat the maybe-lazy-allocation of those inner structs as an implementation detail, and treat the has-bit tracking as an implementation detail. If the default-initialized child struct isn't a valid value, check in your application code against that condition, rather than checking against the hassers (it would practically never matter for "this is a valid value of the child" if the child message field was set but in turn none of the fields on it were set anyway).

With Prost specifically, it may be harder to achieve that pattern naturally since all of the message-typed fields are typed as Option<>, but thats the nature of it and there's some tradeoffs Prost makes in turn for other ergonomic benefits. The V4 official Rust impl is in beta right now, it is an opaque api instead of open structs and so doesn't have the same implication, but it does end up being a design choice that will come with a number of other tradeoffs (some of which are already detailed in our beta documentation here ).

Yordis Prieto

unread,
Jun 3, 2025, 1:12:29 PM Jun 3
to Protocol Buffers
Personally, I do not mind the builder pattern or dealing with getters/setters; if that is the answer, I am OK with it, my primary concern is when things are combined and I can construct broken messages (bypassing builder, and setters) and the builder pattern.

Extra context, I am optimizing for LLM effectiveness, after doing some polyglot work around TypeScript, Go, and Rust. I found that Rust type system avoid hallucination to a point that isn't easy to ignore. For that to happen, the types coming out of the protobuf and its API needs to avoid silly mistakes. The same mistakes that some developer make based on suboptimal assumptions (context).

Without sidetracking too much, the biggest pain in the Go version is that everything is a pointer; we can not know if the reason for the pointer is because we actually want `nil` values or because of memory layout or technical concern. I wish an Option type existed in Go or added to the protobuf for this same reason. And well, defaults .... I digress.

I can strategize around the package, in the worst case, hoping that the protoc plugin allows some policies to change the behaviour. The interesting situation is that, reading about the required deprecation a bit more, definitely wouldn't like stale proto files to blow up if new information is added but my Processor does't care since it doesn't make use of the info, and therefore, it doesn't need the new proto file.
Purely speaking about Event Sourcing so my events are immutable, never breaking change messages.

What do you recommend me to do here? I need to find a way to test the v4 asap, I feel it is prudent to move to it

Yordis Prieto

unread,
Jun 3, 2025, 10:21:17 PM Jun 3
to Protocol Buffers
To add extra context, I am speaking purely from the end-user perspective experience, I am sure they are technical concerns to take into consideration as someone pointed out, I am bringing my naive experience with the intent to figure out whatever optimal strategy would be.

Em Rauch

unread,
Jun 4, 2025, 9:01:47 AM Jun 4
to Yordis Prieto, Protocol Buffers
Lets move this discussion to the associated GH Issue that you opened here https://github.com/protocolbuffers/protobuf/issues/22075

Thanks!

Yordis Prieto

unread,
Jun 4, 2025, 1:32:25 PM Jun 4
to Protocol Buffers
I regret opening the issue ... my intent was to discuss things, what happened is that I created the post here and it disappeared, well turns out that eventually it was here all along ... anyway, moving the conversation there then
Reply all
Reply to author
Forward
0 new messages