Question:
This derives from the discussions on issue
The commit applies the changes suggested in the previous reviews. With those changes, I think we have a good basis for discussing a unified solution in the working group.
Thanks NAME for the review. I have applied the editorial changes. You are asking for something more regarding the "specific logic" required when supporting zero-length CID. There are in fact two pieces to that: logic at the receiver, i.e., the node that chose to receive packets with zero-length CID; and logic at the sender, i.e., the node that accepts to send data on multiple paths towards a node that uses zero-length CID. This is explained in details in the section "Using Zero-Length connection ID", so I guess what we need is a reference to that section.
Comment by NAME (on PR ): I think we expand this section a bit as we found getting the delay right was non-trivial. QUIC time-stamp would work for sure. But if one decides not to use QUIC timestamp, we probably want to explain what is needed to be done here.
I have submitted PR for this one.
Closing as got merged.
In section 3.1, we wrote When the multipath option is negotiated, clients that want to use an additional path MUST first initiate the Address Validation procedure with PATHCHALLENGE and PATHRESPONSE frames described in of ]. After receiving packets from the client on the new paths, the servers MAY in turn attempt to validate these paths using the same mechanisms. With multipath, we cannot simply write that the server MAY validate the new path. There is a risk of a malicious client that operates as follows: client creates path from its own address, A. Path is validate by server client starts long download from server client creates a second path to server using a spoofed address, B client starts address validation for B but does not care about the response from the server client stops acknowledging packets over the first path to force the server to move packets to the second one and then opportunistically acknowledges packets that should have been sent by the server over that second packet (SPNS would probably make such an attack simpler than MPNS) We probably need to start working on the security considerations section of the draft.
This is already covered in the QUIC threat model. The server is supposed to validate new paths by sending a "path challenge" on that path, then waiting for a challenge response. If the client spoofs a path, it will never receive the challenge, and thus the server will never receive the response and will not validate the path. See section 8.2 of RFC 9000, path validation. Security section 21.1.3. Connection Migration states that "Path validation () establishes that a peer is both willing and able to receive packets sent on a particular path."
So yes, the current text should be "the servers MUST in turn attempt to validate these paths using the same mechanisms, as specified in {{ Section 8.2 of RFC9000 }}."
No, this should not be a MUST. It is true that the path MUST be validated before it can be used. However, just because the serve receives a new packet on a path, that doesn't automatically mean that the server has to start path validation. It could e.g. also decide to simply withdraw that packet.
Actually, this is also not right. This is what RFC9000 says (section 9): So I guess we should maybe not use normative language here but refer to section 9 in RFC9000 instead...?
Just submitted PR for this issue.
This draft initially originates from a merging effort of previous Multipath proposals. While many points were found to be common between them, there remains one design point that still requires consensus: the number of packet number spaces that a Multipath QUIC connection should support (i.e., one for the whole connection vs. one per path). The current draft enables experimentation with both variants, but in the final version we will certainly need to choose between one of the versions.
The main issues mentioned so far: Efficiency. One number space per path should be more efficient, because implementations can directly reuse the loss-recovery logic specified for QUIC ACK size. Single space leads to lots of out of order delivery, which causes ACK sizes to grow Simplicity of code. Single space can be implemented without adding mane new code paths to uni-path QUIC Shorter header. Single space works well with NULL length CID The picoquic implementation shows that "efficiency" and "ack size" issues of single space implementations can be mitigated. However, that required significant improvements in the code: to obtain good loss efficiency, picoquic remembers not just the path on which a packet was sent, but also the order of the packet on this path, i.e. a "virtual sequence number". The loss detection logic then operates on that virtual sequence number. to contain ACK size, picoquic implements a prioritization logic to select the most important ranges in an ACK, avoid acking the same range more than 3 or 4 times, and keep knowledge of already acknowledged ranges so range coalescing works. I think these improvements are good in general, and I will keep them in the implementations whether we go for single space or not. The virtual sequence number is for example useful if the CID changes for reasons not related to path changes in multiple number space variants. It is also useful in unipath variants to avoid interference between sequence numbers used in probes and the RACK logic. The ACK size improvements do reduce the size of ACKs in presence of out of order delivery, e.g., if the network is doing some kind of internal load balancing. On the other hand, the improvements are somewhat complex, would need to be described in separate drafts, and pretty much contradicts the "simplicity of code" argument. So we are left with the "Null length CID" issue. I see for cases: Client CID Priority ----------------------- long Used by many implementations NULL Preferred configuration of many big deployments long Rarely used, server load balancing does not work NULL Only mentioned in some P2P deployments The point here is that it is somewhat hard to deploy a large server with NULL CID and use server load balancing. This configuration is mostly favored by planned P2P deployments.
The big debate is for the configuration with NULL CID on client, long CID on server. The packets from server to client do not carry a CID, and only the last bytes of the sequence number. The client will need some logic to infer the full sequence number before decrypting the packet. The client could maybe use the incoming 5 tuple as part of the logic, but it is not obvious. It is much simpler to assume a direct map from destination CID to number space. That means, if a peer uses a NULL CID, all packets sent to that peer are in the same number space.
Revised table: Client CID What -- -- long Multiple number space NULL Multiple number spaces on client side (one per CID), single space on server side long Multiple number spaces on server side (one per CID), single space on client side NULL single number space on each side If a node advertises both NULL CID and multipath support, they SHOULD have logic to contain the size of ACK. If a node engages in multipath with a NULL CID peer, they SHOULD have special logic to make loss recovery work well.
I think the above points to a possible "unified solution": if multipath negotiated, tie number spaces to destination CID, use default (N=0) if NULL CID. use multipath ack; in multipath ACK, identify number space by sequence number of DCID in received packets. Default that number to zero if received with NULL CID. in ABANDON path, identify path either by DCID of received packets, SCID of sent packet, or "this path" (sending path) if CID is NULL.
I like the proposal of the "unified solution". I think the elegance lies in the fact that it allows us to automatically cover all four cases listed above. The previous dilemma for me was that on one hand we have some use cases where we need to support more than two paths and separate PN makes the job easier, but on the other hand, I think we should not ignore the NULL CID use cases as it is also important. Now, with this proposal, a big part of the problem is solved. The rest of challenge is to make sure single PN remains efficient in terms of ACK and loss recovery. On that part, we plan to do an A/B test and would love to share the results when we get them. There is one more problem as pointed in issue , when we want to take hardware offloads into account. In such a case, we may still need single PN for long server CID. However, if hardware supports nonce modification, this problem can be addressed with the proposed "unified solution".
If the APi does not support 96 bit sequence numbers, it should always be possible to create an encryption context per number space, using Key=current key and ID = current-ID + CID sequence number. Of course, that forces creation of multiple context, and that makes key rotation a bit harder to manage. But still, that should be possible.
Thanks for the summary NAME I think one point is missing in your list which is related to issue . Use of a single packet number space might not support ECN. Regarding the unified solution: I think what you actually say is that we would specify both solutions and everybody would need to implement both logics. At least regarding the "simplicity of code" argument, that would be the worst choice. If we can make the multiple packet number spaces solution work with one-sided CIDs, I'm tending toward such an approach. Use of multiple packet number spaces avoids ambiguity/needed "smartness" in ACK handling and packet scheduling which, as you say above, can make the implementation more complex and, moreover, wrong decisions may have large impact on efficient (both local processing and on-the-wire). I don't think we want to leave these things to each implementation individually.
The summary Christian made above about design comparison sounds indeed quite accurate. Besides ECN, my other concern about single packet number is that it require cleverness from the receiver side if you want to be performant in some scenarios. At the sender-side, you need to consider a path-relative packet number threshold instead of an absolute one to avoid spurious losses. Just a point I think we did not mentioned yet is that there can be some interactions between Multipath out-of-order number delivery and incoming packet number duplicate detection. This requires maintaining some state at the receiver side, as described by URL With single packet number space, the receiver should take extra care when updating the "minimum packet number below which all packets are immediately dropped". Otherwise, in presence of paths with very different latencies, the receiver might end up discarding packets from a (slower) path. I'm also preferring the multiple packet number spaces solution for the above reasons. I'm not against thinking for a "unified" solution (the proposal sounds interesting), but I wonder how much complexity this would add compared to requiring end hosts to use one-byte CIDs.
I think the issue is not really so much "single vs multiple number space" as "support for multipath and NULL CID". As noted by Mirja, there is an implementation cost there. My take is: If a node uses NULL CID and proposes support of multipath, then that node MUST implement code to deal with size of acknowledgements. If a node sees a peer using NULL CID and supporting multipath, then that node MUST either use only one path at a time or implement code to deal with impact on loss detection due to out of order arrivals, and impact on congestion control including ambiguity of ECN signals. Then add sections on what it means to deal with the side of acknowledgements, out of order arrivals, and congestion control. I think this approach ends up with the correct compromises: It keeps the main case simple. Nodes that rely on multipath just use long CID and one number space per CID. Nodes that use long CID never have to implement the ACK length minimization algorithms. Nodes that see a peer using NULL CID have a variety of implementation choices, e.g., sending on just one path at a time, sending on mostly one path at a time in "make or break" transitions, or implementing sophisticated algorithms. If we do that, we can get rid of the single vs multiple space discussion, and end up with a single solution addressing all use cases.
Looking at all the discussions here and in other issues such as ECN, I think that we should try to write two different versions of section 7: one covering all the aspects of single packet number space (RTT estimation, retransmissions, ECN, handling of CIDs, ...) one covering all the aspects of multiple packet number space (RTT estimation, retransmissions, ECN, handling of CIDs, ...) There would be some overlap between these two sections and also some differences that would become clear as we specify them. At the end of this writing, we'll know whether it is possible to support/unify both or we need to recommend a single one. The other parts of the document are almost independent of that and can evolve in parallel with these two sections. However, I don't think that such a change would be possible by Monday
I don't think we should rush changes before we have agreed on the final vision.
It might be helpful to have these options as PRs (without merging) them, so people can understand all details of each approach.
I agree with NAME that supporting multi-path with null CIDs is a more fundamental issue than the efficiency comparison between single PN and separate PN, as it would ultimately impact the application scope of multipath QUIC. But indeed, we might want to implement proposed solution first and then decide if we want to adopt such a unified approach.
Since NAME prodded me, we now have a PR for the "unified" proposal.
I totally agree with Christian that the issue is about "support for multipath and NULL CID", and the solution that Christian suggested looks really great! It both takes advantage of multiple spaces, and support NULL CID users without affect the efficiency of ACK arrangements. Besides, the solution is more convenient for implementations, because If both endpoints uses non-zero length cids, endpoints only need to support multiple spaces, and if one of the endpoints use NULL CID, it could use single pn space in one direction and could support NULL CID and multipath at the same time.
Very high level summary of IETF-113 discussion seems that there is interest and likely support for the unified solution (review minutes for further details).

multipath negotiation is ambiguous, they MUST be interpreted as acknowledging packets sent on path 0. <del> Endpoints negotiate the use of one packet number space for all paths or separate packet number spaces per path during the connection handshake nego.  While separate packet number spaces allow for more efficient ACK encoding, especially when paths have highly different latencies, this approach requires the use of a connection ID. Therefore use of a single number space can be beneficial when endpoints use zero-length connection ID for less overhead. </del> 7.1. <del> If the multipath option is negotiated to use one packet number space for all paths, the packet sequence numbers are allocated from the common number space, so that, for example, packet number N could be sent on one path and packet number N+1 on another. ACK frames report the numbers of packets that have been received so far, regardless of the path on which they have been received.  That means the senders needs to maintain an association between sent packet numbers and the path over which these packets were sent.  This is necessary to implement per path congestion control. When a packet is acknowledged, the state of the congestion control MUST be updated for the path where the acknowledged packet was originally sent.  The RTT is calculated based on the delay between the transmission of that packet and its first acknowledgement (see compute-rtt) and is used to update the RTT statistics for the sending path. Also loss detection MUST be adapted to allow for different RTTs on different paths.  For example, timer computations should take into account the RTT of the path on which a packet was sent.  Detections based on packet numbers shall compare a given packet number to the highest packet number received for that path. </del> <ins> If a zero-length connection ID is used, one packet number space for all paths.  That means the packet sequence numbers are allocated from the common number space, so that, for example, packet number N could be sent on one path and packet number N+1 on another. In this case, ACK frames report the numbers of packets that have been received so far, regardless of the path on which they have been received.  That means the senders needs to maintain an association between sent packet numbers and the path over which these packets were sent.  This is necessary to implement per path congestion control, as explained in zero-length-cid-loss-and-congestion. Further, the receiver of packets with zero-length connection IDs should implement handling of acknowledgements as defined in sending- acknowledgements-and-handling-ranges. ECN handing is specified in ecn-handling, and mitigation of the RTT measurement is further explained in ack-delay-and-zero-length-cid- considerations. If a node does not want to implement this logic, it MAY instead limit its use of multiple paths as explained in restricted-sending-to-zero- length-cid-peer. </ins> 7.1.1. <ins> If zero-length CID and therefore also a single packet number space is used by the sender, the receiver MAY send ACK frames instead of ACK_MP frames to reduce overhead as the additional path ID field will anyway always carry the same value. </ins> If senders decide to send packets on paths with different transmission delays, some packets will very likely be received out of order.  This will cause the ACK frames to carry multiple ranges of


Question:
This derives from the discussions on issue
The commit applies the changes suggested in the previous reviews. With those changes, I think we have a good basis for discussing a unified solution in the working group.
Thanks NAME for the review. I have applied the editorial changes. You are asking for something more regarding the "specific logic" required when supporting zero-length CID. There are in fact two pieces to that: logic at the receiver, i.e., the node that chose to receive packets with zero-length CID; and logic at the sender, i.e., the node that accepts to send data on multiple paths towards a node that uses zero-length CID. This is explained in details in the section "Using Zero-Length connection ID", so I guess what we need is a reference to that section.
Comment by NAME (on PR ): I think we expand this section a bit as we found getting the delay right was non-trivial. QUIC time-stamp would work for sure. But if one decides not to use QUIC timestamp, we probably want to explain what is needed to be done here.
I have submitted PR for this one.
Closing as got merged.
In section 3.1, we wrote When the multipath option is negotiated, clients that want to use an additional path MUST first initiate the Address Validation procedure with PATHCHALLENGE and PATHRESPONSE frames described in of ]. After receiving packets from the client on the new paths, the servers MAY in turn attempt to validate these paths using the same mechanisms. With multipath, we cannot simply write that the server MAY validate the new path. There is a risk of a malicious client that operates as follows: client creates path from its own address, A. Path is validate by server client starts long download from server client creates a second path to server using a spoofed address, B client starts address validation for B but does not care about the response from the server client stops acknowledging packets over the first path to force the server to move packets to the second one and then opportunistically acknowledges packets that should have been sent by the server over that second packet (SPNS would probably make such an attack simpler than MPNS) We probably need to start working on the security considerations section of the draft.
This is already covered in the QUIC threat model. The server is supposed to validate new paths by sending a "path challenge" on that path, then waiting for a challenge response. If the client spoofs a path, it will never receive the challenge, and thus the server will never receive the response and will not validate the path. See section 8.2 of RFC 9000, path validation. Security section 21.1.3. Connection Migration states that "Path validation () establishes that a peer is both willing and able to receive packets sent on a particular path."
So yes, the current text should be "the servers MUST in turn attempt to validate these paths using the same mechanisms, as specified in {{ Section 8.2 of RFC9000 }}."
No, this should not be a MUST. It is true that the path MUST be validated before it can be used. However, just because the serve receives a new packet on a path, that doesn't automatically mean that the server has to start path validation. It could e.g. also decide to simply withdraw that packet.
Actually, this is also not right. This is what RFC9000 says (section 9): So I guess we should maybe not use normative language here but refer to section 9 in RFC9000 instead...?
Just submitted PR for this issue.
This draft initially originates from a merging effort of previous Multipath proposals. While many points were found to be common between them, there remains one design point that still requires consensus: the number of packet number spaces that a Multipath QUIC connection should support (i.e., one for the whole connection vs. one per path). The current draft enables experimentation with both variants, but in the final version we will certainly need to choose between one of the versions.
The main issues mentioned so far: Efficiency. One number space per path should be more efficient, because implementations can directly reuse the loss-recovery logic specified for QUIC ACK size. Single space leads to lots of out of order delivery, which causes ACK sizes to grow Simplicity of code. Single space can be implemented without adding mane new code paths to uni-path QUIC Shorter header. Single space works well with NULL length CID The picoquic implementation shows that "efficiency" and "ack size" issues of single space implementations can be mitigated. However, that required significant improvements in the code: to obtain good loss efficiency, picoquic remembers not just the path on which a packet was sent, but also the order of the packet on this path, i.e. a "virtual sequence number". The loss detection logic then operates on that virtual sequence number. to contain ACK size, picoquic implements a prioritization logic to select the most important ranges in an ACK, avoid acking the same range more than 3 or 4 times, and keep knowledge of already acknowledged ranges so range coalescing works. I think these improvements are good in general, and I will keep them in the implementations whether we go for single space or not. The virtual sequence number is for example useful if the CID changes for reasons not related to path changes in multiple number space variants. It is also useful in unipath variants to avoid interference between sequence numbers used in probes and the RACK logic. The ACK size improvements do reduce the size of ACKs in presence of out of order delivery, e.g., if the network is doing some kind of internal load balancing. On the other hand, the improvements are somewhat complex, would need to be described in separate drafts, and pretty much contradicts the "simplicity of code" argument. So we are left with the "Null length CID" issue. I see for cases: Client CID Priority ----------------------- long Used by many implementations NULL Preferred configuration of many big deployments long Rarely used, server load balancing does not work NULL Only mentioned in some P2P deployments The point here is that it is somewhat hard to deploy a large server with NULL CID and use server load balancing. This configuration is mostly favored by planned P2P deployments.
The big debate is for the configuration with NULL CID on client, long CID on server. The packets from server to client do not carry a CID, and only the last bytes of the sequence number. The client will need some logic to infer the full sequence number before decrypting the packet. The client could maybe use the incoming 5 tuple as part of the logic, but it is not obvious. It is much simpler to assume a direct map from destination CID to number space. That means, if a peer uses a NULL CID, all packets sent to that peer are in the same number space.
Revised table: Client CID What -- -- long Multiple number space NULL Multiple number spaces on client side (one per CID), single space on server side long Multiple number spaces on server side (one per CID), single space on client side NULL single number space on each side If a node advertises both NULL CID and multipath support, they SHOULD have logic to contain the size of ACK. If a node engages in multipath with a NULL CID peer, they SHOULD have special logic to make loss recovery work well.
I think the above points to a possible "unified solution": if multipath negotiated, tie number spaces to destination CID, use default (N=0) if NULL CID. use multipath ack; in multipath ACK, identify number space by sequence number of DCID in received packets. Default that number to zero if received with NULL CID. in ABANDON path, identify path either by DCID of received packets, SCID of sent packet, or "this path" (sending path) if CID is NULL.
I like the proposal of the "unified solution". I think the elegance lies in the fact that it allows us to automatically cover all four cases listed above. The previous dilemma for me was that on one hand we have some use cases where we need to support more than two paths and separate PN makes the job easier, but on the other hand, I think we should not ignore the NULL CID use cases as it is also important. Now, with this proposal, a big part of the problem is solved. The rest of challenge is to make sure single PN remains efficient in terms of ACK and loss recovery. On that part, we plan to do an A/B test and would love to share the results when we get them. There is one more problem as pointed in issue , when we want to take hardware offloads into account. In such a case, we may still need single PN for long server CID. However, if hardware supports nonce modification, this problem can be addressed with the proposed "unified solution".
If the APi does not support 96 bit sequence numbers, it should always be possible to create an encryption context per number space, using Key=current key and ID = current-ID + CID sequence number. Of course, that forces creation of multiple context, and that makes key rotation a bit harder to manage. But still, that should be possible.
Thanks for the summary NAME I think one point is missing in your list which is related to issue . Use of a single packet number space might not support ECN. Regarding the unified solution: I think what you actually say is that we would specify both solutions and everybody would need to implement both logics. At least regarding the "simplicity of code" argument, that would be the worst choice. If we can make the multiple packet number spaces solution work with one-sided CIDs, I'm tending toward such an approach. Use of multiple packet number spaces avoids ambiguity/needed "smartness" in ACK handling and packet scheduling which, as you say above, can make the implementation more complex and, moreover, wrong decisions may have large impact on efficient (both local processing and on-the-wire). I don't think we want to leave these things to each implementation individually.
The summary Christian made above about design comparison sounds indeed quite accurate. Besides ECN, my other concern about single packet number is that it require cleverness from the receiver side if you want to be performant in some scenarios. At the sender-side, you need to consider a path-relative packet number threshold instead of an absolute one to avoid spurious losses. Just a point I think we did not mentioned yet is that there can be some interactions between Multipath out-of-order number delivery and incoming packet number duplicate detection. This requires maintaining some state at the receiver side, as described by URL With single packet number space, the receiver should take extra care when updating the "minimum packet number below which all packets are immediately dropped". Otherwise, in presence of paths with very different latencies, the receiver might end up discarding packets from a (slower) path. I'm also preferring the multiple packet number spaces solution for the above reasons. I'm not against thinking for a "unified" solution (the proposal sounds interesting), but I wonder how much complexity this would add compared to requiring end hosts to use one-byte CIDs.
I think the issue is not really so much "single vs multiple number space" as "support for multipath and NULL CID". As noted by Mirja, there is an implementation cost there. My take is: If a node uses NULL CID and proposes support of multipath, then that node MUST implement code to deal with size of acknowledgements. If a node sees a peer using NULL CID and supporting multipath, then that node MUST either use only one path at a time or implement code to deal with impact on loss detection due to out of order arrivals, and impact on congestion control including ambiguity of ECN signals. Then add sections on what it means to deal with the side of acknowledgements, out of order arrivals, and congestion control. I think this approach ends up with the correct compromises: It keeps the main case simple. Nodes that rely on multipath just use long CID and one number space per CID. Nodes that use long CID never have to implement the ACK length minimization algorithms. Nodes that see a peer using NULL CID have a variety of implementation choices, e.g., sending on just one path at a time, sending on mostly one path at a time in "make or break" transitions, or implementing sophisticated algorithms. If we do that, we can get rid of the single vs multiple space discussion, and end up with a single solution addressing all use cases.
Looking at all the discussions here and in other issues such as ECN, I think that we should try to write two different versions of section 7: one covering all the aspects of single packet number space (RTT estimation, retransmissions, ECN, handling of CIDs, ...) one covering all the aspects of multiple packet number space (RTT estimation, retransmissions, ECN, handling of CIDs, ...) There would be some overlap between these two sections and also some differences that would become clear as we specify them. At the end of this writing, we'll know whether it is possible to support/unify both or we need to recommend a single one. The other parts of the document are almost independent of that and can evolve in parallel with these two sections. However, I don't think that such a change would be possible by Monday
I don't think we should rush changes before we have agreed on the final vision.
It might be helpful to have these options as PRs (without merging) them, so people can understand all details of each approach.
I agree with NAME that supporting multi-path with null CIDs is a more fundamental issue than the efficiency comparison between single PN and separate PN, as it would ultimately impact the application scope of multipath QUIC. But indeed, we might want to implement proposed solution first and then decide if we want to adopt such a unified approach.
Since NAME prodded me, we now have a PR for the "unified" proposal.
I totally agree with Christian that the issue is about "support for multipath and NULL CID", and the solution that Christian suggested looks really great! It both takes advantage of multiple spaces, and support NULL CID users without affect the efficiency of ACK arrangements. Besides, the solution is more convenient for implementations, because If both endpoints uses non-zero length cids, endpoints only need to support multiple spaces, and if one of the endpoints use NULL CID, it could use single pn space in one direction and could support NULL CID and multipath at the same time.
Very high level summary of IETF-113 discussion seems that there is interest and likely support for the unified solution (review minutes for further details).

7.1.2. <ins> When sending to a zero-length CID receiver, senders may receive acknowledgements that combine packet numbers received over multiple paths.  However, even if one packet number space is used on multiple path the sender MUST maintain separate congestion control state for each path.  Therefore, senders MUST be able to infer the sending path from the acknowledged packet numbers, for example by remembering which packet was sent on what path.  The senders MUST use that information to perform congestion control on the relevant paths, and to correctly estimate the transmission delays on each path.  (See ack-delay-and-zero-length-cid-considerations for specific considerations about using the ACK Delay field of ACK frames, and ecn-handling for issues on using ECN marks.) Loss detection as specified in QUIC-RECOVERY uses algorithms based on timers and on sequence numbers.  When packets are sent over multiple paths, loss detection must be adapted to allow for different RTTs on different paths.  When sending to zero-length CID receivers, packets sent on different paths may be received out of order.  Therefore senders cannot directly use the packet sequence numbers to compute the Packet Thresholds defined in Section 6.1.1 of QUIC-RECOVERY. Relying only on Time Thresholds produces correct results, but is somewhat suboptimal.  Some implementations have been getting good results by not just remembering the path over which a packet was sent, but also maintaining an order list of packets sent on each path.  That ordered list can then be used to compute acknowledgement gaps per path in Packet Threshold tests. 7.1.3. </ins> The ACK Delay field of the ACK frame is relative to the largest acknowledged packet number (see Section 13.2.5 of QUIC-TRANSPORT). When using paths with different transmission delays, the reported


Question:
This derives from the discussions on issue
The commit applies the changes suggested in the previous reviews. With those changes, I think we have a good basis for discussing a unified solution in the working group.
Thanks NAME for the review. I have applied the editorial changes. You are asking for something more regarding the "specific logic" required when supporting zero-length CID. There are in fact two pieces to that: logic at the receiver, i.e., the node that chose to receive packets with zero-length CID; and logic at the sender, i.e., the node that accepts to send data on multiple paths towards a node that uses zero-length CID. This is explained in details in the section "Using Zero-Length connection ID", so I guess what we need is a reference to that section.
Comment by NAME (on PR ): I think we expand this section a bit as we found getting the delay right was non-trivial. QUIC time-stamp would work for sure. But if one decides not to use QUIC timestamp, we probably want to explain what is needed to be done here.
I have submitted PR for this one.
Closing as got merged.
In section 3.1, we wrote When the multipath option is negotiated, clients that want to use an additional path MUST first initiate the Address Validation procedure with PATHCHALLENGE and PATHRESPONSE frames described in of ]. After receiving packets from the client on the new paths, the servers MAY in turn attempt to validate these paths using the same mechanisms. With multipath, we cannot simply write that the server MAY validate the new path. There is a risk of a malicious client that operates as follows: client creates path from its own address, A. Path is validate by server client starts long download from server client creates a second path to server using a spoofed address, B client starts address validation for B but does not care about the response from the server client stops acknowledging packets over the first path to force the server to move packets to the second one and then opportunistically acknowledges packets that should have been sent by the server over that second packet (SPNS would probably make such an attack simpler than MPNS) We probably need to start working on the security considerations section of the draft.
This is already covered in the QUIC threat model. The server is supposed to validate new paths by sending a "path challenge" on that path, then waiting for a challenge response. If the client spoofs a path, it will never receive the challenge, and thus the server will never receive the response and will not validate the path. See section 8.2 of RFC 9000, path validation. Security section 21.1.3. Connection Migration states that "Path validation () establishes that a peer is both willing and able to receive packets sent on a particular path."
So yes, the current text should be "the servers MUST in turn attempt to validate these paths using the same mechanisms, as specified in {{ Section 8.2 of RFC9000 }}."
No, this should not be a MUST. It is true that the path MUST be validated before it can be used. However, just because the serve receives a new packet on a path, that doesn't automatically mean that the server has to start path validation. It could e.g. also decide to simply withdraw that packet.
Actually, this is also not right. This is what RFC9000 says (section 9): So I guess we should maybe not use normative language here but refer to section 9 in RFC9000 instead...?
Just submitted PR for this issue.
This draft initially originates from a merging effort of previous Multipath proposals. While many points were found to be common between them, there remains one design point that still requires consensus: the number of packet number spaces that a Multipath QUIC connection should support (i.e., one for the whole connection vs. one per path). The current draft enables experimentation with both variants, but in the final version we will certainly need to choose between one of the versions.
The main issues mentioned so far: Efficiency. One number space per path should be more efficient, because implementations can directly reuse the loss-recovery logic specified for QUIC ACK size. Single space leads to lots of out of order delivery, which causes ACK sizes to grow Simplicity of code. Single space can be implemented without adding mane new code paths to uni-path QUIC Shorter header. Single space works well with NULL length CID The picoquic implementation shows that "efficiency" and "ack size" issues of single space implementations can be mitigated. However, that required significant improvements in the code: to obtain good loss efficiency, picoquic remembers not just the path on which a packet was sent, but also the order of the packet on this path, i.e. a "virtual sequence number". The loss detection logic then operates on that virtual sequence number. to contain ACK size, picoquic implements a prioritization logic to select the most important ranges in an ACK, avoid acking the same range more than 3 or 4 times, and keep knowledge of already acknowledged ranges so range coalescing works. I think these improvements are good in general, and I will keep them in the implementations whether we go for single space or not. The virtual sequence number is for example useful if the CID changes for reasons not related to path changes in multiple number space variants. It is also useful in unipath variants to avoid interference between sequence numbers used in probes and the RACK logic. The ACK size improvements do reduce the size of ACKs in presence of out of order delivery, e.g., if the network is doing some kind of internal load balancing. On the other hand, the improvements are somewhat complex, would need to be described in separate drafts, and pretty much contradicts the "simplicity of code" argument. So we are left with the "Null length CID" issue. I see for cases: Client CID Priority ----------------------- long Used by many implementations NULL Preferred configuration of many big deployments long Rarely used, server load balancing does not work NULL Only mentioned in some P2P deployments The point here is that it is somewhat hard to deploy a large server with NULL CID and use server load balancing. This configuration is mostly favored by planned P2P deployments.
The big debate is for the configuration with NULL CID on client, long CID on server. The packets from server to client do not carry a CID, and only the last bytes of the sequence number. The client will need some logic to infer the full sequence number before decrypting the packet. The client could maybe use the incoming 5 tuple as part of the logic, but it is not obvious. It is much simpler to assume a direct map from destination CID to number space. That means, if a peer uses a NULL CID, all packets sent to that peer are in the same number space.
Revised table: Client CID What -- -- long Multiple number space NULL Multiple number spaces on client side (one per CID), single space on server side long Multiple number spaces on server side (one per CID), single space on client side NULL single number space on each side If a node advertises both NULL CID and multipath support, they SHOULD have logic to contain the size of ACK. If a node engages in multipath with a NULL CID peer, they SHOULD have special logic to make loss recovery work well.
I think the above points to a possible "unified solution": if multipath negotiated, tie number spaces to destination CID, use default (N=0) if NULL CID. use multipath ack; in multipath ACK, identify number space by sequence number of DCID in received packets. Default that number to zero if received with NULL CID. in ABANDON path, identify path either by DCID of received packets, SCID of sent packet, or "this path" (sending path) if CID is NULL.
I like the proposal of the "unified solution". I think the elegance lies in the fact that it allows us to automatically cover all four cases listed above. The previous dilemma for me was that on one hand we have some use cases where we need to support more than two paths and separate PN makes the job easier, but on the other hand, I think we should not ignore the NULL CID use cases as it is also important. Now, with this proposal, a big part of the problem is solved. The rest of challenge is to make sure single PN remains efficient in terms of ACK and loss recovery. On that part, we plan to do an A/B test and would love to share the results when we get them. There is one more problem as pointed in issue , when we want to take hardware offloads into account. In such a case, we may still need single PN for long server CID. However, if hardware supports nonce modification, this problem can be addressed with the proposed "unified solution".
If the APi does not support 96 bit sequence numbers, it should always be possible to create an encryption context per number space, using Key=current key and ID = current-ID + CID sequence number. Of course, that forces creation of multiple context, and that makes key rotation a bit harder to manage. But still, that should be possible.
Thanks for the summary NAME I think one point is missing in your list which is related to issue . Use of a single packet number space might not support ECN. Regarding the unified solution: I think what you actually say is that we would specify both solutions and everybody would need to implement both logics. At least regarding the "simplicity of code" argument, that would be the worst choice. If we can make the multiple packet number spaces solution work with one-sided CIDs, I'm tending toward such an approach. Use of multiple packet number spaces avoids ambiguity/needed "smartness" in ACK handling and packet scheduling which, as you say above, can make the implementation more complex and, moreover, wrong decisions may have large impact on efficient (both local processing and on-the-wire). I don't think we want to leave these things to each implementation individually.
The summary Christian made above about design comparison sounds indeed quite accurate. Besides ECN, my other concern about single packet number is that it require cleverness from the receiver side if you want to be performant in some scenarios. At the sender-side, you need to consider a path-relative packet number threshold instead of an absolute one to avoid spurious losses. Just a point I think we did not mentioned yet is that there can be some interactions between Multipath out-of-order number delivery and incoming packet number duplicate detection. This requires maintaining some state at the receiver side, as described by URL With single packet number space, the receiver should take extra care when updating the "minimum packet number below which all packets are immediately dropped". Otherwise, in presence of paths with very different latencies, the receiver might end up discarding packets from a (slower) path. I'm also preferring the multiple packet number spaces solution for the above reasons. I'm not against thinking for a "unified" solution (the proposal sounds interesting), but I wonder how much complexity this would add compared to requiring end hosts to use one-byte CIDs.
I think the issue is not really so much "single vs multiple number space" as "support for multipath and NULL CID". As noted by Mirja, there is an implementation cost there. My take is: If a node uses NULL CID and proposes support of multipath, then that node MUST implement code to deal with size of acknowledgements. If a node sees a peer using NULL CID and supporting multipath, then that node MUST either use only one path at a time or implement code to deal with impact on loss detection due to out of order arrivals, and impact on congestion control including ambiguity of ECN signals. Then add sections on what it means to deal with the side of acknowledgements, out of order arrivals, and congestion control. I think this approach ends up with the correct compromises: It keeps the main case simple. Nodes that rely on multipath just use long CID and one number space per CID. Nodes that use long CID never have to implement the ACK length minimization algorithms. Nodes that see a peer using NULL CID have a variety of implementation choices, e.g., sending on just one path at a time, sending on mostly one path at a time in "make or break" transitions, or implementing sophisticated algorithms. If we do that, we can get rid of the single vs multiple space discussion, and end up with a single solution addressing all use cases.
Looking at all the discussions here and in other issues such as ECN, I think that we should try to write two different versions of section 7: one covering all the aspects of single packet number space (RTT estimation, retransmissions, ECN, handling of CIDs, ...) one covering all the aspects of multiple packet number space (RTT estimation, retransmissions, ECN, handling of CIDs, ...) There would be some overlap between these two sections and also some differences that would become clear as we specify them. At the end of this writing, we'll know whether it is possible to support/unify both or we need to recommend a single one. The other parts of the document are almost independent of that and can evolve in parallel with these two sections. However, I don't think that such a change would be possible by Monday
I don't think we should rush changes before we have agreed on the final vision.
It might be helpful to have these options as PRs (without merging) them, so people can understand all details of each approach.
I agree with NAME that supporting multi-path with null CIDs is a more fundamental issue than the efficiency comparison between single PN and separate PN, as it would ultimately impact the application scope of multipath QUIC. But indeed, we might want to implement proposed solution first and then decide if we want to adopt such a unified approach.
Since NAME prodded me, we now have a PR for the "unified" proposal.
I totally agree with Christian that the issue is about "support for multipath and NULL CID", and the solution that Christian suggested looks really great! It both takes advantage of multiple spaces, and support NULL CID users without affect the efficiency of ACK arrangements. Besides, the solution is more convenient for implementations, because If both endpoints uses non-zero length cids, endpoints only need to support multiple spaces, and if one of the endpoints use NULL CID, it could use single pn space in one direction and could support NULL CID and multipath at the same time.
Very high level summary of IETF-113 discussion seems that there is interest and likely support for the unified solution (review minutes for further details).

latency.  To collect ACK delays on all the paths, hosts should rely on time stamps as described in QUIC-Timestamp. <ins> 7.1.4. ECN feedback in QUIC is provided based on counters in the ACK frame (see Section 19.3.2. of QUIC-TRANSPORT).  That means if an ACK frame acknowledges multiple packets, the ECN feedback cannot be accounted to a specific packet. There are separate counters for each packet number space.  However, sending to zero-length CID receivers, the same number space is used for multiple paths.  Respectively, if an ACK frames acknowledges multiple packets from different paths, the ECN feedback cannot unambiguously be assigned to a path. If the sender marks its packets with the ECN capable flags, the network will expect standard reactions to ECN marks, such as slowing down transmission on the sending path.  If zero-length CID is used, the sending path is however ambiguous.  Therefore, the sender MUST treat a CE marking as a congestion signal on all sending paths that have been by a packet that was acknowledged in the ACK frame signaling the CE counter increase. A host that is sending over multiple paths to a zero-length CID receiver MAY disable ECN marking and send all subsequent packets as Not-ECN capable. 7.1.5. Hosts that are designed to support multipath using multiple number spaces MAY adopt a conservative posture after negotiating multipath support with a peer using zero-length CID.  The simplest posture is to only send data on one path at a time, while accepting packets on all acceptable paths.  In that case: the attribution of packets to path discussed in zero-length-cid- loss-and-congestion are easy to solve because packets are sent on a single path, the ACK Delays are correct, the vast majority of ECN marks relate to the current sending path. Of course, the hosts will only take limited advantage from the multipath capability in these scenarios.  Support for "make before break" migrations will improve, but load sharing between multiple paths will not work. </ins> 7.2. <del> If the multipath option is enabled with a value of 2, each path has its own packet number space for transmitting 1-RTT packets and a new ACK frame format is used as specified in ack-mp-frame.  Compared to the QUIC version 1 ACK frame, the ACK_MP frames additionally contains a Packet Number Space Identifier (PN Space ID).  The PN Space ID used to distinguish packet number spaces for different paths and is simply </del> <ins> If packets contain a non-zero CID, each path has its own packet number space for transmitting 1-RTT packets and a new ACK frame format is used as specified in ack-mp-frame.  Compared to the QUIC version 1 ACK frame, the ACK_MP frames additionally contains a Packet Number Space Identifier (PN Space ID).  The PN Space ID used to distinguish packet number spaces for different paths and is simply </ins> derived from the sequence number of Destination Connection ID. Therefore, the packet number space for 1-RTT packets can be identified based on the Destination Connection ID in each packets.


Question:
This PR is to resolve issue . The description of the path-closing figure is rephrased for better clarity. The pathidtype field in the PATHABANDON frame is also added in the figure to give an idea how the type is used. Moreover, one case for single PN is added (where the client receives zero-length CID while the server receives long CID). According to previous discussion, in the single PN case, only the client needs to send out a PATHABANDON. The server SHOULD stop sending new data on the path indicated by the PATHABANDON frame after receiving it. However, The client may want to repeat the PATHABANDON frame if it sees the server continuing to send data. It is optional for the server to respond with a PATHABANDON after it receives a PATHABANDON frame from the client.
There is one part I am afraid people might feel a little bit hard to follow: the three types of path identifiers in the PATH_STATUS frame. Those three types of path identifiers were introduced in order to be compatible with zero-length CID use cases, so I think we might need two additional example figures (like figure 3 in the current draft), one for (non-zero length CID, zero-length CID) and the other one for (zero-length CID & zero-length CID). Do you think that is a good idea? I can work on those figures if it doesn't complicate the flow of the draft.
This might be indeed useful in the "Example" section.
Yes, please create a PR!
NAME I am editing the path closing figures with zero-length CID and single PN. Is it similar to what you have in mind? {: #fig-example-path-close2 title="Example of closing a path when the server chooses to use zero-length CID"} {: #fig-example-path-close3 title="Example of closing a path when the both client and server choose to use zero-length CIDs."}
In the case of single number space, we use ACK, not ACKMP. But in fact you do not need to draw the ACKs there, there is no special processing of ACKs required. Also, there is no need to wait for an idle timeout, which typically is 10 to 30 seconds. That's too large. Do we really need to send "abandon path" in both directions? If I think about it, the real concern is that client will send a PATHABANDON message, but some time later, due to differences in delay, etc., the server receive data on the same four tuples. How does the server know whether it should create a new path or not? In the single space case, the logical test may be to check the sequence numbers of the packet. If the sequence number of the new packet is greater than that of the path_abandon, then the client is reactivating a path that was abandoned previously. Maybe the client connected again to the same network, after some disconnected interval. The server can start sending on the reactivated path. If the packet number is lower than that of the path abandon, then the server should not reactivate the path. In the multiple space case, if the CID is defined, there is no ambiguity.
ACK works for me. I think we might also want to add a sentence in the Sec. of "one packet number space" explicitly specifying that "for single PN, use ACK instead of ACKMP", and change some texts in the ACKMP frame section that is related to zero-length CID (the current draft says "if the endpoint receives 1-RTT packets with 0-length Connection ID, it SHOULD use Packet Number Space Identifier 0 in ACKMP frames". ). I think in single PN, sending "abandon path" in both directions is helpful. When the client sends "PATHABANDON", it promises not to send non-probing packets in the current activity cycle. It could send probing packets to reactivate the path a moment later if the client thinks the network environment has changed. After sending out "PATHABANDON", the client should still be able to receive packets from the server for a while on that path because there may be the server's inflight packets on that path. It was not until the client sees the server's "PATHABANDON", the client then knows that no more server's data is to be expected except for some out-of-order packets. So after receiving "PATHABANDON", the endpoint can start preparing to free memories and clear path stats allocated for that path. The time interval from seeing "PATHABANDON" to actually releasing relevant resources is going to be an important parameter, and yes, 10-30 seconds is too long. What would be a more appropriate number for this interval? Does it imply that we need to track the packet number of the PATH_ABANDON for a path that is closed? When a path is in a closed state, the server may reject any newly received packets unless it is a probing packet (back to our path initialization process). In this way, it should be OK to forget any info related to that path in the last activity cycle.
In most similar situations, RFC 9000 uses a delay of 3xRTO. Note that in the "simple" case, the client can receive packets from the server on any tuple. There are no resource allocated per path for receiving packets. There will be local resource allocated for sending packets, congestion context for example. But not on for receiving: all packets can be decrypted and processed. If the client knows that it will not send, it can discard the resource allocated for sending without risking a side effect.
I changed the timeout to 3RTO. As the problem is similar to FIN-ACK&FIN-ACK when TCP closes, I kept the ACK in the graph (because the pathidtype=2 refers to the path where the packet is traveling, we want to make sure it is acked before we close the path.). Regarding the resource, I think if we keep a unified receive record, which is not per-path, yes, we can discard the sending resources. But if we have some path stats (e.g. how many packets received in total in this current activity cycle), we need to keep these data structures until we are sure that the peer is not going to send more packets on that path. Indeed, it depends on the implementation, but I feel it is generally helpful to have PATH_ABANDON sent in both directions. What do you think? {: #fig-example-path-close2 title="Example of closing a path when the server chooses to use zero-length CID"} {: #fig-example-path-close3 title="Example of closing a path when both client and server choose to use zero-length CIDs."}
I submitted PR . Let's first have sth that we can start modifying.
addressed by PR
I think there is still a mismatch between the proposed examples and the definition of the pathidtype in Section 11.1 (definition of PATH_ABANDON frame).

threshold, or the quality of RTT or loss rate is becoming worse) and wants to close the initial path. <del> In Figure fig-example-path-close the server's 1-RTT packets use DCID C1, which has a sequence number of 1, for the first path; the client's 1-RTT packets use DCID S2, which has a sequence number of 2. For the second path, the server's 1-RTT packets use DCID C2, which has a sequence number of 2; the client's 1-RTT packets use CID S3, which has a sequence number of 3.  Note that two paths use different packet number space. The client initiates the path closure for the path with ID 1 by sending a packet with an PATH_ABANDON frame.  When the server received the PATH_ABANDON frame, it also sends an PATH_ABANDON frame in the next packet.  Afterwards the connection IDs in both directions can be retired using the RETIRE_CONNECTION_ID frame. </del> <ins> fig-example-path-close1 illustrates an example of path closing when both the client and the server use non-zero-length CIDs.  For the first path, the server's 1-RTT packets use DCID C1, which has a sequence number of 1; the client's 1-RTT packets use DCID S2, which has a sequence number of 2.  For the second path, the server's 1-RTT packets use DCID C2, which has a sequence number of 2; the client's 1-RTT packets use DCID S3, which has a sequence number of 3.  Note that the paths use different packet number spaces.  In this case, the client is going to close the first path.  It identifies the path by the sequence number of the received packet's DCID over that path (path identifier type 0x00), hence using the path_id 1.  Optionally, the server confirms the path closure by sending an PATH_ABANDON frame using the sequence number of the received packet's DCID over that path (path identifier type 0x00) as path identifier, which corresponds to the path_id 2.  Both the client and the server can close the path after receiving the RETIRE_CONNECTION_ID frame for that path. fig-example-path-close2 illustrates an example of path closing when the client chooses to receive zero-length CIDs while the server chooses to receive non-zero-length CIDs.  Because there is a zero- length CID in one direction, single packet number spaces are used. For the first path, the client's 1-RTT packets use DCID S2, which has a sequence number of 2.  For the second path, the client's 1-RTT packets use DCID S3, which has a sequence number of 3.  Again, in this case, the client is going to close the first path.  Because the client now receives zero-length CID packets, it needs to use path identifier type 0x01, which identifies a path by the DCID sequence number of the packets it sends over that path, and hence, it uses a path_id 2 in its PATH_ABANDON frame.  The server SHOULD stop sending new data on the path indicated by the PATH_ABANDON frame after receiving it.  However, The client may want to repeat the PATH_ABANDON frame if it sees the server continuing to send data. When the client's PATH_ABANDON frame is acknowledged, it sends out a RETIRE_CONNECTION_ID frame for the CID used on the first path.  The server can readily close the first path when it receives the RETIRE_CONNECTION_ID frame from the client.  However, since the client will not receive a RETIRE_CONNECTION_ID frame, after sending out the RETIRE_CONNECTION_ID frame, the client waits for 3 RTO before closing the path. </ins> 11.


Question:
Fix issue
In section 3.1, we wrote When the multipath option is negotiated, clients that want to use an additional path MUST first initiate the Address Validation procedure with PATHCHALLENGE and PATHRESPONSE frames described in of ]. After receiving packets from the client on the new paths, the servers MAY in turn attempt to validate these paths using the same mechanisms. With multipath, we cannot simply write that the server MAY validate the new path. There is a risk of a malicious client that operates as follows: client creates path from its own address, A. Path is validate by server client starts long download from server client creates a second path to server using a spoofed address, B client starts address validation for B but does not care about the response from the server client stops acknowledging packets over the first path to force the server to move packets to the second one and then opportunistically acknowledges packets that should have been sent by the server over that second packet (SPNS would probably make such an attack simpler than MPNS) We probably need to start working on the security considerations section of the draft.
This is already covered in the QUIC threat model. The server is supposed to validate new paths by sending a "path challenge" on that path, then waiting for a challenge response. If the client spoofs a path, it will never receive the challenge, and thus the server will never receive the response and will not validate the path. See section 8.2 of RFC 9000, path validation. Security section 21.1.3. Connection Migration states that "Path validation () establishes that a peer is both willing and able to receive packets sent on a particular path."
So yes, the current text should be "the servers MUST in turn attempt to validate these paths using the same mechanisms, as specified in {{ Section 8.2 of RFC9000 }}."
No, this should not be a MUST. It is true that the path MUST be validated before it can be used. However, just because the serve receives a new packet on a path, that doesn't automatically mean that the server has to start path validation. It could e.g. also decide to simply withdraw that packet.
Actually, this is also not right. This is what RFC9000 says (section 9): So I guess we should maybe not use normative language here but refer to section 9 in RFC9000 instead...?
Just submitted PR for this issue.

When the multipath option is negotiated, clients that want to use an additional path MUST first initiate the Address Validation procedure <del> with PATH_CHALLENGE and PATH_RESPONSE frames described in Section 8 of QUIC-TRANSPORT.  After receiving packets from the client on the new paths, the servers MAY in turn attempt to validate these paths using the same mechanisms. </del> <ins> with PATH_CHALLENGE and PATH_RESPONSE frames described in Section 8.2 of QUIC-TRANSPORT.  After receiving packets from the client on a new path, if the server decides to use the new path, the server MUST perform path validation Section 8.2 of QUIC-TRANSPORT unless it has previously validated that address. </ins> If validation succeed, the client can send non-probing, 1-RTT packets on the new paths.  In contrast with the specification in Section 9 of


Question:
I believe we can merge this now as it is. If we want to add more guidance later we should open a separate issue and another PR.
This issue came up during the discussion of issue . We say that ACK or ACK_MP frames can be send over any path, however, maybe we should discuss some guidance and trade-offs. E.g. sending on the same path as the packets received also for RTT measurements; using the shortest delay path might have some performance benefits...?
I agree. We should add some guidance here. We did some experiment regarding this issue. Our finding is returning ACK on the shortest path is helpful. But the improvement is only more significant when the path rtt ratio is larger than 4:1.

provides several general-purpose packet schedulers depending on the application goals. <ins> Note that the receiver could use a different scheduling strategy to send ACK(_MP) frames.  The recommended default behaviour consists in sending ACK(_MP) frames on the path they acknowledge packets.  Other scheduling strategies, such as sending ACK(_MP) frames on the lowest latency path, might be considered, but they could impact the sender with side effects on, e.g., the RTT estimation or the congestion control scheme.  When adopting such asymetrical acknowledgment scheduling, the receiver should at least ensure that the sender negotiated one-way delay calculation mechanism (e.g., QUIC- Timestamp). </ins> 8. Simultaneous use of multiple paths enables different retransmission


Question:
Fix issue
In section 3, we have However, the error code does not exist (anymore), and I'm not sure we need such a specific error code anyway. Why not raising a in such case?
It's reasonable to use TRANSPORTPARAMETERERROR. Submitted PR for this issue.

If endpoint receives an unexpected value for the transport parameter "enable_multipath", it MUST treat this as a connection error of type <del> MP_CONNECTION_ERROR and close the connection. </del> <ins> TRANSPORT_PARAMETER_ERROR (specified in Section 20.1 of QUIC- TRANSPORT) and close the connection. </ins> This extension does not change the definition of any transport parameter defined in Section 18.2. of QUIC-TRANSPORT.


Question:
as this is already stated normatively in RFC900.
Currently the draft say: However, we should not use normative language in this draft but just point to RFC9000 instead as things are defined normatively there already. Btw. this reference should point to section 13.3 ("Upon detecting losses, a sender MUST take appropriate congestion control action.") and 9.4 ("Packets sent on the old path MUST NOT contribute to congestion control or RTT estimation for the new path.") of RFC9000 specifically and maybe also to RFC9002 directly.
LGTMLGTM

7.1. <del> Senders MUST manage per-path congestion status, and MUST NOT send more data on a given path than congestion control on that path allows.  This is already a requirement of QUIC-TRANSPORT. </del> <ins> When the QUIC multipath extension is used, senders manage per-path congestion status as required in Section 9.4 of QUIC-TRANSPORT. However, in QUIC-TRANSPORT only one active path is assumed and as such the requirement is to reset the congestion control status on path migration.  With the multipath extension, multiple paths can be used simultaneously, therefore separate congestion control state is maintained for each path.  This means a sender is not allowed to send more data on a given path than congestion control for that path indicates. </ins> When a Multipath QUIC connection uses two or more paths, there is no guarantee that these paths are fully disjoint.  When two (or more


Question:
Not sure to understand why the last part of the sentence is an issue. The use of zero-length connection IDs is problematic anyway in setups with NATs, for example.
I agree that in many deployments you want at least one CID from the client to server. Multiple packet numbers as currently defined however requires CIDs in both direction; there is separate issue to discuss that. The next sentence also further explains that there might be other deployments though, e.g. in constraint networks, that might also benefit from saving the bits of the CID. This one sentence is of course only a very brief summary of the pros and cons of the different approach and more discussion is needed in the group. So I don't expect to have this part in the draft like this when we finally publish it as RFC. Is there anything in the issue that we need to address now? Or can we close this issue?
Thanks, Mirja. That's what I expected, but I do still think more elaboration is needed for this text to be useful. I would suggest to make this modification and leave the discussion of pros/cons to be adequately documented in Section 7: OLD: This proposal supports the negotiation of either the use of one packet number space for all paths or the use of separate packet number spaces per path. While separate packet number spaces allow for more efficient ACK encoding, especially when paths have highly different latencies, this approach requires the use of a connection ID. Therefore use of a single number space can be beneficial in highly constrained networks that do not benefit from exposing the connection ID in the header. While both approaches are supported by the specification in this version of the document, the intention for the final publication of a multipath extension for QUIC is to choose one option in order to avoid incompatibility. More evaluation and implementation experience is needed to select one approach before final publication. Some discussion about pros and cons can be found here: URL URL NEW: This proposal supports the negotiation of either the use of one packet number space for all paths or the use of separate packet number spaces per path. While both approaches are supported by the specification in this version of the document, the intention for the final publication of a multipath extension for QUIC is to choose one option in order to avoid incompatibility. More evaluation and implementation experience is needed to select one approach before final publication. Some discussion about pros and cons can be found here: URL URL
You are suggesting to strike out the sentences that try to summarize the differences, "While separate packet number spaces allow for more efficient ACK encoding, especially when paths have highly different latencies, this approach requires the use of a connectionID. Therefore use of a single number space can be beneficial in highly constrained networks that do not benefit from exposing the connection ID in the header." I think that's a good idea. The sentences are a short summary of the arguments presented in the powerpoint, which are not limited to "highly constrained networks". For example, the Chrome browser, by default, uses NULL CID. I may be placing words in the mouth of Google developers, but I believe their goal is to reduce the overhead of the QUIC header compared to TCP/TLS. This is related to a general quest for performance, not to network constraints.

This proposal supports the negotiation of either the use of one packet number space for all paths or the use of separate packet <del> number spaces per path.  While separate packet number spaces allow for more efficient ACK encoding, especially when paths have highly different latencies, this approach requires the use of a connection ID.  Therefore use of a single number space can be beneficial in highly constrained networks that do not benefit from exposing the connection ID in the header.  While both approaches are supported by the specification in this version of the document, the intention for the final publication of a multipath extension for QUIC is to choose one option in order to avoid incompatibility.  More evaluation and </del> <ins> number spaces per path.  While both approaches are supported by the specification in this version of the document, the intention for the final publication of a multipath extension for QUIC is to choose one option in order to avoid incompatibility.  More evaluation and </ins> implementation experience is needed to select one approach before final publication.  Some discussion about pros and cons can be found here: https://github.com/mirjak/draft-lmbdhk-quic-


Question:
Not sure to understand why the last part of the sentence is an issue. The use of zero-length connection IDs is problematic anyway in setups with NATs, for example.
I agree that in many deployments you want at least one CID from the client to server. Multiple packet numbers as currently defined however requires CIDs in both direction; there is separate issue to discuss that. The next sentence also further explains that there might be other deployments though, e.g. in constraint networks, that might also benefit from saving the bits of the CID. This one sentence is of course only a very brief summary of the pros and cons of the different approach and more discussion is needed in the group. So I don't expect to have this part in the draft like this when we finally publish it as RFC. Is there anything in the issue that we need to address now? Or can we close this issue?
Thanks, Mirja. That's what I expected, but I do still think more elaboration is needed for this text to be useful. I would suggest to make this modification and leave the discussion of pros/cons to be adequately documented in Section 7: OLD: This proposal supports the negotiation of either the use of one packet number space for all paths or the use of separate packet number spaces per path. While separate packet number spaces allow for more efficient ACK encoding, especially when paths have highly different latencies, this approach requires the use of a connection ID. Therefore use of a single number space can be beneficial in highly constrained networks that do not benefit from exposing the connection ID in the header. While both approaches are supported by the specification in this version of the document, the intention for the final publication of a multipath extension for QUIC is to choose one option in order to avoid incompatibility. More evaluation and implementation experience is needed to select one approach before final publication. Some discussion about pros and cons can be found here: URL URL NEW: This proposal supports the negotiation of either the use of one packet number space for all paths or the use of separate packet number spaces per path. While both approaches are supported by the specification in this version of the document, the intention for the final publication of a multipath extension for QUIC is to choose one option in order to avoid incompatibility. More evaluation and implementation experience is needed to select one approach before final publication. Some discussion about pros and cons can be found here: URL URL
You are suggesting to strike out the sentences that try to summarize the differences, "While separate packet number spaces allow for more efficient ACK encoding, especially when paths have highly different latencies, this approach requires the use of a connectionID. Therefore use of a single number space can be beneficial in highly constrained networks that do not benefit from exposing the connection ID in the header." I think that's a good idea. The sentences are a short summary of the arguments presented in the powerpoint, which are not limited to "highly constrained networks". For example, the Chrome browser, by default, uses NULL CID. I may be placing words in the mouth of Google developers, but I believe their goal is to reduce the overhead of the QUIC header compared to TCP/TLS. This is related to a general quest for performance, not to network constraints.

multipath negotiation is ambiguous, they MUST be interpreted as acknowledging packets sent on path 0. <ins> Endpoints negotiate the use of one packet number space for all paths or separate packet number spaces per path during the connection handshake nego.  While separate packet number spaces allow for more efficient ACK encoding, especially when paths have highly different latencies, this approach requires the use of a connection ID. Therefore use of a single number space can be beneficial when endpoints use zero-length connection ID for less overhead. </ins> 7.1. If the multipath option is negotiated to use one packet number space


Question:
NAME you wrote: That sounds like you want to restrict the multipath extension to only one server address (but potentially different paths from different clients address)...? I think this restriction is unnecessary. Yes we don't specify a way to learn server addresses but that doesn't means that the client doesn't know (e.g. based on higher layer information) multiple addresses of the server. What should we restrict that?
However, can we maybe merge this PR as it is for now and you open a separate issue for your additional point NAME ?
Mirja, That was my understanding of the sentences in the PR. Maybe I got it wrong. I think that we should discuss the role of preferred address in section 3 where we explain how paths can be opened.
Fine, let's do it this way.
RFC9000 defines a list of transport parameters. 0x01 0x02 0x03 0x04 0x05 0x06 0x07 0x08 0x09 0x0a 0x0b 0x0c 0x0d 0x0e 0x0f 0x10 [QUICWG] These transport parameters apply to all the paths that compose a Multipath QUIC connection. We should probably confirm this in the mpquic draft to avoid any misinterpretation (e.g. for maxidletimeout or ack delays). There are two transport parameters that could warrant some discussion: disableactivemigration : This transport parameter is incompatible with enablemultipath. An endpoint that receives both this transport parameter and a non zero enablemultipath parameter MUST treat this as a connection error of type MPCONNECTIONERROR and close the connection. preferredaddress: RFC9000, section 9.2.1 contains the following: Once the handshake is confirmed, the client SHOULD select one of the two addresses provided by the server and initiate path validation (see Section 8.2). A client constructs packets using any previously unused active connection ID, taken from either the preferredaddress transport parameter or a NEWCONNECTION_ID frame. As soon as path validation succeeds, the client SHOULD begin sending all future packets to the new server address using the new connection ID and discontinue use of the old server address. If path validation fails, the client MUST continue sending all future packets to the server's original IP address. For multipath, do we want to be stronger than SHOULD select one ? Can a client create news paths towards the initial address or only to one of the preferred addresses ? Should the client use both addresses (IPv4 and IPv6) if non-empty ?
Good point about "disableactivemigration". I would read it as also disabling multipath. We do not need to say that "transport parameters apply to all the paths that compose a Multipath QUIC connection". We are already inheriting that property from compatibility with path migration in RFC 9000. Multipath support is an extension to QUIC v1, not a new protocol, so we do not need to specify all the defaults. Same goes for the preferred address -- meaning does not change just because the nodes negotiate multipath. I would avoid debating "which server addresses the client can try" in the base multipath document. If we want the server to signal plausible addresses to the client, let's have a specific document for that.
I would maybe add a sentence that this draft does not change the meaning of preferred_address and if that address could also be used for multipath is out of scope for this extension.
Regarding disableactivemigration: this cannot be a connection error, as you can also send preferred address and after migration to the preferred address you can still use migration or multipath (if negotiated). I think the definition in RFC9000 is still valid; it means you can't send any packets from another address than the one used in the handshake. Which effectively means disabling multipath as well as Christian noted (if not moved to another preferred address). I guess we could explicitly note that this draft does not change the definition of disableactivemigration. For your reference this is what RFC9000 says:
I created a PR in . Is this sufficient to address this issue or do we need to say more? NAME ?
The proposed PR sounds good to me, with the slight addition I mentioned.

this limit even applies when no connection ID is exposed in the QUIC header. <ins> This extension does not change the definition of any transport parameter defined in Section 18.2. of QUIC-TRANSPORT.  Inline with the definition in QUIC-TRANSPORT disable_active_migration also disables multipath support, except "after a client has acted on a preferred_address transport parameter" Section 18.2. of QUIC- TRANSPORT.  Further, it is out of scope for this document to specify if the old address can also be used for multipath after a client has migrated to the address provided in the preferred_address transport parameter.  However, it SHOULD NOT be assumed that it is possible to use both addresses simultaneously without further confirmation from the other host. </ins> 3. After completing the handshake, endpoints have agreed to enable


Question:
This seems fine to me, but doesn't seem to include URL
NAME after trying to work that text in it became clear (to me) that a better outcome was modify the existing text we have about anonymity set reduction. So I went with that approach rather than open the can of consistency worms that NAME alluded to.
Let's say HPKE public key A was valid from 2022-07-01 to 2022-07-02. The client encapsulates a request X using pubkey A, but buffers it because of network issues, in order to retry later. Now it rotates and public key B is valid as of 2022-07-03. X encapsulated using pubkey A would not be decryptable by pubkey B. So if you receive X on 2022-07-03, then you know that X was generated between 2022-07-01 and 2022-07-02.
I'm not inclined to add this. "Don't do that" seems pretty obvious to me, especially when we recommend that requests include a Date.
While this advice seems fine, I agree that it's not really necessary. It also brings up more questions to me — what stops any layer below the OHTTP layer or an intermediary from arbitrarily buffering messages? Maybe just have a note that the key config that's used as well as explicit date markings can leak the time that the request was generated?
This seems like a reasonable way to deal with the problem. Liveness information can leak in a number of places, including through noting which public key was used. It might also leak via the date. Maybe this can become part of the differential treatment section?
That sounds good to me
I think that while this text is true, I believe that it is also unnecessary. Discussion of key liveness are dealt with in the key-consistency draft, so this is really about requests that are transmitted well after they are constructed. If we start to document that, there are lots of other things that start to come up.
I agree, but the content I'm proposing here relates to how liveness affects differential treatment. We've already noted a few key ways in which this can happen, so it seems pretty harmless to me to include one more. I don't think we need to say anything more about how to deal with liveness or how it pertains to consistency. I'll propose some text to make this more clear. No more than one sentence.

A relay MAY add information to requests if the client is aware of the nature of the information that could be added.  The client does not need to be aware of the exact value added for each request, but needs <del> to know the range of possible values the relay might use.  It is important to note that information added by the relay can reduce the </del> <ins> to know the range of possible values the relay might use. Importantly, information added by the relay - beyond what is already revealed through encapsulated requests from clients - can reduce the </ins> size of the anonymity set of clients at a gateway. Moreover, relays MAY apply differential treatment to clients that


Question:
This seems fine to me, but doesn't seem to include URL
NAME after trying to work that text in it became clear (to me) that a better outcome was modify the existing text we have about anonymity set reduction. So I went with that approach rather than open the can of consistency worms that NAME alluded to.
Let's say HPKE public key A was valid from 2022-07-01 to 2022-07-02. The client encapsulates a request X using pubkey A, but buffers it because of network issues, in order to retry later. Now it rotates and public key B is valid as of 2022-07-03. X encapsulated using pubkey A would not be decryptable by pubkey B. So if you receive X on 2022-07-03, then you know that X was generated between 2022-07-01 and 2022-07-02.
I'm not inclined to add this. "Don't do that" seems pretty obvious to me, especially when we recommend that requests include a Date.
While this advice seems fine, I agree that it's not really necessary. It also brings up more questions to me — what stops any layer below the OHTTP layer or an intermediary from arbitrarily buffering messages? Maybe just have a note that the key config that's used as well as explicit date markings can leak the time that the request was generated?
This seems like a reasonable way to deal with the problem. Liveness information can leak in a number of places, including through noting which public key was used. It might also leak via the date. Maybe this can become part of the differential treatment section?
That sounds good to me
I think that while this text is true, I believe that it is also unnecessary. Discussion of key liveness are dealt with in the key-consistency draft, so this is really about requests that are transmitted well after they are constructed. If we start to document that, there are lots of other things that start to come up.
I agree, but the content I'm proposing here relates to how liveness affects differential treatment. We've already noted a few key ways in which this can happen, so it seems pretty harmless to me to include one more. I don't think we need to say anything more about how to deal with liveness or how it pertains to consistency. I'll propose some text to make this more clear. No more than one sentence.

request that is made to the target.  This means that only the gateway or target are in a position to identify abuse.  A gateway MAY send signals toward the relay to provide feedback about specific requests. <del> A relay that acts on this feedback could - either inadvertently or by design - lead to clients being deanonymized. </del> <ins> For example, a gateway could respond differently to requests it cannot decapsulate, as mentioned in errors.  A relay that acts on this feedback could - either inadvertently or by design - lead to client deanonymization. </ins> 7.2.2.


Question: Section 5 details HTTP usage. It seems it's technically correct but the section closes with a sequence of paragraphs that jump back and forth between statements about how an Oblivious Gateway Resource treats requests or responses. Perhaps these could be consolidated to present requests first consistently. I.e. swap the final two paragraphs around

Oblivious Relay Resource also needs to observe the guidance in relay- responsibilities. <ins> An Oblivious Gateway Resource acts as a gateway for requests to the Target Resource (see HTTP).  The one exception is that any information it might forward in a response MUST be encapsulated, unless it is responding to errors it detects before removing encapsulation of the request; see errors. </ins> An Oblivious Gateway Resource, if it receives any response from the Target Resource, sends a single 200 response containing the encapsulated response.  Like the request from the client, this


Question: Section 5 details HTTP usage. It seems it's technically correct but the section closes with a sequence of paragraphs that jump back and forth between statements about how an Oblivious Gateway Resource treats requests or responses. Perhaps these could be consolidated to present requests first consistently. I.e. swap the final two paragraphs around

information that does not reveal information about the encapsulated response. <del> An Oblivious Gateway Resource acts as a gateway for requests to the Target Resource (see HTTP).  The one exception is that any information it might forward in a response MUST be encapsulated, unless it is responding to errors it detects before removing encapsulation of the request; see errors. </del> In order to achieve the privacy and security goals of the protocol an Oblivious Gateway Resource also needs to observe the guidance in server-responsibilities.


Question:
We debated this at some length. This defines an optional-to-use signaling method for the gateway so that it can inform the client when the key configuration is not acceptable. Clients can use this as a prompt to refresh their configuration.
(Originally posted by NAME in URL): Any lifetime information that comes with a key is helpful, but in practice is insufficient for clients to drive key updates. Key rotation may not occur on a fixed schedule and it may not occur synchronously (some gateway instances might not get the key at the right time. This means that clients cannot distinguish between failures due to configuration mismatch or request tampering (or whatever else might cause the gateway to reply without an encapsulated response). If the client can't distinguish then the client's recovery implementation is a lot more complicated. Does it always try to forcibly update its configuration? Does it implement some sort of backoff? Can a relay abuse the client's recovery implementation to somehow put the client into a state where it is stuck with an old configuration? It seems vastly simpler to just be clear that the configuration is wrong as a signal to the client that it's time to update. This was done in ODoH with a 401 response code. proposed a new problem statement, which I think is the right way to deal with this. Yes, there are potential privacy problems based on automated updates, but those depend on the client's consistency story and is therefore not something new a signal would introduce.
This looks great -- thanks!

indicate that the client configuration used to construct the request is incorrect or out of date. <ins> 5.3. The problem type PROBLEM of "https://iana.org/assignments/http- problem-types#ohttp-key" is defined.  An MAY use this problem type in a response to indicate that an used an outdated or incorrect . fig-key-problem shows an example response in HTTP/1.1 format. As this response cannot be encrypted, it might not reach the .  A cannot rely on the using this problem type.  A might also be configured to disregard responses that are not encapsulated on the basis that they might be subject to observation or modification by an .  A might manage the risk of an outdated using a heuristic approach whereby it periodically refreshes its if it receives a response with an error status code that has not been encapsulated. </ins> 6. In this design, a wishes to make a request of a server that is


Question:
We debated this at some length. This defines an optional-to-use signaling method for the gateway so that it can inform the client when the key configuration is not acceptable. Clients can use this as a prompt to refresh their configuration.
(Originally posted by NAME in URL): Any lifetime information that comes with a key is helpful, but in practice is insufficient for clients to drive key updates. Key rotation may not occur on a fixed schedule and it may not occur synchronously (some gateway instances might not get the key at the right time. This means that clients cannot distinguish between failures due to configuration mismatch or request tampering (or whatever else might cause the gateway to reply without an encapsulated response). If the client can't distinguish then the client's recovery implementation is a lot more complicated. Does it always try to forcibly update its configuration? Does it implement some sort of backoff? Can a relay abuse the client's recovery implementation to somehow put the client into a state where it is stuck with an old configuration? It seems vastly simpler to just be clear that the configuration is wrong as a signal to the client that it's time to update. This was done in ODoH with a 401 response code. proposed a new problem statement, which I think is the right way to deal with this. Yes, there are potential privacy problems based on automated updates, but those depend on the client's consistency story and is therefore not something new a signal would introduce.
This looks great -- thanks!

9. Please update the "Media Types" registry at <del> https://www.iana.org/assignments/media-types [1] for the media types </del> <ins> https://iana.org/assignments/media-types [1] for the media types </ins> "application/ohttp-keys" (iana-keys), "message/ohttp-req" (iana-req), and "message/ohttp-res" (iana-res). <ins> Please update the "HTTP Problem Types" registry at https://iana.org/assignments/http-problem-types [2] for the types "date" (iana-problem-date) and "ohttp-key" (iana-problem-ohttp-key). </ins> 9.1. The "application/ohttp-keys" media type identifies a used by


Question:
We debated this at some length. This defines an optional-to-use signaling method for the gateway so that it can inform the client when the key configuration is not acceptable. Clients can use this as a prompt to refresh their configuration.
(Originally posted by NAME in URL): Any lifetime information that comes with a key is helpful, but in practice is insufficient for clients to drive key updates. Key rotation may not occur on a fixed schedule and it may not occur synchronously (some gateway instances might not get the key at the right time. This means that clients cannot distinguish between failures due to configuration mismatch or request tampering (or whatever else might cause the gateway to reply without an encapsulated response). If the client can't distinguish then the client's recovery implementation is a lot more complicated. Does it always try to forcibly update its configuration? Does it implement some sort of backoff? Can a relay abuse the client's recovery implementation to somehow put the client into a state where it is stuck with an old configuration? It seems vastly simpler to just be clear that the configuration is wrong as a signal to the client that it's time to update. This was done in ODoH with a 401 response code. proposed a new problem statement, which I think is the right way to deal with this. Yes, there are potential privacy problems based on automated updates, but those depend on the client's consistency story and is therefore not something new a signal would introduce.
This looks great -- thanks!

IANA are requested to create a new entry in the "HTTP Problem Type" registry established by PROBLEM. <ins> 9.5. IANA are requested to create a new entry in the "HTTP Problem Type" registry established by PROBLEM. </ins> 10.  References 10.1.  URIs <del> [1] https://www.iana.org/assignments/media-types </del> <ins> [1] https://iana.org/assignments/media-types [2] https://iana.org/assignments/http-problem-types </ins> Index


Question: From NAME Can we (additionally or in replacement) point to the "HPKE KEM Identifiers" IANA registry created from this table instead? Same for this and "HPKE AEAD Identifiers" registry.

constructing and processing an . The Nenc parameter corresponding to the KEM used in HPKE can be found <del> in HPKE.  Nenc refers to the size of the encapsulated KEM shared secret, in bytes. </del> <ins> in HPKE or the HPKE KEM IANA registry [1].  Nenc refers to the size of the encapsulated KEM shared secret, in bytes. </ins> 4.2.


Question: From NAME Can we (additionally or in replacement) point to the "HPKE KEM Identifiers" IANA registry created from this table instead? Same for this and "HPKE AEAD Identifiers" registry.

constructing and processing an . The Nn and Nk values correspond to parameters of the AEAD used in <del> HPKE, which is defined in HPKE.  Nn and Nk refer to the size of the AEAD nonce and key respectively, in bytes.  The nonce length is set to the larger of these two lengths, i.e., max(Nn, Nk). </del> <ins> HPKE, which is defined in HPKE or the HPKE AEAD IANA registry [2]. Nn and Nk refer to the size of the AEAD nonce and key respectively, in bytes.  The nonce length is set to the larger of these two lengths, i.e., max(Nn, Nk). </ins> 4.3.


Question: From NAME Can we (additionally or in replacement) point to the "HPKE KEM Identifiers" IANA registry created from this table instead? Same for this and "HPKE AEAD Identifiers" registry.

9. Please update the "Media Types" registry at <del> https://iana.org/assignments/media-types [1] for the media types </del> <ins> https://iana.org/assignments/media-types [3] for the media types </ins> "application/ohttp-keys" (iana-keys), "message/ohttp-req" (iana-req), and "message/ohttp-res" (iana-res). Please update the "HTTP Problem Types" registry at <del> https://iana.org/assignments/http-problem-types [2] for the types </del> <ins> https://iana.org/assignments/http-problem-types [4] for the types </ins> "date" (iana-problem-date) and "ohttp-key" (iana-problem-ohttp-key). 9.1.


Question: From NAME Can we (additionally or in replacement) point to the "HPKE KEM Identifiers" IANA registry created from this table instead? Same for this and "HPKE AEAD Identifiers" registry.

10.1.  URIs <del> [1] https://iana.org/assignments/media-types </del> <ins> [1] https://www.iana.org/assignments/hpke/hpke.xhtml#hpke-kem-ids </ins> <del> [2] https://iana.org/assignments/http-problem-types </del> <ins> [2] https://www.iana.org/assignments/hpke/hpke.xhtml#hpke-aead-ids [3] https://iana.org/assignments/media-types [4] https://iana.org/assignments/http-problem-types </ins> Index