Question:
The endpoint sends frames, which carry -- not send -- identifier values.
The way it's intended to parse, "An endpoint ... MUST NOT increase the value they send," is probably incorrect in number or gender. But yes, changing the verb to reflect that the subject is the frames instead of the endpoint is also an option.
OK. I have no preference how this sentence is fixed -- as long as it's fixed.
This looks good. While we are at this paragraph, why say "client?" GOAWAY can be sent by either side now.
It can, but a server isn't going to retry push streams on a new connection; it's going to decide de novo whether to push there. Arguably, nothing breaks if the client increases the Push ID on the server after the fact.
BTW, thanks for catching this. I appreciate how many of these ambiguities you're winkling out.
NAME NAME OK, lgtm, then.
Maybe, given the ambiguity, the right answer is to get rid of "they" in the first place?

A client that is unable to retry requests loses all requests that are in flight when the server closes the connection.  An endpoint MAY <del> send multiple GOAWAY frames indicating different identifiers, but MUST NOT increase the identifier value they send, since clients might already have retried unprocessed requests on another connection. Receiving a GOAWAY containing a larger identifier than previously received MUST be treated as a connection error of type H3_ID_ERROR. </del> <ins> send multiple GOAWAY frames indicating different identifiers, but the identifier in each frame MUST NOT be greater than the identifier in any previous frame, since clients might already have retried unprocessed requests on another connection.  Receiving a GOAWAY containing a larger identifier than previously received MUST be treated as a connection error of type H3_ID_ERROR. </ins> An endpoint that is attempting to gracefully shut down a connection can send a GOAWAY frame with a value set to the maximum possible


Question:
This PR adds explicit context around the choice of ACK frequency in the draft. It also takes in Bob Briscoe's suggestion in , of calling it a guidance in light of the best information we have. Also does some editorial cleanup of text blocks that were not in the right sections.
Thanks NAME NAME -- comments incorporated.
As currently specified, the default QUIC ACK policy models TCP’s ACK policy, but consumes significantly more return path capacity. This can impact resource usage of the return channel, can constrain forward throughput, and can impact the performance of other sessions sharing an uplink (draft-fairhurst-quic-ack-scaling). This was not discussed in Zurich - where discussion moved instead to the separate topic of draft-iyengar-quic-delayed-ack. In many current network segments (e.g., cable networks, mobile cellular, WiFi and other radio tech), the effects of asymetry are presently mitigated by network devices RFC3449 that rely upon observing TCP headers. This results in ossification problems, but does practically have beneit. It reduces the TCP ACK rate over the constrained link, it reduces consumption of "radio" transmission opportunities, and reduces traffic volume from ACKs. These mitigations are not possible, nor desirable for QUIC. However, the default performance of QUIC is therefore much worse than for TCP. In QUIC, the default ACK Ratio has to set by the endpoints implementing the protocol because the sender and receiver have no information about the "cost" of sending an ACK using a link along an asymmetric path. These endpoints can also choose a non-default value to enable a new CC, scale to higher rates, etc (e.g. draft-iyengar-quic-delayed-ack). We have shown there can be significant benefit from an ACK Ratio of 1:10 and this is expected to alos have benefit for higher levels of asymmetry. The proposed change from 1:2 to 1:10 is predicated on QUIC's design decisions: starting a connection with an Initial Window of 10 is acceptable, QUIC uses PTO as the loss detection method (a predicate to issue ), and QUIC has some form of pacing at the sender, at least to mitigate IW bursts.
Even though we've avoided straying this far from the TCP spec, I think allowing this would be good for the internet. It should probably be documented for TCP, too. How does ack stretching of this nature work for BBR bandwidth estimation? It's harder to use the ack pacing, no?
I don't believe QUIC ACKs are significantly larger. In fact, I believe the ACK frame itself is typically smaller than TCP ACKs, but the savings there are used on QUIC's 16-byte auth hash and likely a non-zero-length CID. Do you have a size analysis written up somewhere? I think this is the core issue. It's not that QUIC ACKs are really consuming significantly more capacity in terms of bytes on the wire, it's that middleboxes can't decide to thin the ACKs, because they don't know which packets are ACK-only packets. Did you implement the optimization already described in the transport draft?: "As an optimization, a receiver MAY process multiple packets before sending any ACK frames in response. In this case the receiver can determine whether an immediate or delayed acknowledgement should be generated after processing incoming packets." NAME did and I believe found it hugely reduced the number of ACKs, sometimes too much. As I've said before, I would really prefer not specify an approach in QUIC transport that is a heuristic like does. In particular, on lower bandwidth networks, 10 packets may be the entire congestion window and is too large a value. On higher bandwidth networks, every 20 packets has shown to be slightly better than 10, at least when using BBR. I wrote to add some editorial text about why every 2 packets is a SHOULD, not a MUST. Chrome did not default enable the ACK strategy recommends until BBR was launched, because there was a performance regression with Cubic. I'd rather not assume everyone is using BBR or a BBR-like congestion controller. NAME BBR does fine with stretch ACKs. Cubic and Reno suffer in some cases, however.
QUIC ACKs are larger! The smallest IPv4 QUIC ACK packet is 51 bytes. The ACK frame is 4 bytes, but is sent using a QUIC packet. For IPv4 this is: 20 (IP) + 8 (UDP) +3 (1+1+1 QUIC header ... assuming a 1 B CID) + 4 (minimum ACK frame size)+ 16 (crypto). Other types of frames may also be sent in the same packet, increasing this size. That size difference matters. The point is that QUIC ACKs are consuming more capacity in terms of bytes on the wire, and often also important: in more transmission opportunities at the physical layer. That makes QUIC perform more poorly than it needs to across some paths. The text "As an optimization, a receiver MAY process multiple packets before sending any ACK frames in response. In this case the receiver can determine whether an immediate or delayed acknowledgement should be generated after processing incoming packets." Is permissive to do something different - alas the effect of ordering of the processing depends on rate, timing, etc - The present text is not a recommendation to do something appropriate, which I think it needs to. If this were to describe that a receiver SHOULD reduce the ACK to data packet ratio to 1:10, then that would help. On low bandwidth links, then I agree that waiting for 10 packets would not be suitable, but then wouldn't the rule 4 ACKs/RTT still cover this case? (I have no problems believing every an ACK 20 packets can be slightly better than 10 in some cases, but then we have chosen IW at 10 and that relates to bustiness and responsiveness of the basic spec, so I think 1:10 makes more sense as the recommendation. If the CC and path information suggest 1:20 - perhaps as the rate increases, then a sender can "tune" as per the transport parameter proposal. That I STILL assert that this is a different issue from the expected default.)
Would it help to give a recommendation like, if the congestion window is large, the ACK radio should be reduced? Not sure if we can easily give a threshold for the window but to align with the 4 ACKs per RTT rule we could say 40 or to be more conservation maybe 100?
If the receiver knew the cwnd... we could add that. The current proposal we made is here (along with some of the arguments about why QUIC and TCP are slightly different): draft-fairhurst-quic-ack-scaling
So the min TCP ack is 40 bytes vs QUIC at 50(0 byte CIDs are common, so I'll assume that), 25% larger in the minimum case. However, with a few SACK blocks, TCP ACKs are likely to be larger than QUIC ACKs, since SACK consumes 10 bytes for the first block and 8 for each additional block. By the time the second block is added, there's a good chance QUIC ACKs are smaller and it's almost certainly true with 3 SACK blocks(which I think is a typical number). So I'll go back to my previous statement that the real issue is that the network can't thin QUIC ACKs effectively.
May be "yes", may be "no" on a larger size after loss - since QUIC ACK ranges are often larger - but that depends on loss pattern, etc. In our case, we saw more bytes. But "yes" on me saying there is a need for QUIC to make good default decisions about the ACK rate, because QUIC becomes responsible for transport working across heterogeneous paths and we need to get good defaults rather than rely on "ACK-Thining" and other network modification.
Anybody has a gut feeling how very low latency congestion control fares with sparse ACKs?. I have experienced that DCTCP is more picky with ACK aggregation than e.g Reno. I suspect that ACK-thinning can have the give the issue as well. A too sparse ACK spacing can also give problems. Another aspect is how e.g. TCP Prague with paced chirps work with sparse ACKs ?. To me the RTT/4 rule makes sense but I feel that the best alternative depends a lot on both the congestion control and the source material. Large file transfers are more trivial than sources that are occasionally idle.
I think this isn't the correct question. This is about the default policy. The default policy is about what a receiver does by "default" - the current text uses a value of ACK every other packet, 1:2. TCP specified this when transmission rates were low, and TCP depended on individual ACKs to clock data (no pacing, IW=2), detect loss (no PTO), increase cwnd (per ACK packet), etc. In response, TCP causes ACK congestion and strange interactions with radio access. Performance over WiFi, DOCSIS etc suffers unless the network tries to address this. I don't see why QUIC, which doesn't rely on any of these, needs to set a DEFAULT of 1:2. This is literally a few lines of code difference, and we could use 1:10. These CC's you mention may have different needs and that needs to be handled by telling the remote that your CC has these specific needs - which I think is what Jana's extension is about.
I am kind of the donkey in the middle... ACK thinning in networks should be discouraged. Standardization of ACK thinning mechanisms was proposed in 3GPP RAN2, this request was turned down mainly with reference to that it can harm transport protocol performance and does not work safely with QUIC. Nothing stops UE (terminal and modem) vendors from implement proprietary ACK-thinning but I certainly hope that 5G UE vendors don't attempt to thin QUIC-ACKs as this is a slippery slope. High ACK rates is a 5G problem mainly for high band NR (=5G at e.g 28GHz). It is mainly a coverage issue as UEs have a limited output power due to practical and regulatory requirements. At the same time one need to address that end users don't only download movies today and even VoD streaming implies uplink traffic in the form of chunk requests. And then we have an increasing amount of people doing uplink streaming or upload an infinite amount of pictures of food plates and cats :-) Operators and 5G vendors have to deal with this and solve the problem with deployment of 5G access in various frequency bands to get the best downlink and uplink performance. Reduced ACK rates is of course welcome but I don't currently see a problem that the initial ACK ratio is 1:2 , to increase to 1:? as the congestion window grows, bounded by e.g an RTT/4 rule. But here I need to admit that I at least I have not studied this in detail. Ergo.. I am not convinced that it is necessary to hardwire the QUIC spec to 1:10 , but I also cannot say that it should not be done.
This is a small enough change. However, there is still a bar for those, and the current level of discussion makes me believe we should close this with no action?
Just so that it's easy to find, there is data and a continued discussion at URL
Folks who care about a change here need to prioritize progressing this issue. We're close to being able to last-call the specs, and would like this resolved in time.
We’ve completed a round of updated analysis of the proposed change to the QUIC spec, using quicly (changing to a 1:10 policy), and chromium (which currently does 1:10 by default). Our hope is that this change to the spec, to actually make standard end-to-end QUIC work better than TCP over asymmetric paths! This is a short slide deck about the performance of the proposed change: URL There have been a number of other change made by the QUIC ID editors relating to ACK processing, so we wanted to be sure these results were up to date. (These results do not consider additional ACKs after re-ordering - reflecting the latest text removing that.) We also really hope this will motivate a change the practice of forcing of QUIC fallback to TCP (with PEP/Thining) over highly asymmetric paths. People may also be interested in work for L4S to evaluate the impact of ACK Thining for TCP-Prague. Please do provide feedback, or any questions on this - as Lars notes, we think the time is running out for considering new changes, even relatively simple ones like this. Gorry P.S. First email: URL
I believe that it is beneficial to use a 1:10 policy, I understand that this means that a 1:2 policy that still be used in the initial phase (flow start) or does the timer mechanisms kick in to avoid that ACKs are transmitted too sparsely in this case ?.
Yes, our proposal counted ACK-eliciting packets at the receiver (because the receiver does not in general know the congestion state, so you need a heuristic), and we finally chose the same method in currect QUIC Chromium - i.e. for the first 100 packets we do not use ACK 1:10, changing only after 100 packets.
OK, it sounds like a good idea to use 1:10 after 100 packets or so. Just one potentially stupid question, Is there any special treatment necessary after an RTO ?
[This seems to be a forwarded comment from NAME Hi, Gorry Thank you for sharing your experiments. I'm delighted to see results showing that use of ack10 gives us comparable throughput to ack2. Recently, I have been looking into how ack-frequency affects the computational efficiency of the server [1]. Using the (almost) same setup, I have today run a benchmark comparing three ACK strategies: ack2, ack10, ack 1/8cwnd. ack2: 334Mbps ack10: 414Mbps ack 1/8cwnd: 487Mbps As can be seen, ack10 is better than ack2, but lags behind ack1/8cwnd. That's because there are still frequent ACKs. They take considerable amount of CPU cycles. If we want to see QUIC being as lightweight as TLS over TCP, it is critically important to reduce ACK frequency as much as possible. This is because the cost of processing ACK in QUIC is much more expensive than TCP, due QUIC being a userspace transport. Based on this, if we are to change what we have in the base drafts, I'd much prefer adopting what we have in the proposed delayed ack extension [2], rather than just changing the default ack policy from ack2 to ack10. By adopting what we have in the delayed ack extension, we can expect better performance, and the endpoints would have the freedom to change the behavior when the default ack policy is deemed insufficient (regardless of the default being ack2 or ack10). [1] URL [2] URL 2020428() 20:37 Gorry Fairhurst : issue. We're close to being able to last-call the specs, and would like this resolved in time. QUIC spec, using quicly (changing to a 1:10 policy), and chromium (which currently does 1:10 by default). Our hope is that this change to the spec, to actually make standard end-to-end QUIC work better than TCP over asymmetric paths! relating to ACK processing, so we wanted to be sure these results were up to date. (These results do not consider additional ACKs after re-ordering - reflecting the latest text removing that.) of QUIC fallback to TCP (with PEP/Thining) over highly asymmetric paths. People may also be interested in work for L4S to evaluate the impact of ACK Thining for TCP-Prague. think the time is running out for considering new changes, even relatively simple ones like this. URL
These results/article on computational efficiency are really interesting. Please see below. This is useful, we see here there's a clear benefit here in using a larger ACK Ratio also in terms of server load. We can see already the benefit at 1:10 and GSO helps as expected. We think that this is also really true. QUIC ACKs are also larger than TCP ACKs (see URL). I don't think the two approaches conflict. The difference between the two approaches is that the small change to the base specification to use an ACK Ratio 1:10 ensures the default return path is at least as good as TCP for common traffic (even with common TCP ACK Thinning). Without this change, the end user could experience significant differences in performance between TCP and between different QUIC versions over asymmetric paths. Whereas to me, the transport parameter extension looks like an exciting way to enable many more things, but there are also many ways this could usefully evolve based on CC, offload, pacing etec and I think this will take much more to work-out what is needed, safe and best ... then to reach consensus. We still also need to specify a default ACK Ratio in this case as well. Best wishes, Gorry
I'm greatly in favour of draft-iyengar-quic-delayed-ack-00 (BTW, where would you prefer me to give review comments about that?) I am much less keen on any default delayed ACK policy, whether 1:2 as now in quic-transport, or the 2-10/100 numbers proposed in draft-fairhurst-quic-ack-scaling-02 My concerns would melt away if the default policy was explicitly /implementation-dependent/ and the draft instead gave /guidance/ to implementers. The draft could still give the same 2-10/100 policy, but as /guidance/ for the current public Internet. The draft should also state the conditions under which this guidance was determined (e.g. Internet paths with BDP between x and y, or whatever). What's the difference? Stating a default policy implies this is a matter of interoperability. It's not. If we change to giving guidance, I'm sure most implementers will just use the guidance. However, implementers will be free to evolve their default as they gain experience. This reduces the possibility that other behaviours will evolve around any default like 2-10/100. For instance, network monitoring using it as a signature. Or congestion controls optimizing around it. In turn, this will reduce the chance of ossification around the first numbers we thought of. Senders can always use the ACK frequency frame to ask a receiver to change from its default. Then receivers will gradually learn what senders prefer them to do, and change their default accordingly. IMO, a continually improving default is important for short flows, so senders can avoid requesting a different ACK frequency right from the start.
Need to admit that I have not had too much time to follow the QUIC work lately. I too believe that draft-iyengar-quic-delayed-ack-00 is a good long term solution that allows endpoint to set a desired ACK frequency that is not more than necessary. As experiments show there are good incentives to keep the ACK frequency as low as possible in terms of client/server load and asymmetric links limitations. An ACK-FREQUENCY frame will allow to find the proper balance between the requirements given by the congestion control, the traffic type and the above given limitations. I am not too concerned if it takes an extra development/standards cycle to have that in place in the standard and in running code as long as the QUIC community shows other SDOs that the challenge with QUIC ACKs is being addressed.
NAME NAME could you please trim the quoted text when you reply by email?
(I trimmed the comments, but forwarding inline replies to email messages doesn't work well.)
Given that this is an area where there is still obviously experimentation and refinement, NAME comment seems on point; baking a specific algorithm as requirements into the specifications doesn't seem like a good idea (although putting constraints on endpoints might make sense). Can we focus on general guidance rather than documenting a specific algorithm? What would that PR look like?
+1 to NAME suggestion. If we want to make changes for v1, they need to happen now, and they must be in a form that can let us get consensus.
To be clear, when I said that having a default policy is not a matter of interoperability, that's only strictly true if QUIC supports something like draft-iyengar-quic-delayed-ack for the sender to ask the receiver to use a certain policy. Bob
I've proposed more context and text around our current guidance in the draft, see .
I believe that the text around asymmetric links look fine, not really necessary to write up much more. What can be good to know that for the 5G case with limited UL (uplink = modem->base station transmission) , the pain that a flow with an unnecessary high ACK rate causes is on the flow itself or other flows sharing the same access.
There are some different points of pain here, the default and the need to adapt to the endpoint: An ACK Ratio (AR 1:1): is the finest granuality of feedback. A QUIC AR of 1:1 would consume return path capacity proportional to about 6% of the forward traffic (assuming no loss, etc). This is more than TCP (about 3%). An ACK Ratio (AR 1:2): For a return path bottleneck, QUIC with an AR of 1:2 will still have more impact on sharing flows than TCP would with the same AR 1:2 (because QUIC ACKs are larger, and because networks do not currently thin QUIC ACKs). So QUIC needs to choose the default - is it 1:2, 1:4, 1:8, 1:10? A QUIC AR 1:10 results in an ACK traffic volume of about 1% of the forward traffic. To me, that's a small proportion, which is why I argued that 1:10 is a good recommendation. The default is important when: the endpoints have no knowledge of the asymetry or the impact they have on a link's layer 2 loading, or the endpoints need to reduce the collateral damage to other IP flows sharing the return path capacity. Negotiation: The AR can be driven by the application or endpoint - e.g., a higher rate connection may well have a benefit from an AR of 1:10, 1:20, 1:100, etc - but such ARs are seldom chosen to reduce the impact on sharing a return bottleneck, more likely they are chosen to benefit the performance of an application or endpoint. That consideration is different, and likely depends on the application or ednpoint - which to me is where the flexibility offered by a transport parameter is so important. draft-iyengar-quic-delayed-ack will still need guidance on the "default". I argued that the default policy impacts the congestion behaviour when there is appreciable asymmetry this causes at layer 2 or the collateral damage (congestion impact on sharing flows). Sure, the choice of 1% is just a number I said was reasonable, but I've seldom encountered L2 technologies where less than 1% of forward traffic would be significantly better.
NAME : I understand your position. While I agree that ack thinning happens with TCP, it is not what I expect to find on a common network path. And as far as I know, the endpoints still implement the RFC 5681 recommendation of acking every other packet. What you are arguing is that with ack thinners in the network, TCP's behavior on those network paths is different. However, I do not believe we understand the performance effects of these middleboxes, that is, how they might reduce connection throughput when the asymmetry is not a problem. Importantly, I don't think we should be specifying as default what some middleboxes might be doing without fully understanding the consequences of doing this. Please bear in mind that Chrome's ack policy of 1:10 works with BBR, not with Cubic or Reno. I do not believe there is a magic number at the moment, and as NAME noted, even 1:10 is not a small enough ratio under some conditions. Given this, and given that we know 1:2 is used by TCP endpoints, it makes the most sense to specify that as the suggested guidance. The new text proposed in describes the tradeoff involved, and the rationale for using 1:2. I am strongly against doing anything substantially different than what we have in the draft now without overwhelming information supporting a different position.
Agree with NAME 1:2 seems a reasonable default, and there seems to be considerable interest in draft-iyengar-quic-delayed-ack, which has a better proposal for stacks that care.
I think the new ACK text is improving the specification, but I'd like to be sure we have thought about this decision. Let me try to carefully respond on-list here, to see where we agree/disagree: I disagree, I think we shouldn't be perpetuating a myth that a sender receives a TCP ACK for every other packet. Even in the 90's implementations TCP stacks often generated stretch-ACKs for 3 segments. Since then, limited byte counting was introduced and significantly improved the sender's response to stretch ACKs, and that has been good for everyone. Senders still rely on byte-counting in TCP to handle Stretch-ACKs, and this need is not decreasing: new network cards reduce per-packet receive processing using Large Receive Offload (LRO) or Generic Receiver Offload (GRO). I am saying many TCP senders actually do today see stretch-ACKs. There are papers that have shown significant presence of stretch ACKs, e.g., (H. Ding and M. Rabinovich. TCP stretch acknowledgements and timestamps: findings and implications for passive RTT measurement. ACM SIGCOMM CCR, 45(3):20–27, 2015.). A TCP Stretch-ACK of 3 or 4 was common in their datasets (about 10% of cases). In some networks, the proportion of Stretch-ACKs will be much much higher, since ACK Thining is now widely deployed in WiFi drivers as well as cable, satellite and other access technologies - often reducing ACK size/rate by a factor of two. I am intrigued and would like to know more, in case I missed something? Typical algorithms I am aware of track a queue at the "bottleneck" and then make a decision based on upon the contents of the queue. If there is no asymmtery, there isn't normally a queue and the algorithm won't hunt to discard an "ACK". But, maybe, there is some truth in being wary: Encrypted flows can also build return path queues (as does QUIC traffic). Without an ability to interpret the transport data (which nobody, including me, wants for QUIC), a bottleneck router is forced to either use another rule to drop packets from the queue (such as the smallest packets) or to allow a queue to grow, limiting forward direction throughput. I would say neither of these outcomes are great because the router is trying to fix the transport's asymetry. However, if QUIC causes this problem, I suggest some method will proliferate as QUIC traffic increases. I didn't find something in QUIC spec. that was a major concern. It looked to me like QUIC had learned from TCP how to handle stretch-ACKs. After that, we changed quicly to use 1:10 with Reno, and things worked well. Didn't NAME also use Reno? Sure, the deisgn of TCP recognised that AR 1:1 generated too much traffic, AR 1:2 for TCP resulted in an Forward:Return traffic ratio of ~1.5%, and QUIC increases this ~3%. However, as network link speeds increased, this proved too low for many network links and so, TCP ACK thinning came into being. This often reduces the TCP AR to around 1:4 ( and will do this without bias, whatever the outcome is.
NAME Just to clarify, in our experiment, we reduced the ACK frequency from the default (2) to after leaving slow start, then updated it every 4 PTO. I think that would work fine on a high-bandwidth, stable network. But I am not sure if it would work as well as ack2 on all network conditions. Anyways, I'd explain our experiment as an optimization of ack 2, as it was reduction of ack rate after leaving slow start. I think that that was a sensible choice, because IIUC having frequent acks during slow start is essential. Assuming that we'd prefer using ack 2 during slow start, the question is if we need to hard-code in the base draft how to optimize after leaving slow start. While I agree that continuing to use ack 2 after slow start is concerning for certain networks, I am reluctant to spending time on looking for such a fixed set of values across diverse networks (including much slower links or lossy ones). Rather, my preference goes to shipping the base-draft as-is, then moving on to finalize the ack-frequency frame. I think we can assume most if not all of the popular clients to support the ack-frequency frame. Then, it becomes less of a question what the default in the base-draft is.
Thanks kazuho, 1/8 or 1/4 cwnd seems reasonable, as does using Ack 2 for the first part of slow start ... that closed to what we proposed. And yes there will anyway be cases where any default does not work. If QUIC decides to retain Ack 2, can we at least capture that Ack 2 with QUIC doesn't have the same return path performance as Ack 2 with TCP - so that people know that less frequent Acts are something to think about from the start? I'm doing a talk in PANRG this week, and have decided to talk-through this topic (I was already going to talk about things related to return paths). That'll give people a view of some data - and a little thought about the implications this has for QUIC, and whether there is an incentive for network device to thin QUIC to restore performance.
NAME : I'd be in support of saying that ack thinning happens for TCP and cite those studies, and that in the absence of that, there might be throughput consequences for QUIC. I think we've said exactly that in other words in , but if we want to say more there, that's fine with me. That would make this an editorial issue, but I'm entirely happy to do more words there. I am against changing the recommendation here for another arbitrary recommendation. I think the right approach is to allow endpoints to specify this during a connection, which is where the extension is helpful. As NAME notes, our hope is that the extension gets widely deployed. If you want to make the argument for the extension to become part of the base draft, that's a different conversation, but that's not the one we're having here.
Attempting to summarise the current status of this issue as a chair: there appears to have been a fair amount of discussion about making a design change, with representatives both in favour and against. It appears there is no omnipotent default policy, choosing an optimal one depends on many factors. The editors have landed an editorial change that attempts to better describe the rationale behind the QUICS's current design and highlights some of the factors. There may be further editorial improvements but they no not need to block the design issue. The proposal therefore is to close this issue with no design change. We had prepared some slides for the Vancouver IETD to discuss QUIC's ACKs, and how to make QUIC work better than TCP in common asymmetric paths. This is a short slide deck that summarises what we have learned about ACKs in QUIC: URL It compares the ACK traffic of TCP with that of QUIC from the point of view of a path where QUIC's return traffic is somehow constrained or could impact other user traffic. We think that ACKs are a concern for various shared capacity return link (e.g., radio-based methods in WiFi or mobile cellular were ACKs consume spectrum/radio resource or battery; shared capacity satellite broadband return link or the uplink of a cable system). For TCP, such systems commonly involve a PEP or an ACK Filtering mechanism (which clearly isn't wanted or even possible for QUIC). The baseline QUIC performance (as per current spec) suffers over these paths. So our first question is: Do other people think it is important that the default specified for QUIC works well with asymmetric paths ? Gorry

receipt by sending one or more ACK frames containing the packet number of the received packet. <ins> An endpoint SHOULD treat receipt of an acknowledgment for a packet it did not send as a connection error of type PROTOCOL_VIOLATION, if it is able to detect the condition. </ins> 13.2. Endpoints acknowledge all packets they receive and process.  However,


Question:
This PR adds explicit context around the choice of ACK frequency in the draft. It also takes in Bob Briscoe's suggestion in , of calling it a guidance in light of the best information we have. Also does some editorial cleanup of text blocks that were not in the right sections.
Thanks NAME NAME -- comments incorporated.
As currently specified, the default QUIC ACK policy models TCP’s ACK policy, but consumes significantly more return path capacity. This can impact resource usage of the return channel, can constrain forward throughput, and can impact the performance of other sessions sharing an uplink (draft-fairhurst-quic-ack-scaling). This was not discussed in Zurich - where discussion moved instead to the separate topic of draft-iyengar-quic-delayed-ack. In many current network segments (e.g., cable networks, mobile cellular, WiFi and other radio tech), the effects of asymetry are presently mitigated by network devices RFC3449 that rely upon observing TCP headers. This results in ossification problems, but does practically have beneit. It reduces the TCP ACK rate over the constrained link, it reduces consumption of "radio" transmission opportunities, and reduces traffic volume from ACKs. These mitigations are not possible, nor desirable for QUIC. However, the default performance of QUIC is therefore much worse than for TCP. In QUIC, the default ACK Ratio has to set by the endpoints implementing the protocol because the sender and receiver have no information about the "cost" of sending an ACK using a link along an asymmetric path. These endpoints can also choose a non-default value to enable a new CC, scale to higher rates, etc (e.g. draft-iyengar-quic-delayed-ack). We have shown there can be significant benefit from an ACK Ratio of 1:10 and this is expected to alos have benefit for higher levels of asymmetry. The proposed change from 1:2 to 1:10 is predicated on QUIC's design decisions: starting a connection with an Initial Window of 10 is acceptable, QUIC uses PTO as the loss detection method (a predicate to issue ), and QUIC has some form of pacing at the sender, at least to mitigate IW bursts.
Even though we've avoided straying this far from the TCP spec, I think allowing this would be good for the internet. It should probably be documented for TCP, too. How does ack stretching of this nature work for BBR bandwidth estimation? It's harder to use the ack pacing, no?
I don't believe QUIC ACKs are significantly larger. In fact, I believe the ACK frame itself is typically smaller than TCP ACKs, but the savings there are used on QUIC's 16-byte auth hash and likely a non-zero-length CID. Do you have a size analysis written up somewhere? I think this is the core issue. It's not that QUIC ACKs are really consuming significantly more capacity in terms of bytes on the wire, it's that middleboxes can't decide to thin the ACKs, because they don't know which packets are ACK-only packets. Did you implement the optimization already described in the transport draft?: "As an optimization, a receiver MAY process multiple packets before sending any ACK frames in response. In this case the receiver can determine whether an immediate or delayed acknowledgement should be generated after processing incoming packets." NAME did and I believe found it hugely reduced the number of ACKs, sometimes too much. As I've said before, I would really prefer not specify an approach in QUIC transport that is a heuristic like does. In particular, on lower bandwidth networks, 10 packets may be the entire congestion window and is too large a value. On higher bandwidth networks, every 20 packets has shown to be slightly better than 10, at least when using BBR. I wrote to add some editorial text about why every 2 packets is a SHOULD, not a MUST. Chrome did not default enable the ACK strategy recommends until BBR was launched, because there was a performance regression with Cubic. I'd rather not assume everyone is using BBR or a BBR-like congestion controller. NAME BBR does fine with stretch ACKs. Cubic and Reno suffer in some cases, however.
QUIC ACKs are larger! The smallest IPv4 QUIC ACK packet is 51 bytes. The ACK frame is 4 bytes, but is sent using a QUIC packet. For IPv4 this is: 20 (IP) + 8 (UDP) +3 (1+1+1 QUIC header ... assuming a 1 B CID) + 4 (minimum ACK frame size)+ 16 (crypto). Other types of frames may also be sent in the same packet, increasing this size. That size difference matters. The point is that QUIC ACKs are consuming more capacity in terms of bytes on the wire, and often also important: in more transmission opportunities at the physical layer. That makes QUIC perform more poorly than it needs to across some paths. The text "As an optimization, a receiver MAY process multiple packets before sending any ACK frames in response. In this case the receiver can determine whether an immediate or delayed acknowledgement should be generated after processing incoming packets." Is permissive to do something different - alas the effect of ordering of the processing depends on rate, timing, etc - The present text is not a recommendation to do something appropriate, which I think it needs to. If this were to describe that a receiver SHOULD reduce the ACK to data packet ratio to 1:10, then that would help. On low bandwidth links, then I agree that waiting for 10 packets would not be suitable, but then wouldn't the rule 4 ACKs/RTT still cover this case? (I have no problems believing every an ACK 20 packets can be slightly better than 10 in some cases, but then we have chosen IW at 10 and that relates to bustiness and responsiveness of the basic spec, so I think 1:10 makes more sense as the recommendation. If the CC and path information suggest 1:20 - perhaps as the rate increases, then a sender can "tune" as per the transport parameter proposal. That I STILL assert that this is a different issue from the expected default.)
Would it help to give a recommendation like, if the congestion window is large, the ACK radio should be reduced? Not sure if we can easily give a threshold for the window but to align with the 4 ACKs per RTT rule we could say 40 or to be more conservation maybe 100?
If the receiver knew the cwnd... we could add that. The current proposal we made is here (along with some of the arguments about why QUIC and TCP are slightly different): draft-fairhurst-quic-ack-scaling
So the min TCP ack is 40 bytes vs QUIC at 50(0 byte CIDs are common, so I'll assume that), 25% larger in the minimum case. However, with a few SACK blocks, TCP ACKs are likely to be larger than QUIC ACKs, since SACK consumes 10 bytes for the first block and 8 for each additional block. By the time the second block is added, there's a good chance QUIC ACKs are smaller and it's almost certainly true with 3 SACK blocks(which I think is a typical number). So I'll go back to my previous statement that the real issue is that the network can't thin QUIC ACKs effectively.
May be "yes", may be "no" on a larger size after loss - since QUIC ACK ranges are often larger - but that depends on loss pattern, etc. In our case, we saw more bytes. But "yes" on me saying there is a need for QUIC to make good default decisions about the ACK rate, because QUIC becomes responsible for transport working across heterogeneous paths and we need to get good defaults rather than rely on "ACK-Thining" and other network modification.
Anybody has a gut feeling how very low latency congestion control fares with sparse ACKs?. I have experienced that DCTCP is more picky with ACK aggregation than e.g Reno. I suspect that ACK-thinning can have the give the issue as well. A too sparse ACK spacing can also give problems. Another aspect is how e.g. TCP Prague with paced chirps work with sparse ACKs ?. To me the RTT/4 rule makes sense but I feel that the best alternative depends a lot on both the congestion control and the source material. Large file transfers are more trivial than sources that are occasionally idle.
I think this isn't the correct question. This is about the default policy. The default policy is about what a receiver does by "default" - the current text uses a value of ACK every other packet, 1:2. TCP specified this when transmission rates were low, and TCP depended on individual ACKs to clock data (no pacing, IW=2), detect loss (no PTO), increase cwnd (per ACK packet), etc. In response, TCP causes ACK congestion and strange interactions with radio access. Performance over WiFi, DOCSIS etc suffers unless the network tries to address this. I don't see why QUIC, which doesn't rely on any of these, needs to set a DEFAULT of 1:2. This is literally a few lines of code difference, and we could use 1:10. These CC's you mention may have different needs and that needs to be handled by telling the remote that your CC has these specific needs - which I think is what Jana's extension is about.
I am kind of the donkey in the middle... ACK thinning in networks should be discouraged. Standardization of ACK thinning mechanisms was proposed in 3GPP RAN2, this request was turned down mainly with reference to that it can harm transport protocol performance and does not work safely with QUIC. Nothing stops UE (terminal and modem) vendors from implement proprietary ACK-thinning but I certainly hope that 5G UE vendors don't attempt to thin QUIC-ACKs as this is a slippery slope. High ACK rates is a 5G problem mainly for high band NR (=5G at e.g 28GHz). It is mainly a coverage issue as UEs have a limited output power due to practical and regulatory requirements. At the same time one need to address that end users don't only download movies today and even VoD streaming implies uplink traffic in the form of chunk requests. And then we have an increasing amount of people doing uplink streaming or upload an infinite amount of pictures of food plates and cats :-) Operators and 5G vendors have to deal with this and solve the problem with deployment of 5G access in various frequency bands to get the best downlink and uplink performance. Reduced ACK rates is of course welcome but I don't currently see a problem that the initial ACK ratio is 1:2 , to increase to 1:? as the congestion window grows, bounded by e.g an RTT/4 rule. But here I need to admit that I at least I have not studied this in detail. Ergo.. I am not convinced that it is necessary to hardwire the QUIC spec to 1:10 , but I also cannot say that it should not be done.
This is a small enough change. However, there is still a bar for those, and the current level of discussion makes me believe we should close this with no action?
Just so that it's easy to find, there is data and a continued discussion at URL
Folks who care about a change here need to prioritize progressing this issue. We're close to being able to last-call the specs, and would like this resolved in time.
We’ve completed a round of updated analysis of the proposed change to the QUIC spec, using quicly (changing to a 1:10 policy), and chromium (which currently does 1:10 by default). Our hope is that this change to the spec, to actually make standard end-to-end QUIC work better than TCP over asymmetric paths! This is a short slide deck about the performance of the proposed change: URL There have been a number of other change made by the QUIC ID editors relating to ACK processing, so we wanted to be sure these results were up to date. (These results do not consider additional ACKs after re-ordering - reflecting the latest text removing that.) We also really hope this will motivate a change the practice of forcing of QUIC fallback to TCP (with PEP/Thining) over highly asymmetric paths. People may also be interested in work for L4S to evaluate the impact of ACK Thining for TCP-Prague. Please do provide feedback, or any questions on this - as Lars notes, we think the time is running out for considering new changes, even relatively simple ones like this. Gorry P.S. First email: URL
I believe that it is beneficial to use a 1:10 policy, I understand that this means that a 1:2 policy that still be used in the initial phase (flow start) or does the timer mechanisms kick in to avoid that ACKs are transmitted too sparsely in this case ?.
Yes, our proposal counted ACK-eliciting packets at the receiver (because the receiver does not in general know the congestion state, so you need a heuristic), and we finally chose the same method in currect QUIC Chromium - i.e. for the first 100 packets we do not use ACK 1:10, changing only after 100 packets.
OK, it sounds like a good idea to use 1:10 after 100 packets or so. Just one potentially stupid question, Is there any special treatment necessary after an RTO ?
[This seems to be a forwarded comment from NAME Hi, Gorry Thank you for sharing your experiments. I'm delighted to see results showing that use of ack10 gives us comparable throughput to ack2. Recently, I have been looking into how ack-frequency affects the computational efficiency of the server [1]. Using the (almost) same setup, I have today run a benchmark comparing three ACK strategies: ack2, ack10, ack 1/8cwnd. ack2: 334Mbps ack10: 414Mbps ack 1/8cwnd: 487Mbps As can be seen, ack10 is better than ack2, but lags behind ack1/8cwnd. That's because there are still frequent ACKs. They take considerable amount of CPU cycles. If we want to see QUIC being as lightweight as TLS over TCP, it is critically important to reduce ACK frequency as much as possible. This is because the cost of processing ACK in QUIC is much more expensive than TCP, due QUIC being a userspace transport. Based on this, if we are to change what we have in the base drafts, I'd much prefer adopting what we have in the proposed delayed ack extension [2], rather than just changing the default ack policy from ack2 to ack10. By adopting what we have in the delayed ack extension, we can expect better performance, and the endpoints would have the freedom to change the behavior when the default ack policy is deemed insufficient (regardless of the default being ack2 or ack10). [1] URL [2] URL 2020428() 20:37 Gorry Fairhurst : issue. We're close to being able to last-call the specs, and would like this resolved in time. QUIC spec, using quicly (changing to a 1:10 policy), and chromium (which currently does 1:10 by default). Our hope is that this change to the spec, to actually make standard end-to-end QUIC work better than TCP over asymmetric paths! relating to ACK processing, so we wanted to be sure these results were up to date. (These results do not consider additional ACKs after re-ordering - reflecting the latest text removing that.) of QUIC fallback to TCP (with PEP/Thining) over highly asymmetric paths. People may also be interested in work for L4S to evaluate the impact of ACK Thining for TCP-Prague. think the time is running out for considering new changes, even relatively simple ones like this. URL
These results/article on computational efficiency are really interesting. Please see below. This is useful, we see here there's a clear benefit here in using a larger ACK Ratio also in terms of server load. We can see already the benefit at 1:10 and GSO helps as expected. We think that this is also really true. QUIC ACKs are also larger than TCP ACKs (see URL). I don't think the two approaches conflict. The difference between the two approaches is that the small change to the base specification to use an ACK Ratio 1:10 ensures the default return path is at least as good as TCP for common traffic (even with common TCP ACK Thinning). Without this change, the end user could experience significant differences in performance between TCP and between different QUIC versions over asymmetric paths. Whereas to me, the transport parameter extension looks like an exciting way to enable many more things, but there are also many ways this could usefully evolve based on CC, offload, pacing etec and I think this will take much more to work-out what is needed, safe and best ... then to reach consensus. We still also need to specify a default ACK Ratio in this case as well. Best wishes, Gorry
I'm greatly in favour of draft-iyengar-quic-delayed-ack-00 (BTW, where would you prefer me to give review comments about that?) I am much less keen on any default delayed ACK policy, whether 1:2 as now in quic-transport, or the 2-10/100 numbers proposed in draft-fairhurst-quic-ack-scaling-02 My concerns would melt away if the default policy was explicitly /implementation-dependent/ and the draft instead gave /guidance/ to implementers. The draft could still give the same 2-10/100 policy, but as /guidance/ for the current public Internet. The draft should also state the conditions under which this guidance was determined (e.g. Internet paths with BDP between x and y, or whatever). What's the difference? Stating a default policy implies this is a matter of interoperability. It's not. If we change to giving guidance, I'm sure most implementers will just use the guidance. However, implementers will be free to evolve their default as they gain experience. This reduces the possibility that other behaviours will evolve around any default like 2-10/100. For instance, network monitoring using it as a signature. Or congestion controls optimizing around it. In turn, this will reduce the chance of ossification around the first numbers we thought of. Senders can always use the ACK frequency frame to ask a receiver to change from its default. Then receivers will gradually learn what senders prefer them to do, and change their default accordingly. IMO, a continually improving default is important for short flows, so senders can avoid requesting a different ACK frequency right from the start.
Need to admit that I have not had too much time to follow the QUIC work lately. I too believe that draft-iyengar-quic-delayed-ack-00 is a good long term solution that allows endpoint to set a desired ACK frequency that is not more than necessary. As experiments show there are good incentives to keep the ACK frequency as low as possible in terms of client/server load and asymmetric links limitations. An ACK-FREQUENCY frame will allow to find the proper balance between the requirements given by the congestion control, the traffic type and the above given limitations. I am not too concerned if it takes an extra development/standards cycle to have that in place in the standard and in running code as long as the QUIC community shows other SDOs that the challenge with QUIC ACKs is being addressed.
NAME NAME could you please trim the quoted text when you reply by email?
(I trimmed the comments, but forwarding inline replies to email messages doesn't work well.)
Given that this is an area where there is still obviously experimentation and refinement, NAME comment seems on point; baking a specific algorithm as requirements into the specifications doesn't seem like a good idea (although putting constraints on endpoints might make sense). Can we focus on general guidance rather than documenting a specific algorithm? What would that PR look like?
+1 to NAME suggestion. If we want to make changes for v1, they need to happen now, and they must be in a form that can let us get consensus.
To be clear, when I said that having a default policy is not a matter of interoperability, that's only strictly true if QUIC supports something like draft-iyengar-quic-delayed-ack for the sender to ask the receiver to use a certain policy. Bob
I've proposed more context and text around our current guidance in the draft, see .
I believe that the text around asymmetric links look fine, not really necessary to write up much more. What can be good to know that for the 5G case with limited UL (uplink = modem->base station transmission) , the pain that a flow with an unnecessary high ACK rate causes is on the flow itself or other flows sharing the same access.
There are some different points of pain here, the default and the need to adapt to the endpoint: An ACK Ratio (AR 1:1): is the finest granuality of feedback. A QUIC AR of 1:1 would consume return path capacity proportional to about 6% of the forward traffic (assuming no loss, etc). This is more than TCP (about 3%). An ACK Ratio (AR 1:2): For a return path bottleneck, QUIC with an AR of 1:2 will still have more impact on sharing flows than TCP would with the same AR 1:2 (because QUIC ACKs are larger, and because networks do not currently thin QUIC ACKs). So QUIC needs to choose the default - is it 1:2, 1:4, 1:8, 1:10? A QUIC AR 1:10 results in an ACK traffic volume of about 1% of the forward traffic. To me, that's a small proportion, which is why I argued that 1:10 is a good recommendation. The default is important when: the endpoints have no knowledge of the asymetry or the impact they have on a link's layer 2 loading, or the endpoints need to reduce the collateral damage to other IP flows sharing the return path capacity. Negotiation: The AR can be driven by the application or endpoint - e.g., a higher rate connection may well have a benefit from an AR of 1:10, 1:20, 1:100, etc - but such ARs are seldom chosen to reduce the impact on sharing a return bottleneck, more likely they are chosen to benefit the performance of an application or endpoint. That consideration is different, and likely depends on the application or ednpoint - which to me is where the flexibility offered by a transport parameter is so important. draft-iyengar-quic-delayed-ack will still need guidance on the "default". I argued that the default policy impacts the congestion behaviour when there is appreciable asymmetry this causes at layer 2 or the collateral damage (congestion impact on sharing flows). Sure, the choice of 1% is just a number I said was reasonable, but I've seldom encountered L2 technologies where less than 1% of forward traffic would be significantly better.
NAME : I understand your position. While I agree that ack thinning happens with TCP, it is not what I expect to find on a common network path. And as far as I know, the endpoints still implement the RFC 5681 recommendation of acking every other packet. What you are arguing is that with ack thinners in the network, TCP's behavior on those network paths is different. However, I do not believe we understand the performance effects of these middleboxes, that is, how they might reduce connection throughput when the asymmetry is not a problem. Importantly, I don't think we should be specifying as default what some middleboxes might be doing without fully understanding the consequences of doing this. Please bear in mind that Chrome's ack policy of 1:10 works with BBR, not with Cubic or Reno. I do not believe there is a magic number at the moment, and as NAME noted, even 1:10 is not a small enough ratio under some conditions. Given this, and given that we know 1:2 is used by TCP endpoints, it makes the most sense to specify that as the suggested guidance. The new text proposed in describes the tradeoff involved, and the rationale for using 1:2. I am strongly against doing anything substantially different than what we have in the draft now without overwhelming information supporting a different position.
Agree with NAME 1:2 seems a reasonable default, and there seems to be considerable interest in draft-iyengar-quic-delayed-ack, which has a better proposal for stacks that care.
I think the new ACK text is improving the specification, but I'd like to be sure we have thought about this decision. Let me try to carefully respond on-list here, to see where we agree/disagree: I disagree, I think we shouldn't be perpetuating a myth that a sender receives a TCP ACK for every other packet. Even in the 90's implementations TCP stacks often generated stretch-ACKs for 3 segments. Since then, limited byte counting was introduced and significantly improved the sender's response to stretch ACKs, and that has been good for everyone. Senders still rely on byte-counting in TCP to handle Stretch-ACKs, and this need is not decreasing: new network cards reduce per-packet receive processing using Large Receive Offload (LRO) or Generic Receiver Offload (GRO). I am saying many TCP senders actually do today see stretch-ACKs. There are papers that have shown significant presence of stretch ACKs, e.g., (H. Ding and M. Rabinovich. TCP stretch acknowledgements and timestamps: findings and implications for passive RTT measurement. ACM SIGCOMM CCR, 45(3):20–27, 2015.). A TCP Stretch-ACK of 3 or 4 was common in their datasets (about 10% of cases). In some networks, the proportion of Stretch-ACKs will be much much higher, since ACK Thining is now widely deployed in WiFi drivers as well as cable, satellite and other access technologies - often reducing ACK size/rate by a factor of two. I am intrigued and would like to know more, in case I missed something? Typical algorithms I am aware of track a queue at the "bottleneck" and then make a decision based on upon the contents of the queue. If there is no asymmtery, there isn't normally a queue and the algorithm won't hunt to discard an "ACK". But, maybe, there is some truth in being wary: Encrypted flows can also build return path queues (as does QUIC traffic). Without an ability to interpret the transport data (which nobody, including me, wants for QUIC), a bottleneck router is forced to either use another rule to drop packets from the queue (such as the smallest packets) or to allow a queue to grow, limiting forward direction throughput. I would say neither of these outcomes are great because the router is trying to fix the transport's asymetry. However, if QUIC causes this problem, I suggest some method will proliferate as QUIC traffic increases. I didn't find something in QUIC spec. that was a major concern. It looked to me like QUIC had learned from TCP how to handle stretch-ACKs. After that, we changed quicly to use 1:10 with Reno, and things worked well. Didn't NAME also use Reno? Sure, the deisgn of TCP recognised that AR 1:1 generated too much traffic, AR 1:2 for TCP resulted in an Forward:Return traffic ratio of ~1.5%, and QUIC increases this ~3%. However, as network link speeds increased, this proved too low for many network links and so, TCP ACK thinning came into being. This often reduces the TCP AR to around 1:4 ( and will do this without bias, whatever the outcome is.
NAME Just to clarify, in our experiment, we reduced the ACK frequency from the default (2) to after leaving slow start, then updated it every 4 PTO. I think that would work fine on a high-bandwidth, stable network. But I am not sure if it would work as well as ack2 on all network conditions. Anyways, I'd explain our experiment as an optimization of ack 2, as it was reduction of ack rate after leaving slow start. I think that that was a sensible choice, because IIUC having frequent acks during slow start is essential. Assuming that we'd prefer using ack 2 during slow start, the question is if we need to hard-code in the base draft how to optimize after leaving slow start. While I agree that continuing to use ack 2 after slow start is concerning for certain networks, I am reluctant to spending time on looking for such a fixed set of values across diverse networks (including much slower links or lossy ones). Rather, my preference goes to shipping the base-draft as-is, then moving on to finalize the ack-frequency frame. I think we can assume most if not all of the popular clients to support the ack-frequency frame. Then, it becomes less of a question what the default in the base-draft is.
Thanks kazuho, 1/8 or 1/4 cwnd seems reasonable, as does using Ack 2 for the first part of slow start ... that closed to what we proposed. And yes there will anyway be cases where any default does not work. If QUIC decides to retain Ack 2, can we at least capture that Ack 2 with QUIC doesn't have the same return path performance as Ack 2 with TCP - so that people know that less frequent Acts are something to think about from the start? I'm doing a talk in PANRG this week, and have decided to talk-through this topic (I was already going to talk about things related to return paths). That'll give people a view of some data - and a little thought about the implications this has for QUIC, and whether there is an incentive for network device to thin QUIC to restore performance.
NAME : I'd be in support of saying that ack thinning happens for TCP and cite those studies, and that in the absence of that, there might be throughput consequences for QUIC. I think we've said exactly that in other words in , but if we want to say more there, that's fine with me. That would make this an editorial issue, but I'm entirely happy to do more words there. I am against changing the recommendation here for another arbitrary recommendation. I think the right approach is to allow endpoints to specify this during a connection, which is where the extension is helpful. As NAME notes, our hope is that the extension gets widely deployed. If you want to make the argument for the extension to become part of the base draft, that's a different conversation, but that's not the one we're having here.
Attempting to summarise the current status of this issue as a chair: there appears to have been a fair amount of discussion about making a design change, with representatives both in favour and against. It appears there is no omnipotent default policy, choosing an optimal one depends on many factors. The editors have landed an editorial change that attempts to better describe the rationale behind the QUICS's current design and highlights some of the factors. There may be further editorial improvements but they no not need to block the design issue. The proposal therefore is to close this issue with no design change. We had prepared some slides for the Vancouver IETD to discuss QUIC's ACKs, and how to make QUIC work better than TCP in common asymmetric paths. This is a short slide deck that summarises what we have learned about ACKs in QUIC: URL It compares the ACK traffic of TCP with that of QUIC from the point of view of a path where QUIC's return traffic is somehow constrained or could impact other user traffic. We think that ACKs are a concern for various shared capacity return link (e.g., radio-based methods in WiFi or mobile cellular were ACKs consume spectrum/radio resource or battery; shared capacity satellite broadband return link or the uplink of a cable system). For TCP, such systems commonly involve a PEP or an ACK Filtering mechanism (which clearly isn't wanted or even possible for QUIC). The baseline QUIC performance (as per current spec) suffers over these paths. So our first question is: Do other people think it is important that the default specified for QUIC works well with asymmetric paths ? Gorry

receiver's max_ack_delay value in determining timeouts for timer- based retransmission, as detailed in Section 5.2.1 of QUIC-RECOVERY. <del> An ACK frame SHOULD be generated for at least every second ack- eliciting packet.  This recommendation is in keeping with standard practice for TCP RFC5681.  A receiver could decide to send an ACK frame less frequently if it has information about how frequently the sender's congestion controller needs feedback, or if the receiver is CPU or bandwidth constrained. </del> <ins> Since packets containing only ACK frames are not congestion controlled, an endpoint MUST NOT send more than one such packet in response to receiving an ack-eliciting packet. An endpoint MUST NOT send a non-ack-eliciting packet in response to a non-ack-eliciting packet, even if there are packet gaps which precede the received packet.  This avoids an infinite feedback loop of acknowledgements, which could prevent the connection from ever becoming idle.  Non-ack-eliciting packets are eventually acknowledged when the endpoint sends an ACK frame in response to other events. </ins> In order to assist loss detection at the sender, an endpoint SHOULD send an ACK frame immediately on receiving an ack-eliciting packet


Question:
This PR adds explicit context around the choice of ACK frequency in the draft. It also takes in Bob Briscoe's suggestion in , of calling it a guidance in light of the best information we have. Also does some editorial cleanup of text blocks that were not in the right sections.
Thanks NAME NAME -- comments incorporated.
As currently specified, the default QUIC ACK policy models TCP’s ACK policy, but consumes significantly more return path capacity. This can impact resource usage of the return channel, can constrain forward throughput, and can impact the performance of other sessions sharing an uplink (draft-fairhurst-quic-ack-scaling). This was not discussed in Zurich - where discussion moved instead to the separate topic of draft-iyengar-quic-delayed-ack. In many current network segments (e.g., cable networks, mobile cellular, WiFi and other radio tech), the effects of asymetry are presently mitigated by network devices RFC3449 that rely upon observing TCP headers. This results in ossification problems, but does practically have beneit. It reduces the TCP ACK rate over the constrained link, it reduces consumption of "radio" transmission opportunities, and reduces traffic volume from ACKs. These mitigations are not possible, nor desirable for QUIC. However, the default performance of QUIC is therefore much worse than for TCP. In QUIC, the default ACK Ratio has to set by the endpoints implementing the protocol because the sender and receiver have no information about the "cost" of sending an ACK using a link along an asymmetric path. These endpoints can also choose a non-default value to enable a new CC, scale to higher rates, etc (e.g. draft-iyengar-quic-delayed-ack). We have shown there can be significant benefit from an ACK Ratio of 1:10 and this is expected to alos have benefit for higher levels of asymmetry. The proposed change from 1:2 to 1:10 is predicated on QUIC's design decisions: starting a connection with an Initial Window of 10 is acceptable, QUIC uses PTO as the loss detection method (a predicate to issue ), and QUIC has some form of pacing at the sender, at least to mitigate IW bursts.
Even though we've avoided straying this far from the TCP spec, I think allowing this would be good for the internet. It should probably be documented for TCP, too. How does ack stretching of this nature work for BBR bandwidth estimation? It's harder to use the ack pacing, no?
I don't believe QUIC ACKs are significantly larger. In fact, I believe the ACK frame itself is typically smaller than TCP ACKs, but the savings there are used on QUIC's 16-byte auth hash and likely a non-zero-length CID. Do you have a size analysis written up somewhere? I think this is the core issue. It's not that QUIC ACKs are really consuming significantly more capacity in terms of bytes on the wire, it's that middleboxes can't decide to thin the ACKs, because they don't know which packets are ACK-only packets. Did you implement the optimization already described in the transport draft?: "As an optimization, a receiver MAY process multiple packets before sending any ACK frames in response. In this case the receiver can determine whether an immediate or delayed acknowledgement should be generated after processing incoming packets." NAME did and I believe found it hugely reduced the number of ACKs, sometimes too much. As I've said before, I would really prefer not specify an approach in QUIC transport that is a heuristic like does. In particular, on lower bandwidth networks, 10 packets may be the entire congestion window and is too large a value. On higher bandwidth networks, every 20 packets has shown to be slightly better than 10, at least when using BBR. I wrote to add some editorial text about why every 2 packets is a SHOULD, not a MUST. Chrome did not default enable the ACK strategy recommends until BBR was launched, because there was a performance regression with Cubic. I'd rather not assume everyone is using BBR or a BBR-like congestion controller. NAME BBR does fine with stretch ACKs. Cubic and Reno suffer in some cases, however.
QUIC ACKs are larger! The smallest IPv4 QUIC ACK packet is 51 bytes. The ACK frame is 4 bytes, but is sent using a QUIC packet. For IPv4 this is: 20 (IP) + 8 (UDP) +3 (1+1+1 QUIC header ... assuming a 1 B CID) + 4 (minimum ACK frame size)+ 16 (crypto). Other types of frames may also be sent in the same packet, increasing this size. That size difference matters. The point is that QUIC ACKs are consuming more capacity in terms of bytes on the wire, and often also important: in more transmission opportunities at the physical layer. That makes QUIC perform more poorly than it needs to across some paths. The text "As an optimization, a receiver MAY process multiple packets before sending any ACK frames in response. In this case the receiver can determine whether an immediate or delayed acknowledgement should be generated after processing incoming packets." Is permissive to do something different - alas the effect of ordering of the processing depends on rate, timing, etc - The present text is not a recommendation to do something appropriate, which I think it needs to. If this were to describe that a receiver SHOULD reduce the ACK to data packet ratio to 1:10, then that would help. On low bandwidth links, then I agree that waiting for 10 packets would not be suitable, but then wouldn't the rule 4 ACKs/RTT still cover this case? (I have no problems believing every an ACK 20 packets can be slightly better than 10 in some cases, but then we have chosen IW at 10 and that relates to bustiness and responsiveness of the basic spec, so I think 1:10 makes more sense as the recommendation. If the CC and path information suggest 1:20 - perhaps as the rate increases, then a sender can "tune" as per the transport parameter proposal. That I STILL assert that this is a different issue from the expected default.)
Would it help to give a recommendation like, if the congestion window is large, the ACK radio should be reduced? Not sure if we can easily give a threshold for the window but to align with the 4 ACKs per RTT rule we could say 40 or to be more conservation maybe 100?
If the receiver knew the cwnd... we could add that. The current proposal we made is here (along with some of the arguments about why QUIC and TCP are slightly different): draft-fairhurst-quic-ack-scaling
So the min TCP ack is 40 bytes vs QUIC at 50(0 byte CIDs are common, so I'll assume that), 25% larger in the minimum case. However, with a few SACK blocks, TCP ACKs are likely to be larger than QUIC ACKs, since SACK consumes 10 bytes for the first block and 8 for each additional block. By the time the second block is added, there's a good chance QUIC ACKs are smaller and it's almost certainly true with 3 SACK blocks(which I think is a typical number). So I'll go back to my previous statement that the real issue is that the network can't thin QUIC ACKs effectively.
May be "yes", may be "no" on a larger size after loss - since QUIC ACK ranges are often larger - but that depends on loss pattern, etc. In our case, we saw more bytes. But "yes" on me saying there is a need for QUIC to make good default decisions about the ACK rate, because QUIC becomes responsible for transport working across heterogeneous paths and we need to get good defaults rather than rely on "ACK-Thining" and other network modification.
Anybody has a gut feeling how very low latency congestion control fares with sparse ACKs?. I have experienced that DCTCP is more picky with ACK aggregation than e.g Reno. I suspect that ACK-thinning can have the give the issue as well. A too sparse ACK spacing can also give problems. Another aspect is how e.g. TCP Prague with paced chirps work with sparse ACKs ?. To me the RTT/4 rule makes sense but I feel that the best alternative depends a lot on both the congestion control and the source material. Large file transfers are more trivial than sources that are occasionally idle.
I think this isn't the correct question. This is about the default policy. The default policy is about what a receiver does by "default" - the current text uses a value of ACK every other packet, 1:2. TCP specified this when transmission rates were low, and TCP depended on individual ACKs to clock data (no pacing, IW=2), detect loss (no PTO), increase cwnd (per ACK packet), etc. In response, TCP causes ACK congestion and strange interactions with radio access. Performance over WiFi, DOCSIS etc suffers unless the network tries to address this. I don't see why QUIC, which doesn't rely on any of these, needs to set a DEFAULT of 1:2. This is literally a few lines of code difference, and we could use 1:10. These CC's you mention may have different needs and that needs to be handled by telling the remote that your CC has these specific needs - which I think is what Jana's extension is about.
I am kind of the donkey in the middle... ACK thinning in networks should be discouraged. Standardization of ACK thinning mechanisms was proposed in 3GPP RAN2, this request was turned down mainly with reference to that it can harm transport protocol performance and does not work safely with QUIC. Nothing stops UE (terminal and modem) vendors from implement proprietary ACK-thinning but I certainly hope that 5G UE vendors don't attempt to thin QUIC-ACKs as this is a slippery slope. High ACK rates is a 5G problem mainly for high band NR (=5G at e.g 28GHz). It is mainly a coverage issue as UEs have a limited output power due to practical and regulatory requirements. At the same time one need to address that end users don't only download movies today and even VoD streaming implies uplink traffic in the form of chunk requests. And then we have an increasing amount of people doing uplink streaming or upload an infinite amount of pictures of food plates and cats :-) Operators and 5G vendors have to deal with this and solve the problem with deployment of 5G access in various frequency bands to get the best downlink and uplink performance. Reduced ACK rates is of course welcome but I don't currently see a problem that the initial ACK ratio is 1:2 , to increase to 1:? as the congestion window grows, bounded by e.g an RTT/4 rule. But here I need to admit that I at least I have not studied this in detail. Ergo.. I am not convinced that it is necessary to hardwire the QUIC spec to 1:10 , but I also cannot say that it should not be done.
This is a small enough change. However, there is still a bar for those, and the current level of discussion makes me believe we should close this with no action?
Just so that it's easy to find, there is data and a continued discussion at URL
Folks who care about a change here need to prioritize progressing this issue. We're close to being able to last-call the specs, and would like this resolved in time.
We’ve completed a round of updated analysis of the proposed change to the QUIC spec, using quicly (changing to a 1:10 policy), and chromium (which currently does 1:10 by default). Our hope is that this change to the spec, to actually make standard end-to-end QUIC work better than TCP over asymmetric paths! This is a short slide deck about the performance of the proposed change: URL There have been a number of other change made by the QUIC ID editors relating to ACK processing, so we wanted to be sure these results were up to date. (These results do not consider additional ACKs after re-ordering - reflecting the latest text removing that.) We also really hope this will motivate a change the practice of forcing of QUIC fallback to TCP (with PEP/Thining) over highly asymmetric paths. People may also be interested in work for L4S to evaluate the impact of ACK Thining for TCP-Prague. Please do provide feedback, or any questions on this - as Lars notes, we think the time is running out for considering new changes, even relatively simple ones like this. Gorry P.S. First email: URL
I believe that it is beneficial to use a 1:10 policy, I understand that this means that a 1:2 policy that still be used in the initial phase (flow start) or does the timer mechanisms kick in to avoid that ACKs are transmitted too sparsely in this case ?.
Yes, our proposal counted ACK-eliciting packets at the receiver (because the receiver does not in general know the congestion state, so you need a heuristic), and we finally chose the same method in currect QUIC Chromium - i.e. for the first 100 packets we do not use ACK 1:10, changing only after 100 packets.
OK, it sounds like a good idea to use 1:10 after 100 packets or so. Just one potentially stupid question, Is there any special treatment necessary after an RTO ?
[This seems to be a forwarded comment from NAME Hi, Gorry Thank you for sharing your experiments. I'm delighted to see results showing that use of ack10 gives us comparable throughput to ack2. Recently, I have been looking into how ack-frequency affects the computational efficiency of the server [1]. Using the (almost) same setup, I have today run a benchmark comparing three ACK strategies: ack2, ack10, ack 1/8cwnd. ack2: 334Mbps ack10: 414Mbps ack 1/8cwnd: 487Mbps As can be seen, ack10 is better than ack2, but lags behind ack1/8cwnd. That's because there are still frequent ACKs. They take considerable amount of CPU cycles. If we want to see QUIC being as lightweight as TLS over TCP, it is critically important to reduce ACK frequency as much as possible. This is because the cost of processing ACK in QUIC is much more expensive than TCP, due QUIC being a userspace transport. Based on this, if we are to change what we have in the base drafts, I'd much prefer adopting what we have in the proposed delayed ack extension [2], rather than just changing the default ack policy from ack2 to ack10. By adopting what we have in the delayed ack extension, we can expect better performance, and the endpoints would have the freedom to change the behavior when the default ack policy is deemed insufficient (regardless of the default being ack2 or ack10). [1] URL [2] URL 2020428() 20:37 Gorry Fairhurst : issue. We're close to being able to last-call the specs, and would like this resolved in time. QUIC spec, using quicly (changing to a 1:10 policy), and chromium (which currently does 1:10 by default). Our hope is that this change to the spec, to actually make standard end-to-end QUIC work better than TCP over asymmetric paths! relating to ACK processing, so we wanted to be sure these results were up to date. (These results do not consider additional ACKs after re-ordering - reflecting the latest text removing that.) of QUIC fallback to TCP (with PEP/Thining) over highly asymmetric paths. People may also be interested in work for L4S to evaluate the impact of ACK Thining for TCP-Prague. think the time is running out for considering new changes, even relatively simple ones like this. URL
These results/article on computational efficiency are really interesting. Please see below. This is useful, we see here there's a clear benefit here in using a larger ACK Ratio also in terms of server load. We can see already the benefit at 1:10 and GSO helps as expected. We think that this is also really true. QUIC ACKs are also larger than TCP ACKs (see URL). I don't think the two approaches conflict. The difference between the two approaches is that the small change to the base specification to use an ACK Ratio 1:10 ensures the default return path is at least as good as TCP for common traffic (even with common TCP ACK Thinning). Without this change, the end user could experience significant differences in performance between TCP and between different QUIC versions over asymmetric paths. Whereas to me, the transport parameter extension looks like an exciting way to enable many more things, but there are also many ways this could usefully evolve based on CC, offload, pacing etec and I think this will take much more to work-out what is needed, safe and best ... then to reach consensus. We still also need to specify a default ACK Ratio in this case as well. Best wishes, Gorry
I'm greatly in favour of draft-iyengar-quic-delayed-ack-00 (BTW, where would you prefer me to give review comments about that?) I am much less keen on any default delayed ACK policy, whether 1:2 as now in quic-transport, or the 2-10/100 numbers proposed in draft-fairhurst-quic-ack-scaling-02 My concerns would melt away if the default policy was explicitly /implementation-dependent/ and the draft instead gave /guidance/ to implementers. The draft could still give the same 2-10/100 policy, but as /guidance/ for the current public Internet. The draft should also state the conditions under which this guidance was determined (e.g. Internet paths with BDP between x and y, or whatever). What's the difference? Stating a default policy implies this is a matter of interoperability. It's not. If we change to giving guidance, I'm sure most implementers will just use the guidance. However, implementers will be free to evolve their default as they gain experience. This reduces the possibility that other behaviours will evolve around any default like 2-10/100. For instance, network monitoring using it as a signature. Or congestion controls optimizing around it. In turn, this will reduce the chance of ossification around the first numbers we thought of. Senders can always use the ACK frequency frame to ask a receiver to change from its default. Then receivers will gradually learn what senders prefer them to do, and change their default accordingly. IMO, a continually improving default is important for short flows, so senders can avoid requesting a different ACK frequency right from the start.
Need to admit that I have not had too much time to follow the QUIC work lately. I too believe that draft-iyengar-quic-delayed-ack-00 is a good long term solution that allows endpoint to set a desired ACK frequency that is not more than necessary. As experiments show there are good incentives to keep the ACK frequency as low as possible in terms of client/server load and asymmetric links limitations. An ACK-FREQUENCY frame will allow to find the proper balance between the requirements given by the congestion control, the traffic type and the above given limitations. I am not too concerned if it takes an extra development/standards cycle to have that in place in the standard and in running code as long as the QUIC community shows other SDOs that the challenge with QUIC ACKs is being addressed.
NAME NAME could you please trim the quoted text when you reply by email?
(I trimmed the comments, but forwarding inline replies to email messages doesn't work well.)
Given that this is an area where there is still obviously experimentation and refinement, NAME comment seems on point; baking a specific algorithm as requirements into the specifications doesn't seem like a good idea (although putting constraints on endpoints might make sense). Can we focus on general guidance rather than documenting a specific algorithm? What would that PR look like?
+1 to NAME suggestion. If we want to make changes for v1, they need to happen now, and they must be in a form that can let us get consensus.
To be clear, when I said that having a default policy is not a matter of interoperability, that's only strictly true if QUIC supports something like draft-iyengar-quic-delayed-ack for the sender to ask the receiver to use a certain policy. Bob
I've proposed more context and text around our current guidance in the draft, see .
I believe that the text around asymmetric links look fine, not really necessary to write up much more. What can be good to know that for the 5G case with limited UL (uplink = modem->base station transmission) , the pain that a flow with an unnecessary high ACK rate causes is on the flow itself or other flows sharing the same access.
There are some different points of pain here, the default and the need to adapt to the endpoint: An ACK Ratio (AR 1:1): is the finest granuality of feedback. A QUIC AR of 1:1 would consume return path capacity proportional to about 6% of the forward traffic (assuming no loss, etc). This is more than TCP (about 3%). An ACK Ratio (AR 1:2): For a return path bottleneck, QUIC with an AR of 1:2 will still have more impact on sharing flows than TCP would with the same AR 1:2 (because QUIC ACKs are larger, and because networks do not currently thin QUIC ACKs). So QUIC needs to choose the default - is it 1:2, 1:4, 1:8, 1:10? A QUIC AR 1:10 results in an ACK traffic volume of about 1% of the forward traffic. To me, that's a small proportion, which is why I argued that 1:10 is a good recommendation. The default is important when: the endpoints have no knowledge of the asymetry or the impact they have on a link's layer 2 loading, or the endpoints need to reduce the collateral damage to other IP flows sharing the return path capacity. Negotiation: The AR can be driven by the application or endpoint - e.g., a higher rate connection may well have a benefit from an AR of 1:10, 1:20, 1:100, etc - but such ARs are seldom chosen to reduce the impact on sharing a return bottleneck, more likely they are chosen to benefit the performance of an application or endpoint. That consideration is different, and likely depends on the application or ednpoint - which to me is where the flexibility offered by a transport parameter is so important. draft-iyengar-quic-delayed-ack will still need guidance on the "default". I argued that the default policy impacts the congestion behaviour when there is appreciable asymmetry this causes at layer 2 or the collateral damage (congestion impact on sharing flows). Sure, the choice of 1% is just a number I said was reasonable, but I've seldom encountered L2 technologies where less than 1% of forward traffic would be significantly better.
NAME : I understand your position. While I agree that ack thinning happens with TCP, it is not what I expect to find on a common network path. And as far as I know, the endpoints still implement the RFC 5681 recommendation of acking every other packet. What you are arguing is that with ack thinners in the network, TCP's behavior on those network paths is different. However, I do not believe we understand the performance effects of these middleboxes, that is, how they might reduce connection throughput when the asymmetry is not a problem. Importantly, I don't think we should be specifying as default what some middleboxes might be doing without fully understanding the consequences of doing this. Please bear in mind that Chrome's ack policy of 1:10 works with BBR, not with Cubic or Reno. I do not believe there is a magic number at the moment, and as NAME noted, even 1:10 is not a small enough ratio under some conditions. Given this, and given that we know 1:2 is used by TCP endpoints, it makes the most sense to specify that as the suggested guidance. The new text proposed in describes the tradeoff involved, and the rationale for using 1:2. I am strongly against doing anything substantially different than what we have in the draft now without overwhelming information supporting a different position.
Agree with NAME 1:2 seems a reasonable default, and there seems to be considerable interest in draft-iyengar-quic-delayed-ack, which has a better proposal for stacks that care.
I think the new ACK text is improving the specification, but I'd like to be sure we have thought about this decision. Let me try to carefully respond on-list here, to see where we agree/disagree: I disagree, I think we shouldn't be perpetuating a myth that a sender receives a TCP ACK for every other packet. Even in the 90's implementations TCP stacks often generated stretch-ACKs for 3 segments. Since then, limited byte counting was introduced and significantly improved the sender's response to stretch ACKs, and that has been good for everyone. Senders still rely on byte-counting in TCP to handle Stretch-ACKs, and this need is not decreasing: new network cards reduce per-packet receive processing using Large Receive Offload (LRO) or Generic Receiver Offload (GRO). I am saying many TCP senders actually do today see stretch-ACKs. There are papers that have shown significant presence of stretch ACKs, e.g., (H. Ding and M. Rabinovich. TCP stretch acknowledgements and timestamps: findings and implications for passive RTT measurement. ACM SIGCOMM CCR, 45(3):20–27, 2015.). A TCP Stretch-ACK of 3 or 4 was common in their datasets (about 10% of cases). In some networks, the proportion of Stretch-ACKs will be much much higher, since ACK Thining is now widely deployed in WiFi drivers as well as cable, satellite and other access technologies - often reducing ACK size/rate by a factor of two. I am intrigued and would like to know more, in case I missed something? Typical algorithms I am aware of track a queue at the "bottleneck" and then make a decision based on upon the contents of the queue. If there is no asymmtery, there isn't normally a queue and the algorithm won't hunt to discard an "ACK". But, maybe, there is some truth in being wary: Encrypted flows can also build return path queues (as does QUIC traffic). Without an ability to interpret the transport data (which nobody, including me, wants for QUIC), a bottleneck router is forced to either use another rule to drop packets from the queue (such as the smallest packets) or to allow a queue to grow, limiting forward direction throughput. I would say neither of these outcomes are great because the router is trying to fix the transport's asymetry. However, if QUIC causes this problem, I suggest some method will proliferate as QUIC traffic increases. I didn't find something in QUIC spec. that was a major concern. It looked to me like QUIC had learned from TCP how to handle stretch-ACKs. After that, we changed quicly to use 1:10 with Reno, and things worked well. Didn't NAME also use Reno? Sure, the deisgn of TCP recognised that AR 1:1 generated too much traffic, AR 1:2 for TCP resulted in an Forward:Return traffic ratio of ~1.5%, and QUIC increases this ~3%. However, as network link speeds increased, this proved too low for many network links and so, TCP ACK thinning came into being. This often reduces the TCP AR to around 1:4 ( and will do this without bias, whatever the outcome is.
NAME Just to clarify, in our experiment, we reduced the ACK frequency from the default (2) to after leaving slow start, then updated it every 4 PTO. I think that would work fine on a high-bandwidth, stable network. But I am not sure if it would work as well as ack2 on all network conditions. Anyways, I'd explain our experiment as an optimization of ack 2, as it was reduction of ack rate after leaving slow start. I think that that was a sensible choice, because IIUC having frequent acks during slow start is essential. Assuming that we'd prefer using ack 2 during slow start, the question is if we need to hard-code in the base draft how to optimize after leaving slow start. While I agree that continuing to use ack 2 after slow start is concerning for certain networks, I am reluctant to spending time on looking for such a fixed set of values across diverse networks (including much slower links or lossy ones). Rather, my preference goes to shipping the base-draft as-is, then moving on to finalize the ack-frequency frame. I think we can assume most if not all of the popular clients to support the ack-frequency frame. Then, it becomes less of a question what the default in the base-draft is.
Thanks kazuho, 1/8 or 1/4 cwnd seems reasonable, as does using Ack 2 for the first part of slow start ... that closed to what we proposed. And yes there will anyway be cases where any default does not work. If QUIC decides to retain Ack 2, can we at least capture that Ack 2 with QUIC doesn't have the same return path performance as Ack 2 with TCP - so that people know that less frequent Acts are something to think about from the start? I'm doing a talk in PANRG this week, and have decided to talk-through this topic (I was already going to talk about things related to return paths). That'll give people a view of some data - and a little thought about the implications this has for QUIC, and whether there is an incentive for network device to thin QUIC to restore performance.
NAME : I'd be in support of saying that ack thinning happens for TCP and cite those studies, and that in the absence of that, there might be throughput consequences for QUIC. I think we've said exactly that in other words in , but if we want to say more there, that's fine with me. That would make this an editorial issue, but I'm entirely happy to do more words there. I am against changing the recommendation here for another arbitrary recommendation. I think the right approach is to allow endpoints to specify this during a connection, which is where the extension is helpful. As NAME notes, our hope is that the extension gets widely deployed. If you want to make the argument for the extension to become part of the base draft, that's a different conversation, but that's not the one we're having here.
Attempting to summarise the current status of this issue as a chair: there appears to have been a fair amount of discussion about making a design change, with representatives both in favour and against. It appears there is no omnipotent default policy, choosing an optimal one depends on many factors. The editors have landed an editorial change that attempts to better describe the rationale behind the QUICS's current design and highlights some of the factors. There may be further editorial improvements but they no not need to block the design issue. The proposal therefore is to close this issue with no design change. We had prepared some slides for the Vancouver IETD to discuss QUIC's ACKs, and how to make QUIC work better than TCP in common asymmetric paths. This is a short slide deck that summarises what we have learned about ACKs in QUIC: URL It compares the ACK traffic of TCP with that of QUIC from the point of view of a path where QUIC's return traffic is somehow constrained or could impact other user traffic. We think that ACKs are a concern for various shared capacity return link (e.g., radio-based methods in WiFi or mobile cellular were ACKs consume spectrum/radio resource or battery; shared capacity satellite broadband return link or the uplink of a cable system). For TCP, such systems commonly involve a PEP or an ACK Filtering mechanism (which clearly isn't wanted or even possible for QUIC). The baseline QUIC performance (as per current spec) suffers over these paths. So our first question is: Do other people think it is important that the default specified for QUIC works well with asymmetric paths ? Gorry

codepoint in the IP header SHOULD be acknowledged immediately, to reduce the peer's response time to congestion events. <del> As an optimization, a receiver MAY process multiple packets before sending any ACK frames in response.  In this case the receiver can determine whether an immediate or delayed acknowledgement should be generated after processing incoming packets. Packets containing PADDING frames are considered to be in flight for congestion control purposes QUIC-RECOVERY.  Sending only PADDING frames might cause the sender to become limited by the congestion controller with no acknowledgments forthcoming from the receiver. Therefore, a sender SHOULD ensure that other frames are sent in addition to PADDING frames to elicit acknowledgments from the receiver. </del> <ins> The algorithms in QUIC-RECOVERY are expected to be resilient to receivers that do not follow guidance offered above.  However, an implementer should only deviate from these requirements after careful consideration of the performance implications of doing so. </ins> An endpoint that is only sending ACK frames will not receive acknowledgments from its peer unless those acknowledgements are