- 25 Jul, 2022 26 commits
-
-
Pavel Begunkov authored
It's not clear how widely used IOSQE_CQE_SKIP_SUCCESS is, and how often ->flush_cqes flag prevents from completion being flushed. Sometimes it's high level of concurrency that enables it at least for one CQE, but sometimes it doesn't save much because nobody waiting on the CQ. Remove ->flush_cqes flag and the optimisation, it should benefit the normal use case. Note, that there is no spurious eventfd problem with that as checks for spuriousness were incorporated into io_eventfd_signal(). Signed-off-by:
Pavel Begunkov <asml.silence@gmail.com> Link: https://lore.kernel.org/r/692e81eeddccc096f449a7960365fa7b4a18f8e6.1655637157.git.asml.silence@gmail.com [axboe: remove now dead state->flush_cqes variable] Signed-off-by:
Jens Axboe <axboe@kernel.dk>
-
Pavel Begunkov authored
It's a good idea to first do forward declarations and then inline helpers, otherwise there will be keep stumbling on dependencies between them. Signed-off-by:
-
Pavel Begunkov authored
Move io_uring types to linux/include, need them public so tracing can see the definitions and we can clean trace/events/io_uring.h Signed-off-by:
-
Pavel Begunkov authored
With IORING_SETUP_CQE32 ->cqe_cached doesn't store a real address but rather an implicit offset into cqes. Store the real cqe pointer and increment it accordingly if CQE32. Signed-off-by:
-
Pavel Begunkov authored
Deduplicate calls to io_get_cqe() from __io_fill_cqe_req(). Signed-off-by:
-
Pavel Begunkov authored
Deduplicate two trace_io_uring_complete() calls in __io_fill_cqe_req(). Signed-off-by:
-
Pavel Begunkov authored
__io_fill_cqe_req() is hot and inlined, we want it to be as small as possible. Add io_req_cqe_overflow() accepting only a request and doing all overflow accounting, and replace with it two calls to 6 argument io_cqring_event_overflow(). Signed-off-by:
-
Pavel Begunkov authored
__io_get_cqe() is not as hot as io_get_cqe(), no need to inline it, it sheds ~500B from the binary. Signed-off-by:
-
Pavel Begunkov authored
Deduplicate some code and add a helper for filling an aux CQE, locking and notification. Signed-off-by:
-
Pavel Begunkov authored
REQ_F_COMPLETE_INLINE is only needed to delay queueing into the completion list to io_queue_sqe() as __io_req_complete() is inlined and we don't want to bloat the kernel. As now we complete in a more centralised fashion in io_issue_sqe() we can get rid of the flag and queue to the list directly. Signed-off-by:
Hao Xu <howeyxu@tencent.com> Signed-off-by:
-
Pavel Begunkov authored
There is a bunch of inline helpers that will be useful not only to the core of io_uring, move them to headers. Signed-off-by:
-
Jens Axboe authored
Signed-off-by: -
Jens Axboe authored
Signed-off-by: -
Jens Axboe authored
Signed-off-by: -
Jens Axboe authored
Move both the opcodes related to it, and the internals code dealing with it. Signed-off-by: -
Jens Axboe authored
This also helps cleanup the io_uring.h cancel parts, as we can make things static in the cancel.c file, mostly. Signed-off-by: -
Jens Axboe authored
Add a io_poll_issue() rather than export the general task_work locking and io_issue_sqe(), and put the io_op_defs definition and structure into a separate header file so that poll can use it. Signed-off-by: -
Jens Axboe authored
Signed-off-by: -
Jens Axboe authored
Signed-off-by: -
Jens Axboe authored
Signed-off-by: -
Jens Axboe authored
While at it, convert the handlers to just use io_eopnotsupp_prep() if CONFIG_NET isn't set. Signed-off-by: -
Jens Axboe authored
Signed-off-by: -
Jens Axboe authored
Signed-off-by: -
Jens Axboe authored
This splits out splice and tee support. Signed-off-by: -
Jens Axboe authored
Normally request handlers complete requests themselves, if they don't return an error. For the latter case, the core will complete it for them. This is unhandy for pushing opcode handlers further out, as we don't want a bunch of inline completion code and we don't want to make the completion path slower than it is now. Let the core handle any completion, unless the handler explicitly asks us not to. Signed-off-by: -
Jens Axboe authored
Signed-off-by:
-
