Do note that server sessions are text only so the data needs to be serialized to json and back on retrieval. Not sure if we have formatters for this.
Also there might be a limit of the session object length. And you have to handle the case that sessions is expired and retrieve fresh.
So at the end you might be better if in just storing the record Id and do the query when you need it later on. It will go just as fast.