C++: correct way to have heterogenous containers

Question

I am currently working on a basic language parser. For the tokenization process, the tokens can be of different types. So I need a heterogenous container to hold those tokens.

1. What to choose - std::variant or base class pointers?
From this thread on StackOverflow,
https://stackoverflow.com/questions/59784261/stdvariant-vs-pointer-to-base-class-for-heterogeneous-containers-in-c

I found that the main argument against using base class pointers is doing dynamic memory allocations and handling their pointers. But in my case, I'll need to put the tokens on the heap anyways because the nunber of tokens can be huge and putting them on the stack can lead to problems.

So in my particular case, which way is better?

[point no. 2 as answer in thread because I'm reaching character limit]

Answer 1

Maybe this is a too simplistic way of thinking, but inheritance would only be beneficial if you have a common interface for your tokens to operate on. After all, if you store them together in base pointers, you will only have access to the (virtual) base class methods.
Since it sounds like you can't come up with a reasonable interface, it would make me think to go with std::variant<>, where you could at least write a visitor to dispatch whatever you need based on the actual type of the token via template techniques.

Then again, the two times I've been operating on parsers so far were both with inheritance structures. However, in both those cases there was a fixed set of operations each expression in the final abstract syntax tree had to implement, e.g. type checking and evaluation.

Answer 2

[continued from question]
2. If I choose base class pointers, how do I make the base class polymorphic?
The problem is that the base class `Token` will have pretty much nothing in common with the derived types. So how do I make it virtual? I know I could make the destructor virtual, but is it a good practise to have inheritance solely for the purpose of polymorphism?

TL;DR:
1. std::variant vs base clas pointers for heterogenous containers when the individual objects will be dynamically allocated.
2. Is it good practise to inherit from a base class only to enable polymorphism, i.e. when the purpose of inheritance is not to actually inherit something, but to just allow the derived class to be accessible from a base class pointer

Answer 3

Shadow
Thanks.

"you will only have access to the base class methods"
In my mind, I was thinking about having an enum field in the base class that describes which type of token it is, and according to that, casting the pointer to the corresponding token type, and then doing operations on it. But I guess that would have a lot of overhead compared to std::variant.

"in both those cases there was a fixed set of operations...."
In my case, the parser is actually for HTML. When a token is formed, it is passed to the tree constructor directly so there is no operation to be done on that.

So I guess I'll stick to std::variant. Thanks again.